Comprehensive data protection for all workloads
Post Reply
m.novelli
Veeam ProPartner
Posts: 355
Liked: 40 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Slow VM backup with SQL Server 2008 R2

Post by m.novelli » Jun 15, 2010 7:44 am

Hi friends, I've installed a new Vmware infrastructure with two Dell R610 servers connected to a Dell MD3000i storage (iSCSI)

I'm running vSphere 4.0 U1 and Veeam 4.1.1 with CBT

All VM are backupping at high speed (from 100 to 200 MB/sec after initial full backup) except one: a Windows 2008 R2 + SQL Server 2008 R2
This VM is backupping at 30/40 MB/sec and need from 2 to 3 hours to complete an incremental backup. During Veeam backup the SQL Server is idle. This machine produce every day about 60 GB of backups on local disks, but it's a normal quantity.

Anyone experienced something similar?

Maybe Veeam need to update the VSS integration for SQL Server 2008 R2 as they did for Exchange 2010?

Thanks for any help,

Marco

Gostev
SVP, Product Management
Posts: 24789
Liked: 3522 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow VM backup with SQL Server 2008 R2

Post by Gostev » Jun 15, 2010 10:20 am

Hi Marco, this could be normal for highly transactional Exchange and SQL servers to have slower incremental backup speed, because there are much more incremental data to process. This is caused by the fact that these applications use transaction logs which change a lot of disk blocks, and all these blocks have to be processed with incremental backup. Large amounts of data obviously take longer to update synthethic full with. You can find existing threads on Exchange backups about this forum as well, so this is not some specific to you. So this is something you just have to live this, obviously VMs with large amounts of changed data will always backup slower than VMs with small amount of changes (all other VMs you have).

m.novelli
Veeam ProPartner
Posts: 355
Liked: 40 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: Slow VM backup with SQL Server 2008 R2

Post by m.novelli » Jun 26, 2010 9:18 am

I've finally solved this issue. It was a configuration issue on the storage Dell MD3000i

I have only one Disk Group (7 SAS disks x 450 GB 15k) divided in two LUN

One LUN is dedicated to SQL Server VM and one LUN is for all other VM

One LUN was owned by one RAID Controller Module (aka SP) and one LUN was owned by the other RAID Controller Module

Setting both LUN owned by the same RAID Controlled Module give me lighthing speed also on the SQL Server VM

So: for Dell MD3000i the LUNs inside a Disk Group should be owned by the same RAID Controller Module

Marco

Vitaliy S.
Product Manager
Posts: 22984
Liked: 1556 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow VM backup with SQL Server 2008 R2

Post by Vitaliy S. » Jun 26, 2010 9:20 am

Marco, thank you for sharing! This information is very useful.

Tom Bale
Novice
Posts: 8
Liked: never
Joined: Mar 08, 2011 10:11 am
Full Name: Thomas Bale
Contact:

Slow Exchange Backups, is this normal?

Post by Tom Bale » May 10, 2011 9:04 am

[merged into existing discussion]

Hi All,

We have now got to a point where our Veeam backup system is running without much interaction which is great but backups for our Exchange server seem to take much longer than expected. This isn't really a support issue because I want to get a feel for how others are finding this and how they have it setup.

I wouldn't say that this is a particularly heavily used exchange server (Windows 2008 R2). but backups are taking roughly 12 hours daily which means that backups are sometimes over-running into our normal opening hours.

We are using reverse incremental with optimal compression and change block tracking, is this a normal setup for an exchange server? Maybe we should not be using change block tracking?

Thanks in advance.

Vitaliy S.
Product Manager
Posts: 22984
Liked: 1556 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow VM backup with SQL Server 2008 R2

Post by Vitaliy S. » May 10, 2011 9:47 am

Hi Thomas,

I would say it is expected to have slower backups for VMs with lots of changed blocks during the day. By the way could you tell me what is your destination target for an Exchange backup job?

Thanks.

Tom Bale
Novice
Posts: 8
Liked: never
Joined: Mar 08, 2011 10:11 am
Full Name: Thomas Bale
Contact:

Re: Slow VM backup with SQL Server 2008 R2

Post by Tom Bale » May 10, 2011 10:30 am

Hi Vitaliy,

The backups are being stored onto local disks on the backup server. It is a physical dedicated backup server if that makes any difference. The machine is running a RAID 5 with roughly 6 disks but other back jobs are much faster and not running at the same time.

Ta

Vitaliy S.
Product Manager
Posts: 22984
Liked: 1556 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow VM backup with SQL Server 2008 R2

Post by Vitaliy S. » May 10, 2011 2:31 pm

Thomas, you may want to try switching to Forward Incremental backup mode, should improve the overall performance. Moreover, taking into consideration that CBT is enabled and only changed blocks are transferred, I can say that your Exchange server is fairly busy.

Could you tell me please the size of the rollbacks for this VM? By the way, you may want to check our your backup server resources/performance... What is the CPU usage during the job? If it hits the limit, you should either add more CPU resources (vCPUs) or configure this job with less compression settings, will help.

Anyway, reading back through Anton's reply above, it is expected to have slower performance rates for such VMs as Exchange/SQL Servers.

Tom Bale
Novice
Posts: 8
Liked: never
Joined: Mar 08, 2011 10:11 am
Full Name: Thomas Bale
Contact:

Re: Slow VM backup with SQL Server 2008 R2

Post by Tom Bale » May 11, 2011 3:05 pm

OK, Forward Incremental may be worth trying, would you expect to use more disk space this way?

Rollbacks are between 25GB and 40GB, is this considered large?

In terms of system resources with the Exchange jobs running it is generally sat at around 50%

Thanks

Vitaliy S.
Product Manager
Posts: 22984
Liked: 1556 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow VM backup with SQL Server 2008 R2

Post by Vitaliy S. » May 11, 2011 4:09 pm

Tom Bale wrote:OK, Forward Incremental may be worth trying, would you expect to use more disk space this way?
Yes, depending on your retention policy settings you will have at least two full backups (VBKs), but on the other hand the overall job performance should be much higher due to the fact that with forward incrementals you have three times less I/O, and thus much lower IOPS.
Tom Bale wrote:Rollbacks are between 25GB and 40GB, is this considered large?
No, that is not much for an Exchange server.

slidek9
Enthusiast
Posts: 44
Liked: never
Joined: Apr 20, 2010 5:00 pm
Full Name: Leon
Contact:

Re: Slow VM backup with SQL Server 2008 R2

Post by slidek9 » Jul 01, 2011 3:43 pm

Linking my thread here as we saw the same and this was my solution..
A script for backing up the latest .VBK file

stephengough
Lurker
Posts: 1
Liked: never
Joined: Jul 13, 2011 2:43 am
Full Name: Stephen Gough
Contact:

Slow Backups - With Virtual RDM

Post by stephengough » Jul 13, 2011 3:06 am

[merged]

Setup - Veeam installed on a physical machine, single quad core processor, iSCSI HBA, separate SAN switches, iSCSI SAN.
Backing up Exchange server with virtual RDM taking up to 3 days to complete.
Backup using SAN mode, CBT is enabled and the backup job does complete.
Snapshots a set to be created on separate volume with 4mb Block size
My thoughts are that it is attempting a full backup each run, but not sure how to debug further. any thoughts would be appreciated

Vitaliy S.
Product Manager
Posts: 22984
Liked: 1556 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow VM backup with SQL Server 2008 R2

Post by Vitaliy S. » Jul 13, 2011 8:49 am

Stephen,

Please look through this topic for more details why this behavior might be considered as expected one.

Also here is a good description of basic general troubleshooting steps that should be taken if you have a performance issue:
New user, need help optimizing backups

Thanks.

smgj
Lurker
Posts: 2
Liked: never
Joined: Jul 19, 2011 7:25 am
Full Name: Selma M. Gjeset
Contact:

Very slow performance on specific servers

Post by smgj » Jul 22, 2011 10:16 am

[merged]

I have a backup job with 7 VMs that have the same OS (2008 R2 installed on C: drive), all are exchange servers and all have a D: drive with exchange store and a E: drive with the daily store dump - store.bkf. D and E: drives are typically large drives due to large amount of mail data.

So - I have servers which are similar according to the C: drive, but that are dissimilare according to D: and E:. And the data on E: changes daily.

Which options would be best for the performance for this job (I mainly wish a faster backup, but I also do not want bloating):
- enable or disable storage deduplication? (If enabled - which compression and optimization would be best?)
- enable or disable vSphere changed block tracking?

Vitaliy S.
Product Manager
Posts: 22984
Liked: 1556 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Very slow performance on specific servers

Post by Vitaliy S. » Jul 22, 2011 10:41 am

smgj wrote:- enable or disable storage deduplication? (If enabled - which compression and optimization would be best?)
You should be using the default settings to get best performance for your jobs (deduplication enabled, compression optimal).
smgj wrote:- enable or disable vSphere changed block tracking?
In order to have smaller backup windows you definitely want to enable vSphere Changed Block Tracking.

smgj
Lurker
Posts: 2
Liked: never
Joined: Jul 19, 2011 7:25 am
Full Name: Selma M. Gjeset
Contact:

Re: Slow VM backup with SQL Server 2008 R2

Post by smgj » Jul 22, 2011 10:50 am

Thanks. We are running with those enabled now.

We're running reversed incrementals. Would it be beneficial to run standard "forward" incrementals?
What about doing a periodically full backup - would that decrease the size/run time for folowing incrementals? I don't quite understand how all those daily changing data (especially on the E: drives) affect this form of backup.

Vitaliy S.
Product Manager
Posts: 22984
Liked: 1556 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow VM backup with SQL Server 2008 R2

Post by Vitaliy S. » Jul 22, 2011 11:05 am

Yes, switching to forward incremental mode should give better performance rates. With forward incremental mode, there is no difference (for subsequent incremental pass) whether you perform active or synthetic full periodically.

kurt
Enthusiast
Posts: 65
Liked: 2 times
Joined: Jul 07, 2010 9:03 pm
Full Name: Robert
Contact:

1 VM slow backup

Post by kurt » Jul 22, 2011 3:59 pm

[merged]

I have a reverse incremental backup job which runs nightly. Every VM takes less than 45 minutes except for one which takes almost 4 hours and I'm not sure why. It does run sql server but so do two of my other VM's and they don't have this issue. Processing rate was 14MB/s. It's on the same LUN as other VMs which backup much faster.

Is it really because of the amount of changed data or something else? How can I determine what is causing this VM to take so much longer than the others?

kurt
Enthusiast
Posts: 65
Liked: 2 times
Joined: Jul 07, 2010 9:03 pm
Full Name: Robert
Contact:

Re: Slow VM backup with SQL Server 2008 R2

Post by kurt » Jul 22, 2011 4:11 pm

I think I know what my issue is. The server runs a script every night to backup autodesk vault sql and data and it's 23 gig. The script just backs up to the C drive and over writes the previous backup everyday. That data is then copied off the server nightly. I'm guessing that amount of changed data is causing the backup to take so long.

Post Reply

Who is online

Users browsing this forum: No registered users and 29 guests