Slow VM backup with SQL Server 2008 R2

[m.novelli]

Hi friends, I've installed a new Vmware infrastructure with two Dell R610 servers connected to a Dell MD3000i storage (iSCSI)

I'm running vSphere 4.0 U1 and Veeam 4.1.1 with CBT

All VM are backupping at high speed (from 100 to 200 MB/sec after initial full backup) except one: a Windows 2008 R2 + SQL Server 2008 R2
This VM is backupping at 30/40 MB/sec and need from 2 to 3 hours to complete an incremental backup. During Veeam backup the SQL Server is idle. This machine produce every day about 60 GB of backups on local disks, but it's a normal quantity.

Anyone experienced something similar?

Maybe Veeam need to update the VSS integration for SQL Server 2008 R2 as they did for Exchange 2010?

Thanks for any help,

Marco

[Gostev]

Hi Marco, this could be normal for highly transactional Exchange and SQL servers to have slower incremental backup speed, because there are much more incremental data to process. This is caused by the fact that these applications use transaction logs which change a lot of disk blocks, and all these blocks have to be processed with incremental backup. Large amounts of data obviously take longer to update synthethic full with. You can find existing threads on Exchange backups about this forum as well, so this is not some specific to you. So this is something you just have to live this, obviously VMs with large amounts of changed data will always backup slower than VMs with small amount of changes (all other VMs you have).

[m.novelli]

I've finally solved this issue. It was a configuration issue on the storage Dell MD3000i

I have only one Disk Group (7 SAS disks x 450 GB 15k) divided in two LUN

One LUN is dedicated to SQL Server VM and one LUN is for all other VM

One LUN was owned by one RAID Controller Module (aka SP) and one LUN was owned by the other RAID Controller Module

Setting both LUN owned by the same RAID Controlled Module give me lighthing speed also on the SQL Server VM

So: for Dell MD3000i the LUNs inside a Disk Group should be owned by the same RAID Controller Module

Marco

[Vitaliy S.]

Marco, thank you for sharing! This information is very useful.

[Tom Bale]

[merged into existing discussion]

Hi All,

We have now got to a point where our Veeam backup system is running without much interaction which is great but backups for our Exchange server seem to take much longer than expected. This isn't really a support issue because I want to get a feel for how others are finding this and how they have it setup.

I wouldn't say that this is a particularly heavily used exchange server (Windows 2008 R2). but backups are taking roughly 12 hours daily which means that backups are sometimes over-running into our normal opening hours.

We are using reverse incremental with optimal compression and change block tracking, is this a normal setup for an exchange server? Maybe we should not be using change block tracking?

Thanks in advance.

[Vitaliy S.]

Hi Thomas,

I would say it is expected to have slower backups for VMs with lots of changed blocks during the day. By the way could you tell me what is your destination target for an Exchange backup job?

Thanks.

[Tom Bale]

Hi Vitaliy,

The backups are being stored onto local disks on the backup server. It is a physical dedicated backup server if that makes any difference. The machine is running a RAID 5 with roughly 6 disks but other back jobs are much faster and not running at the same time.

Ta

[Vitaliy S.]

Thomas, you may want to try switching to Forward Incremental backup mode, should improve the overall performance. Moreover, taking into consideration that CBT is enabled and only changed blocks are transferred, I can say that your Exchange server is fairly busy.

Could you tell me please the size of the rollbacks for this VM? By the way, you may want to check our your backup server resources/performance... What is the CPU usage during the job? If it hits the limit, you should either add more CPU resources (vCPUs) or configure this job with less compression settings, will help.

Anyway, reading back through Anton's reply above, it is expected to have slower performance rates for such VMs as Exchange/SQL Servers.

[Tom Bale]

OK, Forward Incremental may be worth trying, would you expect to use more disk space this way?

Rollbacks are between 25GB and 40GB, is this considered large?

In terms of system resources with the Exchange jobs running it is generally sat at around 50%

Thanks

[Vitaliy S.]

OK, Forward Incremental may be worth trying, would you expect to use more disk space this way?

Yes, depending on your retention policy settings you will have at least two full backups (VBKs), but on the other hand the overall job performance should be much higher due to the fact that with forward incrementals you have three times less I/O, and thus much lower IOPS.

Rollbacks are between 25GB and 40GB, is this considered large?

No, that is not much for an Exchange server.

[slidek9]

Linking my thread here as we saw the same and this was my solution..
A script for backing up the latest .VBK file

[stephengough]

[merged]

Setup - Veeam installed on a physical machine, single quad core processor, iSCSI HBA, separate SAN switches, iSCSI SAN.
Backing up Exchange server with virtual RDM taking up to 3 days to complete.
Backup using SAN mode, CBT is enabled and the backup job does complete.
Snapshots a set to be created on separate volume with 4mb Block size
My thoughts are that it is attempting a full backup each run, but not sure how to debug further. any thoughts would be appreciated

[Vitaliy S.]

Stephen,

Please look through this topic for more details why this behavior might be considered as expected one.

Also here is a good description of basic general troubleshooting steps that should be taken if you have a performance issue:
New user, need help optimizing backups

Thanks.

[smgj]

[merged]

I have a backup job with 7 VMs that have the same OS (2008 R2 installed on C: drive), all are exchange servers and all have a D: drive with exchange store and a E: drive with the daily store dump - store.bkf. D and E: drives are typically large drives due to large amount of mail data.

So - I have servers which are similar according to the C: drive, but that are dissimilare according to D: and E:. And the data on E: changes daily.

Which options would be best for the performance for this job (I mainly wish a faster backup, but I also do not want bloating):
- enable or disable storage deduplication? (If enabled - which compression and optimization would be best?)
- enable or disable vSphere changed block tracking?

[Vitaliy S.]

- enable or disable storage deduplication? (If enabled - which compression and optimization would be best?)

You should be using the default settings to get best performance for your jobs (deduplication enabled, compression optimal).

- enable or disable vSphere changed block tracking?

In order to have smaller backup windows you definitely want to enable vSphere Changed Block Tracking.

[smgj]

Thanks. We are running with those enabled now.

We're running reversed incrementals. Would it be beneficial to run standard "forward" incrementals?
What about doing a periodically full backup - would that decrease the size/run time for folowing incrementals? I don't quite understand how all those daily changing data (especially on the E: drives) affect this form of backup.

[Vitaliy S.]

Yes, switching to forward incremental mode should give better performance rates. With forward incremental mode, there is no difference (for subsequent incremental pass) whether you perform active or synthetic full periodically.

[kurt]

[merged]

I have a reverse incremental backup job which runs nightly. Every VM takes less than 45 minutes except for one which takes almost 4 hours and I'm not sure why. It does run sql server but so do two of my other VM's and they don't have this issue. Processing rate was 14MB/s. It's on the same LUN as other VMs which backup much faster.

Is it really because of the amount of changed data or something else? How can I determine what is causing this VM to take so much longer than the others?

[kurt]

I think I know what my issue is. The server runs a script every night to backup autodesk vault sql and data and it's 23 gig. The script just backs up to the C drive and over writes the previous backup everyday. That data is then copied off the server nightly. I'm guessing that amount of changed data is causing the backup to take so long.