Improving VM Backup Times

[kurt2439]

Wondering if anyone has any tips for speeding up windows backups. Perhaps it is Windows 2003 specifically, but backups seems to take an inexplicably long time for these machines compared to our linux machines and even our Windows 2008 machines. I have tried using/not-using VSS and moving the backup times around so that there would be very little IO on both the VM and the datastore. But I can't get the backup times to improve. I have also noticed this in two different VMWare environments so I don't think it's our SAN or network. The linux machines and server 2008 backup in minutes (in general)! Here are some examples of problematic Windows 2003 VMs:

a SQL Server
Size 334GB
Read 22.9 GB
Transferred 4GB
Duration 3:03:49 (yes 3 hours)

another SQL Server
Size 155GB
Read 153 GB
Transferred 2.6GB
Duration 1:44:31

IIS Server (extremely low use)
Size: 60GB
Read 58GB
Transferred 1GB
Duration 48:38

Is there some issue with Windows 2003?

[Vitaliy S.]

Hi James,

This performance degradation has nothing to do with Windows version deployed on your VMs. What is important here is the application which is installed on the VM you're backing up. Lower job processing rate on incremental runs for highly transaction applications (like SQL Server) is expected, please take a look at this topic for more information: Slow VM backup with SQL Server 2008 R2

Btw, what backup mode (reversed or forward incremental) have you configured for these VMs?

Thanks!

[Gostev]

It is important to understand that the content of the backed up VM disks completely does not matter for image-level backup process. Makes no difference if the disks have some OS, or no OS installed, or even simply stuffed by meaningless random data.

You should be looking at the bottleneck statistics for the problematic VMs to determine the root cause for backup slowness.

[kurt2439]

SQL server I can somewhat understand and I'll look into that article, but the other machine only has IIS installed and stil takes 50 minutes. I have another machine that is just IIS that takes 50 minutes as well.

I'm using reverse incrementals

Gostev, that's not really true if you're using VSS.

[Vitaliy S.]

If you switch to forward incremental you will improve your current backup job performance rates. Here is an existing topic for further reading:
Reversed incrementals vs. Fwd w/ DAILY Synthetic + Transform

Hope this helps!

[Gostev]

Gostev, that's not really true if you're using VSS.

VSS is completely irrelevant to the data moving process (which statistics you have posted). All VSS processing happens before the VM snapshot is created, and VM data processing starts after that.

Also, I noticed that "another SQL Server" and "IIS Server" do not use changed block tracking according to what you have posted above (specifically, read counter). This is the first issue you should be looking to resolve to improve performance. The "a SQL Server" clearly does use CBT though, and for that one you should be looking at the bottleneck statistics for this VM to determine the root cause for backup slowness.

[kurt2439]

Also, I noticed that "another SQL Server" and "IIS Server" do not use changed block tracking according to what you have posted above (specifically, read counter). This is the first issue you should be looking to resolve to improve performance. The "a SQL Server" clearly does use CBT though, and for that one you should be looking at the bottleneck statistics for this VM to determine the root cause for backup slowness.

Interesting idea. I checked the settings and CBT is indeed active all of those VM's. Is there some other reason that would cause Veeam to read through so much data?

[kurt2439]

If you switch to forward incremental you will improve your current backup job performance rates.

OK, I will give this a shot on one of my VMs and see how this works out!

[Gostev]

When something updates each and every disk block on those 2 VMs every single day, making CBT mark every block as changed
Just to check, what exactly do you mean when you say "CBT is indeed active all of those VM's"?

[kurt2439]

Lol, Hmm. That would be quite the process.

I went into Veeam and checked the Advanced settings for the backup job and made sure it was using CBT for the backups. Is it possible it's disabled at the VMware level somehow? Let's see. Ahhh, yes it didn't have a ctk file! Strange. I wonder how that got disabled. Thanks for pointing me in that direction!

I'll see how my VM backups go now. I changed the machine that was taking 3 hours to backup to use forward incremental, and I enabled CBT on the machines that you suggested were not using this.

[Gostev]

Whether CBT is enabled on a VM or in the job settings, what you are really want to see is if CBT data is actually being leveraged for the specific virtual disk of the specific VM. This information is available in the corresponding job session's details. Expand this dialog to "show more", select the VM to the left, and look for [CBT] mark next to each virtual disk in the log to the right. If you don't see these letters, then CBT data is not being leveraged.

For example, if you have just enabled CBT on a VM, the very first job run after that will not leverage CBT yet (but the following runs will).

[kurt2439]

K. I see what you mean and I will use that as my marker for success in the following backups.

[chrisdearden]

Which version of Virtual hardware are those Machines on? 4 or 7?

[kurt2439]

Another good point. They are version 4. I hope my manually enabling CBT doesn't cause corruption. Guess I can just restore them from a backup if so -- they are just static file servers.

[Vitaliy S.]

Oh, that explains why your jobs are so slow. Be aware that your VMs should be running at least on hardware version 7 to leverage VMware changed block tracking mechanism.

[kurt2439]

It explains some of them for sure. The SQL Server that takes 3 - 5 hours to backup is V7 and does use CBT.

[Vitaliy S.]

....but it does change a lot of small block on its virtual disks Try forward incremental backup mode, should help.

[chrisdearden]

It explains some of them for sure. The SQL Server that takes 3 - 5 hours to backup is V7 and does use CBT.

Its still possible that CBT isn't working for some reason - especially on the web server you should not have that level of changed blocks.

have a look at the following Vmware Kb on CBT and check if you have the relevant entries in your VMX and VMDK files.
http://kb.vmware.com/selfservice/micros ... Id=1020128

[Gostev]

If he can see [CBT] next to the virtual disk in the session log, than it is working for sure.

[pendragoncrw]

On our Veeam's, I have an automated script that runs before each full-backup which has reduced our total backup sizes and times significantly.

The script does the following things:
1. Runs the command line version of Auslogics on all fixed drives in "defrag and optimize"
2. Runs sdelete in the virtual machine optimization mode on all fixed drives.
Note: This should not be used on thin-provisioned disks because it will inflate them to the fully alloted size.

Generally, we have seen reductions of 6-8 hours and 150-200gb on a full backup of 25 VM's.

[scott_mac]

I had exactly the same issue with regard to slow times, it was down to version 4 hardware for the VM - but we were told to use this on SBS2003 rather than version 7 as apparently that can cause issues

Hardly a practical solution I know but an upgrade to 2008 does make a massive difference... and don't forget that a single Server 2008 license can legitimately be used to license several servers running in a virtual environment - Microsoft in being kind for once!

[Vitaliy S.]

but we were told to use this on SBS2003 rather than version 7 as apparently that can cause issues

Never heard about this limitation of SBS 2003, how did you end up with this? Are you still running this server on hardware version 4?

[scott_mac]

Never heard about this limitation of SBS 2003, how did you end up with this? Are you still running this server on hardware version 4?

Sorry I should have expanded on that... I never found any information to back that up either but every time we tried to upgrade the hardware to version 7, the system failed to boot. It may have been a local problem with our installation.

Thankfully we've now upgraded to SBS2011 and CBT moves things along nice and quickly.