Comprehensive data protection for all workloads
Post Reply
kurt2439
Influencer
Posts: 13
Liked: never
Joined: Apr 19, 2012 5:14 pm
Full Name: James Chase
Contact:

Improving VM Backup Times

Post by kurt2439 » May 30, 2012 11:44 am

Wondering if anyone has any tips for speeding up windows backups. Perhaps it is Windows 2003 specifically, but backups seems to take an inexplicably long time for these machines compared to our linux machines and even our Windows 2008 machines. I have tried using/not-using VSS and moving the backup times around so that there would be very little IO on both the VM and the datastore. But I can't get the backup times to improve. I have also noticed this in two different VMWare environments so I don't think it's our SAN or network. The linux machines and server 2008 backup in minutes (in general)! Here are some examples of problematic Windows 2003 VMs:

a SQL Server
Size 334GB
Read 22.9 GB
Transferred 4GB
Duration 3:03:49 (yes 3 hours)

another SQL Server
Size 155GB
Read 153 GB
Transferred 2.6GB
Duration 1:44:31

IIS Server (extremely low use)
Size: 60GB
Read 58GB
Transferred 1GB
Duration 48:38

Is there some issue with Windows 2003?

Vitaliy S.
Product Manager
Posts: 22976
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Improving Windows Server 2003 Backup Times

Post by Vitaliy S. » May 30, 2012 12:54 pm

Hi James,

This performance degradation has nothing to do with Windows version deployed on your VMs. What is important here is the application which is installed on the VM you're backing up. Lower job processing rate on incremental runs for highly transaction applications (like SQL Server) is expected, please take a look at this topic for more information: Slow VM backup with SQL Server 2008 R2

Btw, what backup mode (reversed or forward incremental) have you configured for these VMs?

Thanks!

Gostev
SVP, Product Management
Posts: 24788
Liked: 3521 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Improving Windows Server 2003 Backup Times

Post by Gostev » May 30, 2012 12:57 pm

It is important to understand that the content of the backed up VM disks completely does not matter for image-level backup process. Makes no difference if the disks have some OS, or no OS installed, or even simply stuffed by meaningless random data.

You should be looking at the bottleneck statistics for the problematic VMs to determine the root cause for backup slowness.

kurt2439
Influencer
Posts: 13
Liked: never
Joined: Apr 19, 2012 5:14 pm
Full Name: James Chase
Contact:

Re: Improving Windows Server 2003 Backup Times

Post by kurt2439 » May 30, 2012 12:59 pm

SQL server I can somewhat understand and I'll look into that article, but the other machine only has IIS installed and stil takes 50 minutes. I have another machine that is just IIS that takes 50 minutes as well.

I'm using reverse incrementals

Gostev, that's not really true if you're using VSS.

Vitaliy S.
Product Manager
Posts: 22976
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Improving Windows Server 2003 Backup Times

Post by Vitaliy S. » May 30, 2012 1:04 pm

If you switch to forward incremental you will improve your current backup job performance rates. Here is an existing topic for further reading:
Reversed incrementals vs. Fwd w/ DAILY Synthetic + Transform

Hope this helps!

Gostev
SVP, Product Management
Posts: 24788
Liked: 3521 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Improving Windows Server 2003 Backup Times

Post by Gostev » May 30, 2012 1:05 pm

kurt2439 wrote:Gostev, that's not really true if you're using VSS.
VSS is completely irrelevant to the data moving process (which statistics you have posted). All VSS processing happens before the VM snapshot is created, and VM data processing starts after that.

Also, I noticed that "another SQL Server" and "IIS Server" do not use changed block tracking according to what you have posted above (specifically, read counter). This is the first issue you should be looking to resolve to improve performance. The "a SQL Server" clearly does use CBT though, and for that one you should be looking at the bottleneck statistics for this VM to determine the root cause for backup slowness.

kurt2439
Influencer
Posts: 13
Liked: never
Joined: Apr 19, 2012 5:14 pm
Full Name: James Chase
Contact:

Re: Improving Windows Server 2003 Backup Times

Post by kurt2439 » May 30, 2012 1:09 pm

Gostev wrote:Also, I noticed that "another SQL Server" and "IIS Server" do not use changed block tracking according to what you have posted above (specifically, read counter). This is the first issue you should be looking to resolve to improve performance. The "a SQL Server" clearly does use CBT though, and for that one you should be looking at the bottleneck statistics for this VM to determine the root cause for backup slowness.
Interesting idea. I checked the settings and CBT is indeed active all of those VM's. Is there some other reason that would cause Veeam to read through so much data?

kurt2439
Influencer
Posts: 13
Liked: never
Joined: Apr 19, 2012 5:14 pm
Full Name: James Chase
Contact:

Re: Improving Windows Server 2003 Backup Times

Post by kurt2439 » May 30, 2012 1:13 pm

Vitaliy S. wrote:If you switch to forward incremental you will improve your current backup job performance rates.
OK, I will give this a shot on one of my VMs and see how this works out!

Gostev
SVP, Product Management
Posts: 24788
Liked: 3521 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Improving Windows Server 2003 Backup Times

Post by Gostev » May 30, 2012 1:13 pm

When something updates each and every disk block on those 2 VMs every single day, making CBT mark every block as changed ;)
Just to check, what exactly do you mean when you say "CBT is indeed active all of those VM's"?

kurt2439
Influencer
Posts: 13
Liked: never
Joined: Apr 19, 2012 5:14 pm
Full Name: James Chase
Contact:

Re: Improving Windows Server 2003 Backup Times

Post by kurt2439 » May 30, 2012 1:36 pm

Lol, Hmm. That would be quite the process.

I went into Veeam and checked the Advanced settings for the backup job and made sure it was using CBT for the backups. Is it possible it's disabled at the VMware level somehow? Let's see. Ahhh, yes it didn't have a ctk file! Strange. I wonder how that got disabled. Thanks for pointing me in that direction!

I'll see how my VM backups go now. I changed the machine that was taking 3 hours to backup to use forward incremental, and I enabled CBT on the machines that you suggested were not using this.

Gostev
SVP, Product Management
Posts: 24788
Liked: 3521 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Improving Windows Server 2003 Backup Times

Post by Gostev » May 30, 2012 1:50 pm

Whether CBT is enabled on a VM or in the job settings, what you are really want to see is if CBT data is actually being leveraged for the specific virtual disk of the specific VM. This information is available in the corresponding job session's details. Expand this dialog to "show more", select the VM to the left, and look for [CBT] mark next to each virtual disk in the log to the right. If you don't see these letters, then CBT data is not being leveraged.

For example, if you have just enabled CBT on a VM, the very first job run after that will not leverage CBT yet (but the following runs will).

kurt2439
Influencer
Posts: 13
Liked: never
Joined: Apr 19, 2012 5:14 pm
Full Name: James Chase
Contact:

Re: Improving VM Backup Times

Post by kurt2439 » May 30, 2012 2:11 pm

K. I see what you mean and I will use that as my marker for success in the following backups.

chrisdearden
Expert
Posts: 1530
Liked: 225 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: Improving VM Backup Times

Post by chrisdearden » May 30, 2012 4:32 pm

Which version of Virtual hardware are those Machines on? 4 or 7?

kurt2439
Influencer
Posts: 13
Liked: never
Joined: Apr 19, 2012 5:14 pm
Full Name: James Chase
Contact:

Re: Improving VM Backup Times

Post by kurt2439 » May 30, 2012 5:24 pm

Another good point. They are version 4. I hope my manually enabling CBT doesn't cause corruption. Guess I can just restore them from a backup if so -- they are just static file servers.

Vitaliy S.
Product Manager
Posts: 22976
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Improving VM Backup Times

Post by Vitaliy S. » May 30, 2012 5:57 pm

Oh, that explains why your jobs are so slow. Be aware that your VMs should be running at least on hardware version 7 to leverage VMware changed block tracking mechanism.

kurt2439
Influencer
Posts: 13
Liked: never
Joined: Apr 19, 2012 5:14 pm
Full Name: James Chase
Contact:

Re: Improving VM Backup Times

Post by kurt2439 » May 30, 2012 6:10 pm

It explains some of them for sure. The SQL Server that takes 3 - 5 hours to backup is V7 and does use CBT.

Vitaliy S.
Product Manager
Posts: 22976
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Improving VM Backup Times

Post by Vitaliy S. » May 30, 2012 6:39 pm

....but it does change a lot of small block on its virtual disks ;) Try forward incremental backup mode, should help.

chrisdearden
Expert
Posts: 1530
Liked: 225 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: Improving VM Backup Times

Post by chrisdearden » May 30, 2012 8:04 pm

kurt2439 wrote:It explains some of them for sure. The SQL Server that takes 3 - 5 hours to backup is V7 and does use CBT.
Its still possible that CBT isn't working for some reason - especially on the web server you should not have that level of changed blocks.

have a look at the following Vmware Kb on CBT and check if you have the relevant entries in your VMX and VMDK files.
http://kb.vmware.com/selfservice/micros ... Id=1020128

Gostev
SVP, Product Management
Posts: 24788
Liked: 3521 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Improving VM Backup Times

Post by Gostev » May 31, 2012 4:44 pm

If he can see [CBT] next to the virtual disk in the session log, than it is working for sure.

pendragoncrw
Enthusiast
Posts: 38
Liked: 3 times
Joined: Jun 14, 2010 3:06 am
Full Name: C White
Contact:

Re: Improving VM Backup Times

Post by pendragoncrw » Jun 04, 2012 2:53 am

On our Veeam's, I have an automated script that runs before each full-backup which has reduced our total backup sizes and times significantly.

The script does the following things:
1. Runs the command line version of Auslogics on all fixed drives in "defrag and optimize"
2. Runs sdelete in the virtual machine optimization mode on all fixed drives.
Note: This should not be used on thin-provisioned disks because it will inflate them to the fully alloted size.

Generally, we have seen reductions of 6-8 hours and 150-200gb on a full backup of 25 VM's.

scott_mac
Enthusiast
Posts: 31
Liked: 1 time
Joined: Aug 18, 2011 2:35 pm
Full Name: Scott Mckenzie
Contact:

Re: Improving VM Backup Times

Post by scott_mac » Jun 07, 2012 8:11 am

I had exactly the same issue with regard to slow times, it was down to version 4 hardware for the VM - but we were told to use this on SBS2003 rather than version 7 as apparently that can cause issues :(

Hardly a practical solution I know but an upgrade to 2008 does make a massive difference... and don't forget that a single Server 2008 license can legitimately be used to license several servers running in a virtual environment - Microsoft in being kind for once!

Vitaliy S.
Product Manager
Posts: 22976
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Improving VM Backup Times

Post by Vitaliy S. » Jun 07, 2012 8:44 am

scott_mac wrote:but we were told to use this on SBS2003 rather than version 7 as apparently that can cause issues
Never heard about this limitation of SBS 2003, how did you end up with this? Are you still running this server on hardware version 4?

scott_mac
Enthusiast
Posts: 31
Liked: 1 time
Joined: Aug 18, 2011 2:35 pm
Full Name: Scott Mckenzie
Contact:

Re: Improving VM Backup Times

Post by scott_mac » Jun 07, 2012 11:49 am

Vitaliy S. wrote: Never heard about this limitation of SBS 2003, how did you end up with this? Are you still running this server on hardware version 4?
Sorry I should have expanded on that... I never found any information to back that up either but every time we tried to upgrade the hardware to version 7, the system failed to boot. It may have been a local problem with our installation.

Thankfully we've now upgraded to SBS2011 and CBT moves things along nice and quickly.

Post Reply

Who is online

Users browsing this forum: Bing [Bot], Hucketr88 and 28 guests