Backup Slowdown Over Time

VMware specific discussions

Backup Slowdown Over Time

Veeam Logoby stmux » Fri Jul 17, 2015 11:23 pm

2-ESXi 5.5.0.2718055 Hosts
vCenter Server 5.5.0.2646482
2-Intel 82571 EB Dual NICs in each host
Veeam 8.0.0.2029


Each ESXi host has four active VMs all of which are being backed up with Veeam B&R. When the VM Hosts are first booted backup speeds for each VM are anywhere from 50MB/s to 80MB/s. After the hosts have been running for 60 days the backup speed drops to below 25MB/s. I have seen some backing up as slow as 9MB/s. Reboot the hosts and backup speeds go back to the high numbers. FYI, Scheduled backups are set to run sequentially with only one VM being backed up at a time so there is no slowdown due to multiple backups at the same time. Any thoughts about what might be going on?
Thanks,
mux
stmux
Enthusiast
 
Posts: 29
Liked: never
Joined: Fri May 24, 2013 2:51 pm
Full Name: ja

Re: Backup Slowdown Over Time

Veeam Logoby Andreas Neufert » Sat Jul 18, 2015 8:44 pm

Please check the Job Statistics. Which Backup Mode do you run? NBD/HotAdd/DirectSAN?
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: Backup Slowdown Over Time

Veeam Logoby Vitaliy S. » Sun Jul 19, 2015 9:28 pm

Also what's your destination target and retention policy for the backup job?
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Backup Slowdown Over Time

Veeam Logoby stmux » Mon Jul 20, 2015 2:46 am

I don't see any of these things in the Job Properties. NBD/HotAdd/DirectSAN
stmux
Enthusiast
 
Posts: 29
Liked: never
Joined: Fri May 24, 2013 2:51 pm
Full Name: ja

Re: Backup Slowdown Over Time

Veeam Logoby stmux » Mon Jul 20, 2015 2:48 am

Destination repository is a local drive and retention is 14 days.
stmux
Enthusiast
 
Posts: 29
Liked: never
Joined: Fri May 24, 2013 2:51 pm
Full Name: ja

Re: Backup Slowdown Over Time

Veeam Logoby Andreas Neufert » Mon Jul 20, 2015 6:55 am

Pleas go to the job. right click on it - Statistics - then select a vm - on each disk you can see on of the 3 mode NBD/HotAdd/DirectSAN.

Maybe it is a good idea to speak with one of our Gold Partners and let them review the installation.
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: Backup Slowdown Over Time

Veeam Logoby Vitaliy S. » Mon Jul 20, 2015 11:57 am

And bottleneck stats would be helpful as well for the run when you have decent performance and when it is slowed down.
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Backup Slowdown Over Time

Veeam Logoby PTide » Mon Jul 20, 2015 12:12 pm

I'd also check the performance tab during last month on both of your hosts to see if there is a slope starting somewhere toward present day.
By the way - have you checked if VM hard resetting makes backups run faster for that VM?

Thank you.
PTide
Veeam Software
 
Posts: 3021
Liked: 246 times
Joined: Tue May 19, 2015 1:46 pm

Re: Backup Slowdown Over Time

Veeam Logoby Vitaliy S. » Mon Jul 20, 2015 1:47 pm

If you have Veeam ONE deployed, then generating backup job historical information would be useful in our troubleshooting process.
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Backup Slowdown Over Time

Veeam Logoby Andreas Neufert » Mon Jul 20, 2015 3:08 pm

Or maybe contact our support and let them have a look.
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: Backup Slowdown Over Time

Veeam Logoby stmux » Mon Jul 27, 2015 12:41 am

Andreas Neufert wrote:Pleas go to the job. right click on it - Statistics - then select a vm - on each disk you can see on of the 3 mode NBD/HotAdd/DirectSAN.

Maybe it is a good idea to speak with one of our Gold Partners and let them review the installation.



Below is from properties of a Server 2k8 vm. Looks like NBD. Bottleneck shows source on this one, however, some show bottleneck as target, as well. Note, this is with host uptime at 11 days. I will track these every week to see if there is a pattern. I will also check host performance over time. Thing is, this occurs on two hosts that are identical, hardware and OS/current vmware updates. The only difference is the vm's that run on the hosts. This host was running during backup. Some of the vm's are down during backup and even those slow down over time, if that means anything.
I appreciate the feedback and suggestions. mux
Code: Select all
7/25/2015 8:30:52 PM :: Queued for processing at 7/25/2015 8:30:52 PM
7/25/2015 8:30:53 PM :: Required backup infrastructure resources have been assigned
7/25/2015 8:30:53 PM :: VM processing started at 7/25/2015 8:30:53 PM
7/25/2015 8:30:53 PM :: VM size: 60.0 GB
7/25/2015 8:30:54 PM :: Getting VM info from vSphere
7/25/2015 8:31:00 PM :: Inventorying guest system
7/25/2015 8:31:02 PM :: Preparing guest for hot backup
7/25/2015 8:31:05 PM :: Creating snapshot
7/25/2015 8:31:09 PM :: Releasing guest
7/25/2015 8:31:17 PM :: Indexing guest file system
7/25/2015 8:31:22 PM :: Saving [esxi51-2-01] SF2k8DC-2/SF2k8DC-2.vmx
7/25/2015 8:31:23 PM :: Saving [esxi51-2-01] SF2k8DC-2/SF2k8DC-2.vmxf
7/25/2015 8:31:24 PM :: Saving [esxi51-2-01] SF2k8DC-2/SF2k8DC-2.nvram
7/25/2015 8:31:26 PM :: Using backup proxy VMware Backup Proxy for disk Hard disk 1 [nbd]
7/25/2015 8:31:28 PM :: Hard disk 1 (60.0 GB) 2.6 GB read at 51 MB/s [CBT]
7/25/2015 8:31:47 PM :: Getting list of guest file system local users
7/25/2015 8:32:36 PM :: Saving GuestIndexData.zip
7/25/2015 8:32:37 PM :: Saving GuestMembers.xml
7/25/2015 8:32:39 PM :: Removing VM snapshot
7/25/2015 8:33:05 PM :: Publishing guest file system index to catalog
7/25/2015 8:33:07 PM :: Finalizing
7/25/2015 8:33:07 PM :: Truncating transaction logs
7/25/2015 8:33:09 PM :: Truncating SQL server transaction logs
7/25/2015 8:33:19 PM :: Busy: Source 84% > Proxy 43% > Network 34% > Target 80%
7/25/2015 8:33:19 PM :: Primary bottleneck: Source
7/25/2015 8:33:19 PM :: Network traffic verification detected no corrupted blocks
7/25/2015 8:33:19 PM :: Processing finished at 7/25/2015 8:33:19 PM
stmux
Enthusiast
 
Posts: 29
Liked: never
Joined: Fri May 24, 2013 2:51 pm
Full Name: ja

Re: Backup Slowdown Over Time

Veeam Logoby Andreas Neufert » Fri Jul 31, 2015 2:41 pm

Hi Jay,

I am just guessing.
Did you use forward incremental without syntetic full? If so, you will have smaller runs till you reached the point where oldest restore point needs to be transformed.

As well I can see that you use NBD mode. This mode can work from VMware sied up to 40% of the VMwere VMKernel interface. Do you have 10GbE on the VMKernel interface?

From Statistics you can see that Target has as well a high load. What do you use as target with what Raid Configuration and Disks.

You see that it is really hard to gather all the needed information together in this forum format. Maybe better contact one of our more experienced partners and they can help you with a analyse.
Or contact your local Veeam SE or Inside SE.
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM


Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 13 guests