Backup Slowdown Over Time

[stmux]

2-ESXi 5.5.0.2718055 Hosts
vCenter Server 5.5.0.2646482
2-Intel 82571 EB Dual NICs in each host
Veeam 8.0.0.2029


Each ESXi host has four active VMs all of which are being backed up with Veeam B&R. When the VM Hosts are first booted backup speeds for each VM are anywhere from 50MB/s to 80MB/s. After the hosts have been running for 60 days the backup speed drops to below 25MB/s. I have seen some backing up as slow as 9MB/s. Reboot the hosts and backup speeds go back to the high numbers. FYI, Scheduled backups are set to run sequentially with only one VM being backed up at a time so there is no slowdown due to multiple backups at the same time. Any thoughts about what might be going on?
Thanks,
mux

[Andreas Neufert]

Please check the Job Statistics. Which Backup Mode do you run? NBD/HotAdd/DirectSAN?

[Vitaliy S.]

Also what's your destination target and retention policy for the backup job?

[stmux]

I don't see any of these things in the Job Properties. NBD/HotAdd/DirectSAN

[stmux]

Destination repository is a local drive and retention is 14 days.

[Andreas Neufert]

Pleas go to the job. right click on it - Statistics - then select a vm - on each disk you can see on of the 3 mode NBD/HotAdd/DirectSAN.

Maybe it is a good idea to speak with one of our Gold Partners and let them review the installation.

[Vitaliy S.]

And bottleneck stats would be helpful as well for the run when you have decent performance and when it is slowed down.

[PTide]

I'd also check the performance tab during last month on both of your hosts to see if there is a slope starting somewhere toward present day.
By the way - have you checked if VM hard resetting makes backups run faster for that VM?

Thank you.

[Vitaliy S.]

If you have Veeam ONE deployed, then generating backup job historical information would be useful in our troubleshooting process.

[Andreas Neufert]

Or maybe contact our support and let them have a look.

[stmux]

Pleas go to the job. right click on it - Statistics - then select a vm - on each disk you can see on of the 3 mode NBD/HotAdd/DirectSAN.

Maybe it is a good idea to speak with one of our Gold Partners and let them review the installation.

Below is from properties of a Server 2k8 vm. Looks like NBD. Bottleneck shows source on this one, however, some show bottleneck as target, as well. Note, this is with host uptime at 11 days. I will track these every week to see if there is a pattern. I will also check host performance over time. Thing is, this occurs on two hosts that are identical, hardware and OS/current vmware updates. The only difference is the vm's that run on the hosts. This host was running during backup. Some of the vm's are down during backup and even those slow down over time, if that means anything.
I appreciate the feedback and suggestions. mux

Code: Select all

7/25/2015 8:30:52 PM :: Queued for processing at 7/25/2015 8:30:52 PM 
7/25/2015 8:30:53 PM :: Required backup infrastructure resources have been assigned 
7/25/2015 8:30:53 PM :: VM processing started at 7/25/2015 8:30:53 PM 
7/25/2015 8:30:53 PM :: VM size: 60.0 GB 
7/25/2015 8:30:54 PM :: Getting VM info from vSphere 
7/25/2015 8:31:00 PM :: Inventorying guest system 
7/25/2015 8:31:02 PM :: Preparing guest for hot backup 
7/25/2015 8:31:05 PM :: Creating snapshot 
7/25/2015 8:31:09 PM :: Releasing guest 
7/25/2015 8:31:17 PM :: Indexing guest file system 
7/25/2015 8:31:22 PM :: Saving [esxi51-2-01] SF2k8DC-2/SF2k8DC-2.vmx 
7/25/2015 8:31:23 PM :: Saving [esxi51-2-01] SF2k8DC-2/SF2k8DC-2.vmxf 
7/25/2015 8:31:24 PM :: Saving [esxi51-2-01] SF2k8DC-2/SF2k8DC-2.nvram 
7/25/2015 8:31:26 PM :: Using backup proxy VMware Backup Proxy for disk Hard disk 1 [nbd] 
7/25/2015 8:31:28 PM :: Hard disk 1 (60.0 GB) 2.6 GB read at 51 MB/s [CBT]
7/25/2015 8:31:47 PM :: Getting list of guest file system local users 
7/25/2015 8:32:36 PM :: Saving GuestIndexData.zip 
7/25/2015 8:32:37 PM :: Saving GuestMembers.xml 
7/25/2015 8:32:39 PM :: Removing VM snapshot 
7/25/2015 8:33:05 PM :: Publishing guest file system index to catalog 
7/25/2015 8:33:07 PM :: Finalizing 
7/25/2015 8:33:07 PM :: Truncating transaction logs 
7/25/2015 8:33:09 PM :: Truncating SQL server transaction logs 
7/25/2015 8:33:19 PM :: Busy: Source 84% > Proxy 43% > Network 34% > Target 80% 
7/25/2015 8:33:19 PM :: Primary bottleneck: Source 
7/25/2015 8:33:19 PM :: Network traffic verification detected no corrupted blocks 
7/25/2015 8:33:19 PM :: Processing finished at 7/25/2015 8:33:19 PM

[Andreas Neufert]

Hi Jay,

I am just guessing.
Did you use forward incremental without syntetic full? If so, you will have smaller runs till you reached the point where oldest restore point needs to be transformed.

As well I can see that you use NBD mode. This mode can work from VMware sied up to 40% of the VMwere VMKernel interface. Do you have 10GbE on the VMKernel interface?

From Statistics you can see that Target has as well a high load. What do you use as target with what Raid Configuration and Disks.

You see that it is really hard to gather all the needed information together in this forum format. Maybe better contact one of our more experienced partners and they can help you with a analyse.
Or contact your local Veeam SE or Inside SE.