Full Active Backup slowing to a crawl

[tk330]

I am backing up a vSphere 5.5 environment on an EMC SAN using Veeam (9.5.01038) to a Synology RS816 NAS over CIFS (gigabit). The Veeam proxy is virtual and is the same as the Veeam server. It has 4 CPUs with 24GB of RAM.

The backups are running fine, however, we have a large 2008R2 file server (3.4TB) that is taking forever to complete. We start an Active Full Backup on Saturday evenings expecting it to be finished by Monday in the AM. The job is running up to 80-90 hours and goes into mid-week. The daily incrementals never take longer than 15-20 minutes.

The VM of the file server contains 6 disks of different sizes (80GB, 300GB, 1.3TB, 1.0TB 100GB, 750GB). The 80GB OS disk finishes fast and reads at 100+MB/sec. As the backup moves to the next disk, the read speed cuts in half (50-75MB/sec) and by the 5th and 6th disks, the read is down to 5-10MB/sec, which is why the backup is taking forever to finish.

I opened a case with support and we have not found a solution yet. I have reset the CBT on the VMs, turned off indexing and created brand new backup jobs. Support had me create 3 new jobs with 2 disks in each job using exclusions. Surprisingly, in these 3 jobs, the first disk always had a read speed of 80-100+MB/sec and the second disk slowed to a crawl (5-10MB/sec).

We have another SAN and VMware/Veeam/Synology install at another location and are not experiencing these slowdowns. I am hesitant to blame the Synology NAS since the speed is fine elsewhere with other VMs. If there was a networking/SAN issue, why is the first disk and other VMs backing up much faster? It is using the same storage, SAN, networking, etc...

The 3.4TB file share is by far our largest server. This job used to finish under 30 hours a few months ago and we're clueless on why it is now crawling.

Any idea why a job would get progressively worse as the backup runs? New disks have not been added or extended.

I was considering changing over from CIFS to iSCSI on the Synology but I do not have the space available to relocate the existing backups and repurpose the volume. My next is possibly creating 6 new jobs with one disk in each to hopefully get a fast read speed but that seems more like a band aid than a fix.

[Dima P.]

Hi tk330,

I opened a case with support and we have not found a solution yet.

Can you please update this thread with your case ID? Thanks in advance.

[tk330]

Case 02404987 - Performance issue on a backup

[Gostev]

As far as I remember, the most typical root cause for this behavior is backup server being installed on an old Windows Server version, which handles system cache poorly. There's a big existing topic here with more details... if this is not the case, then wait to see what our support comes back with. Thanks!

[tk330]

The server is on 2008 R2 so it is older

I just found out that if I browse to the CIFS share where the backup is located and press F5 a few times to refresh, the read speed in Veeam jumps back up to 60-80MB/sec. I had a constant ping to the NAS and had no network drops and there are no power/hibernation settings in effect. I'm not sure if this is a nuance of using CIFS or not.