Very slow backup - throttling is off

[solmssen]

Hi all - using Veeam here and there on servers and clients to backup to local drives and it's been good. I installed it on this machine (Dell R720xd, Windows 2008 R2, Hyper-V, PERC H710Mini). Machine is running one VM at this point, a file server that's not terribly busy. All drives are SAS, C is two 15K 300GBs RAID1, the others are two 7.2K 4TB RAID1. Backing up to an SMB share on a Synology RackStation over 1GB Ethernet. I turned throttling off during the job, is it immediate or does it wait until after the currently executing job completes.

Perc H710 mini info Firmware Version 21.3.4-0001 Driver Version 6.805.03.00 Storport Driver Version 6.1.7601.23403

Performance is wacky. Was using Symantec System Recovery 2013 and had pretty good performance - a full backup was about 30-40 hours. Changed to Veeam because it's on other machines at this site and trying to standardize, and got this on first backup, which is still not done.



Any thoughts? The VM is on 4 VHD files on D and E, which are the slow ones. Resource Monitor says the disk and network are nowhere near saturated. Veeam backups on a very similarly configured R420 and R530 running busy VMs to the same or an identical Synology were fine, about 110-115 MB/s - wire speed, really. I'd love some guidance before I revert to SSR.

[Gostev]

The first experiment I would do is trying to copy something big off of the E: drive as it is being processed, to another fast disk on the same system. If that copy completes fast, then the only remaining suspects will be CPU, path to the target storage, or the target itself. CPU can be excluded by looking at the Task Scheduler, then the remaining test will be to copy that same test file now from the fast disk over to the backup target, again in parallel with backup.

In case all of that does not show any issues, which is unlikely, then we will need to enable the internal performance metrics for further diagnostics.

One potential area of concern is Windows 2008, which has some issues with system cache logic that used to impact B&R performance in a weird way in early days. And in case of the backup target being file share, both data movers are started on the backed up server itself, and may be a subject to this issue. Which I remember very vaguely, and it could have been specific to non-R2 boxes only anyway.

[Gostev]

Oops, just noticed this - no, disabling the throttling is not immediate, as all data movers have already been started with the lowest priority. This is likely the culprit.

[solmssen]

Thanks! It did finally finish after about 112 hours. The incremental last night was much faster - about 6.5 hours. What does [CBT] after the performance numbers mean?

6/26/2019 2:09:19 AM :: Initializing
6/26/2019 2:09:29 AM :: Preparing for backup
6/26/2019 2:11:00 AM :: Creating VSS snapshot
6/26/2019 2:20:11 AM :: Calculating digests
6/26/2019 2:20:51 AM :: System Reserved (disk 0) (100.0 MB) 13.0 MB read at 3 MB/s [CBT]
6/26/2019 2:20:53 AM :: VD0-PD00-PD01 (C:) (278.8 GB) 16.2 GB read at 113 MB/s [CBT]
6/26/2019 2:23:56 AM :: VD2-PD06-PD07 (F:) (3.6 TB) 213.0 MB read at 204 MB/s [CBT]
6/26/2019 2:24:21 AM :: VD1-PD03-PD04 (D:) (3.6 TB) 2.4 TB read at 217 MB/s [CBT]
6/26/2019 5:41:31 AM :: VD3-PD09-PD10 (E:) (3.6 TB) 2.3 TB read at 216 MB/s [CBT]
6/26/2019 8:44:18 AM :: Finalizing
6/26/2019 8:44:25 AM :: Incremental backup created
6/26/2019 8:45:04 AM :: Processing finished at 6/26/2019 8:45:04 AM

[Gostev]

Yep, now read performance is much better and also consistent between disks.

CBT tag means that Changed Block Tracking data is used for the given incremental run. This data enables the job to only read those blocks that we know have changed since the last run, which shortens the backup time and reduces load on the production storage.

[solmssen]

ok - thanks for your help!