VBR10 using backup copy job to create replica seed is SLOW reading from source?

[cerberus]

Running into a very strange issue using backup copy jobs to create replication seed files that contain VBK/VBM.

When I run a backup copy job to create the seed from existing backups on disk, copying some VMs is VERY slow (taking 24-48hrs) while others are fast.

As an example, running a backup copy job on our exchange VM is capping out at 80MB/s with source being the bottleneck. Using similar settings to copy other large non-Exchange VMs we get expected speed of 500MB/s.

In both test cases the backup copy job is pulling from local RAID6 SSD storage and saving to a separate local RAID10 storage.

I have done numerous perf tests against both source and destination disk subsystem, these should not be the bottleneck. Our nightly reverse incremental B2D2T jobs that run (backup copy job is using this as the source) offload VBKs to tape every night, using two LTO7 drives we are able to saturate each drive at close to 300MB/s; so the source read/write performance should not be the bottleneck when reading the VBKs from disk.

Windows 2019 with ReFS 64K, RAID6 SSD (source) and RAID10 spinning (target); reverse incremental

Code: Select all

Slow backup copy job stats (ended up cancelling it after 30min):
Hard disk 3 (1.9 TB) 157 GB read at 79 MB/s	34:05
Busy: Source 99% > Proxy 5% > Network 6% > Target 0% 	

Fast backup copy job stats:
Hard disk 1 (100 GB) 40.8 GB read at 976 MB/s	00:43
Hard disk 2 (4.5 TB) 3.8 TB read at 587 MB/s	01:54:26
Busy: Source 69% > Proxy 11% > Network 84% > Target 17% 	

The only thing that makes sense right now is perhaps heavy fragmentation at play for the Exchange VMs? Strange because if that was the case, I would expect both VMs to be slow as they would have started backing up around the same date. Also wouldn't the B2T be slow since its also reading the same source? Unless the "backup copy job" is synthesizing an active full where as the B2T is doing something different....

We don't do any active full backup "maintenance" on the RI B2D2T jobs that were setup over a year ago, my understanding is that would do very little for VBKs sitting on ReFS where block cloning is used.

Long term, I plan to switch this to reverse incremental forever (read plenty of posts from gostev/foggy where this is suggested and the benefits); right now trying to get a seed offloaded relatively quick, the slow backup copy job is defeating the purpose of the seed.

Case #04763284

[cerberus]

Can't edit original post, typo in the last sentence; Long term I plan to switch away from reverse incremental to forward incremental forever.

[mkretzer]

Is your Tape job running at the same time, reading the same files as the copy job? If yes this is a problem of V10: Tape uses a different IO mode (unbuffered) that collides with every other type of job (buffered).
If this is your issue 79 MB/s is even very fast - with our spinning disks performance went down to 10 MB/s!

[cerberus]

The source where the backup VBKs are is SSDs, the destination of the backup copy job is spinning disks; source being the bottleneck. I confirmed that no other jobs are running while the backup copy job is running.

It starts off running at about 500MB/s for a solid few minutes, then drops off to 80MB/s. I only see this behavior on the Exchange VMs; backup copy of other VMs runs at 500MB/s through.

Logs from working backup copy job:

Code: Select all

Copying restore point 4/18/2021 8:16:34 PM from backup repository MD1420-SSD 	
Queued for processing at 4/19/2021 12:51:13 PM 	
Required backup infrastructure resources have been assigned 	
VM processing started at 4/19/2021 12:51:13 PM 	
VM size: 4.1 TB 	
Saving marketing.vmx 	00:00
Saving marketing.vmxf 	00:00
Saving marketing.nvram 	00:00
Saving FsAwareMeta:10e1a762-31cc-4f9f-8398-d7e7685d1845:2000 	00:00
Saving FsAwareMeta:10e1a762-31cc-4f9f-8398-d7e7685d1845:2001 	00:00
Saving GuestIndexData.zip 	00:00
Saving GuestMembers.xml 	00:00
Saving BackupComponents.xml 	00:00
Hard disk 1 (100 GB) 40.8 GB read at 976 MB/s	00:43
Hard disk 2 (4.5 TB) 3.8 TB read at 587 MB/s	01:54:26
Finalizing 	00:00
Busy: Source 69% > Proxy 11% > Network 84% > Target 17% 	
Primary bottleneck: Network 	
Network traffic verification detected no corrupted blocks 	
Processing finished at 4/19/2021 2:47:33 PM

Logs from non-working backup copy job:

Code: Select all

Copying restore point 4/19/2021 7:46:19 PM from backup repository MD1420-SSD 	
Queued for processing at 4/20/2021 7:20:04 AM 	
Required backup infrastructure resources have been assigned 	
VM processing started at 4/20/2021 7:20:04 AM 	
VM size: 4.4 TB 	
Saving Mail03.vmx 	00:00
Saving Mail03.vmxf 	00:00
Saving Mail03.nvram 	00:00
Saving FsAwareMeta:4fe68448-844d-4e19-ab37-86a1c4cce1b1:2003 	00:00
Saving FsAwareMeta:4fe68448-844d-4e19-ab37-86a1c4cce1b1:2002 	00:00
Saving FsAwareMeta:4fe68448-844d-4e19-ab37-86a1c4cce1b1:2000 	00:00
Saving FsAwareMeta:4fe68448-844d-4e19-ab37-86a1c4cce1b1:2001 	00:00
Saving GuestIndexData.zip 	00:00
Saving GuestMembers.xml 	00:00
Saving BackupComponents.xml 	00:00
Hard disk 3 (1.9 TB) 62.5 GB read at 78 MB/s	13:55

[cerberus]

Turned out this was VBK fragmentation at play, updated our B2D reverse incremental jobs to perform active full backups once a month.