VCC Backup - Slow throughput

[preecyp]

We have two HyperV standalone servers hosting 12 vm's. One of the vm's is our Veeam server (running B&R v9.4 u4). We have a single backup job which performs daily backups to a Buffalo NAS which is situated on the same 1Gbit LAN. The NAS repository is accessed via CIFS, not iSCSI. The backup job achieves a processing rate of 82MB/s.

We have just setup a backup copy job to a Veeam Cloud Connect service provider. We're only getting a possessing rate 3-4MB/s. The bottleneck shows:

- source 68%
- proxy 18%
- network 33%
- target 0%

We have a 100Mbps up & down internet connection. We have tried enabling and disabling WAN acceleration but this does not improve the processing rate of the copy job.

Any suggestions would be appreciated.

[PetrM]

Hi Phil!

Bottleneck source means that source Data Mover spends most of time on reading data from the NAS.

I'd suggest to check the following:
1. The gateway server which is specified at the level of repository settings.
In case of automatic selection, source Data Mover could be started on a remote server and read data from the NAS over a slow link.

2. Full backup file could be heavily fragmented if you used forever forward incremental (no active full scheduled) or reversed incremental chain.
This kind of fragmentation could affect read speed. You may schedule compact or active full to decrease fragmentation level (if you have enough free space on repository for these tasks).

If the suggestions above don't help to resolve the issue, I recommend to open a support case.

Thanks!

[preecyp]

Hi PetrM

1. The gateway server, for the NAS repository, is set to the Veeam server. Other than 'automatic' i can see both HyperV hosts are available in the drop down selection box.

2. Our backup job is configured using incremental with a weekly synthetic full. I'm assuming the copy backup job will be the same. I will check the fragmentation level to see if this improves matters.

[PetrM]

Thanks for reply!

I was talking about fragmentation of full backup file produced by primary backup job but if you have synthetic full scheduled, the issue won't be related to backup file fragmentation.
Please open a support request and provide us with case number.

Thanks!

[foggy]

Backup copy job performs random reads on source repository and your NAS might not be good enough at that.

[preecyp]

Turns out someone enabled a global throttling policy on the firewall. We've white listed the Veeam server and now get 18-22MB/s, however, we now have a different issue! The vcc backup copy job sets off with throughput around the 20MB/s mark for an hour or two but then drops off. The job throughput display shows 0KB/s for an hour or so then spikes again for short while.

The copy job just says 'working' and the status shows:

Processing VMNAME
Waiting for backup infrastructure resources availability

Running a speed tests from the Veeam server shows our link is still hitting 100Mbps.

Any thoughts?

[PetrM]

Hi Phil!

Most likely the job is waiting for available slot on repository, take a look at this option, read operations consume slots as well.
You can also ask your provider about possible limit of concurrent tasks on his side.

On the other hand, it's always better to contact our support team in order to investigate the issue.

Thanks!

[matthewkwarburton]

Turns out someone enabled a global throttling policy on the firewall. We've white listed the Veeam server and now get 18-22MB/s, however, we now have a different issue! The vcc backup copy job sets off with throughput around the 20MB/s mark for an hour or two but then drops off. The job throughput display shows 0KB/s for an hour or so then spikes again for short while.

The copy job just says 'working' and the status shows:

Processing VMNAME
Waiting for backup infrastructure resources availability

Running a speed tests from the Veeam server shows our link is still hitting 100Mbps.

Any thoughts?

Case # 03866987

I have a very similar issue. Job starts off fine but then after 2 - 3 hours drops to 0 when syncing to cloud repo via VCC. Occasionally going up to 1Mbps for a short stint. Nothing has changed on our network since it synced the full backups fine a few month back. Have changed onsite repo to rule out source, changed over to backup 100/100Mbps WAN, fully patched Server 2016 host, disabled all QoS, Server resources look fine whilst it's stuck on 0 Mbps. Bottleneck shows 99% network but everything on the network looks fine. Running out of ideas with it!