Slow copy jobs over WAN

[dkleeman]

We have an enterprise environment running version 9.5.0.1922 (Update 3a). We are experiencing relatively slow backup speeds over a high-speed WAN.

Source: Proxy holding backups on local disk
Target: Proxy with iSCSI volume NAS connected on 1G LAN
Network: VPN which we have tested up to 220 Mbps on simple file transfer with latency of around 7 ms

Backup copy speeds are around 14 MB/S processing rate with busy loads: Source 7% > Proxy 19% > Network 99% > Target 0%

WAN speed only rises to average 70 Mbps with peaks at 115 Mbps. We know it can run at double this speed. The LAN to the NAS (iSCSI protocol) runs at around 70 Mbps and we believe this can run up to 1 Gbps.

What can we do to troubleshoot these speeds? We have good network support at each end of the link.

[Rick.Vanover]

Welcome Daniel to the Veeam Forums.

Is the proxy doing the backup at the source data side? (In the backup job you can specify which proxy does the backup - vs automatic)

Also, have incrementals been run yet? If so, did they speed up?

I'm also curious if a backups is done locally - does it go the speed you want? (Meaning - are the proxies on the source side doing what they need to do)

I'd also wonder if you have any encryption, compression or dedupe settings on the job - they can all effect the runtime. (They are in the storage option of the job)

[UBX_Cloud_Steve]

Hi Daniel,

How many VMs are apart of each backup copy job? We see best WAN performance when the copy jobs contain multiple jobs with 4-5 VMs per job.

Are you taking advantage of WAN Acceleration? You might want to try it and see if it gets you the speed you are expecting.

[Rick.Vanover]

Good question UBX - is Daniel doing backup - or backup copy jobs. That makes a huge difference. The first line, 2nd sentence made me think backup - but later on, copies are mentioned.

[dkleeman]

Thank you for your constructive comments and questions. I will try to clarify here:

1) I am referring to backup copy jobs. These are the only ones we run over the WAN. The first line of my post is misleading.

2) We have multiple backup primary jobs - maybe 20. These all run from a VM backup server and use a real Windows server with local storage as the proxy/repository. There is no WAN element in these jobs, everything is on the LAN.

3) backup jobs run at much higher speeds. Examples of average processing rates are between 48 and 120 MB/S.

3) I have tried increasing the 'multiple upload streams per job' from 5 to 10. This has had no noticeable effect.

4) The single backup copy job we are investigating covers all the VMs being backed up: 45 in total.

5) because it is a backup copy job there is no selection of source proxy. Data transfer is 'direct' (we do not have the enterprise plus licence).

6) Currently we are running a full copy that is taking a very long time and running at 14 MB/S. Incremental sessions run at similar speeds.

7) The storage options of the copy job are
- Inline data deduplication: enabled
- Compression level: high
- Encryption: disabled

[foggy]

Source: Proxy holding backups on local disk
Target: Proxy with iSCSI volume NAS connected on 1G LAN

These are different physical servers located on different sides of the WAN link, right? Both repositories are added as Windows servers?

[dkleeman]

Both servers are added as Windows servers (one is true local disk and one is iSCSI disk on NAS). The former is a real server and the latter is a VM on decent hardware.

[foggy]

Network 99% means that the link between these two servers is the bottleneck here, so I'd look here more closely.