Backup slow on 10Gbps network, Bottleneck Target 99%

[lakecountydot]

Case #04667119

We have two HPE servers running VMWare ESXi, one 6.5U2 and one 7.0U1. Both are connected to a Cisco L3 on 10Gbps connections on the management port. Both servers are also running SSD in RAID5. When I run a replication task or quick migration, the speed maxes at 110MB/s with Bottleneck=Target 99%.

We have several Windows Server VMs connected to the 10Gbps network and can transfer files between them at 700MB/s. Additionally, running iperf3 between the machines yields 1GB/s between them.

I have tried the replication or quick migration in both Network Mode and Virtual Appliance (HotAdd) and they produce the same result, nothing over 110MB/s. The replication window shows the proxies being added before transfer.

I have Veeam B&R running on one Windows Server VM (10Gbps connection), with a backup proxy on the same machine. I have another backup proxy on the other host machine (10Gbps connection).

Note: the speed of 110MB/s is during the initial replication copy. I did notice one incremental backup (of only 90MB) peak at 1GB/s. I have not tested this again, so I dont know if this was an error in reporting or true, but I would think any copy should be able to reach 500+MB/s

I also noticed on the servers that the DISK for read + write never go over 200MB/s during the replication.

[foggy]

Hi Ryan, can you confirm that the target proxy is effectively using hotadd transport mode?

[lakecountydot]

Sure, where should I look for that? The status window said that each proxy was being added as hotadd

[Gostev]

In the backup job action log. You need to select the particular VM to the left to see its log to the right. This log will indicate the processing mode next to each virtual disk, in brackets.

[lakecountydot]

Yes, it reads :
Using source proxy name1 for disk Hard disk 1 [hotadd]
Using target proxy name2 for disk Hard disk 1 [hotadd]
...
Busy: Srouce 2% > Proxy 14% > Network 1% > Target 99%
Primary bottleneck: Target

[foggy]

Does network mode provide the same performance? Have you checked the latency on the target side? Overall, I would let our engineers take a look and review the setup closer - since the bottleneck is reported as a target, it means that the job spent most of the time writing the data to the target, which could be either a result of storage performance issues (which is likely not, based on your other tests) or some environmental stuff.

[lakecountydot]

Thanks, I have a case open. We'll see where it goes. I included logs, etc. I'll also test network mode. Do I need proxies on each end for that? I don't think that matters because network mode just uses the management ports, correct?

[foggy]

For network mode, it is not a requirement, right. Veeam B&R will select the proxy from the same subnet, if available, but can use servers from other networks otherwise.

[s2cbackup]

I'm seeing a similar issue. I have two servers connected by Private Lan at 10 Gbps, and the backups are running at 100 Mbps to 280 Mbps. Theoretically, how fast can Veeam process the data? Should this bandwidth be close to 10 Gbps? What is the goal speed for a Veeam Backup in transit?

[Gostev]

Theoretically, there's no limit. Veeam can saturate any network so long as source and target storage can keep up. For example, Veeam processed data at 11.4 GB/s (so over 90 Gbps) in this V11 test.

[pirx]

I'm seeing a similar issue. I have two servers connected by Private Lan at 10 Gbps, and the backups are running at 100 Mbps to 280 Mbps. Theoretically, how fast can Veeam process the data? Should this bandwidth be close to 10 Gbps? What is the goal speed for a Veeam Backup in transit?

nbd or hot add mode?