Network Mode much slower as HotAdd mode

[stevenrodenburg1]

In a small environment, B&R 9.5 U4b, running 4 all-flash ESXi vSAN nodes and 10gig connectivity between the nodes, we have two proxy VM's which are configured to handle 6 VMDK's each. In backup jobs, we see the expected 12 VMDK's, divided over several VM's (some have more than one disk) being worked on at the same time and backups are working without problems.

We noticed that, when the proxies are set to Network mode, the throughput of individual VMDK's is very low. Example:
We see a fileserver, with 3 disks (50gig, 80gig and 900gig) being backed-up and the console says something like:

Harddisk 1 (50GB) 945MB read at 11 MB/s CBT
Harddisk 2 (80GB) 671MB read at 13 MB/s CBT
Harddisk 3 (900GB) 354.3GB read at 10 MB/s CBT

This means that harddisk 3 takes many hours to complete.
This fileserver has a high change-rate on that disk 3 by the way.
We observed this behaviour (very low read speeds of 10 to 15 MB/s) for several weeks.

Then, out of curiosity, we changed those 2 proxies to Virtual Appliance Mode (hotadd). The Job did not change. Since then we see something like this everyday:

Harddisk 1 (50GB) 441MB read at 341 MB/s CBT
Harddisk 2 (80GB) 468MB read at 298 MB/s CBT
Harddisk 3 (900GB) 414.6GB read at 411 MB/s CBT

Needles to say, backups are massively faster. Done in a jiffy.

So this begs the question: why these super slow read speeds in Network mode? All is well. No errors. Just super slow compared to HotAdd.
The hardware is really powerful. If i'm not mistaken, the documentation even recommends Network mode on high bandwidth networks.

[Egor Yakovlev]

Hi Steven,
vSAN storage spans nodes and you cannot predict which node will have majority of protected VMDK data blocks, that's why Network mode is slow(overload on vSAN network to retrieve data) and why it is recommended to be used with Hot-Add transport by Veeam Best Practices for optimal data traffic. If you have Proxy per vSAN node, we will choose best suited proxy for each VM!
/Thanks!

[Andreas Neufert]

Some additional explanation.
https://www.veeambp.com/proxy_servers_i ... twork_mode

NBD mode is always slower for many reasons.
Throttling from VMware. VMware reserves throughput for the regular VMware backend processing. You can get up to 40% of your network speed with some parallelization.

With HotAdd we are actually reading the data and use multiple additional optimizations like async IO processing which allows the storage system to optimize the reads and writes.
Veeam has added many non standard optimizations into our HotAdd processing which makes it the fastest protocol for backup and restore processing.

In case of vSAN it is best practice to use HotAdd.
https://www.veeambp.com/proxy_servers_i ... mendations

[jamcool]

Is there any adjustments that can be made in VMWARE ESXi management port to speed up NBD (actual NBDSSL). We have 10 Gbps networking and we just upgraded our SAN storage to NVME flash from 15K RPM based disk. No changes in backups speeds and still only get at most 20 MB/s to 60 MB/s even if just one machine being backed up. Backups jobs still say bottleneck is "Source. We have enjoyed NBD over Hot Add as it is was proving to be about 3x faster for our incremental backups with not having to add disk to proxy servers and Hot add was occasionally not freeing up a disk on a proxy server.

[Andreas Neufert]

Likely not as it is limited by VMware. You can check that the connections are handele by the correct interfaces in Veeam and VMware. Check for example that you do not use a 1 GB/s network for your primary VMKernel interface. Potentially open a support ticket to check if there is a problem noted in logs additionally.

[jamcool]

we are all dual port 10 Gbps on VMWARE and the Veeam Repository servers (physical severs). I did open a suppose case with Veeam but was told that because we are using NBD and the bottleneck is at Source, they cannot help us. I guess I can open a case with VMWARE to see if they have any recommendation. I was kind of hoping that if anybody else was using NBD had similar speeds or maybe had recommendations in Veeam or VMWARE that I can do. Thanks.

[Andreas Neufert]

NBD scales to some hundred MB/s per host only with parallelization (backup of multiple disks in parallel).