Discussions specific to the VMware vSphere hypervisor
Post Reply
stevenrodenburg1
Expert
Posts: 132
Liked: 20 times
Joined: May 31, 2011 9:11 am
Full Name: Steven Rodenburg
Location: Switzerland
Contact:

Network Mode much slower as HotAdd mode

Post by stevenrodenburg1 »

In a small environment, B&R 9.5 U4b, running 4 all-flash ESXi vSAN nodes and 10gig connectivity between the nodes, we have two proxy VM's which are configured to handle 6 VMDK's each. In backup jobs, we see the expected 12 VMDK's, divided over several VM's (some have more than one disk) being worked on at the same time and backups are working without problems.

We noticed that, when the proxies are set to Network mode, the throughput of individual VMDK's is very low. Example:
We see a fileserver, with 3 disks (50gig, 80gig and 900gig) being backed-up and the console says something like:
Harddisk 1 (50GB) 945MB read at 11 MB/s CBT
Harddisk 2 (80GB) 671MB read at 13 MB/s CBT
Harddisk 3 (900GB) 354.3GB read at 10 MB/s CBT
This means that harddisk 3 takes many hours to complete.
This fileserver has a high change-rate on that disk 3 by the way.
We observed this behaviour (very low read speeds of 10 to 15 MB/s) for several weeks.

Then, out of curiosity, we changed those 2 proxies to Virtual Appliance Mode (hotadd). The Job did not change. Since then we see something like this everyday:
Harddisk 1 (50GB) 441MB read at 341 MB/s CBT
Harddisk 2 (80GB) 468MB read at 298 MB/s CBT
Harddisk 3 (900GB) 414.6GB read at 411 MB/s CBT
Needles to say, backups are massively faster. Done in a jiffy.

So this begs the question: why these super slow read speeds in Network mode? All is well. No errors. Just super slow compared to HotAdd.
The hardware is really powerful. If i'm not mistaken, the documentation even recommends Network mode on high bandwidth networks.

Egor Yakovlev
Veeam Software
Posts: 1528
Liked: 346 times
Joined: Jun 14, 2013 9:30 am
Full Name: Egor Yakovlev
Contact:

Re: Network Mode much slower as HotAdd mode

Post by Egor Yakovlev »

Hi Steven,
vSAN storage spans nodes and you cannot predict which node will have majority of protected VMDK data blocks, that's why Network mode is slow(overload on vSAN network to retrieve data) and why it is recommended to be used with Hot-Add transport by Veeam Best Practices for optimal data traffic. If you have Proxy per vSAN node, we will choose best suited proxy for each VM!
/Thanks!

Andreas Neufert
VP, Product Management
Posts: 4521
Liked: 857 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Network Mode much slower as HotAdd mode

Post by Andreas Neufert »

Some additional explanation.
https://www.veeambp.com/proxy_servers_i ... twork_mode

NBD mode is always slower for many reasons.
Throttling from VMware. VMware reserves throughput for the regular VMware backend processing. You can get up to 40% of your network speed with some parallelization.

With HotAdd we are actually reading the data and use multiple additional optimizations like async IO processing which allows the storage system to optimize the reads and writes.
Veeam has added many non standard optimizations into our HotAdd processing which makes it the fastest protocol for backup and restore processing.

In case of vSAN it is best practice to use HotAdd.
https://www.veeambp.com/proxy_servers_i ... mendations

jamcool
Enthusiast
Posts: 38
Liked: 4 times
Joined: Feb 02, 2018 7:56 pm
Full Name: Jason Mount
Contact:

Re: Network Mode much slower as HotAdd mode

Post by jamcool »

Is there any adjustments that can be made in VMWARE ESXi management port to speed up NBD (actual NBDSSL). We have 10 Gbps networking and we just upgraded our SAN storage to NVME flash from 15K RPM based disk. No changes in backups speeds and still only get at most 20 MB/s to 60 MB/s even if just one machine being backed up. Backups jobs still say bottleneck is "Source. We have enjoyed NBD over Hot Add as it is was proving to be about 3x faster for our incremental backups with not having to add disk to proxy servers and Hot add was occasionally not freeing up a disk on a proxy server.

Andreas Neufert
VP, Product Management
Posts: 4521
Liked: 857 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Network Mode much slower as HotAdd mode

Post by Andreas Neufert »

Likely not as it is limited by VMware. You can check that the connections are handele by the correct interfaces in Veeam and VMware. Check for example that you do not use a 1 GB/s network for your primary VMKernel interface. Potentially open a support ticket to check if there is a problem noted in logs additionally.

jamcool
Enthusiast
Posts: 38
Liked: 4 times
Joined: Feb 02, 2018 7:56 pm
Full Name: Jason Mount
Contact:

Re: Network Mode much slower as HotAdd mode

Post by jamcool »

we are all dual port 10 Gbps on VMWARE and the Veeam Repository servers (physical severs). I did open a suppose case with Veeam but was told that because we are using NBD and the bottleneck is at Source, they cannot help us. I guess I can open a case with VMWARE to see if they have any recommendation. I was kind of hoping that if anybody else was using NBD had similar speeds or maybe had recommendations in Veeam or VMWARE that I can do. Thanks.

Andreas Neufert
VP, Product Management
Posts: 4521
Liked: 857 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Network Mode much slower as HotAdd mode

Post by Andreas Neufert »

NBD scales to some hundred MB/s per host only with parallelization (backup of multiple disks in parallel).

Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 11 guests