Discussions specific to the VMware vSphere hypervisor
Post Reply
Posts: 132
Liked: 20 times
Joined: May 31, 2011 9:11 am
Full Name: Steven Rodenburg
Location: Switzerland

Network Mode much slower as HotAdd mode

Post by stevenrodenburg1 »

In a small environment, B&R 9.5 U4b, running 4 all-flash ESXi vSAN nodes and 10gig connectivity between the nodes, we have two proxy VM's which are configured to handle 6 VMDK's each. In backup jobs, we see the expected 12 VMDK's, divided over several VM's (some have more than one disk) being worked on at the same time and backups are working without problems.

We noticed that, when the proxies are set to Network mode, the throughput of individual VMDK's is very low. Example:
We see a fileserver, with 3 disks (50gig, 80gig and 900gig) being backed-up and the console says something like:
Harddisk 1 (50GB) 945MB read at 11 MB/s CBT
Harddisk 2 (80GB) 671MB read at 13 MB/s CBT
Harddisk 3 (900GB) 354.3GB read at 10 MB/s CBT
This means that harddisk 3 takes many hours to complete.
This fileserver has a high change-rate on that disk 3 by the way.
We observed this behaviour (very low read speeds of 10 to 15 MB/s) for several weeks.

Then, out of curiosity, we changed those 2 proxies to Virtual Appliance Mode (hotadd). The Job did not change. Since then we see something like this everyday:
Harddisk 1 (50GB) 441MB read at 341 MB/s CBT
Harddisk 2 (80GB) 468MB read at 298 MB/s CBT
Harddisk 3 (900GB) 414.6GB read at 411 MB/s CBT
Needles to say, backups are massively faster. Done in a jiffy.

So this begs the question: why these super slow read speeds in Network mode? All is well. No errors. Just super slow compared to HotAdd.
The hardware is really powerful. If i'm not mistaken, the documentation even recommends Network mode on high bandwidth networks.

Egor Yakovlev
Veeam Software
Posts: 1464
Liked: 331 times
Joined: Jun 14, 2013 9:30 am
Full Name: Egor Yakovlev

Re: Network Mode much slower as HotAdd mode

Post by Egor Yakovlev »

Hi Steven,
vSAN storage spans nodes and you cannot predict which node will have majority of protected VMDK data blocks, that's why Network mode is slow(overload on vSAN network to retrieve data) and why it is recommended to be used with Hot-Add transport by Veeam Best Practices for optimal data traffic. If you have Proxy per vSAN node, we will choose best suited proxy for each VM!

Andreas Neufert
VP, Product Management
Posts: 4352
Liked: 814 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany

Re: Network Mode much slower as HotAdd mode

Post by Andreas Neufert »

Some additional explanation.
https://www.veeambp.com/proxy_servers_i ... twork_mode

NBD mode is always slower for many reasons.
Throttling from VMware. VMware reserves throughput for the regular VMware backend processing. You can get up to 40% of your network speed with some parallelization.

With HotAdd we are actually reading the data and use multiple additional optimizations like async IO processing which allows the storage system to optimize the reads and writes.
Veeam has added many non standard optimizations into our HotAdd processing which makes it the fastest protocol for backup and restore processing.

In case of vSAN it is best practice to use HotAdd.
https://www.veeambp.com/proxy_servers_i ... mendations

Post Reply

Who is online

Users browsing this forum: adb98, cekame9953, Google [Bot], nurislam and 44 guests