Slow NBD Backup over 10Gbit Network

[agrob]

Good Day

We did recently change our network infrastructure to 10Gbit. I've read that in a 10Gbit network the nbd backup mode should be more performant than the hotadd mode. But i only get a bit more speed in nbd with the 10gbit network compared to 1Gbit.

There is a prod cluster, one host is already connected with 10Gbit (all 1 Gbit adapters are removed from the vswitch config). On this host is running the test client i use for backup testing and the proxy appliance.
The Backup Server (also the repository server) is a vm running in another building on a esxi host. this esxi host is also connected with 10gbit.

if i start a active full backup job and use the proxy vm in "virtual appliance" mode i get good results, so 10gbit is working here well:
-> Festplatte 1 (40,0 GB) 19,2 GB read at 448 MB/s [CBT]

If i change the backup job to use the backup vm as proxy (so proxy, repository is now the same server on the other esx host) i get poor results:
-> Festplatte 1 (40,0 GB) 19,2 GB read at 42 MB/s [CBT]

If i do a file copy job inside the vms (between vm on the prod esxi host and the vm on the backup esxi host, then i get good results as well with about 250MB/s > regarding to the disk subsystem this is well)

I would expect at least the same speed results in nbd mode as with hot add mode. any idea what to check?

thanks

[foggy]

If i change the backup job to use the backup vm as proxy (so proxy, repository is now the same server on the other esx host) i get poor results:
-> Festplatte 1 (40,0 GB) 19,2 GB read at 42 MB/s [CBT]

Can you confirm that [nbd] is selected as transport mode instead of [hotadd]? You should see this tag right after the proxy server name in the job stats window.

[dellock6]

The issue seems more related to the use of the VBR server to execute operations rather than the transport mode (hotadd vs nbd). Are you sure that VBR is linked at 10gpbs? It may happen that the underlying port group is using 10GB link, but the VM has the vmxnet3 network card? Otherwise it's limited to only 1 GB. Also, the management link of the ESXi is on a 10GB link? I've seen cases where the production vm network was at 10Gb , but the management network was only 1GB.
I can confirm you that nbd over 10gb, if properly configured, can go really fast, in our labs I tested recently our Nimble storage, and nbd is able to push data at around 300 MBs.

[agrob]

thanks for your Feedback!
@foggy, yes i have configured NBD as transportmode for the vbr Server. Here is the Output from another test:
***
Using backup proxy VMware Backup Proxy for disk Festplatte 1 [nbd]
Festplatte 1 (40,0 GB) 19,2 GB read at 55 MB/s [CBT]
***

@luca, the vbr vm has vmxnet3 (changed it from e1000 as soon as the host was 10gbit connected). the "backup" esx host on which vbr vm is running is only connected over a 10gbit nic. the 1gbit nics are not configured to a vswitch anymore.

i'll do some more testing and research

[agrob]

Hi Again

Made some more testings. This is very interessting...
Veeam Backup Server (VBR), Proxy, Repository is the Same VM on a dedicated ESXI Host. 10Gbit NIC for Management, vMotion, VM Traffic
Client (the machine configured in the test backup) is running on the production ESXI Host. 10Gbit NIC for Management, vMotion, VM Traffic
Backup job is using NBD as Transport mode (because proxy is the VBR Server on another Host which has no access to the Storage where the Client is running on)

Test 1: Client is on the local disk (2x 300 GB SAS RAID1) on the ESXI host
01.12.2016 12:23:21 :: Using backup proxy VMware Backup Proxy for disk Festplatte 1 [nbd]
01.12.2016 12:23:22 :: Festplatte 1 (40,0 GB) 19,2 GB read at 267 MB/s [CBT]
01.12.2016 12:24:53 :: Removing VM snapshot

Test 2: Client is moved to the shared 3Par Storage (56x SAS 450GB), connected with 2x 8Gbit FC to the ESXI Host
01.12.2016 12:30:44 :: Using backup proxy VMware Backup Proxy for disk Festplatte 1 [nbd]
01.12.2016 12:30:46 :: Festplatte 1 (40,0 GB) 19,2 GB read at 74 MB/s [CBT]
01.12.2016 12:35:30 :: Removing VM snapshot

Backup from local Disk is about 3.5 times faster compared to the shared storage...
Ok now one could say the shared storage is the Limitation (maybe too much load on it), BUT why is the Performance in the following test here so much faster in hotadd mode?
Test 3: Proxy is now a virtuall appliance on the same esxi host where the client is running. client is still on the shared storage, same as in Test 2.

01.12.2016 12:38:12 :: Using backup proxy leistersrv186.leister.com for disk Festplatte 1 [hotadd]
01.12.2016 12:38:39 :: Festplatte 1 (40,0 GB) 19,2 GB read at 387 MB/s [CBT]
01.12.2016 12:39:53 :: Removing VM snapshot

At the moment i don't see why Test 2 (nbd, client on shared storage) and Test 3 (hotadd, client on shared storage) have so much different performance. if share storage would be the Problem, then i should not get 387MB/s in hotadd and only 74MB in NBD. Even if we proof in Test 1 (nbd, local disk) that nbd can be fast as well, but not when the backup source is the shared storage....

can someone explain this?

many thanks

[tsightler]

We did recently change our network infrastructure to 10Gbit. I've read that in a 10Gbit network the nbd backup mode should be more performant than the hotadd mode.

Where did you read this? Below in the thread I see you are testing mostly full backups. I'm not aware of any scenario where NBD mode should provide more throughput than hotadd. NBD mode cans sometimes be faster for incremental backups due to significantly less overhead as hotadd requires lots of attaching/detaching operations and other commands which typically take around 2 minutes per VM, although running mulitple VMs in parallel does mitigate this somewhat.

At the moment i don't see why Test 2 (nbd, client on shared storage) and Test 3 (hotadd, client on shared storage) have so much different performance. if share storage would be the Problem, then i should not get 387MB/s in hotadd and only 74MB in NBD. Even if we proof in Test 1 (nbd, local disk) that nbd can be fast as well, but not when the backup source is the shared storage....

I'm assuming you are testing with 9.5? With NBD mode I/O is performed via the VDDK libraries which provides some limitation, primarily impacted by latency of the underlying storage vs total throughput. Even a few ms of additional latency can drastically reduce the throughput of NBD mode. However, when using hotadd mode (as well as Backup from Storage Snapshot and Direct NFS) Veeam 9.5 is able to use our the new Advanced Data Fetcher in 9.5. This offers significant performance advantages when reading from many storage devices. This is highlighted in the very first section of the v9.5 What's New document.

https://www.veeam.com/pdf/new/veeam_bac ... new_en.pdf

[agrob]

Hi Tom
Thanks for your answer. I read this on different sites/forums including veeam Forum. yes i did test with active full because it is the easiest way to have always the same test Szenario.
i read about the data fecher. it is just "funny" that in nbd mode an Array with 2 disk in raid1 is able to deliver much more Speed than an Enterprise class Array with 56 disk. okay there is much more going on on the Shared storage compared to the local disk...

it is just strange to see that a full backup in nbd is 3.5 times slower than in hotadd . i'm not sure if i really have a Problem about the "slow" Speed in nbd mode when reading from Shared storage or is it just "normal". I mean the backup Performance is great with hotadd and full backup...

[tsightler]

If you monitor the read latency of the datastore during the backup, and share those graphs (or even just the result), that would probably be telling.

[agrob]

Read Latency is between 0-2ms on the datastore during backup in nbd. i think this is in a normal range...?

[Andreas Neufert]

NBD is somehow limited within the ESXi itself. I would say to ~40% of the link speed of the VMware NIC. But these 40% can be likely achived only with multiple streams (multiple VM disk backups in parallel). I wonder if you had tested only with a single vms and if the VM on the local storage had more disks => better processing speed.

In many cases I sas a lot of unexpected throughput limitations when it comes to shared/virtual NIC usage in BladeCenter and other server were 2x 10Gb Cards are used for FCoE/FC and virtual NICs. VMware Management Interfaces get often low priorities with limited throughput settings. As you saw less performance when you access the data on you primary storage, this is maybe the case here (shared usage).

Regarding faster as HotAdd... Sometime NBD is "faster" than hotadd for incrmental processing with low change rates, as HotAdd disk map/unmap need time and in this timeframe NBD can transport data.

Maybe the speed difference in your environment come from the fact that hotadd read from the storage and compress data directly, while NBD transport the data through the network stack.

[agrob]

Thank you for the Feedback. I know about the limitation in esx about Network traffic in case of nbd backup. but even if it is limitted to 40%, this would be arround 400MB/s on a 10Gbit nic.
Nic is not shared, only used for networktraffic and not for traffic to the storage because it is attached over "traditional" fibre channel.

I'll monitor how the Performance is when a Job with more than 1 vm is processed.

but i think i'll stay with hotadd. Overall (even with the time needed to add the disks to the virtual appliance) the Performance seems faster

[Andreas Neufert]

With v9.5 HotAdd processing is one of the fastest backup and restore modes as it uses now asynchronous read (and write).

Only if you use NFS datastores, I highly recommend you to use DirectNFS to avoid "HotAdd with NFS" typical problems.

[agrob]

it is indeed very fast. did test full backup with 4 vms on different esx host on the same shared storage... you have to watch out on your storage read read latency.

[Andreas Neufert]

Thanks, you can enable Latecy Control within Veeam Options to monitor this for you. We will reduce our processing speed then if we affect the storage latency.

[lando_uk]

I was ultimately disappointed when moving to 10Gbe Management, after testing I confirmed that NBD mode only goes at about 100MB/s per VM max (some esxi related throttling), but when you have more VM's in a Job (all on the same host), combined they can achieve much faster speeds, going up to 500MB/s and beyond, ultimately beating hotadd timings when processing lots.

[Andreas Neufert]

Thanks, this matches my experience. This is a limitation of VMware... nothing we can do about at NBD at the moment.