Solution for poor NBD performance

[kratos]

Hello everyone,

I wanted to share some insights about our experience using NBD as a transmission method, as we encountered poor performance while currently implementing Veeam. We primarily rely on Storage Snapshots, but some of our ESXi hosts utilize local storage. We aimed to avoid using Proxy Appliances.

Despite our efforts, diverting the traffic away from the VMkernel Management interface to a separate interface didn't yield any improvement. In fact, upgrading from vSphere 7 to vSphere 8 even slightly worsened the performance:

vSphere 7 NBD over VMkernel Management: 222 MB/s
vSphere 7 NBD over separate VMkernel vSphere Backup NFC: 227 MB/s
vSphere 8 NBD over VMkernel Management: 210 MB/s
vSphere 8 NBD over separate VMkernel vSphere Backup NFC: 215 MB/s

However, after some research, we finally stumbled upon a solution on the following page: https://tecblog.au.de/veeam-v12-12-1-do ... rformance/

The information provided on that page proved to be invaluable. We didn't double, we quadrupled the throughput:

vSphere 8 NBD over separate VMkernel vSphere Backup NFC: 795 MB/s
vSphere 7 NBD over VMkernel Management: 833 MB/s

Create a text file with the following values on the vSphere backup proxy:

Code: Select all

VixDiskLib.nfcAio.Session.BufSizeIn64KB=16
vixDiskLib.nfcAio.Session.BufCount=4

Create the corresponding registry keys:

Code: Select all

[HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication]
"VddkConfigPath"="c:\temp\vddkkeys.txt"

[HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication]
"VddkExtraFlags"=dword:00000001

If any of you are encountering similar challenges with NBD performance, I highly recommend checking out the provided solution. Additionally, if anyone has alternative methods or insights to share, we would greatly appreciate it.

Best regards,
kratos

[Sturniolo]

Hi Karatos

Great info to share…

Yes we are aware of these VDDK buffer settings.

[DataDefender]

Tried that in my lab setup a few days ago and noticed quite the performance increase. If you're experiencing slow backup speeds, I recommend giving it a try.

Bye, DD

[Sturniolo]

Veeam 12.1.2 is now available - https://www.veeam.com/kb4510

"Improved VMware NBD (Network Transport Mode) transport mode performance by up to 2x."

[FrenchBlue]

Hi,

I just upgraded 2 VBR instances to 12.1.2.172, and I can confirm there is a great improvement in NBD speed. I now easily peak around 350-360 MB/s for a single VM backup, where previously it was rather around 200-220 like mentioned in the initial post here, without doing anything else. The tuning described above is even better, but I'm afraid it could saturate the network and slow down VMs traffic since we don't have dedicated nics for this

[Gostev]

Manual tuning is not something I would recommend because RAM consumption per NBD connection must be coordinated with the maximum NBD connections limit. Otherwise your jobs will start experiencing random failures due to OOM errors. We've seen this a few times at customers who beta tested this tuning.