New NBD-Multithreading Feature v11

[RedVision81]

Hey,
i was wondering if anyone already tried the new NBD Multithreading Feature in v11?
When i am right, i just need to add the registry dword key VMwareNBDConnectionsPerDisk under HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
and 1, 2, 3 ... depending on threads i want to start, right?

Regards

Peter

[PetrM]

Hi Peter,

Yes, that's right. By the way, the best performance is achieved with two NBD connections per disk according to our internal testing. Please don't forget to share your results with us, it will help us to decide on enabling this functionality by default in future updates.

Thanks!

[RedVision81]

Is there an interference with the backup proxys concurrent tasks? currently we use 3 backup proxys with 20 concurrent tasks in total.
making NBD Multithread with 2, they will be 40 "streams" then right?

[Gostev]

Yes and no.

In general, NBD streams are completely unrelated to Veeam tasks. With the above registry value, you are basically controlling the number of NBD streams per task (where task = virtual disk, just as a reminder).

However, max number of concurrent Veeam tasks may still be affected implicitly because NBD connections limit per ESXi host is reached more quickly (our scheduler tracks those).

[slaburda]

Hi, this feature would be suitable for turning on individual VMs or per job, preferably for VMs with large disks eg 500 GB +.

When I back up by 2 threads I also restore by 2 threads?

[Gostev]

Yes, it also applies to restores.

[slaburda]

Where can I find in the log that multi streaming is actually turned on? I added a value in the register, but I don't see in the GUI log that multi streaming would be used.

Can multithreding also be used with NBDSSL mode?

[Gostev]

You will not see it in the GUI for sure... I will ask devs what debug log file to look in.

Yes, it works for NBDSSL.

[slaburda]

I just did a few tests and there is a improvement.

[PetrM]

Hello,

You can search for this line: "Non-default option 'VMwareNBDConnectionsPerDisk':".

It must be in the following log on the proxy server:
C:\ProgramData\Veeam\Backup\"JobName"\Agent."JobName".Source."VMName"."DiskName".vmdk.log (/var/log/VeeamBackup/ on Linux proxy)

Thanks!

[tsightler]

Another place where you can monitor this is one of my favorite logs in all of VBR, RTS.ResourcesUsage.log (typically C:\ProgramData\Veeam\Backup\RTS.ResourcesUsage.log). Here's an example from my system when running a backup with VMwareNBDConnectionsPerDisk = 2:

Code: Select all

****************************************************************************************************
|--------------------------------------------------------------------------------------------------|
| Resource name                  | Usage | Id                                                      |
|-------------------------------------------------------------------------------- vSphere Proxies -|
| proxy01.notyourdomain.com      | 1     | {c4807832-de0c-49df-87e7-cf359a8025e7}                  |
| proxy02.notyourdomain.com      | 6     | {ef09e5f0-6613-4a17-962d-ef991f7d3ed1}                  |
|-------------------------------------------------------------------------------- Nfc Connections -|
| esx04.notyourdomain.com        | 2     | {80298232-020c-4339-acf1-154fa8daf058}                  |
| esx01.notyourdomain.com        | 2     | {2f0aab15-4a82-4116-9f32-194f32d3c43f}                  |
| esx02.notyourdomain.com        | 2     | {9c5d85ce-4f95-429a-923d-cf072d9db54a}                  |
| esx03.notyourdomain.com        | 8     | {18ff604d-54f0-4ee9-8f95-e2594e8cc361}                  |
|----------------------------------------------------------------------------------- Repositories -|
| Repo01 Extent                  | 1     | {0490c7a8-aef2-4027-96ae-115f6005d6ca}                  |
| Repo02 Extent                  | 6     | {8fab886a-656a-4d89-86ed-4b7643f53af6}                  |
|--------------------------------------------------------------------------------------------------|
****************************************************************************************************

As you can see in this screen there are a total of 7 backup tasks in process, one task using proxy01/repo01 and the other six using proxy02/repo02 (I have proxy affinity between proxy/repo combos so it's expected that they should be matched). However, looking at the NFC connections you can see that there are 14 connections to the ESXi hosts, so two connections per active backup tasks, exactly as expected.

You can watch this log for updates during backups with something simple like:

Code: Select all

get-content .\RTS.ResourcesUsage.log -tail 25 -wait

[soncscy]

Just a small hack for everyone, the Veeam.Backup.Manager.exe from the program files directory has lots of little flags you can launch it with, one of which is -SHOWPROXYUSAGES

I got this from support awhile back and it seems harmless, but appears to do what Tom's log does, but live from Powershell or Command Prompt.

[d3nz]

Is it really 10% performance profit of two NBD connections per disk?

[Gostev]

Thing is, when backing up multiple VMs concurrently, there are usually enough NBD connection to saturate some part of the backup infrastructure regardless.

[slaburda]

I made some tests. Backup one VM with 3 disks, processed 610 GB, read 527,3 GB, transferred 388,6 GB. Mode NBD.

# threads Processing rate MB/s Duration hour:min:sec
1----------- 84----------------------- 01:48:00
2-----------112-----------------------01:22:00
3-----------153-----------------------01:00:30
4-----------154-----------------------01:00:11

3 threads looks like the best.
Nice feature to backup individual VMs with large disks.

[tsightler]

Yep, nice result. I've see a few others get improvements in that 40% range, based on the specific network and hardware setup, others don't benefit as much, but nice to see some good results from this feature as it never hurts to have some options for those large VMs that are out there!

[mcz]

...and the hunting goes on

Just wannted to test this key as well but didn't see any improvements yet. But at least I've found a bug: With the active key, it works for most of the vm's, but it fails on one disk of one vm. I've created a case: #04729857

Let's see what they find out...

[jamcool]

are there exact steps on how to setup NBD multi-threading? I saw "add the registry dword key VMwareNBDConnectionsPerDisk under HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
and 1, 2, 3 ...". Do you do this on every Windows OS based proxy server?

[soncscy]

Any chance this registry value can be improved in the future to be per virtual machine (or even a special processing option in the UI)? I'd imagine I guess that it takes a multi-string value and matches on the machine refid, and relevant jobs would try initiate a multithreaded connection for just the matching virtual machine. I would imagine that it can be set per connection to each VM, right? Or is there some reason it must be global? (I get it's a "v1" of the feature so just having it is great, just a dream for the future )

The justification for my request is that as I get it, it's global, and potentially can overwhelm NBD mode if there are too many NBD connections, right? I imagine most people don't need their little 12 GB web servers /appliances to be backed up multithreaded, but they don't want to remake jobs just to isolate their 10 TB+ servers to toggle the registry value on/off (this is how we're doing it with a few clients that want the benefit without it being global -- a pre-job script is applied for all jobs that sets the value appropriately based on whether or not the VMs in the job really need the extra oomph, and VMs are organized into jobs based on size + role)

But it'd be great if this was per machine option, via registry or even in the UI/Powershell

[Gostev]

@jamcool it's documented in the V11 What's New.

@soncscy honestly, your suggested approach would not be manageable in any decent size environment where VMs come and go in dozens daily. I'd imagine a universal disk size threshold would be the better approach to achieve the same w/out requiring constant manual edits.

However, the answer to your question is no - there's no overwhelming host with too many NBD connections in any case, because our scheduler is aware of NBD multi-threading and won't create more NBD connections than the host can handle. Actually, this is also specifically documented in the V11 What's New

My only concern would be some OTHER software creating additional NBD connections. But if you only use Veeam, you should be safe to enable NBD multi-threading enabled. We did not make it the default only because it means changing the behavior of good 300K+ backup server to a brand new logic. This is just too much risk for a relatively small gain. We'd rather make it the default setting later, when it is more proven in the field.

[jamcool]

Gostev,
thank you. The what's new (https://www.veeam.com/veeam_backup_11_0 ... new_wn.pdf) does state "VMwareNBDConnectionsPerDisk (DWORD) registry value on the backup server" on page 11 but did not give the complete registry path. I do see it at the top of this thread. Just wondering if was actually posted somewhere with the complete registry path. Seems like that would be some key information for those that want to try it out.

On a side note, I am really lookin forward to try this out as performance via NBD has been a struggle and I appreciate Veeam trying to find ways to improve performance. Maybe in next release this can be something you can enable/configure via Console and not have to do registry entries.

[Gostev]

Yes, all Veeam registry values go to this same key.

[mcz]

Guys, I've created the case 04729857 because I had issues in combination with this key. Now support told me that it should get fixed in the next patch but it doesn't (I've installed the latest patches). So either they haven't fixed it yet or they have and it still does not work on my side. Could you please have a look and tell me if the bug should have been fixed yet - thanks!

[Gostev]

Yes, according to your case it causes some issue with replication jobs specifically. But I think you misunderstood each other with the support engineer, because in the internal case note back from April he clearly documented that this is "planned to be fixed in V11a".

[mcz]

Thanks Anton, obviously I had the next "patch" in mind but 11a is a bit a different story...

[d3nz]

Hello.
As I suppose this registry key should be added to VBR?
What's about this feature at v12? It also works with registry key?

[RedVision81]

the registry key did not work for me, neither in v11 nor v12/12.1 instead i used this blog, which a technician told me yesterday, and this works great.
https://tecblog.au.de/veeam-v12-12-1-do ... rformance/

reduced incrementel time by 50%
its in german but i think this should be no problem

[Gostev]

Right, we're currently piloting this with some customers in the field and plan to include in 12.1.2 by default if all is well.

[neilp]

Hello. I would like to test this. Please confirm that '"add the registry dword key VMwareNBDConnectionsPerDisk under HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication" should be done on the Veeam backup and replication server and not the Proxies. Thanks!