Host-based backup of VMware vSphere VMs.
Post Reply
RedVision81
Enthusiast
Posts: 39
Liked: 2 times
Joined: Jan 14, 2016 8:02 am
Full Name: Peter
Contact:

New NBD-Multithreading Feature v11

Post by RedVision81 »

Hey,
i was wondering if anyone already tried the new NBD Multithreading Feature in v11?
When i am right, i just need to add the registry dword key VMwareNBDConnectionsPerDisk under HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
and 1, 2, 3 ... depending on threads i want to start, right?

Regards

Peter
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: New NBD-Multithreading Feature v11

Post by PetrM »

Hi Peter,

Yes, that's right. By the way, the best performance is achieved with two NBD connections per disk according to our internal testing. Please don't forget to share your results with us, it will help us to decide on enabling this functionality by default in future updates.

Thanks!
RedVision81
Enthusiast
Posts: 39
Liked: 2 times
Joined: Jan 14, 2016 8:02 am
Full Name: Peter
Contact:

Re: New NBD-Multithreading Feature v11

Post by RedVision81 »

Is there an interference with the backup proxys concurrent tasks? currently we use 3 backup proxys with 20 concurrent tasks in total.
making NBD Multithread with 2, they will be 40 "streams" then right?
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: New NBD-Multithreading Feature v11

Post by Gostev » 2 people like this post

Yes and no.

In general, NBD streams are completely unrelated to Veeam tasks. With the above registry value, you are basically controlling the number of NBD streams per task (where task = virtual disk, just as a reminder).

However, max number of concurrent Veeam tasks may still be affected implicitly because NBD connections limit per ESXi host is reached more quickly (our scheduler tracks those).
slaburda
Enthusiast
Posts: 39
Liked: 4 times
Joined: Apr 08, 2011 1:24 pm
Full Name: Stanislav Laburda
Location: Prague
Contact:

Re: New NBD-Multithreading Feature v11

Post by slaburda »

Hi, this feature would be suitable for turning on individual VMs or per job, preferably for VMs with large disks eg 500 GB +.

When I back up by 2 threads I also restore by 2 threads?
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: New NBD-Multithreading Feature v11

Post by Gostev » 3 people like this post

Yes, it also applies to restores.
slaburda
Enthusiast
Posts: 39
Liked: 4 times
Joined: Apr 08, 2011 1:24 pm
Full Name: Stanislav Laburda
Location: Prague
Contact:

Re: New NBD-Multithreading Feature v11

Post by slaburda »

Where can I find in the log that multi streaming is actually turned on? I added a value in the register, but I don't see in the GUI log that multi streaming would be used.

Can multithreding also be used with NBDSSL mode?
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: New NBD-Multithreading Feature v11

Post by Gostev »

You will not see it in the GUI for sure... I will ask devs what debug log file to look in.

Yes, it works for NBDSSL.
slaburda
Enthusiast
Posts: 39
Liked: 4 times
Joined: Apr 08, 2011 1:24 pm
Full Name: Stanislav Laburda
Location: Prague
Contact:

Re: New NBD-Multithreading Feature v11

Post by slaburda »

I just did a few tests and there is a improvement.
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: New NBD-Multithreading Feature v11

Post by PetrM »

Hello,

You can search for this line: "Non-default option 'VMwareNBDConnectionsPerDisk':".

It must be in the following log on the proxy server:
C:\ProgramData\Veeam\Backup\"JobName"\Agent."JobName".Source."VMName"."DiskName".vmdk.log (/var/log/VeeamBackup/ on Linux proxy)

Thanks!
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: New NBD-Multithreading Feature v11

Post by tsightler » 3 people like this post

Another place where you can monitor this is one of my favorite logs in all of VBR, RTS.ResourcesUsage.log (typically C:\ProgramData\Veeam\Backup\RTS.ResourcesUsage.log). Here's an example from my system when running a backup with VMwareNBDConnectionsPerDisk = 2:

Code: Select all

****************************************************************************************************
|--------------------------------------------------------------------------------------------------|
| Resource name                  | Usage | Id                                                      |
|-------------------------------------------------------------------------------- vSphere Proxies -|
| proxy01.notyourdomain.com      | 1     | {c4807832-de0c-49df-87e7-cf359a8025e7}                  |
| proxy02.notyourdomain.com      | 6     | {ef09e5f0-6613-4a17-962d-ef991f7d3ed1}                  |
|-------------------------------------------------------------------------------- Nfc Connections -|
| esx04.notyourdomain.com        | 2     | {80298232-020c-4339-acf1-154fa8daf058}                  |
| esx01.notyourdomain.com        | 2     | {2f0aab15-4a82-4116-9f32-194f32d3c43f}                  |
| esx02.notyourdomain.com        | 2     | {9c5d85ce-4f95-429a-923d-cf072d9db54a}                  |
| esx03.notyourdomain.com        | 8     | {18ff604d-54f0-4ee9-8f95-e2594e8cc361}                  |
|----------------------------------------------------------------------------------- Repositories -|
| Repo01 Extent                  | 1     | {0490c7a8-aef2-4027-96ae-115f6005d6ca}                  |
| Repo02 Extent                  | 6     | {8fab886a-656a-4d89-86ed-4b7643f53af6}                  |
|--------------------------------------------------------------------------------------------------|
****************************************************************************************************
As you can see in this screen there are a total of 7 backup tasks in process, one task using proxy01/repo01 and the other six using proxy02/repo02 (I have proxy affinity between proxy/repo combos so it's expected that they should be matched). However, looking at the NFC connections you can see that there are 14 connections to the ESXi hosts, so two connections per active backup tasks, exactly as expected.

You can watch this log for updates during backups with something simple like:

Code: Select all

get-content .\RTS.ResourcesUsage.log -tail 25 -wait
soncscy
Veteran
Posts: 643
Liked: 312 times
Joined: Aug 04, 2019 2:57 pm
Full Name: Harvey
Contact:

Re: New NBD-Multithreading Feature v11

Post by soncscy » 3 people like this post

Just a small hack for everyone, the Veeam.Backup.Manager.exe from the program files directory has lots of little flags you can launch it with, one of which is -SHOWPROXYUSAGES

I got this from support awhile back and it seems harmless, but appears to do what Tom's log does, but live from Powershell or Command Prompt.
d3nz
Expert
Posts: 130
Liked: 14 times
Joined: Mar 20, 2018 12:47 pm
Contact:

Re: New NBD-Multithreading Feature v11

Post by d3nz »

Is it really 10% performance profit of two NBD connections per disk?
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: New NBD-Multithreading Feature v11

Post by Gostev »

Thing is, when backing up multiple VMs concurrently, there are usually enough NBD connection to saturate some part of the backup infrastructure regardless.
slaburda
Enthusiast
Posts: 39
Liked: 4 times
Joined: Apr 08, 2011 1:24 pm
Full Name: Stanislav Laburda
Location: Prague
Contact:

Re: New NBD-Multithreading Feature v11

Post by slaburda » 2 people like this post

I made some tests. Backup one VM with 3 disks, processed 610 GB, read 527,3 GB, transferred 388,6 GB. Mode NBD.

# threads Processing rate MB/s Duration hour:min:sec
1----------- 84----------------------- 01:48:00
2-----------112-----------------------01:22:00
3-----------153-----------------------01:00:30
4-----------154-----------------------01:00:11

3 threads looks like the best.
Nice feature to backup individual VMs with large disks.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: New NBD-Multithreading Feature v11

Post by tsightler »

Yep, nice result. I've see a few others get improvements in that 40% range, based on the specific network and hardware setup, others don't benefit as much, but nice to see some good results from this feature as it never hurts to have some options for those large VMs that are out there!
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: New NBD-Multithreading Feature v11

Post by mcz »

...and the hunting goes on ;)

Just wannted to test this key as well but didn't see any improvements yet. But at least I've found a bug: With the active key, it works for most of the vm's, but it fails on one disk of one vm. I've created a case: #04729857

Let's see what they find out...
jamcool
Enthusiast
Posts: 67
Liked: 11 times
Joined: Feb 02, 2018 7:56 pm
Full Name: Jason Mount
Contact:

Re: New NBD-Multithreading Feature v11

Post by jamcool »

are there exact steps on how to setup NBD multi-threading? I saw "add the registry dword key VMwareNBDConnectionsPerDisk under HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
and 1, 2, 3 ...". Do you do this on every Windows OS based proxy server?
soncscy
Veteran
Posts: 643
Liked: 312 times
Joined: Aug 04, 2019 2:57 pm
Full Name: Harvey
Contact:

Re: New NBD-Multithreading Feature v11

Post by soncscy »

Any chance this registry value can be improved in the future to be per virtual machine (or even a special processing option in the UI)? I'd imagine I guess that it takes a multi-string value and matches on the machine refid, and relevant jobs would try initiate a multithreaded connection for just the matching virtual machine. I would imagine that it can be set per connection to each VM, right? Or is there some reason it must be global? (I get it's a "v1" of the feature so just having it is great, just a dream for the future :D)

The justification for my request is that as I get it, it's global, and potentially can overwhelm NBD mode if there are too many NBD connections, right? I imagine most people don't need their little 12 GB web servers /appliances to be backed up multithreaded, but they don't want to remake jobs just to isolate their 10 TB+ servers to toggle the registry value on/off (this is how we're doing it with a few clients that want the benefit without it being global -- a pre-job script is applied for all jobs that sets the value appropriately based on whether or not the VMs in the job really need the extra oomph, and VMs are organized into jobs based on size + role)

But it'd be great if this was per machine option, via registry or even in the UI/Powershell :)
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: New NBD-Multithreading Feature v11

Post by Gostev »

@jamcool it's documented in the V11 What's New.

@soncscy honestly, your suggested approach would not be manageable in any decent size environment where VMs come and go in dozens daily. I'd imagine a universal disk size threshold would be the better approach to achieve the same w/out requiring constant manual edits.

However, the answer to your question is no - there's no overwhelming host with too many NBD connections in any case, because our scheduler is aware of NBD multi-threading and won't create more NBD connections than the host can handle. Actually, this is also specifically documented in the V11 What's New :D

My only concern would be some OTHER software creating additional NBD connections. But if you only use Veeam, you should be safe to enable NBD multi-threading enabled. We did not make it the default only because it means changing the behavior of good 300K+ backup server to a brand new logic. This is just too much risk for a relatively small gain. We'd rather make it the default setting later, when it is more proven in the field.
jamcool
Enthusiast
Posts: 67
Liked: 11 times
Joined: Feb 02, 2018 7:56 pm
Full Name: Jason Mount
Contact:

Re: New NBD-Multithreading Feature v11

Post by jamcool »

Gostev,
thank you. The what's new (https://www.veeam.com/veeam_backup_11_0 ... new_wn.pdf) does state "VMwareNBDConnectionsPerDisk (DWORD) registry value on the backup server" on page 11 but did not give the complete registry path. I do see it at the top of this thread. Just wondering if was actually posted somewhere with the complete registry path. Seems like that would be some key information for those that want to try it out.

On a side note, I am really lookin forward to try this out as performance via NBD has been a struggle and I appreciate Veeam trying to find ways to improve performance. Maybe in next release this can be something you can enable/configure via Console and not have to do registry entries.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: New NBD-Multithreading Feature v11

Post by Gostev »

Yes, all Veeam registry values go to this same key.
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: New NBD-Multithreading Feature v11

Post by mcz »

Guys, I've created the case 04729857 because I had issues in combination with this key. Now support told me that it should get fixed in the next patch but it doesn't (I've installed the latest patches). So either they haven't fixed it yet or they have and it still does not work on my side. Could you please have a look and tell me if the bug should have been fixed yet - thanks!
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: New NBD-Multithreading Feature v11

Post by Gostev »

Yes, according to your case it causes some issue with replication jobs specifically. But I think you misunderstood each other with the support engineer, because in the internal case note back from April he clearly documented that this is "planned to be fixed in V11a".
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: New NBD-Multithreading Feature v11

Post by mcz »

Thanks Anton, obviously I had the next "patch" in mind but 11a is a bit a different story...
d3nz
Expert
Posts: 130
Liked: 14 times
Joined: Mar 20, 2018 12:47 pm
Contact:

Re: New NBD-Multithreading Feature v11

Post by d3nz »

Hello.
As I suppose this registry key should be added to VBR?
What's about this feature at v12? It also works with registry key?
RedVision81
Enthusiast
Posts: 39
Liked: 2 times
Joined: Jan 14, 2016 8:02 am
Full Name: Peter
Contact:

Re: New NBD-Multithreading Feature v11

Post by RedVision81 »

the registry key did not work for me, neither in v11 nor v12/12.1 instead i used this blog, which a technician told me yesterday, and this works great.
https://tecblog.au.de/veeam-v12-12-1-do ... rformance/

reduced incrementel time by 50%
its in german but i think this should be no problem
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: New NBD-Multithreading Feature v11

Post by Gostev »

Right, we're currently piloting this with some customers in the field and plan to include in 12.1.2 by default if all is well.
Post Reply

Who is online

Users browsing this forum: Google Feedfetcher, Mildur and 79 guests