V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

[i2M]

Does anyone know if this issue exists in ESXi 7.0 U1 as well? I have a brand new SimpliVity setup running (where Hot Add is not recommend) and I seem to be running into this, but it's 7.0 U1 not U2. Everything 10 GB of course.

In network mode I get ~ 80 -90 MB/s no matter how many streams, and any single HDD stream will never go above 50 MB/s. With the same job I can get up to 400 MB/s with 200 MB/s write streams by using Hot Add instead.

Network Mode Speeds


Hot add speeds:


Before I spend too much longer digging deeper, does anyone know if U1 has the same issue but possibly to a lesser extent? I notice that the extremely poorly worded VMWare KB mentions it affects 7.0 and up, but it's a such a badly written article I don't know if it's a typo?

[d.artzen]

Hi,

AFAIK all versions of Vsphere 7 have this behavior. We noticed the decrease in speed in our environment after the Upgrade from 6.7 to 7.0U1. I am unsure if this was fixed in 7.0U3c, but until U3b it was still there. We did have even less speed than you though. This was the reason for us to change to Hot-Add Mode, which solved it for us.

[i2M]

Thanks for the info. I guess we'll need to stick with HotAdd or live with the reduced speeds. Or downgrade to 6.7..

[martinhdk]

For what it's worth...
We ran into the same issue after upgrading our Primary and Secondary Sites to vSphere 7.0.

Our Replication Destination is Secondary Site - and it was sufficient for us to just change Transport Mode for the Target Proxies on Secondary Site.
Our Source Proxies on Primary Site still run NBD when processing Production VMs - with no risk of Proxies "hijacking" Production VMDKs.

After the change on Target Proxies, our overall Job Processing Rate went from 30-60 MB/s to 150-250 MB/s.
On individual VMDKs we saw an increase from 5-10 MB/s to 100-200 MB/s.
(The latter being most important since certain VMs were unable to complete replication within a 1 hour window)

Best regards
Martin

[gingerdazza]

Nutanix "strongly discourage the use of virtual appliance backup mode (hot-add). This mode increases the load on vCenter and may lead to orphaned snapshots; also, snapshot commits may stun VMs". How have VMWare not fixed this by now? Poor. Anyone aware of any update from VMWare?

[Gostev]

Indeed these all seem like old issues which VMware has fixed (or Veeam has workarounded) by now. So unless issues are specific to Nutanix architecture, I don't believe the recommendation above is valid. Hot-add backup mode is used extremely widely by Veeam customers.

VMware devs seem to had a breakthrough a few days ago as they reported they found the way to restore NBD performance to original levels. The fix should be included in the next VDDK release.

[lwarner]

Support asked me to post to the forums with our case, 05435549.

We are seeing this behaviour on all replication jobs that run NBD on traditional storage. In our case, the performance hit is pretty severe, with jobs running at about 5-10 MB/s instead of 100's of MB/s with hotadd.

Our situation is that a VMware bug is preventing the use of hotadd for VMs that use custom Storage Policies.
Hopefully this can be resolved because it is now completely impossible to run replication jobs for any larger VM guests.
We are on vSphere 7.0U2d, and vCenter 7.0U3d.

[timchin]

Ugh. We are experiencing this, too.

Has the Veeam dev team seen this article? https://kb.vmware.com/s/article/83401?lang=en_US

[TMC_MG]

Today i have tested a Replication on a new Server with Veeam B&R 11.0.1.1261 P20220302 and Vmware 7.0 Update3 (target).
The replication job rans with < 1mb/s and will take for 50 GB > 24 hours.

I can't believe there hasn't been a solution for months ?!?


has anyone found a solution ??

[foggy]

Have you ever opened a support case to investigate a performance issue? I do not see any case number posted by you in the thread. In your case, the issue is likely environmental so your setup should be closely reviewed by our engineers.

[TMC_MG]

Yes, but the test are with Community Editions and my support case 05118360 is closed without a answer.:

This is Veeam Support with a system generated courtesy notification. Unfortunately, due to high Support Team load we were unable to process your request and it has been archived automatically.

[foggy]

Understood. You can try again later. Anyway, what I meant was it's hard to speak about a solution without the ability to investigate the issue in the first place and especially when the solution hides in the environment itself.

[TMC_MG]

ok. i find a solution or a "workaround" that is ok for us.

I unterstand now that we have to add a proxy to a windows server that is on the vmware server site.
you haven't install full veeam on this server. you can add it by backup proxies / add new vmware backup proxy / add new / windows server.

So now we change (properties in the added proxy) the transport mode to virtual appliance.
now the restore or replication use this option (you can see it by restore live-log, [hot add] and not nbd) .
the speed is up form 800 kb to 180 mb/s and 50 GB restores in a few minutes.
So the only way to NOT use nbd is to install the proxy on the vmware server site on a windows host with access to the datastore.
you can use the recover with select the new proxy and you can chose entire vm restore or instant recovery.
both speeds up.


hallelujah !

[foggy]

Well, that is not actually a workaround but the best practice. Anyway, thanks for getting back and sharing - glad you've nailed it.

[d.artzen]

ok. i find a solution or a "workaround" that is ok for us.

I unterstand now that we have to add a proxy to a windows server that is on the vmware server site.
you haven't install full veeam on this server. you can add it by backup proxies / add new vmware backup proxy / add new / windows server.

So now we change (properties in the added proxy) the transport mode to virtual appliance.
now the restore or replication use this option (you can see it by restore live-log, [hot add] and not nbd) .
the speed is up form 800 kb to 180 mb/s and 50 GB restores in a few minutes.
So the only way to NOT use nbd is to install the proxy on the vmware server site on a windows host with access to the datastore.
you can use the recover with select the new proxy and you can chose entire vm restore or instant recovery.
both speeds up.


hallelujah !

Just as a small correction, the proxy does not have to be on a Windows OS, it also can be a supported Linux Distro (we use Ubuntu 20.04 LTS for this, works great). Only other requirement is that the proxy must reside on a Host that has access to all datastores, where the VMs are located. If using local datastores of the hosts you need a proxy on each Esx Host.
But yeah VMWare broke NBD pretty good with vSphere 7. This was also our reason to switch to Hot-Add.

[mitchberlin]

I've been doing some testing on this issue. We are replicating a single vCenter Server Appliance VM from one host to another using NDB and a local SCSI datastore (RAID 10, SSDs). The source host is vSphere 7.0 U2. When the destination host is vSphere 6.7, the speed is normal. If we rebuild that same destination host with vSphere 7.0 U2, the speed drops to 2-5 MB/s. As others have indicated in this post, it seems we can isolate the issue to vSphere 7.0. My questions are:

1. Is this issue present on all versions of vSphere 7.0 or just 7.0 U2 and higher? It's mentioned in previous posts that it MIGHT be all 7.0, but has anyone validated this?
2. The Hot Add option - is this something people are doing with the vCenter Server Appliance? VMware isn't really endorsing image based backups for vcsa anymore and to now enable hot add seems like we're even going further down the unsupported route.

I do have a Veeam case open on this issue (Case # 05466382). The support engineer contacted a backline team and shared this:
"I inquired about how this issue will be addressed going forward and was informed that any changes in Veeam code will be available only in version 12, and are yet to be determined."

So, no plans for a fix in version 11, but maybe in version 12.

[ascinto]

We've been facing this issue for some time now. Case 05420231.

We tried hot add and it definitely increases throughput. However, we're replicating over 1000 VMs. If there is ever any issue on our side or the tenant side, it will cause a disk to become stuck on the proxy. The job will then subsequently fail until someone manually goes into vcenter and removes the disk from the proxy and possibly consolidate snapshots on the replica. This is a nightmare at scale and not best practice for service providers replicating thousands of disks.

[domsi]

VMware devs seem to had a breakthrough a few days ago as they reported they found the way to restore NBD performance to original levels. The fix should be included in the next VDDK release.

VMware has updated the ticket: https://kb.vmware.com/s/article/86269?l ... urce=email

VMware Engineering has identified the issue only happen for Windows proxy, we have improved the performance for Windows with VDDK 8.0 and VDDK 7.0.3 Patch2

@veeam - when will be an update / hotfix for the improved version available?

[Gostev]

Most likely in V12 at the earliest, as we won't have any release vehicles until then. Updating a VDDK version is not a "hot fix" type of a change unfortunately.

[domsi]

Thank you. Do you have an estimated release date for V12?

[Gostev]

We're aiming to ship at the end of this year.

[e.rottier]

It's mentioned in previous posts that it MIGHT be all 7.0, but has anyone validated this?

Just my 2 cents.

My Veeam version is Build: 11.0.1.1261 P20220302
My vCenter version is Version: 7.0.3, Build: 19480866 (although I'm updating very soon)
My ESXi version is 7.0.3, 19193900 (=7.0 U3c)
My LAn is 10 Gb/s

My last few VMware backup has processing rate where 232, 213, 211 and 219 MB/s.

I am using the GFS-system. Daily incrementals, weekly and monthly full backups, with 3 synthetics and 1 active.
Throughout the past months the speed has been between 193 and 266 MB/s with one or two drops to low speeds like 83 probably to disk size changes.

Every VM-backup log this line: "Using backup proxy VMware Backup Proxy for disk Hard disk <number> [nbd]"
So it's using NDB.

[nettec]

So then let me write down my quick tests I did today, which resulted in exactly the same as all the other 7.0.x versions I have tested before. I updated one ESXi host to the latest build 7.0.3f, 20036589 and left the other hosts on 6.7.0, 15160138. I performed 2 runs for each test, and for each run I copied a 10GB ISO file to a disk on the VM to have some changes to replicate on it.
Normal average write speed is about 40-70MB/s using NBD for us.

You can clearly see the write speed drop for the 3. test where the target host is on the ESXi 7.0 build. So for me the problem occurs when the target host for the replication job is on 7.0. Maybe someone can confirm this? I think this is caused by our source proxy using [san] and target proxy is using [nbd].
Please keep in mind that the exact same host did not have NBD performance issues when it was on 6.7 before I upgraded it to 7.0.3, and it also wont have issues when I downgrade it back to 6.7, because I already did that to another host in the last few months.


1. Test:
Source VM host: ESXi 6.7.0
Target VM host: ESXi 6.7.0
1. Run max write speed: 77MB/s
2. Run max write speed: 46MB/s
Duration: Both runs took about 5-6 minutes to finish.

2. Test:
Source VM Host: ESXi 7.0.3f
Target VM Host: ESXi 6.7.0
1. Run max write speed: 54MB/s
2. Run max write speed: 72MB/s
Duration: Both runs also took about 5-6 minutes to finish.

3. Test:
Source VM Host: ESXi 6.7.0
Target VM Host: ESXi 7.0.3f
1. Run max write speed: 4MB/s
Duration: I stopped the first run manually because after 35 minutes the progress was only at 27% and the speed dropped to 2MB/s. I expected the job to take 60 to 90 minutes to run through, maybe even more.
In addition to that the job took over 1 hour to stop or even removing the snapshots, which is also not normal. Normally jobs are stopping within a few minutes depending on the VM- and snapshot-size.

I didn't bother to start a second run for the 3. test because the results of the first one were very clear and matched the results of my previous tests with earlier 7.0 versions.

Even read speed dropped below 4MB/s for the 3. test.


All I hear from VMware support everytime I ask back is: A solution is going to be implemented "very soon"...

And yeah 6.7 will be end of support by the and of this year.

I hope that the Patch 2 which VMware mentioned will be released soon and hopefully fixes the issue.


Regards, Cedy

[jasbos]

Esxi 8 and V11 still facing the issue.

[Milenco]

Just to follow up here: I've upgraded to Veeam 12 and the NBD restore is still very slow (4MB/s for us), so no improvement here..

[Gostev]

This is likely a different issue. The NBD performance issue discussed in this topic, which was in fact mostly addressed in newer VDDK versions included with V12, relates to the issue when customers would get "only" 200MB/s on a 10Gbps network.

[Andreas Neufert]

It is strange speed limitation that I think is not related. We see 200MB/s and faster with this but not at the previous level when the VMware limtiation is met with it. 4MB/s looks more of a environment limitation. I suggest to reach out to our support.

[Milenco]

I have created a support ticket by your advice, but I believe the issue is related to others here: It started when upgrading the ESXi 7, before on 6.7 we had great restore speeds. The backup speeds are still great (about 103 MB/s over 1Gbit) it's just the restore speed (which defaults to nbd) which is terrible at around 4MB/s. I upgraded our ESXi hypervisors to 7.0U3j yesterday as well, hoping that might solve it but unfortunately the issue remains.

Curious if others here who reported issues have theirs resolved by updating to Veeam 12 with the new VDDK library.

[Andreas Neufert]

If you have a 1GB/s environment I really suggest to create a Linux/Windows VM and use it as HotAdd Proxy.
As we send and receive compressed data to it, your bandwidth will double with this. As well HotAdd will not be limited by any VMware technology other than the disk backend itself. You can easily test this with a disk test tool in a VM to see what the read and write maximum speed is.

[davwalbfs]

We just upgraded to V12 and also still are having this issue. We are doing VM replications from one ESXi 7 host to another ESXi7 host and also only get ~4 to max 10 MB/s since ESXi 7 upgrade. Before upgrade our ESXi hosts were on 6.5 and we had replication speeds of ~60-80 MB/s in a 1 Gb/s network.

@Milenco: You are talking about a new VDDK library, could you please explain what you mean by that? Thanks!