V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

[shob]

Hello,

yesterday, I upgraded our Veeam Backup and Recovery Enterprise from version 10 to version 11.0.0.837 P20210507. Everything it's ok except replications. I manually deleted replica virtual machine and ran replica job to invoke full replica of virtual machine. The replication speed was quite good at first, but gradually decreased to approximately 2-3 MB/s:

Total size: 1,5 TB
Data read: 159,2 GB
Backup size: 132,7 GB
Transferred: 132,7 GB
Botlleneck: Target
Average processing rate: 8 MB/s
Duration: 5:39:41 !!

Our environment:

Vmware vSphere Enterprise Plus version 7.0u2a
SAN Fabric 16Gb
Disc arrays: 2x DELL/EMC Compellent SC v3020
Ethernet 10GbE
Veeam B&R is installed on dedicated physical server

Replication JOB settings:

Replicated virtual machine: VCSA (vCenter Server Appliance)
Source: Compellent SC #1
Destination: Compellent SC #2
CBT: enabled
Data transfer: Direct
Transport mode: Network (NBD)
Align backup file data blocks: enabled
Backup proxy is installed on physical backup server with OS Windows 2012 R2 Standard

On Veeam version 10 with the same configuration, the speed of replication was good and full replica took about 20 minutes.

Fortunately - the increments are quite small, so it takes about 15-20 minutes to replicate daily increment (only about 4 minutes on previosly version of Veeam),
so this issue is not very critical for us ... however, we would like to know what the problem is.


Regards,

L. Dolezal

[foggy]

Hi Lubos, for a closer investigation, please open a case with our technical support. Logs will tell what takes longer than before. Just to make sure, is the same transport mode being utilized while writing to the target datastore?

[shob]

Hi Foggy - unfortunately, I have found that the problem is much more complex: not only replications, but also restore jobs are extremely slow - restore speed is about 2MB/s!

This is already critical for us, co I created a issue ticket num #04807348 on Veeam Support web portal.

Lubos

[Gostev]

There are known NBD/NBDSSL performance issues in vSphere 7.0 U2, the sticky topic mentions that.
I suggest using hot add transport instead (requires backup proxy in a VM) for full VM restores.

Alternatively, you can roll back to an earlier ESXi version until VMware addresses the issue.

[Egor Yakovlev]

Hi Lubos.
Please upload entire VBR log package to the case.
Attached screenshot will not be enough to find a root cause, and logs will be the first thing support will ask from you.
/Thanks!

[shob]

I uploaded logs to the case. According to Mr. Gostev, I also try change type of transport from NBD to hot-add. The writing speed has really increased rapidly, but it is a temporary solution for us, because we strongly prefer NBD transport instead of hot-add.

Ok - I think you may close the issue ticket - I am going to reconfigure our Veeam B&R to hot-add and we will wait to hotfix from VMware or Veeam.

Thank you for your assistance

Kind Regards,

Lubos Dolezal

[mkretzer]

@Gostev: Is the issue only happening with ESXi 7U2 hosts or also with older hosts and 7U2 VCenter?

[Gostev]

NBD connections go direct from ESXi hosts, vCenter is not involved (except in helping a client to establish one).

[jmarkey]

Hi-Have the NBD/NBDSSL performance issues been fixed by VMware yet or will this be addressed in a VEEAM patch? We have temporarily implemented hot-add as suggested, but would definitely prefer to be using NBD as we have for the last several years. Any insight would be greatly appreciated. Thanks!

[Gostev]

Unfortunately we cannot address this on our end, as it is a regression in ESXi itself. So I would not expect any changes until the following ESXi 7.0 update releases.

[ElmerAcme]

Hello,
we run into a similar performance issue and already created a case yesterday.
We only have ESXi 7.0.1 Installed.
Can somebody confirm that this issue is the same there?
Regards

Just to clarify: the performance issue occurs directly after upgrading from V10 to V11 without updating ESXi.

[Gostev]

Please note that this topic is about ESXi 7.0 U2 and the specific issue with it, so let's not derail it with irrelevant issues.

Feel free to create a topic about your issue following the rules displayed when you click New Topic.

[scolledge]

Do you perhaps have a link to this issue on VMware's site where they acknowledge it is a known bug and we can track the fixing of it there? I am finding this issue over hotadd as well not just NBD. And it started literally on the next replication job after upgrading our DR server to ESXi 7.0 U2. Where the job would usually take an hour and a half to run, it now takes about 8 hours at 7MB/s where it used to be around 70-80MB/s.

[Gostev]

I am finding this issue over hotadd as well

There are no known regression in ESXi 7.0 U2 with Hot Add transport, so this one should be as fast as before. Thus, what your are observing can only be caused due to the presence of a bottleneck elsewhere in your backup infrastructure. I recommend you investigate this with our support.

By the way, even NBD performance is not down 10x in U2... no more than 2x from what I remember, depending on the scenario (usually less).

Thanks!

[NightBird]

Do you know if vsphere 7.0U2c resolv this problem ?

[arncat]

Hello,

Do you know if upcoming Veeam update P20210525 or vmmware Update 7.0 u2c or u2d resolved this problem ?

Thanks for the reply

[Gostev]

Hello!

Since the issue is in ESXi, Veeam update can't fix it.

And I have not heard of any recent VMware updates fixing it either.

Thanks!

[gingerdazza]

Has anyone tried with the new 7.0 U3 with NBD and whether that's helped?

[jamcool]

I saw that you mentioned were running P20010507 as of September 28th (3 weeks ago). I would at least apply patch P20010525 (came out end of May) or consider just going to v11a that came out almost 2 weeks ago. ESXi 7.0 U2 is not supported until P20010525.

https://www.veeam.com/kb4126

[TMC_MG]

Hi,

I can confirm that with 7 U3 it is still very slow on two tested systems.
Restore on system A under 1 Mb/s. System B 4mb/s

System A has two esxi 7 and the replication job is also 950kb ...

backup jobs are normal...

System : B+R 11.01.1261
Vmware 7 U3

does someone have a solution ?

[nettec]

I can also confirm the performance is still slow.

ESXi 7.0.3
VEEAM 11.0.1.1261

This is the last update I got from VMware Support:

I hope you are doing well.
Kindly note that I have found an internal Problem Report with the issue and currently our Engineering are working on identifying the root cause.
Their investigations could take some time so if this performance issue is causing any major impacts please feel free to roll back to 6.7.

At least they have acknowledged the issue. Now it's just waitig for the next release or a patch. I don't wanna go back to 6.7.

[nettec]

VMware published an article for the issue now:

https://kb.vmware.com/s/article/86269

[Gostev]

I wonder where did VMware outsource their technical writing...

[TMC_MG]

I recreated it on a test system to test Vmware U3b 18905247
on the productive systems i have a rate by restore or replicate 1mb/s

So i recreated it on a test system. Vmware 7 u2 Veeam 11.01.1261 windows 10 pc with ssd as Proxy . i have a rate by 35mb/s. Qnap Nas (backupfiles) -> Veeam Proxy - Vmware 7 u2
where is the difference ? the test system has fewer resources.. but is much faster ...

the workaround on https://kb.vmware.com/s/article/86269 describes high latency storage ... this is not my problem ... latency is good... also by replicate from vmware server A to B only 1mb/s.

any ideas ?

[TMC_MG]

i have install B&R 10 and test it on both systems with vmware 7 u2 .. restore speeds up to 90MB/s.

the solution for our case is to downgrade B&R to have no sleeples nights anymore

[Gostev]

Your case does appear to have anything to deal with the regression discussed in this topic, as your performance is too slow even in light of this regression, meaning something else was causing it. To share some base lines, in our labs we get over 150MB/s NBD transport performance with 11a, where as before the regression was introduced in ESXi 7.0 U2 it was double of that.

In any case, please note that V10 does not support vSphere 7 U2 in principle, so you're running an unsupported environment.

[robhagopian]

Confirmed slow as molasses (8MB/s) replication over 10Gb between two VRTXs with vmware 7.0u3 and 11.0.1.1261 P20211123. I notice that concurrent streams each go at the same slow speed but total throughput increases perfectly.

I'll try the hot-add stuff - my first time dipping my toes in that <sigh>

[nettec]

I wont use the hot-add method. Had some very bad experience with hot-add where the VMDKs of my VMs were not released anymor after the job finished or failed, which resulted in rebuilding my exchange server because it was dead after some time.

Still waiting for a fix from VMware. If they can't fix this I am clean installing all my servers with ESXi 6.7.

[Gostev]

If your experience with hot-add was a while back, then you should try it again because its reliability was improved significantly over years.

[mkretzer]

I also did not like hotadd in the past but i prefer it over all other methods now, especially with the linux hotadd proxies that are now available.
Its really stable (any i am very paranoid!).