Host-based backup of VMware vSphere VMs.
shob
Novice
Posts: 7
Liked: never
Joined: May 13, 2021 5:22 am
Full Name: Lubos Dolezal
Contact:

V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by shob »

Hello,

yesterday, I upgraded our Veeam Backup and Recovery Enterprise from version 10 to version 11.0.0.837 P20210507. Everything it's ok except replications. I manually deleted replica virtual machine and ran replica job to invoke full replica of virtual machine. The replication speed was quite good at first, but gradually decreased to approximately 2-3 MB/s:

Total size: 1,5 TB
Data read: 159,2 GB
Backup size: 132,7 GB
Transferred: 132,7 GB
Botlleneck: Target
Average processing rate: 8 MB/s
Duration: 5:39:41 !!

Our environment:

Vmware vSphere Enterprise Plus version 7.0u2a
SAN Fabric 16Gb
Disc arrays: 2x DELL/EMC Compellent SC v3020
Ethernet 10GbE
Veeam B&R is installed on dedicated physical server

Replication JOB settings:

Replicated virtual machine: VCSA (vCenter Server Appliance)
Source: Compellent SC #1
Destination: Compellent SC #2
CBT: enabled
Data transfer: Direct
Transport mode: Network (NBD)
Align backup file data blocks: enabled
Backup proxy is installed on physical backup server with OS Windows 2012 R2 Standard

On Veeam version 10 with the same configuration, the speed of replication was good and full replica took about 20 minutes.

Fortunately - the increments are quite small, so it takes about 15-20 minutes to replicate daily increment (only about 4 minutes on previosly version of Veeam),
so this issue is not very critical for us ... however, we would like to know what the problem is.


Regards,

L. Dolezal
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam 11 - extremely slow replication job

Post by foggy »

Hi Lubos, for a closer investigation, please open a case with our technical support. Logs will tell what takes longer than before. Just to make sure, is the same transport mode being utilized while writing to the target datastore?
shob
Novice
Posts: 7
Liked: never
Joined: May 13, 2021 5:22 am
Full Name: Lubos Dolezal
Contact:

Re: Veeam 11 - extremely slow replication job

Post by shob »

Hi Foggy - unfortunately, I have found that the problem is much more complex: not only replications, but also restore jobs are extremely slow - restore speed is about 2MB/s!

This is already critical for us, co I created a issue ticket num #04807348 on Veeam Support web portal.

Lubos
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam 11 - extremely slow replication job

Post by Gostev »

There are known NBD/NBDSSL performance issues in vSphere 7.0 U2, the sticky topic mentions that.
I suggest using hot add transport instead (requires backup proxy in a VM) for full VM restores.

Alternatively, you can roll back to an earlier ESXi version until VMware addresses the issue.
Egor Yakovlev
Veeam Software
Posts: 2536
Liked: 680 times
Joined: Jun 14, 2013 9:30 am
Full Name: Egor Yakovlev
Location: Prague, Czech Republic
Contact:

Re: Veeam 11 - extremely slow replication job

Post by Egor Yakovlev »

Hi Lubos.
Please upload entire VBR log package to the case.
Attached screenshot will not be enough to find a root cause, and logs will be the first thing support will ask from you.
/Thanks!
shob
Novice
Posts: 7
Liked: never
Joined: May 13, 2021 5:22 am
Full Name: Lubos Dolezal
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by shob »

I uploaded logs to the case. According to Mr. Gostev, I also try change type of transport from NBD to hot-add. The writing speed has really increased rapidly, but it is a temporary solution for us, because we strongly prefer NBD transport instead of hot-add.

Ok - I think you may close the issue ticket - I am going to reconfigure our Veeam B&R to hot-add and we will wait to hotfix from VMware or Veeam.

Thank you for your assistance

Kind Regards,

Lubos Dolezal
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by mkretzer »

@Gostev: Is the issue only happening with ESXi 7U2 hosts or also with older hosts and 7U2 VCenter?
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by Gostev »

NBD connections go direct from ESXi hosts, vCenter is not involved (except in helping a client to establish one).
jmarkey
Lurker
Posts: 1
Liked: never
Joined: Jun 11, 2021 3:47 pm
Full Name: Jon Markey
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by jmarkey »

Hi-Have the NBD/NBDSSL performance issues been fixed by VMware yet or will this be addressed in a VEEAM patch? We have temporarily implemented hot-add as suggested, but would definitely prefer to be using NBD as we have for the last several years. Any insight would be greatly appreciated. Thanks!
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by Gostev » 1 person likes this post

Unfortunately we cannot address this on our end, as it is a regression in ESXi itself. So I would not expect any changes until the following ESXi 7.0 update releases.
ElmerAcme
Influencer
Posts: 24
Liked: 6 times
Joined: Apr 24, 2020 6:14 am
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by ElmerAcme »

Hello,
we run into a similar performance issue and already created a case yesterday.
We only have ESXi 7.0.1 Installed.
Can somebody confirm that this issue is the same there?
Regards

Just to clarify: the performance issue occurs directly after upgrading from V10 to V11 without updating ESXi.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by Gostev »

Please note that this topic is about ESXi 7.0 U2 and the specific issue with it, so let's not derail it with irrelevant issues.

Feel free to create a topic about your issue following the rules displayed when you click New Topic.
scolledge
Influencer
Posts: 14
Liked: 2 times
Joined: Dec 05, 2019 10:51 am
Full Name: Sean Colledge
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by scolledge »

Do you perhaps have a link to this issue on VMware's site where they acknowledge it is a known bug and we can track the fixing of it there? I am finding this issue over hotadd as well not just NBD. And it started literally on the next replication job after upgrading our DR server to ESXi 7.0 U2. Where the job would usually take an hour and a half to run, it now takes about 8 hours at 7MB/s where it used to be around 70-80MB/s.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by Gostev »

scolledge wrote: Aug 11, 2021 5:57 amI am finding this issue over hotadd as well
There are no known regression in ESXi 7.0 U2 with Hot Add transport, so this one should be as fast as before. Thus, what your are observing can only be caused due to the presence of a bottleneck elsewhere in your backup infrastructure. I recommend you investigate this with our support.

By the way, even NBD performance is not down 10x in U2... no more than 2x from what I remember, depending on the scenario (usually less).

Thanks!
NightBird
Expert
Posts: 242
Liked: 57 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by NightBird »

Do you know if vsphere 7.0U2c resolv this problem ?
arncat
Lurker
Posts: 1
Liked: never
Joined: Sep 28, 2021 1:47 pm
Full Name: Arnaud Catherine
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by arncat »

Hello,

Do you know if upcoming Veeam update P20210525 or vmmware Update 7.0 u2c or u2d resolved this problem ?

Thanks for the reply
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by Gostev »

Hello!

Since the issue is in ESXi, Veeam update can't fix it.

And I have not heard of any recent VMware updates fixing it either.

Thanks!
gingerdazza
Expert
Posts: 191
Liked: 14 times
Joined: Jul 23, 2013 9:14 am
Full Name: Dazza
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by gingerdazza »

Has anyone tried with the new 7.0 U3 with NBD and whether that's helped?
jamcool
Enthusiast
Posts: 67
Liked: 11 times
Joined: Feb 02, 2018 7:56 pm
Full Name: Jason Mount
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by jamcool » 1 person likes this post

I saw that you mentioned were running P20010507 as of September 28th (3 weeks ago). I would at least apply patch P20010525 (came out end of May) or consider just going to v11a that came out almost 2 weeks ago. ESXi 7.0 U2 is not supported until P20010525.

https://www.veeam.com/kb4126
TMC_MG
Novice
Posts: 6
Liked: 1 time
Joined: Nov 06, 2021 10:48 am
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by TMC_MG »

Hi,

I can confirm that with 7 U3 it is still very slow on two tested systems.
Restore on system A under 1 Mb/s. System B 4mb/s

System A has two esxi 7 and the replication job is also 950kb ...

backup jobs are normal...

System : B+R 11.01.1261
Vmware 7 U3

does someone have a solution ?
nettec
Novice
Posts: 4
Liked: 4 times
Joined: Jul 17, 2015 7:36 am
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by nettec » 3 people like this post

I can also confirm the performance is still slow.

ESXi 7.0.3
VEEAM 11.0.1.1261

This is the last update I got from VMware Support:

I hope you are doing well.
Kindly note that I have found an internal Problem Report with the issue and currently our Engineering are working on identifying the root cause.
Their investigations could take some time so if this performance issue is causing any major impacts please feel free to roll back to 6.7.


At least they have acknowledged the issue. Now it's just waitig for the next release or a patch. I don't wanna go back to 6.7.
nettec
Novice
Posts: 4
Liked: 4 times
Joined: Jul 17, 2015 7:36 am
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by nettec »

VMware published an article for the issue now:

https://kb.vmware.com/s/article/86269
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by Gostev »

I wonder where did VMware outsource their technical writing...
TMC_MG
Novice
Posts: 6
Liked: 1 time
Joined: Nov 06, 2021 10:48 am
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by TMC_MG »

I recreated it on a test system to test Vmware U3b 18905247
on the productive systems i have a rate by restore or replicate 1mb/s

So i recreated it on a test system. Vmware 7 u2 Veeam 11.01.1261 windows 10 pc with ssd as Proxy . i have a rate by 35mb/s. Qnap Nas (backupfiles) -> Veeam Proxy - Vmware 7 u2
where is the difference ? the test system has fewer resources.. but is much faster ...

the workaround on https://kb.vmware.com/s/article/86269 describes high latency storage ... this is not my problem ... latency is good... also by replicate from vmware server A to B only 1mb/s.

any ideas ?
TMC_MG
Novice
Posts: 6
Liked: 1 time
Joined: Nov 06, 2021 10:48 am
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by TMC_MG »

i have install B&R 10 and test it on both systems with vmware 7 u2 .. restore speeds up to 90MB/s.

the solution for our case is to downgrade B&R to have no sleeples nights anymore
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by Gostev »

Your case does appear to have anything to deal with the regression discussed in this topic, as your performance is too slow even in light of this regression, meaning something else was causing it. To share some base lines, in our labs we get over 150MB/s NBD transport performance with 11a, where as before the regression was introduced in ESXi 7.0 U2 it was double of that.

In any case, please note that V10 does not support vSphere 7 U2 in principle, so you're running an unsupported environment.
robhagopian
Lurker
Posts: 1
Liked: never
Joined: Feb 19, 2021 5:56 pm
Full Name: Rob Hagopian
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by robhagopian »

Confirmed slow as molasses (8MB/s) replication over 10Gb between two VRTXs with vmware 7.0u3 and 11.0.1.1261 P20211123. I notice that concurrent streams each go at the same slow speed but total throughput increases perfectly.

I'll try the hot-add stuff - my first time dipping my toes in that <sigh>
nettec
Novice
Posts: 4
Liked: 4 times
Joined: Jul 17, 2015 7:36 am
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by nettec » 1 person likes this post

I wont use the hot-add method. Had some very bad experience with hot-add where the VMDKs of my VMs were not released anymor after the job finished or failed, which resulted in rebuilding my exchange server because it was dead after some time.

Still waiting for a fix from VMware. If they can't fix this I am clean installing all my servers with ESXi 6.7.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by Gostev »

If your experience with hot-add was a while back, then you should try it again because its reliability was improved significantly over years.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V11 + ESXi 7.0 U2: extremely slow replication and restore over NBD

Post by mkretzer » 1 person likes this post

I also did not like hotadd in the past but i prefer it over all other methods now, especially with the linux hotadd proxies that are now available.
Its really stable (any i am very paranoid!).
Post Reply

Who is online

Users browsing this forum: No registered users and 78 guests