Discussions specific to the VMware vSphere hypervisor
Post Reply
foggy
Veeam Software
Posts: 17712
Liked: 1481 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam 9.5 Udpate 3a and vSphere 6.5 Update 2

Post by foggy » Dec 14, 2018 12:36 pm 1 person likes this post

Hi Falko, please review this thread for more information. Thanks!

Yuya
Enthusiast
Posts: 41
Liked: 1 time
Joined: Jan 30, 2013 9:32 am
Contact:

Re: vSphere 6.5U2 support

Post by Yuya » Dec 25, 2018 8:37 am

UPDATE: Our auto-tests have exhibited one potentials issue with 6.5 U2 which may reduce reliability of a number of Veeam Backup & Replication features dependent on the corresponding protocol. Having said that, the issue is only reproducible in our stress testing lab where we emulate a heavy environment load, meaning for those who already upgraded these issue will likely exhibit "random" behavior which is in reality tied up to peak load in the environment. We're opening a support case with VMware - and I will keep you posted on the progress. Thanks!
1)Specifically, what kind of problems do you encounter?
For example, will the backup continue to fail until the load of ESXi goes down?
Or, because there is no reliability of the backup file, is there a possibility that it can not be restored?

2)I understand that VMware support recognizes that the cause of this problem is the vSphere API.
Do you have information on when to be fixed?

3)Is there a workaround for this problem?

I could not understand from the forum, so I'm glad if you tell me.

Thanks,
Yuya

foggy
Veeam Software
Posts: 17712
Liked: 1481 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: vSphere 6.5U2 support

Post by foggy » Dec 26, 2018 12:28 pm 1 person likes this post

I'll try to summarize what you can find in this thread above for you:
Yuya wrote:1)Specifically, what kind of problems do you encounter?
Like mentioned above, almost all product functionality might behave unreliably due to ESXi API failures during periods when the environment operates under heavy load.
Yuya wrote:For example, will the backup continue to fail until the load of ESXi goes down?
The issue has random nature but does not show up under normal load conditions, so if your hosts are not over-provisioned, you should be safe.
Yuya wrote:Or, because there is no reliability of the backup file, is there a possibility that it can not be restored?
This behavior does not cause any data corruptions, you just will not be able to perform some operations and, most likely, to restore from the backup files created during failed jobs. If subsequent job run doesn't produce any errors/warnings, the corresponding restore point should be restorable.
Yuya wrote:Do you have information on when to be fixed?
Currently we do not have any information in this regard.
Yuya wrote:3)Is there a workaround for this problem?
Currently the only workaround is to provide environment operation under normal load.

Yuya
Enthusiast
Posts: 41
Liked: 1 time
Joined: Jan 30, 2013 9:32 am
Contact:

Re: vSphere 6.5U2 support

Post by Yuya » Jan 08, 2019 4:07 am

Thank you for your answer.

This issue can occur when the load on the ESXi host is high.
I understood that this is the ESXi host where the virtual machine to be backed up resides, correct?
Does the vCenter load have an impact?

For example, suppose vCenter is created in a virtual machine and this vCenter VM is in ESXi 6.5 u2.
- The load on the ESXi host (A) where the virtual machine to be backed up resides is low.
- The load on the ESXi host (B) where the vCenter VM resides is high.
In this case, is there a possibility that this issue occurs?

Even if the load of the ESXi host where the vCenter VM resides is high, if the load of the ESXi host where the virtual machine to be backed up resides is low, this issue is unlikely to occur, correct?

Thanks,
Yuya

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » Jan 08, 2019 4:09 pm

Correct.

skrause
Expert
Posts: 381
Liked: 75 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Re: vSphere 6.5U2 support

Post by skrause » Jan 08, 2019 9:08 pm

Is there a ballpark level of host resource consumption (CPU/RAM%) that this happens above but that would likely be safe if consumption stayed below?

We are unable to upgrade to 6.7 due to some of our storage arrays not supporting (or planning on supporting) anything above 6.5.
Steve Krause
Veeam Certified Architect

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » Jan 09, 2019 2:52 am

All I know it's very high load, this was observed in our stress testing lab.

fll
Enthusiast
Posts: 36
Liked: never
Joined: Sep 05, 2016 10:35 am
Contact:

[MERGED] Veeam Backup & Replication 9.5 Update 3a and VMware vSphere 6.5 U2.

Post by fll » Jan 11, 2019 1:43 pm

Hello,
According to KB 2646, there are some problems between Veeam Backup & Replication 9.5 U3a and VSphere 6.5 U2.
"VMware vSphere 6.5 U2." This update introduces preliminary support by addressing all outstanding U2-specific compatibility issues that can be managed from the Veeam side, however, there is a major regression in ESXi 6.5 U2 code that makes the vSphere API fail randomly during high host CPU load periods, consequently impacting a variety of Veeam Backup & Replication functionality VMware is currently troubleshooting this bug, and fixing one will most likely require the new ESXi 6.5 U2 build issued. "
Does the same thing happen in ESXi 6.5 U1?

Thank you.

wishr
Veeam Software
Posts: 592
Liked: 55 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: vSphere 6.5U2 support

Post by wishr » Jan 11, 2019 2:09 pm

Hi Fll,

The given issue was discovered just in 6.5 U2 release bits, so it's safe to install 6.5 U1 for use with B&R 9.5 Update 2 or later versions. You can check that thread for additional information regarding 6.5 U1 support.

I've moved your post to an existing thread covering 6.5 U2 support topic.

Thanks

perez
Lurker
Posts: 1
Liked: 1 time
Joined: Jan 14, 2019 9:14 am
Contact:

Re: vSphere 6.5U2 support

Post by perez » Jan 14, 2019 2:05 pm 1 person likes this post

- What about the possibility of the problem could be caused by the NFC limit, which it is not restricted by the moment in ESXi 6.5U2?
It looks that in ESXi 6.7, there is not created a new thread for each NFC connection, so the thread limitation won't be hit and consequently the problem looks not exist.
- Is it possible that even if VEEAM is NOT using the API, and instead performs a large number of NFC operations, more than 25 simultaneous NFC connections, the problems also occurs, because VEEAM is hitting an NFC limit?
-Has VEEAM tested if the problem persist if VEEAM client does not hit the limit of 25 simultaneous NFC connections, using the API?
-If the problem is finally related to the NFC connections, is it possible that the new VEEAM release, include any fix which could update the client in order to restrict the number of NFC connection, and solve the problem or you think that the fix would have to be in a newest ESXi release?

- And finally, could you please provide me, the SR number of the case you opened to VMWare?
Thanks

wishr
Veeam Software
Posts: 592
Liked: 55 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: vSphere 6.5U2 support

Post by wishr » Jan 14, 2019 4:45 pm

Hi Perez,

Please find the answers below.
perez wrote:What about the possibility of the problem could be caused by the NFC limit, which it is not restricted by the moment in ESXi 6.5U2?
No, unfortunately, the root of the issue cause was located on the VMware VDDK end that is used by B&R along with VMware API for performing backup and restore operations within VMware hypervisors. There were some memory leak issues in one of the VDDK versions used by B&R (just to let you know we deliver several VDDK libraries in each version to ensure compatibility and good customer experience because VDDK compatibility tests performed by VMware are far from being ideal).
perez wrote:Is it possible that even if VEEAM is NOT using the API, and instead performs a large number of NFC operations, more than 25 simultaneous NFC connections, the problems also occurs, because VEEAM is hitting an NFC limit?
No, please see above. NFC connections limit is well known here and always taken into account.
perez wrote:Has VEEAM tested if the problem persist if VEEAM client does not hit the limit of 25 simultaneous NFC connections, using the API?
Sure, this is tested frequently.
perez wrote:If the problem is finally related to the NFC connections, is it possible that the new VEEAM release, include any fix which could update the client in order to restrict the number of NFC connection, and solve the problem or you think that the fix would have to be in a newest ESXi release?
Sure, from one B&R version to another we update VDDK to the most stable versions and always deliver several VDDK versions in a single product bundle. Moreover, we test them internally, because VDDK issues are NOT something unusual. Regardless of that, this particular issue required changes in VDDK itself and on the ESXi end, so our ability to deal with it on our own was extremely limited.

The initial SR had a number 18797316305. After VDDK replacement the issue was fixed, but since that, we started observing another issue that itself disappeared after a couple updates to VDDK and ESXi servers in our labs that obviously required us to extensively communicate the issues, wait until VMware release their patches, test them internally, etc.

We'll update this thread with additional useful information once have it and it should be finally good news.

BR,
Fedor

CPCS
Influencer
Posts: 16
Liked: 3 times
Joined: Mar 01, 2016 1:49 pm
Contact:

Re: vSphere 6.5U2 support

Post by CPCS » Jan 21, 2019 9:57 am

I would like to know what we could expect now from Update4 or Veeam or Vmware? Do we rely on VMware now to fix the left issues? Do everyone of us has to open a case for that so they will fix it or will they FINALLY do this after nearly one year everyone waiting for this?

This is really frustrating, sitting on ESXi 5.5 (already out of Support!) and not able to update (cause hardware is not supporting 6.7) and just looking at vmware/veeam/whoever not getting this fixed AFTER ALL THIS TIME...

foggy
Veeam Software
Posts: 17712
Liked: 1481 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: vSphere 6.5U2 support

Post by foggy » Jan 21, 2019 10:06 am

U4 doesn't change anything in this regard, the ball is on VMware side. Why don't you consider going with whatever version between 5.5 and 6.2U2: 6.0/6.5/6.5U1? All of them are supported and 6.5U1 proved to be rock solid.

CPCS
Influencer
Posts: 16
Liked: 3 times
Joined: Mar 01, 2016 1:49 pm
Contact:

Re: vSphere 6.5U2 support

Post by CPCS » Jan 21, 2019 10:11 am

Because as war as i understood, if you want to install latest security patches (which you should today) it is mandatory to update to 6.5U2 (coming from 6.5(U1), isnt it?

"the ball is on VMware side" does mean what exactly? Veeam does have an open ticket on vmware side for this issue?

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » Jan 21, 2019 8:25 pm

Yes, it is referenced by wishr in the post just above yours.

Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests