Discussions specific to the VMware vSphere hypervisor
Gostev
SVP, Product Management
Posts: 24293
Liked: 3330 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » Oct 21, 2018 9:08 pm

ctsherman wrote:
Oct 21, 2018 12:29 am
In absence of hotadd or proxyVMs, will this regression issue affect us?
Yes. The only thing that seems to matters is your host load.

madhon
Lurker
Posts: 2
Liked: never
Joined: Jun 12, 2017 8:25 am
Full Name: Jonatho Douglas
Contact:

Re: vSphere 6.5U2 support

Post by madhon » Oct 24, 2018 12:35 pm

Just noticed VMware have released a new patch ESXi-6.5.0-20181004002-standard (Build 10390116) which includes the following in the release notes ..

PR 2209189: Heavy I/Os issued to a snapshot virtual machine using the SEsparse format might cause guest OS file system inconsistencies
Heavy I/Os issued to a snapshot virtual machine using the SEsparse format might cause guest OS file system inconsistencies, or data inconsistencies in applications.

Is this the esxi side fix for the regression ?

Gostev
SVP, Product Management
Posts: 24293
Liked: 3330 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » Oct 25, 2018 10:27 am

No, it's a different regression that VMware has first disclosed just last week - we were not aware of it.

bdufour
Expert
Posts: 198
Liked: 29 times
Joined: Nov 01, 2017 8:52 pm
Full Name: blake dufour
Contact:

Re: vSphere 6.5U2 support

Post by bdufour » Oct 25, 2018 4:31 pm

im on 6.5.0 Update 2 with network mode and not experiencing these issues or any other issue related to veeam.

cheers!

bpayne
Enthusiast
Posts: 52
Liked: 12 times
Joined: Jan 20, 2015 2:07 pm
Full Name: Brandon Payne
Contact:

Re: vSphere 6.5U2 support

Post by bpayne » Oct 29, 2018 1:11 pm

rmyers wrote:
Oct 18, 2018 6:59 pm
To the folks who mentioned use network mode over hot add - I would have to better understand if there is a performance hit doing so considering the size of our environment and the number of jobs I have running. I would expect proxy hot add to be the most efficient way to complete the backups.
@rmyers - I have a very similar environment. I bit the bullet last week and switched all my Veeam proxies to Network Mode only. I was pretty nervous doing this given the size of my environment, but all has been well the past 7 days. I did notice a decrease in performance though as anticipated so expect that your backup window might go a bit longer.

lroscoe
Novice
Posts: 8
Liked: 1 time
Joined: May 17, 2017 4:55 pm
Full Name: Leo Roscoe
Contact:

Re: vSphere 6.5U2 support

Post by lroscoe » Nov 15, 2018 3:52 pm

The following issue listed below just started to affect our environment. I see some response in the Veeam forums about a possible VMware patch being released by the end of November to address this issue. We've opened a case with VMware support and Veeam support but haven't gotten a concrete answer of when a patch will be released to address this issue. Any updates?

Backup Proxy VMs goes invalid and terminates during VM backup
https://kb.vmware.com/s/article/56453

Gostev
SVP, Product Management
Posts: 24293
Liked: 3330 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » Nov 21, 2018 4:19 pm

This looks to be a different 6.5 U2 issue from the "main" one that have been discussed in this thread so far. We have not been in contact with VMware on this one, so I don't know much about it at the moment. Except that it is certainly not a common issue in our support.

anpa
Service Provider
Posts: 19
Liked: 9 times
Joined: Oct 27, 2011 8:42 am
Full Name: Andreas Cederlund
Location: Landskrona, Sweden
Contact:

Re: vSphere 6.5U2 support

Post by anpa » Nov 29, 2018 11:43 am

Hi guys,

Have you got any updates on this?
Do you have any figures on how many customers that have 6.5U2 are experiencing problems?
We are running metro cluster setup, so our hosts are never loaded to more than 50%, so if this is triggered by high load perhaps we could be safe?

/Andreas

Gostev
SVP, Product Management
Posts: 24293
Liked: 3330 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » Nov 29, 2018 10:07 pm

Hi - yes, you should be safe with your load. In general, I don't believe we have had many customers report being impacted by the issue. I think this is because when the issue does happen, automated retries just take care of it transparently. Thanks!

anpa
Service Provider
Posts: 19
Liked: 9 times
Joined: Oct 27, 2011 8:42 am
Full Name: Andreas Cederlund
Location: Landskrona, Sweden
Contact:

Re: vSphere 6.5U2 support

Post by anpa » Nov 30, 2018 8:20 am

Hi again, and thanks for the prompt reply!

VMware seems to have released a new version of vCenter recently (well, yesterday);
https://docs.vmware.com/en/VMware-vSphe ... notes.html
There's also a new patched version of ESXi released:
https://docs.vmware.com/en/VMware-vSphe ... 11002.html

Seems like a couple of issues are fixed, not sure if they are affecting Veeam or not tho..

/Andreas

Ollo
Influencer
Posts: 20
Liked: 2 times
Joined: Jan 19, 2018 6:32 am
Contact:

Re: vSphere 6.5U2 support

Post by Ollo » Nov 30, 2018 2:54 pm

Should be fixed.

https://docs.vmware.com/en/VMware-vSphe ... d-resolved

PR 2170126: Backup proxy virtual machines might go to invalid state during backup

mdiver
Service Provider
Posts: 50
Liked: 9 times
Joined: Nov 04, 2009 2:08 pm
Location: Heidelberg, Germany
Contact:

Re: vSphere 6.5U2 support

Post by mdiver » Dec 06, 2018 3:46 pm

Could someone from Veeam please comment on this most recent VMWARE patch.
The changelog really reads as if the load issue has finally been taken care of.

Thanks,
Mike

vDuck
Veeam Software
Posts: 56
Liked: 3 times
Joined: Apr 02, 2018 4:16 am
Full Name: Satoko Takahashi
Contact:

Re: vSphere 6.5U2 support

Post by vDuck » Dec 10, 2018 2:21 am


Ollo
Influencer
Posts: 20
Liked: 2 times
Joined: Jan 19, 2018 6:32 am
Contact:

Re: vSphere 6.5U2 support

Post by Ollo » Dec 10, 2018 1:28 pm

Mermaid, the question was about the Heavy Load issue in overprovisioned environments in U2, not the hotadd issue.

falkob
Service Provider
Posts: 64
Liked: 20 times
Joined: Sep 28, 2017 7:47 am
Full Name: Falko Banaszak
Contact:

[MERGED] Veeam 9.5 Udpate 3a and vSphere 6.5 Update 2

Post by falkob » Dec 14, 2018 8:30 am

Hi community,

i have customer who wants to upgrade his vSphere Infrastructure to vSphere 6.5 Update 2.
He is concerned about the "Known Issues" described in this article: https://www.veeam.com/kb2443

There are two Known Issues with VBR 9.5U3a and vSphere 6.5 Update 2

1. Supported with the exception of issues caused by a regression introduced in ESXi 6.5 U2 that causes its API to fail randomly under heavy host load.
2. There is a known issue where a Veeam Proxy running on a VM may become listed as (invalid) in vSphere. More information here: https://kb.vmware.com/s/article/56453


Issue number 2 won't have an impact on my customer, cause he is using physical Proxies with Direct Storage Access.

But in fact i wanted to know about Issue number 1. How frequent does this "occur" ? Are there any experiences on this ? What does heavy host load mean in this context ?

Thank you so far !

Best Regards

Falko
VCP6.5-DCV, VCP6-DCV, VMCE
https://www.virtualhome.blog

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam 9.5 Udpate 3a and vSphere 6.5 Update 2

Post by foggy » Dec 14, 2018 12:36 pm 1 person likes this post

Hi Falko, please review this thread for more information. Thanks!

Yuya
Enthusiast
Posts: 58
Liked: 1 time
Joined: Jan 30, 2013 9:32 am
Contact:

Re: vSphere 6.5U2 support

Post by Yuya » Dec 25, 2018 8:37 am

UPDATE: Our auto-tests have exhibited one potentials issue with 6.5 U2 which may reduce reliability of a number of Veeam Backup & Replication features dependent on the corresponding protocol. Having said that, the issue is only reproducible in our stress testing lab where we emulate a heavy environment load, meaning for those who already upgraded these issue will likely exhibit "random" behavior which is in reality tied up to peak load in the environment. We're opening a support case with VMware - and I will keep you posted on the progress. Thanks!
1)Specifically, what kind of problems do you encounter?
For example, will the backup continue to fail until the load of ESXi goes down?
Or, because there is no reliability of the backup file, is there a possibility that it can not be restored?

2)I understand that VMware support recognizes that the cause of this problem is the vSphere API.
Do you have information on when to be fixed?

3)Is there a workaround for this problem?

I could not understand from the forum, so I'm glad if you tell me.

Thanks,
Yuya

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: vSphere 6.5U2 support

Post by foggy » Dec 26, 2018 12:28 pm 1 person likes this post

I'll try to summarize what you can find in this thread above for you:
Yuya wrote:1)Specifically, what kind of problems do you encounter?
Like mentioned above, almost all product functionality might behave unreliably due to ESXi API failures during periods when the environment operates under heavy load.
Yuya wrote:For example, will the backup continue to fail until the load of ESXi goes down?
The issue has random nature but does not show up under normal load conditions, so if your hosts are not over-provisioned, you should be safe.
Yuya wrote:Or, because there is no reliability of the backup file, is there a possibility that it can not be restored?
This behavior does not cause any data corruptions, you just will not be able to perform some operations and, most likely, to restore from the backup files created during failed jobs. If subsequent job run doesn't produce any errors/warnings, the corresponding restore point should be restorable.
Yuya wrote:Do you have information on when to be fixed?
Currently we do not have any information in this regard.
Yuya wrote:3)Is there a workaround for this problem?
Currently the only workaround is to provide environment operation under normal load.

Yuya
Enthusiast
Posts: 58
Liked: 1 time
Joined: Jan 30, 2013 9:32 am
Contact:

Re: vSphere 6.5U2 support

Post by Yuya » Jan 08, 2019 4:07 am

Thank you for your answer.

This issue can occur when the load on the ESXi host is high.
I understood that this is the ESXi host where the virtual machine to be backed up resides, correct?
Does the vCenter load have an impact?

For example, suppose vCenter is created in a virtual machine and this vCenter VM is in ESXi 6.5 u2.
- The load on the ESXi host (A) where the virtual machine to be backed up resides is low.
- The load on the ESXi host (B) where the vCenter VM resides is high.
In this case, is there a possibility that this issue occurs?

Even if the load of the ESXi host where the vCenter VM resides is high, if the load of the ESXi host where the virtual machine to be backed up resides is low, this issue is unlikely to occur, correct?

Thanks,
Yuya

Gostev
SVP, Product Management
Posts: 24293
Liked: 3330 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » Jan 08, 2019 4:09 pm

Correct.

skrause
Expert
Posts: 410
Liked: 86 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Re: vSphere 6.5U2 support

Post by skrause » Jan 08, 2019 9:08 pm

Is there a ballpark level of host resource consumption (CPU/RAM%) that this happens above but that would likely be safe if consumption stayed below?

We are unable to upgrade to 6.7 due to some of our storage arrays not supporting (or planning on supporting) anything above 6.5.
Steve Krause
Veeam Certified Architect

Gostev
SVP, Product Management
Posts: 24293
Liked: 3330 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » Jan 09, 2019 2:52 am

All I know it's very high load, this was observed in our stress testing lab.

fll
Enthusiast
Posts: 37
Liked: never
Joined: Sep 05, 2016 10:35 am
Contact:

[MERGED] Veeam Backup & Replication 9.5 Update 3a and VMware vSphere 6.5 U2.

Post by fll » Jan 11, 2019 1:43 pm

Hello,
According to KB 2646, there are some problems between Veeam Backup & Replication 9.5 U3a and VSphere 6.5 U2.
"VMware vSphere 6.5 U2." This update introduces preliminary support by addressing all outstanding U2-specific compatibility issues that can be managed from the Veeam side, however, there is a major regression in ESXi 6.5 U2 code that makes the vSphere API fail randomly during high host CPU load periods, consequently impacting a variety of Veeam Backup & Replication functionality VMware is currently troubleshooting this bug, and fixing one will most likely require the new ESXi 6.5 U2 build issued. "
Does the same thing happen in ESXi 6.5 U1?

Thank you.

wishr
Veeam Software
Posts: 873
Liked: 83 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: vSphere 6.5U2 support

Post by wishr » Jan 11, 2019 2:09 pm

Hi Fll,

The given issue was discovered just in 6.5 U2 release bits, so it's safe to install 6.5 U1 for use with B&R 9.5 Update 2 or later versions. You can check that thread for additional information regarding 6.5 U1 support.

I've moved your post to an existing thread covering 6.5 U2 support topic.

Thanks

perez
Lurker
Posts: 1
Liked: 1 time
Joined: Jan 14, 2019 9:14 am
Contact:

Re: vSphere 6.5U2 support

Post by perez » Jan 14, 2019 2:05 pm 1 person likes this post

- What about the possibility of the problem could be caused by the NFC limit, which it is not restricted by the moment in ESXi 6.5U2?
It looks that in ESXi 6.7, there is not created a new thread for each NFC connection, so the thread limitation won't be hit and consequently the problem looks not exist.
- Is it possible that even if VEEAM is NOT using the API, and instead performs a large number of NFC operations, more than 25 simultaneous NFC connections, the problems also occurs, because VEEAM is hitting an NFC limit?
-Has VEEAM tested if the problem persist if VEEAM client does not hit the limit of 25 simultaneous NFC connections, using the API?
-If the problem is finally related to the NFC connections, is it possible that the new VEEAM release, include any fix which could update the client in order to restrict the number of NFC connection, and solve the problem or you think that the fix would have to be in a newest ESXi release?

- And finally, could you please provide me, the SR number of the case you opened to VMWare?
Thanks

wishr
Veeam Software
Posts: 873
Liked: 83 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: vSphere 6.5U2 support

Post by wishr » Jan 14, 2019 4:45 pm

Hi Perez,

Please find the answers below.
perez wrote:What about the possibility of the problem could be caused by the NFC limit, which it is not restricted by the moment in ESXi 6.5U2?
No, unfortunately, the root of the issue cause was located on the VMware VDDK end that is used by B&R along with VMware API for performing backup and restore operations within VMware hypervisors. There were some memory leak issues in one of the VDDK versions used by B&R (just to let you know we deliver several VDDK libraries in each version to ensure compatibility and good customer experience because VDDK compatibility tests performed by VMware are far from being ideal).
perez wrote:Is it possible that even if VEEAM is NOT using the API, and instead performs a large number of NFC operations, more than 25 simultaneous NFC connections, the problems also occurs, because VEEAM is hitting an NFC limit?
No, please see above. NFC connections limit is well known here and always taken into account.
perez wrote:Has VEEAM tested if the problem persist if VEEAM client does not hit the limit of 25 simultaneous NFC connections, using the API?
Sure, this is tested frequently.
perez wrote:If the problem is finally related to the NFC connections, is it possible that the new VEEAM release, include any fix which could update the client in order to restrict the number of NFC connection, and solve the problem or you think that the fix would have to be in a newest ESXi release?
Sure, from one B&R version to another we update VDDK to the most stable versions and always deliver several VDDK versions in a single product bundle. Moreover, we test them internally, because VDDK issues are NOT something unusual. Regardless of that, this particular issue required changes in VDDK itself and on the ESXi end, so our ability to deal with it on our own was extremely limited.

The initial SR had a number 18797316305. After VDDK replacement the issue was fixed, but since that, we started observing another issue that itself disappeared after a couple updates to VDDK and ESXi servers in our labs that obviously required us to extensively communicate the issues, wait until VMware release their patches, test them internally, etc.

We'll update this thread with additional useful information once have it and it should be finally good news.

BR,
Fedor

CPCS
Influencer
Posts: 16
Liked: 3 times
Joined: Mar 01, 2016 1:49 pm
Contact:

Re: vSphere 6.5U2 support

Post by CPCS » Jan 21, 2019 9:57 am

I would like to know what we could expect now from Update4 or Veeam or Vmware? Do we rely on VMware now to fix the left issues? Do everyone of us has to open a case for that so they will fix it or will they FINALLY do this after nearly one year everyone waiting for this?

This is really frustrating, sitting on ESXi 5.5 (already out of Support!) and not able to update (cause hardware is not supporting 6.7) and just looking at vmware/veeam/whoever not getting this fixed AFTER ALL THIS TIME...

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: vSphere 6.5U2 support

Post by foggy » Jan 21, 2019 10:06 am

U4 doesn't change anything in this regard, the ball is on VMware side. Why don't you consider going with whatever version between 5.5 and 6.2U2: 6.0/6.5/6.5U1? All of them are supported and 6.5U1 proved to be rock solid.

CPCS
Influencer
Posts: 16
Liked: 3 times
Joined: Mar 01, 2016 1:49 pm
Contact:

Re: vSphere 6.5U2 support

Post by CPCS » Jan 21, 2019 10:11 am

Because as war as i understood, if you want to install latest security patches (which you should today) it is mandatory to update to 6.5U2 (coming from 6.5(U1), isnt it?

"the ball is on VMware side" does mean what exactly? Veeam does have an open ticket on vmware side for this issue?

Gostev
SVP, Product Management
Posts: 24293
Liked: 3330 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » Jan 21, 2019 8:25 pm

Yes, it is referenced by wishr in the post just above yours.

Post Reply

Who is online

Users browsing this forum: Baidu [Spider], Google [Bot] and 26 guests