vSphere 6.5U2 support

[Gostev]

No news. I will of course update this topic as soon as we get any material update from VMware.

[law999]

We will need to patch our hosts to the latest due to the "L1 Terminal Fault"

Gostev, What would be the recommendation. go straight to 6.7?

[btmaus]

Hello,

We are currently on vSphere Appliance 6.5 U1 and want to update to U2c.
Our ESXI is currently on 6.5 build 73886907.
We are running Veeam 9.5 3a.

We need to upgrade our VMware ESXI hosts and vCenter servers to the latest version of 6.5.
I would like to confirm whether this is compatible with vSphere vCenter Server Appliance 6.5 U2c build 9451637?
The ESXI version that we want to upgrade to along with vCenter will be 6.5 U2 build 9298722.

I found this article from Veeam:
https://www.veeam.com/kb2443

It means that there is support and compatibility however there is an exception to this.
Can you please provide more information around the exception and confirm whether the above mentioned versions are compatible and supported?

Thanks

Case ID: #03170017

[DGrinev]

Hi Glenn,

There is still an issue with the vSphere 6.5 U2 on the VMware side, it would be better to wait.
Please read this discussion for additional information. Thanks!

[btmaus]

Thanks for the reply DGrinev.

Is there any "official" KB article from Veeam or VMware, that specifics what the problem is and what issues are experienced?
Would it crash a VMware host for example, or does it only affect Veeam services?

I'm trying to find out what exactly "doesn't work" or is affected, as our management team requires the information for CAB changes.

Thanks!

[DGrinev]

Hi Glenn,

I ran through the topic and here you can find the information about the issue:
1. Issues confirmed by tests
2. A brief explanation by Gostev
3. Another description by Gostev

Thanks!

[williampu]

@jamcool
The esxi hosts were rolled back to the previous version and everything worked.
I've asked the VMware guys to delay upgrades to 6.5U2 until I get an all-clear from Veeam.
Hopefully it won't take too long until a hotfix is available.

[late]

Thanks for the clarification DGrinev. We are in a process of updating our hosts from 5.5U2 to a newer version. It would make sense for us to go to version 6.7 but the problem is that we have couple of hosts that only support ESXi up to 6.5U2. So 6.7 isn't really an option for us. About the issues described in this thread: have you been able to resolve any of them or are they all related to VMware and you are waiting for their actions? And have the issues affected multiple Veeam users or are they all more or less individual cases? Thanks!

[Gostev]

Late, the issue is related to a regression in vSphere 6.5 U2, so we're waiting for VMware to recognize and fix one.
The issue is isolated to over-provisioned environments with heavily loaded hosts (so more or less individual cases).

[late]

Hi Gostev and thank you for the information!

[vDuck]

FYI

https://kb.vmware.com/s/article/56453

[bthorne]

Hello Veeam,

Can we please get some offical details on the current issues customers are facing if they update.

Veeam version : 9.5 update 3 and will upgrade to 3a
Vmware version: 6.5

1. since we do SAN direct are we impacted ?
2. if you dont upgrade vCentre but upgrade the hosts to 6.5 U2 are we ok?
3.Have many customers had issues with the upgrade ?
4: is there any more info out there about this ?

Thank you very much

The kb arcticle still only has the original information of https://www.veeam.com/kb2443.

[foggy]

Hi Bernard, please review information in this thread for details. Thanks.

[bhnelson]

FYI

https://kb.vmware.com/s/article/56453

This kb seems to describe the exact problem I'm having. However, it does not sound at all like the issue(s) described in this thread (random failures based on high host load).
It also says it affects both 6.5 U2 and 6.7 while Veeam says 6.7 is fine.
Can someone comment further?

Thanks!

[rimvydukas]

Hi,

Dear Gostev. I just started migration to 6.5 version. I choosed to use 6.5 U1g, build 7388607. Is this build safe from the CPU regression issue? Was it solely brought by 6.5U2?

[Gostev]

Yes, 6.5 U2 is the only version that is known to be affected. Thanks!

[evilaedmin]

Our environment has many disks per VM and we've experienced this known issue with ESXi 6.5 Update 2: https://kb.vmware.com/s/article/2146829 Is anyone else struggling with it? Support says it will be resolved in 6.7 Update 1.

[bpayne]

Accidentally neglected to see this forum thread before building a new large cluster of ESXi hosts at 6.5 Update2c. Also experiencing all of my Veeam Proxy VM's being marked as invalid. Manual procedure: Remove VM from inventory, Re-add VM back to inventory, Edit settings: remove hot-added disks, then I can power on proxy VM. Happening to all my proxy VM's in this cluster once every 1-2 days. Cluster size has several hundred VM's.

We can't go to vCenter/ESXi 6.7 yet. Also getting pushed to patch to Update 2c for "esx.problem.hyperthreading.unmitigated" (L1 Terminal Fault - VMM’ (L1TF - VMM) Speculative-Execution vulnerability, https://kb.vmware.com/s/article/55806). In a really tough spot.

Does anyone know if switching those Proxy VM's from Hot-Add to Network mode yield better results?
Only thing I can think of is to build many more Proxy VM's to give myself a better chance of more proxies to pick from to get through the night.

[bhnelson]

FYI

https://kb.vmware.com/s/article/56453

FYI, I was told this by VMware support:

Regarding the issue that is addressed in the KB 56453, the fix should be out in the next patch, which will be released by the end of November 2018.

[evilaedmin]

Accidentally neglected to see this forum thread before building a new large cluster of ESXi hosts at 6.5 Update2c. Also experiencing all of my Veeam Proxy VM's being marked as invalid.

Have you looked into reverting? If you have done updates as directed and not something funky like we did (attempt the same update twice) the previous version should be right there on disk for you to select from the boot loader: https://kb.vmware.com/s/article/1033604

[jamcool]

2 months ago we switched to network mode from Hot Add for proxy servers and have been very happy. It eliminated the proxy servers going invalid and cut out backup times by as much as 75% (we had two large jobs that went from nearly 8 hours to 2 hour each). our configuration is ESXi 6.5 update2 and Veeam 9.5 with 3a patch since July. I am not certain why if you have 10 Gbps networking that is not highly utilized would not use network mode.

[lroscoe]

Any updates from VMware concerning this issue mention below?

"VMware vSphere 6.5 U2. This update introduces preliminary support by addressing all outstanding U2-specific compatibility issues that can be managed from the Veeam side. However, there is a major regression in ESXi 6.5 U2 code that makes the vSphere API fail randomly during high host CPU load periods, consequently impacting a variety of Veeam Backup & Replication functionality. VMware is currently troubleshooting this bug, and fixing one will most likely require the new ESXi 6.5 U2 build issued."

[Gostev]

No updates so far. Some senior product person from VMware has reached out to me about a month ago with the seemingly strong desire to get to the bottom of this issue, but sadly I have not heard back ever since... so, it is starting to look like upgrading to 6.7 will be the only "solution" to this issue.

[mgd5]

Hi,

We are in the same boat with 6.5 U2 and Veeam U3. We have 6 Proxy servers but one proxy in particular had problems. When it was running jobs it failed 1-5 vm per job but also the other Proxys didnt work well. So i have turned that Proxy off and now almost all jobs run. This weekend i had only one Proxy host go in invalid state and we have atleast 100 backup jobs around 500 vm´s running every night. How can that be ?

/Christer

[jcapone]

We are on Veeam 9.5 U3a and also recently upgraded to ESXi 6.5 U2 and are now kind of stuck here. Our hardware does not support 6.7, so we cannot migrate forward, so unless VMWare locates and resolves the problem with a U2 x patch, we will be forced to downgrade to 6.5 U1. Fortunately we are a small shop (40+ VMs, 2 hosts) and have not seen any problems yet. We are really trying to decide the best course of action here. Any advice? We opened our own ticket with VMWare (case #18947155010) but the engineer indicated that so far they do not see any global issue, that the problem is only seen in individual cases and as such will not likely be patched. Does anyone else have a case open with VMWare that they can share so we can ask the engineer to look at more details of the problem? As I said, since we are fortunate enough to not be experiencing the problem, I have no technical details to supply and don't know if we should stay on U2 or downgrade. Thanks for any input.

[TrevorBell]

I run 6.5U2 VMware ESXi, 6.5.0, 9298722 latest build with 200 VMs 100 per site and dont see any issues yet with backups or cross site replication. I only have 1 dedicated proxy per site win 2016 VM , with 10gig iscsi and Eqallogic 6210 boxes. ( all dell kit )

No issues soo far.. but will report back if i see anything strange

[jamcool]

We are on Veeam 9.5 U3a and also recently upgraded to ESXi 6.5 U2 and are now kind of stuck here. Our hardware does not support 6.7, so we cannot migrate forward, so unless VMWare locates and resolves the problem with a U2 x patch, we will be forced to downgrade to 6.5 U1. Fortunately we are a small shop (40+ VMs, 2 hosts) and have not seen any problems yet. We are really trying to decide the best course of action here. Any advice? We opened our own ticket with VMWare (case #18947155010) but the engineer indicated that so far they do not see any global issue, that the problem is only seen in individual cases and as such will not likely be patched. Does anyone else have a case open with VMWare that they can share so we can ask the engineer to look at more details of the problem? As I said, since we are fortunate enough to not be experiencing the problem, I have no technical details to supply and don't know if we should stay on U2 or downgrade. Thanks for any input.

Have you tried switching to network mode instead instead of Hot add?

[jcapone]

jamcool I saw a mention of that in one of the replies, but I don't know much about that mode, so I have not tried that yet. I'll check the documentation and see if it's something we can use here. Thanks!

[rmyers]

I stumbled upon this thread while researching this issue. I also have a Veeam case opened and will probably open a VMware one as well.

@jcapone - Our environment consists of a couple hundred ESXi hosts, most of which I've just upgraded to 6.5U2 from 5.5U3. Within the last couple months aligning with the upgrade project I started experiencing the exact symptoms described in https://kb.vmware.com/s/article/56453 which I have since confirmed in vkernel.log. We have 22 vproxy servers and I tend to lose anywhere from 0 to 10 a night as backups run. Fixing the proxies is a real hassle (remove/re add to inventory, disconnect hot add disks, power on and answer questions, rescan from veeam, re add proxies to Keep Apart DRS Rules since dropping out of vcenter clears those out).

I recently put a script together to do a vMotion of all the proxy servers at end of day but it's too soon to tell if that's helping although I've had no invalids pop from last nights backup. I'll continue to monitor this thread to see if any other solutions are posted.

To the folks who mentioned use network mode over hot add - I would have to better understand if there is a performance hit doing so considering the size of our environment and the number of jobs I have running. I would expect proxy hot add to be the most efficient way to complete the backups.

[ctsherman]

Just to check/clarify on this...

Is this issue only affecting hotadd/proxyVM environments? I manage the network for a small business (three ESXi servers, one of which is offsite), and we're presently using a combination of NFS and network mode. In absence of hotadd or proxyVMs, will this regression issue affect us?