vSphere 6.5U2 support

[pidthepiper]

Is the issue with vCenter 6.5u2 or esxi 6.5u2?

I have a customer that is currently on 6.5u1 but wants to apply the latest security patches which will take them to u2, but their esxi hosts are on 5.5 and soon to be 6.0u3

[Gostev]

Most likely in ESXi, because it can be observed in standalone ESXi configurations as well.

[pidthepiper]

If that is going to be the case. Could you not support vCenter 6.5u2 with the caveat that ESXi 6.5u2 is unsupported?

[Gostev]

Yes, if VMware confirms through their research that their issue sits in ESXi itself, then we will be able to do that.

[ReealityPrime]

For those that are stuck on vSphere 6.5U2 at the moment, do you recommend sticking it out and waiting for an official update from VMware or is it best to jump to vSphere 6.7 or back rev to 6.5U1?

I am new to Veeam, and didn't think to check compatibility for U2 before doing the update during our maintenance window.

I have about 20-30 VMs spread across 3 non-clustered servers and use embedded platform controller on vCenter 6.5U2.

It sounds like the problems with vSphere 6.5U2 and Veeam typically happen under heavy load. Can you advise on what's considered a "heavy load."

Thanks.

[Gostev]

If you're observing no issues in your environment and granted you have Update 3a installed, you may stick to 6.5U2. You will not be automatically refused technical support for any unrelated issues just because you're on 6.5U2, we will be reasonable of course.

Heavy load means heavily overprovisioned environment. We cannot know which particular aspect of it triggers the issue though - host CPU load, RAM pressure or something else - this is something VMware should be able to understand in their troubleshooting of the issue.

[slackhouse]

I'm currently running Veeam 9.5 update 3. My ESXi hosts are running 6.5u1. I would like to apply the latest security updates. These updates will push my hosts to ESXi 6.5 build 8285314. Is this Build of ESXi supported by Veeam 9.5 Update 3? Or do i need to upgrade to Veeam 9.5 Update 3a first?

[Gostev]

You need Update 3a first.

[rsberzerker]

VMware 6.5 Update2B is out. Anyone know if this is a fixed version and therefore safe to install?

[Gostev]

I assume we would get an update on our support case with VMware if the fix was included... which we did not receive - but we will test regardless.

[timmi2704]

Hey Gostev
Do you have any ID to the VMware case I could refer to with our TAM so that he can assess the impact on our environment and recommend an appropriate vSphere update path while keeping this bug in mind?
Thanks.

[anpa]

The only new 6.5 release I can find is vCenter 6.5 Update 2b, which only seems to be a security update release..

From https://docs.vmware.com/en/VMware-vSphe ... notes.html
"This updated version of vCenter Server provides part of the hypervisor-assisted guest mitigation of CVE-2018-3639 for guest operating systems. For important details on this mitigation, see VMware Security Advisory VMSA-2018-0012.1."

I cannot find any ESXi release later than May 2018..

[Gostev]

@timmi2704 it's 18797316305

[Gostev]

Looks like U2b fixes nothing except 4th Spectre incarnation. I have confirmed that "our" issue is still being researched by VMware support.

[sinatee]

to be clear, is the veeam br issue with esxi or vcenter? is it safe to run vcenter 652b with veeambr 950u3a with older pre 6.x or 5.5 esxi hosts?

thank you!

[egrutman]

Based on what I am following on the thread this is related to ESXi 6.5 u2.
So if you upgrade vcenter and veeam, you will be fine.

[pidthepiper]

Yeah it seems to be an esxi 6.5u2 issue. But Veeam havent stated that directly and have said not to upgrade to it, if they would say you can upgrade vcenter to u2 but not the hosts that would help me with patching my vCenters, at the moment I am keeping them on u1 just to avoid any kind of support issues

[cflaggrun]

Is there anyone here running 6.5 U1 that has seen these issues? We are planning our 5.5 to 6.5 upgrade and will stay 6.5 U1 until the issue is resolved. I spoke to a vmware tech last week about this and he was pushing me to to go 6.0 U3.

Thanks

[Gostev]

6.5 U1 has been rock solid.

[lroscoe]

We are planning on upgrading from vSphere 5.5 to 6.5 U2. We're currently running Veeam Backup and Replication 9.5 U3. Is there a need to update to U3a for Veeam to work with vSphere 6.5 U2? Or can we continue to run VBR 9.5 U3 without issue? I've seen a lot of posts about issues with API failures and hot-add backups failing as well.

[DGrinev]

Hi Leo and welcome to the community!

It's recommended to use the most recent version of the Veeam B&R, however, speaking about vSphere 6.5 U2 we cannot give you guarantees as there is still an issue on the VMware side.
You can review the information in the What's new document regarding 6.5 U2.
Please take a look on this discussion for additional information. Thanks!

[jamcool]

Case ID #03028438

Currently have an issue with vCenter 6.5 Update 2 / ESXi 6.5 Update 2 / Veeam 9.5 Update 3.

vCenter was upgraded to 6.5 Update 2 a fortnight ago and we haven't seen any issues since the upgrade.
The ESXi hosts began upgrades to 6.5 Update 2 late last week and since then, some proxies have become invalid in vCenter.
The proxies which go invalid are in a large cluster of VM's (~1300 VMs) so I assume it would be the major load on the proxies and the ESXi patch causing issues.
Other clusters don't seem to have the issue with less VM's and proxies.
We are only doing VM level backups (no SQL or Oracle level backups yet) so we haven't seen any major issues besides running out of time in the backup window.
We are currently in the progress of rolling back the ESXi update applied to the hosts so I will try and report back after that has been completed and a night's worth of backup has been executed.

Williampu,
did you get any fix for the Invalid proxy servers in large backups (large number of VMs)?

I don't have ask many as you do but on my two largest (each about 500 VMs), I seem to loose every 2-3 days 1-3 proxy servers and have to unregsiter, register and remove the non proxy disk. I have added (doubled) more proxy servers and dropped the threads really low to 6 (4 vCPU and 8 GB RAM), from 12 (6 vCPU and 16 GB RAM) and they never get more 40% CPU usage and memory usage is pretty minimal. I also made sure automount is disabled in diskpart, anti-virus exclusions setup, only using LSI SAS disk controller (we do 2 controllers per proxy for years), VMFS 6 datastores and all 1 MB block size and even setup cluster rule so no more than 1 proxy on a VM Host. Never saw invalid proxy until went to ESXi 6.5 update2 so I agree that it is VMWARE change that is causing problems. We are at ESXi 6.5 update2 with B&R 9.5 with 3a. We just upgraded vCenter appliance too from 6.5 update 1g to update 2. I saw something about update 4 for B&R 9.5 coming out soon so may be it will address some of these issues. Maybe see it before VMWorld in Las Vegas !!!

[shooker]

I'm late to the party but ... having similar issues. I'm running vCenter version 6.5.0.10000 [6.5 u1] and everything worked perfectly until I updated my ESXi hosts from 6.0 u3 to 6.5u2.
I really wish I hadn't done that without doing more research ... I'm going to roll back one of my ESXi hosts [with SQL/Exchange servers] on it this weekend.

My other hosts seem to back up okay they are just slower to start that prior to the update.

For those of you that don't know there is a fairly easy way to roll back to previous VMware host version. If you are at the console you hit Shift+[R] during ESXi host start up. It allows you to select previous version.
*Assuming you chose the upgrade options*

[shooker]

So I just rolled back my VM hosts to 6.0 u3 this morning ... and SQL backups were still failing. I checked my VMware tools as someone had mentioned that in another thread. I was running
tools Version: 10305 esx/6.5u2 10.2.1 8294253

I grabbed the latest and greatest Tools Version: 10336 esx/0.0 10.3.0

*My backups seem to be running properly again*

[chris_lalala]

Are there any updates regarding this Case? I see Veeam V9.5 U3a is supported with ESXi 6.5 U2 https://www.veeam.com/kb2646

But this not my understanding of complete support

"VMware vSphere 6.5 U2. This update introduces preliminary support by addressing all outstanding U2-specific compatibility issues that can be managed from the Veeam side. However, there is a major regression in ESXi 6.5 U2 code that makes the vSphere API fail randomly during high host CPU load periods, consequently impacting a variety of Veeam Backup & Replication functionality. VMware is currently troubleshooting this bug, and fixing one will most likely require the new ESXi 6.5 U2 build issued."

[Gostev]

No updates from VMware yet.

[Frosty]

Glad I read this thread. Was just thinking about updating to 6.5 U2 but I think I will hold off for another 2-3 months and see what transpires. We are on 6.5 U1 at the moment and it is very stable.

[chris_lalala]

Sure 6.5 U1 is stable, but I want to patch Spectre 3a and v4 ...

[francs]

It would seem that even if you installed 6.5U1 from ISO and run Update Manager it will install the U2 patches.
According to this site (https://esxi-patches.v-front.de/ESXi-6.5.0.html), 6.5U2 came out on 2018-05-03
What I ended up doing is to create a baseline and exclude everything after 2018-03-20.
There is another set of patches (Build 8285314), but it has the same date as U2, so I left it out to be on the safe side.

[Samba222]

Are there any news regarding vSphere 6.5U2 support? Or schedule?

Thx!