Host-based backup of VMware vSphere VMs.
pidthepiper
Enthusiast
Posts: 80
Liked: 7 times
Joined: Aug 11, 2015 9:10 am
Full Name: Bilal AHmed
Contact:

Re: vSphere 6.5U2 support

Post by pidthepiper »

Is the issue with vCenter 6.5u2 or esxi 6.5u2?

I have a customer that is currently on 6.5u1 but wants to apply the latest security patches which will take them to u2, but their esxi hosts are on 5.5 and soon to be 6.0u3
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev »

Most likely in ESXi, because it can be observed in standalone ESXi configurations as well.
pidthepiper
Enthusiast
Posts: 80
Liked: 7 times
Joined: Aug 11, 2015 9:10 am
Full Name: Bilal AHmed
Contact:

Re: vSphere 6.5U2 support

Post by pidthepiper »

If that is going to be the case. Could you not support vCenter 6.5u2 with the caveat that ESXi 6.5u2 is unsupported?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » 1 person likes this post

Yes, if VMware confirms through their research that their issue sits in ESXi itself, then we will be able to do that.
ReealityPrime
Novice
Posts: 3
Liked: never
Joined: Jun 26, 2018 7:02 pm
Contact:

Re: vSphere 6.5U2 support

Post by ReealityPrime »

For those that are stuck on vSphere 6.5U2 at the moment, do you recommend sticking it out and waiting for an official update from VMware or is it best to jump to vSphere 6.7 or back rev to 6.5U1?

I am new to Veeam, and didn't think to check compatibility for U2 before doing the update during our maintenance window.

I have about 20-30 VMs spread across 3 non-clustered servers and use embedded platform controller on vCenter 6.5U2.

It sounds like the problems with vSphere 6.5U2 and Veeam typically happen under heavy load. Can you advise on what's considered a "heavy load."

Thanks.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev »

If you're observing no issues in your environment and granted you have Update 3a installed, you may stick to 6.5U2. You will not be automatically refused technical support for any unrelated issues just because you're on 6.5U2, we will be reasonable of course.

Heavy load means heavily overprovisioned environment. We cannot know which particular aspect of it triggers the issue though - host CPU load, RAM pressure or something else - this is something VMware should be able to understand in their troubleshooting of the issue.
slackhouse
Influencer
Posts: 10
Liked: 2 times
Joined: May 21, 2018 12:13 pm
Full Name: Jeremy Slack
Contact:

Re: vSphere 6.5U2 support

Post by slackhouse »

I'm currently running Veeam 9.5 update 3. My ESXi hosts are running 6.5u1. I would like to apply the latest security updates. These updates will push my hosts to ESXi 6.5 build 8285314. Is this Build of ESXi supported by Veeam 9.5 Update 3? Or do i need to upgrade to Veeam 9.5 Update 3a first?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev »

You need Update 3a first.
rsberzerker
Influencer
Posts: 19
Liked: never
Joined: Dec 02, 2015 3:39 pm
Full Name: Robert
Contact:

Re: vSphere 6.5U2 support

Post by rsberzerker »

VMware 6.5 Update2B is out. Anyone know if this is a fixed version and therefore safe to install?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev »

I assume we would get an update on our support case with VMware if the fix was included... which we did not receive - but we will test regardless.
timmi2704
Expert
Posts: 100
Liked: 5 times
Joined: Jan 14, 2014 10:41 am
Full Name: Timo Brandt
Contact:

Re: vSphere 6.5U2 support

Post by timmi2704 »

Hey Gostev :)
Do you have any ID to the VMware case I could refer to with our TAM so that he can assess the impact on our environment and recommend an appropriate vSphere update path while keeping this bug in mind?
Thanks.
anpa
Service Provider
Posts: 28
Liked: 11 times
Joined: Oct 27, 2011 8:42 am
Full Name: Andreas Cederlund
Location: Landskrona, Sweden
Contact:

Re: vSphere 6.5U2 support

Post by anpa »

The only new 6.5 release I can find is vCenter 6.5 Update 2b, which only seems to be a security update release..

From https://docs.vmware.com/en/VMware-vSphe ... notes.html
"This updated version of vCenter Server provides part of the hypervisor-assisted guest mitigation of CVE-2018-3639 for guest operating systems. For important details on this mitigation, see VMware Security Advisory VMSA-2018-0012.1."

I cannot find any ESXi release later than May 2018..
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev »

@timmi2704 it's 18797316305
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » 3 people like this post

Looks like U2b fixes nothing except 4th Spectre incarnation. I have confirmed that "our" issue is still being researched by VMware support.
sinatee
Lurker
Posts: 1
Liked: never
Joined: Jul 17, 2018 9:40 pm
Full Name: sina tee
Contact:

Re: vSphere 6.5U2 support

Post by sinatee »

to be clear, is the veeam br issue with esxi or vcenter? is it safe to run vcenter 652b with veeambr 950u3a with older pre 6.x or 5.5 esxi hosts?

thank you!
egrutman
Novice
Posts: 5
Liked: never
Joined: May 31, 2016 1:22 pm
Full Name: Yevgeniy Grutman
Contact:

Re: vSphere 6.5U2 support

Post by egrutman »

Based on what I am following on the thread this is related to ESXi 6.5 u2.
So if you upgrade vcenter and veeam, you will be fine.
pidthepiper
Enthusiast
Posts: 80
Liked: 7 times
Joined: Aug 11, 2015 9:10 am
Full Name: Bilal AHmed
Contact:

Re: vSphere 6.5U2 support

Post by pidthepiper »

Yeah it seems to be an esxi 6.5u2 issue. But Veeam havent stated that directly and have said not to upgrade to it, if they would say you can upgrade vcenter to u2 but not the hosts that would help me with patching my vCenters, at the moment I am keeping them on u1 just to avoid any kind of support issues
cflaggrun
Lurker
Posts: 1
Liked: never
Joined: Jul 23, 2018 5:35 pm
Full Name: Chris Flagg
Contact:

Re: vSphere 6.5U2 support

Post by cflaggrun »

Is there anyone here running 6.5 U1 that has seen these issues? We are planning our 5.5 to 6.5 upgrade and will stay 6.5 U1 until the issue is resolved. I spoke to a vmware tech last week about this and he was pushing me to to go 6.0 U3.

Thanks
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » 1 person likes this post

6.5 U1 has been rock solid.
lroscoe
Influencer
Posts: 14
Liked: 1 time
Joined: May 17, 2017 4:55 pm
Full Name: Leo Roscoe
Contact:

[MERGED]vSphere 5.5 to vSphere 6.5 U2 upgrade running VBR 9.

Post by lroscoe »

We are planning on upgrading from vSphere 5.5 to 6.5 U2. We're currently running Veeam Backup and Replication 9.5 U3. Is there a need to update to U3a for Veeam to work with vSphere 6.5 U2? Or can we continue to run VBR 9.5 U3 without issue? I've seen a lot of posts about issues with API failures and hot-add backups failing as well.
DGrinev
Veteran
Posts: 1943
Liked: 247 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: vSphere 5.5 to vSphere 6.5 U2 upgrade running VBR 9.5 U3

Post by DGrinev »

Hi Leo and welcome to the community!

It's recommended to use the most recent version of the Veeam B&R, however, speaking about vSphere 6.5 U2 we cannot give you guarantees as there is still an issue on the VMware side.
You can review the information in the What's new document regarding 6.5 U2.
Please take a look on this discussion for additional information. Thanks!
jamcool
Enthusiast
Posts: 67
Liked: 11 times
Joined: Feb 02, 2018 7:56 pm
Full Name: Jason Mount
Contact:

Re: vSphere 6.5U2 support

Post by jamcool »

williampu wrote:Case ID #03028438

Currently have an issue with vCenter 6.5 Update 2 / ESXi 6.5 Update 2 / Veeam 9.5 Update 3.

vCenter was upgraded to 6.5 Update 2 a fortnight ago and we haven't seen any issues since the upgrade.
The ESXi hosts began upgrades to 6.5 Update 2 late last week and since then, some proxies have become invalid in vCenter.
The proxies which go invalid are in a large cluster of VM's (~1300 VMs) so I assume it would be the major load on the proxies and the ESXi patch causing issues.
Other clusters don't seem to have the issue with less VM's and proxies.
We are only doing VM level backups (no SQL or Oracle level backups yet) so we haven't seen any major issues besides running out of time in the backup window.
We are currently in the progress of rolling back the ESXi update applied to the hosts so I will try and report back after that has been completed and a night's worth of backup has been executed.
Williampu,
did you get any fix for the Invalid proxy servers in large backups (large number of VMs)?

I don't have ask many as you do but on my two largest (each about 500 VMs), I seem to loose every 2-3 days 1-3 proxy servers and have to unregsiter, register and remove the non proxy disk. I have added (doubled) more proxy servers and dropped the threads really low to 6 (4 vCPU and 8 GB RAM), from 12 (6 vCPU and 16 GB RAM) and they never get more 40% CPU usage and memory usage is pretty minimal. I also made sure automount is disabled in diskpart, anti-virus exclusions setup, only using LSI SAS disk controller (we do 2 controllers per proxy for years), VMFS 6 datastores and all 1 MB block size and even setup cluster rule so no more than 1 proxy on a VM Host. Never saw invalid proxy until went to ESXi 6.5 update2 so I agree that it is VMWARE change that is causing problems. We are at ESXi 6.5 update2 with B&R 9.5 with 3a. We just upgraded vCenter appliance too from 6.5 update 1g to update 2. I saw something about update 4 for B&R 9.5 coming out soon so may be it will address some of these issues. Maybe see it before VMWorld in Las Vegas !!! :)
shooker
Novice
Posts: 3
Liked: 1 time
Joined: Mar 19, 2016 12:50 pm
Full Name: Shannon Hooker
Contact:

Re: vSphere 6.5U2 support

Post by shooker »

I'm late to the party but ... having similar issues. I'm running vCenter version 6.5.0.10000 [6.5 u1] and everything worked perfectly until I updated my ESXi hosts from 6.0 u3 to 6.5u2.
I really wish I hadn't done that without doing more research ... I'm going to roll back one of my ESXi hosts [with SQL/Exchange servers] on it this weekend.

My other hosts seem to back up okay they are just slower to start that prior to the update.

For those of you that don't know there is a fairly easy way to roll back to previous VMware host version. If you are at the console you hit Shift+[R] during ESXi host start up. It allows you to select previous version.
*Assuming you chose the upgrade options*
shooker
Novice
Posts: 3
Liked: 1 time
Joined: Mar 19, 2016 12:50 pm
Full Name: Shannon Hooker
Contact:

Re: vSphere 6.5U2 support

Post by shooker » 1 person likes this post

So I just rolled back my VM hosts to 6.0 u3 this morning ... and SQL backups were still failing. I checked my VMware tools as someone had mentioned that in another thread. I was running
tools Version: 10305 esx/6.5u2 10.2.1 8294253

I grabbed the latest and greatest Tools Version: 10336 esx/0.0 10.3.0

*My backups seem to be running properly again*
chris_lalala
Influencer
Posts: 10
Liked: 3 times
Joined: Jun 26, 2013 8:12 am
Full Name: Chris
Contact:

Re: vSphere 6.5U2 support

Post by chris_lalala »

Are there any updates regarding this Case? I see Veeam V9.5 U3a is supported with ESXi 6.5 U2 https://www.veeam.com/kb2646

But this not my understanding of complete support :oops:

"VMware vSphere 6.5 U2. This update introduces preliminary support by addressing all outstanding U2-specific compatibility issues that can be managed from the Veeam side. However, there is a major regression in ESXi 6.5 U2 code that makes the vSphere API fail randomly during high host CPU load periods, consequently impacting a variety of Veeam Backup & Replication functionality. VMware is currently troubleshooting this bug, and fixing one will most likely require the new ESXi 6.5 U2 build issued."
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev »

No updates from VMware yet.
Frosty
Expert
Posts: 200
Liked: 43 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

Re: vSphere 6.5U2 support

Post by Frosty » 1 person likes this post

Glad I read this thread. Was just thinking about updating to 6.5 U2 but I think I will hold off for another 2-3 months and see what transpires. We are on 6.5 U1 at the moment and it is very stable.
chris_lalala
Influencer
Posts: 10
Liked: 3 times
Joined: Jun 26, 2013 8:12 am
Full Name: Chris
Contact:

Re: vSphere 6.5U2 support

Post by chris_lalala »

Sure 6.5 U1 is stable, but I want to patch Spectre 3a and v4 ... :?
francs
Influencer
Posts: 17
Liked: 1 time
Joined: Mar 20, 2012 8:54 am
Contact:

Re: vSphere 6.5U2 support

Post by francs »

It would seem that even if you installed 6.5U1 from ISO and run Update Manager it will install the U2 patches.
According to this site (https://esxi-patches.v-front.de/ESXi-6.5.0.html), 6.5U2 came out on 2018-05-03
What I ended up doing is to create a baseline and exclude everything after 2018-03-20.
There is another set of patches (Build 8285314), but it has the same date as U2, so I left it out to be on the safe side.
Samba222
Service Provider
Posts: 79
Liked: 24 times
Joined: Jan 18, 2017 11:54 am
Full Name: Ronald
Contact:

Re: vSphere 6.5U2 support

Post by Samba222 »

Are there any news regarding vSphere 6.5U2 support? Or schedule?

Thx!
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 65 guests