Discussions specific to the VMware vSphere hypervisor
Gostev
SVP, Product Management
Posts: 24474
Liked: 3414 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » May 24, 2018 9:55 pm

Correct, it's reasonably safe to stay on 6.5 U2 if you're not observing intermittent Veeam failures. And in any case, the issue will not cause data corruptions or anything like that. It is just that in heavily over-provisioned environments, our product will be failing left and right on most operations due to underlying vSphere API failures - which is the only reason why we can't officially support it on vSphere 6.5 U2 yet.

Gostev
SVP, Product Management
Posts: 24474
Liked: 3414 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » May 27, 2018 8:37 pm

For those on 6.5U2 already, please be aware that we've now confirmed one other issue that causes hot add transport to fail on certain backup proxies.

williampu
Influencer
Posts: 14
Liked: never
Joined: Sep 08, 2017 1:26 am
Full Name: William Pu
Contact:

Re: vSphere 6.5U2 support

Post by williampu » May 29, 2018 4:07 am

Case ID #03028438

Currently have an issue with vCenter 6.5 Update 2 / ESXi 6.5 Update 2 / Veeam 9.5 Update 3.

vCenter was upgraded to 6.5 Update 2 a fortnight ago and we haven't seen any issues since the upgrade.
The ESXi hosts began upgrades to 6.5 Update 2 late last week and since then, some proxies have become invalid in vCenter.
The proxies which go invalid are in a large cluster of VM's (~1300 VMs) so I assume it would be the major load on the proxies and the ESXi patch causing issues.
Other clusters don't seem to have the issue with less VM's and proxies.
We are only doing VM level backups (no SQL or Oracle level backups yet) so we haven't seen any major issues besides running out of time in the backup window.
We are currently in the progress of rolling back the ESXi update applied to the hosts so I will try and report back after that has been completed and a night's worth of backup has been executed.

Sorenemig
Influencer
Posts: 12
Liked: never
Joined: Jan 05, 2013 4:16 pm
Full Name: Søren Emig

Re: vSphere 6.5U2 support

Post by Sorenemig » Jun 07, 2018 11:03 am

The same think happend in our setup.

We are now using NBD and it seems stable. I don't want to downgrade unless it is absolutely necessary.

@Gostev, any new information on the case Veeam opened with VMware?

jamcool
Influencer
Posts: 19
Liked: 3 times
Joined: Feb 02, 2018 7:56 pm
Full Name: Jason Mount
Contact:

Re: vSphere 6.5U2 support

Post by jamcool » Jun 10, 2018 2:04 am

Our vCenter is still at update 1g but we have started upgrading our VM Host to 6.5 update2 and just started having this problem. Veeam support had me on the proxy server running diskpart disable automount but that did not help. Anybody find a solution yet?

Fred_vBrain
Influencer
Posts: 17
Liked: 6 times
Joined: Sep 26, 2013 6:41 am
Full Name: Manfred Hofer
Location: Austria
Contact:

Re: vSphere 6.5U2 support

Post by Fred_vBrain » Jun 10, 2018 6:08 pm

jamcool,

even if you wouldn't have problem with Veeam you would run an unsupported constellation. Check the Interop Matrix. http://partnerweb.vmware.com/comp_guide ... erop&2=&1=

vCenter 6.5 U1x is not supported to manage 6.5 U2 host. It can work but you definitely run into issues over time.

Cheers,
Fred
Fred | vBrain.info | @Fred_vBrain

VCAP-DCA/DCD/CIA/CID/DTA/DTD, VCIX-NV, VCIX6-DCV/DTM/NV/CMA, VMCE v7-v9, vExpert 2014-18, vExpert NSX 2016-17, Cisco Champion 2015-18, Veeam Vanguard 2016, Dell EMC Elect 2017

Gostev
SVP, Product Management
Posts: 24474
Liked: 3414 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » Jun 10, 2018 10:31 pm 1 person likes this post

Sorenemig wrote:@Gostev, any new information on the case Veeam opened with VMware?
It's now been 1 month since we opened the support case with VMware, and the case has been moving extremely slow. But there's some progress for sure, as apparently we finally pinched through technical support part of the story – because last week, we've been told that PR (Problem Report) has been created and developers are now in the loop. We also have lots of good people from VMware involved at this point, and helping us to push this through in any way they can. So, here's to hope it won't be another month now.

andrie
Influencer
Posts: 10
Liked: never
Joined: Nov 20, 2015 12:37 pm
Full Name: Andy
Contact:

Re: vSphere 6.5U2 support

Post by andrie » Jun 11, 2018 7:37 am

VMware vSphere 6.5 U2 will NOT be officially supported with Update 3a ...
...However, Update 3a does introduce "preliminary" U2 support ...It's just that we are unable to support it officially...
Will there be another update (9.5 U3b?) or hotfix to support vSphere 6.5 U2 or do we have to wait for the next major update?

Sorenemig
Influencer
Posts: 12
Liked: never
Joined: Jan 05, 2013 4:16 pm
Full Name: Søren Emig

Re: vSphere 6.5U2 support

Post by Sorenemig » Jun 11, 2018 9:37 am

Gostev wrote: It's now been 1 month since we opened the support case with VMware, and the case has been moving extremely slow. But there's some progress for sure, as apparently we finally pinched through technical support part of the story – because last week, we've been told that PR (Problem Report) has been created and developers are now in the loop. We also have lots of good people from VMware involved at this point, and helping us to push this through in any way they can. So, here's to hope it won't be another month now.
Thank you. Btw. this is one of the things that I like about Veeam as a company. I appreciate that you are “allowed” to keep us in the loop

Gostev
SVP, Product Management
Posts: 24474
Liked: 3414 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » Jun 11, 2018 9:51 am

andrie wrote:Will there be another update (9.5 U3b?) or hotfix to support vSphere 6.5 U2 or do we have to wait for the next major update?
I have covered this in the last forum digest, reposting here:
Gostev wrote:Update 3a does introduce "preliminary" U2 support by fixing all outstanding U2-specific issues (such as hot add transport failing with certain proxies), so in theory a newer U2 build fixing the API issue should be supported automatically – so long as it does not break something else, of course. And in general, Update 3a will make U2 "safer" to use in average (not heavily overprovisioned) environments which already upgraded to one. It's just that we are unable to support it officially, because any significant load spike will cause vSphere API to start failing randomly, consequently impacting most of the Veeam functionality.

chris.childerhose
Service Provider
Posts: 104
Liked: 12 times
Joined: Aug 13, 2014 6:03 pm
Full Name: Chris Childerhose
Location: Toronto, ON
Contact:

Re: vSphere 6.5U2 support

Post by chris.childerhose » Jun 11, 2018 6:14 pm

Would this bug be seen with snapshot integration with Nimble using the ENT+ licensing? Also based on what I am seeing if the vCenter is upgraded to U2 but the hosts stay on U1x the bug will not arise? In the process of migrating to a new Domain in our environment so I am starting fresh with the VCSA appliances and if U2 is an option (allows embedded PSC in ELM) then I want to take it.
-----------------------
Chris Childerhose
Veeam Vanguard / VMCE v9
VCAP-DCA / VCP6 / MCITP
Personal blog: https://just-virtualization.tech
Twitter: @cchilderhose

Gostev
SVP, Product Management
Posts: 24474
Liked: 3414 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » Jun 11, 2018 8:15 pm

Yes, it will be seen regardless of how you do backups. And as far as your other question, we need to wait for the official conclusion from VMware on whether the root cause for the issue sits in the vCenter Server, or in ESXi.

BurdsJM
Lurker
Posts: 1
Liked: never
Joined: May 15, 2018 8:44 pm
Full Name: Jason Burds
Contact:

Re: vSphere 6.5U2 support

Post by BurdsJM » Jun 15, 2018 2:50 pm

Thanks Gostev for the update.

I may just reinstall ESXi on the 4 boxes that got updated to u2. It's not like they have an upgrade path from 6.5 u2 to 6.7.
Jason Burds
City of Dubuque

popjls
Lurker
Posts: 2
Liked: 1 time
Joined: Jun 25, 2018 3:41 am
Contact:

Re: vSphere 6.5U2 support

Post by popjls » Jun 25, 2018 3:44 am

So it's safe to assume that 6.5U2a will need to be verified as well or will the upcoming patch work ok out of the box?

Gostev
SVP, Product Management
Posts: 24474
Liked: 3414 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » Jun 25, 2018 1:52 pm

Yes - we will need to verify that it does address the regression in question, and does so without introducing any new ones.

l.vinokur
Enthusiast
Posts: 26
Liked: 7 times
Joined: Sep 25, 2017 6:37 am
Full Name: Leonid Vinokur
Contact:

Re: vSphere 6.5U2 support

Post by l.vinokur » Jun 25, 2018 4:31 pm

Fred_vBrain wrote:jamcool,

even if you wouldn't have problem with Veeam you would run an unsupported constellation. Check the Interop Matrix. http://partnerweb.vmware.com/comp_guide ... erop&2=&1=

vCenter 6.5 U1x is not supported to manage 6.5 U2 host. It can work but you definitely run into issues over time.

Cheers,
Fred
Thought so as well, but this is incorrect.
You can manage the ESXi with any VC as long as the major+minor versions are same. So any 6.0 VC can manage any 6.0 host, any 6.5 VC can manage any 6.5 (and some lower) hosts.
From your link you can see that VC 6.5 RTM+ can manage ESXi 6.5u2
Image

nobrell
Lurker
Posts: 2
Liked: 5 times
Joined: Sep 17, 2015 6:22 pm
Full Name: Rikard Nobrell
Contact:

Re: vSphere 6.5U2 support

Post by nobrell » Jun 25, 2018 10:12 pm 2 people like this post

Some quick feedback on 9.5.0.1922(update3a) RTM which I got and installed today :)
SureBackup is now working again on vSphere 6.5.U2. (Also updated the appliance)
Regular Backup is still also working but we didn't have any issued before either.

I'm now looking into updating to vSphere 6.7.

Thanks for this update!

rscale
Novice
Posts: 6
Liked: never
Joined: Feb 13, 2018 6:39 pm
Full Name: Mike Richter
Contact:

Re: vSphere 6.5U2 support

Post by rscale » Jun 26, 2018 1:17 pm

Just to clarify where we are, is it correct to say that Veeam currently does not support/recommend moving to 6.5 u2 on either esxi or vcenter? We've had some internal questions on if we can update vcenter at this time and wait for esxi support. Thanks!!

Gostev
SVP, Product Management
Posts: 24474
Liked: 3414 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » Jun 26, 2018 3:32 pm 1 person likes this post

Correct, 6.5 U2 is not currently supported due to a major regression in ESXi code that makes vSphere API start failing under heavy host load.

l.vinokur
Enthusiast
Posts: 26
Liked: 7 times
Joined: Sep 25, 2017 6:37 am
Full Name: Leonid Vinokur
Contact:

Re: vSphere 6.5U2 support

Post by l.vinokur » Jun 26, 2018 5:43 pm

There was a mention that the reason that update2 is not a regular update is that VMware had ported a bunch of code from 6.7

Is 6.7 currently fully supported or should we expect backup issues under heavy load with vSphere 6.7 as well?

Gostev
SVP, Product Management
Posts: 24474
Liked: 3414 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » Jun 26, 2018 6:24 pm

vSphere 6.7 is fully supported with Update 3a, no issues under heavy load there.

RayEL
Lurker
Posts: 1
Liked: never
Joined: Jan 03, 2013 4:51 pm
Full Name: Ray Love
Contact:

Re: vSphere 6.5U2 support

Post by RayEL » Jun 26, 2018 7:42 pm

I have an open ticket on 6.5 u2 as well. If the 9.5 update 3a is available, even if it is not perfect for 6.5u2 (can we at least access this update to see if this helps resolve some of the issues.

Thanks,
Ray

dgapinski
Enthusiast
Posts: 55
Liked: 2 times
Joined: Dec 05, 2013 8:09 pm
Full Name: Dan Gapinski
Contact:

Re: vSphere 6.5U2 support

Post by dgapinski » Jun 26, 2018 8:08 pm

nobrell wrote:I'm now looking into updating to vSphere 6.7.
Nobrell, I just saw today that upgrading from 6.5U2 to 6.7 is currently not supported (https://my.vmware.com/web/vmware/detail ... rPId=23762). I thought you'd think it's quite relevant if you hadn't seen it before. I'm sure that they'd figure out an upgrade path to 6.7 at some point, but something to keep an eye on.

Gostev
SVP, Product Management
Posts: 24474
Liked: 3414 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » Jun 26, 2018 9:58 pm

RayEL wrote:I have an open ticket on 6.5 u2 as well. If the 9.5 update 3a is available, even if it is not perfect for 6.5u2 (can we at least access this update to see if this helps resolve some of the issues).
Yes, most certainly. Your support engineer will be able to provide you Update 3a bits.

ottl05
Influencer
Posts: 14
Liked: 3 times
Joined: Oct 16, 2014 11:29 am
Contact:

Re: vSphere 6.5U2 support

Post by ottl05 » Jun 29, 2018 8:18 am

Is there a release date for the 6.5U2 support for veeam (problem with heavy host load)

nielsengelen
Veeam Software
Posts: 2613
Liked: 538 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: vSphere 6.5U2 support

Post by nielsengelen » Jun 29, 2018 9:32 am

Currently no ETA as the heavy load issue is still there.
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be

Gostev
SVP, Product Management
Posts: 24474
Liked: 3414 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » Jun 29, 2018 12:55 pm 1 person likes this post

The release date for the 6.5U2 support is basically "never". Veeam will only be able to fully support 6.5U2 deployments starting from some newer buld (like 6.5U2a or something, as VMware usually names them). Basically, whatever ESXi build will bring a fix to the issue, we will be able to immediately support with Update 3a (unless it breaks something else).

ottl05
Influencer
Posts: 14
Liked: 3 times
Joined: Oct 16, 2014 11:29 am
Contact:

Re: vSphere 6.5U2 support

Post by ottl05 » Jun 29, 2018 1:37 pm

we had the problem, that the veeam-proxies on the hosts with 6.5u2, crashed. the disk from the other vms (hot-add) where mounted and the proxies died.....

Njumaen
Influencer
Posts: 10
Liked: 3 times
Joined: Dec 04, 2014 5:28 am
Full Name: Ralf Neumann
Contact:

Re: vSphere 6.5U2 support

Post by Njumaen » Jun 30, 2018 9:30 pm

Gostev wrote:Veeam will only be able to fully support 6.5U2 deployments starting from some newer buld (like 6.5U2a or something, as VMware usually names them).
A new build seems to be available!
https://kb.vmware.com/s/article/55913

Gostev
SVP, Product Management
Posts: 24474
Liked: 3414 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » Jul 01, 2018 8:57 pm

It's a patch that addresses the specific issue, which is clearly not the one in question.

Post Reply

Who is online

Users browsing this forum: DGrinev and 24 guests