vSphere 6.5U2 support

[Gostev]

Correct, it's reasonably safe to stay on 6.5 U2 if you're not observing intermittent Veeam failures. And in any case, the issue will not cause data corruptions or anything like that. It is just that in heavily over-provisioned environments, our product will be failing left and right on most operations due to underlying vSphere API failures - which is the only reason why we can't officially support it on vSphere 6.5 U2 yet.

[Gostev]

For those on 6.5U2 already, please be aware that we've now confirmed one other issue that causes hot add transport to fail on certain backup proxies.

[williampu]

Case ID #03028438

Currently have an issue with vCenter 6.5 Update 2 / ESXi 6.5 Update 2 / Veeam 9.5 Update 3.

vCenter was upgraded to 6.5 Update 2 a fortnight ago and we haven't seen any issues since the upgrade.
The ESXi hosts began upgrades to 6.5 Update 2 late last week and since then, some proxies have become invalid in vCenter.
The proxies which go invalid are in a large cluster of VM's (~1300 VMs) so I assume it would be the major load on the proxies and the ESXi patch causing issues.
Other clusters don't seem to have the issue with less VM's and proxies.
We are only doing VM level backups (no SQL or Oracle level backups yet) so we haven't seen any major issues besides running out of time in the backup window.
We are currently in the progress of rolling back the ESXi update applied to the hosts so I will try and report back after that has been completed and a night's worth of backup has been executed.

[Sorenemig]

The same think happend in our setup.

We are now using NBD and it seems stable. I don't want to downgrade unless it is absolutely necessary.

@Gostev, any new information on the case Veeam opened with VMware?

[jamcool]

Our vCenter is still at update 1g but we have started upgrading our VM Host to 6.5 update2 and just started having this problem. Veeam support had me on the proxy server running diskpart disable automount but that did not help. Anybody find a solution yet?

[Fred_vBrain]

jamcool,

even if you wouldn't have problem with Veeam you would run an unsupported constellation. Check the Interop Matrix. http://partnerweb.vmware.com/comp_guide ... erop&2=&1=

vCenter 6.5 U1x is not supported to manage 6.5 U2 host. It can work but you definitely run into issues over time.

Cheers,
Fred

[Gostev]

@Gostev, any new information on the case Veeam opened with VMware?

It's now been 1 month since we opened the support case with VMware, and the case has been moving extremely slow. But there's some progress for sure, as apparently we finally pinched through technical support part of the story – because last week, we've been told that PR (Problem Report) has been created and developers are now in the loop. We also have lots of good people from VMware involved at this point, and helping us to push this through in any way they can. So, here's to hope it won't be another month now.

[andrie]

VMware vSphere 6.5 U2 will NOT be officially supported with Update 3a ...
...However, Update 3a does introduce "preliminary" U2 support ...It's just that we are unable to support it officially...

Will there be another update (9.5 U3b?) or hotfix to support vSphere 6.5 U2 or do we have to wait for the next major update?

[Sorenemig]

It's now been 1 month since we opened the support case with VMware, and the case has been moving extremely slow. But there's some progress for sure, as apparently we finally pinched through technical support part of the story – because last week, we've been told that PR (Problem Report) has been created and developers are now in the loop. We also have lots of good people from VMware involved at this point, and helping us to push this through in any way they can. So, here's to hope it won't be another month now.

Thank you. Btw. this is one of the things that I like about Veeam as a company. I appreciate that you are “allowed” to keep us in the loop

[Gostev]

Will there be another update (9.5 U3b?) or hotfix to support vSphere 6.5 U2 or do we have to wait for the next major update?

I have covered this in the last forum digest, reposting here:

Update 3a does introduce "preliminary" U2 support by fixing all outstanding U2-specific issues (such as hot add transport failing with certain proxies), so in theory a newer U2 build fixing the API issue should be supported automatically – so long as it does not break something else, of course. And in general, Update 3a will make U2 "safer" to use in average (not heavily overprovisioned) environments which already upgraded to one. It's just that we are unable to support it officially, because any significant load spike will cause vSphere API to start failing randomly, consequently impacting most of the Veeam functionality.

[chris.childerhose]

Would this bug be seen with snapshot integration with Nimble using the ENT+ licensing? Also based on what I am seeing if the vCenter is upgraded to U2 but the hosts stay on U1x the bug will not arise? In the process of migrating to a new Domain in our environment so I am starting fresh with the VCSA appliances and if U2 is an option (allows embedded PSC in ELM) then I want to take it.

[Gostev]

Yes, it will be seen regardless of how you do backups. And as far as your other question, we need to wait for the official conclusion from VMware on whether the root cause for the issue sits in the vCenter Server, or in ESXi.

[BurdsJM]

Thanks Gostev for the update.

I may just reinstall ESXi on the 4 boxes that got updated to u2. It's not like they have an upgrade path from 6.5 u2 to 6.7.

[popjls]

So it's safe to assume that 6.5U2a will need to be verified as well or will the upcoming patch work ok out of the box?

[Gostev]

Yes - we will need to verify that it does address the regression in question, and does so without introducing any new ones.

[l.vinokur]

jamcool,

even if you wouldn't have problem with Veeam you would run an unsupported constellation. Check the Interop Matrix. http://partnerweb.vmware.com/comp_guide ... erop&2=&1=

vCenter 6.5 U1x is not supported to manage 6.5 U2 host. It can work but you definitely run into issues over time.

Cheers,
Fred

Thought so as well, but this is incorrect.
You can manage the ESXi with any VC as long as the major+minor versions are same. So any 6.0 VC can manage any 6.0 host, any 6.5 VC can manage any 6.5 (and some lower) hosts.
From your link you can see that VC 6.5 RTM+ can manage ESXi 6.5u2

[nobrell]

Some quick feedback on 9.5.0.1922(update3a) RTM which I got and installed today
SureBackup is now working again on vSphere 6.5.U2. (Also updated the appliance)
Regular Backup is still also working but we didn't have any issued before either.

I'm now looking into updating to vSphere 6.7.

Thanks for this update!

[rscale]

Just to clarify where we are, is it correct to say that Veeam currently does not support/recommend moving to 6.5 u2 on either esxi or vcenter? We've had some internal questions on if we can update vcenter at this time and wait for esxi support. Thanks!!

[Gostev]

Correct, 6.5 U2 is not currently supported due to a major regression in ESXi code that makes vSphere API start failing under heavy host load.

[l.vinokur]

There was a mention that the reason that update2 is not a regular update is that VMware had ported a bunch of code from 6.7

Is 6.7 currently fully supported or should we expect backup issues under heavy load with vSphere 6.7 as well?

[Gostev]

vSphere 6.7 is fully supported with Update 3a, no issues under heavy load there.

[RayEL]

I have an open ticket on 6.5 u2 as well. If the 9.5 update 3a is available, even if it is not perfect for 6.5u2 (can we at least access this update to see if this helps resolve some of the issues.

Thanks,
Ray

[dgapinski]

I'm now looking into updating to vSphere 6.7.

Nobrell, I just saw today that upgrading from 6.5U2 to 6.7 is currently not supported (https://my.vmware.com/web/vmware/detail ... rPId=23762). I thought you'd think it's quite relevant if you hadn't seen it before. I'm sure that they'd figure out an upgrade path to 6.7 at some point, but something to keep an eye on.

[Gostev]

I have an open ticket on 6.5 u2 as well. If the 9.5 update 3a is available, even if it is not perfect for 6.5u2 (can we at least access this update to see if this helps resolve some of the issues).

Yes, most certainly. Your support engineer will be able to provide you Update 3a bits.

[ottl05]

Is there a release date for the 6.5U2 support for veeam (problem with heavy host load)

[nielsengelen]

Currently no ETA as the heavy load issue is still there.

[Gostev]

The release date for the 6.5U2 support is basically "never". Veeam will only be able to fully support 6.5U2 deployments starting from some newer buld (like 6.5U2a or something, as VMware usually names them). Basically, whatever ESXi build will bring a fix to the issue, we will be able to immediately support with Update 3a (unless it breaks something else).

[ottl05]

we had the problem, that the veeam-proxies on the hosts with 6.5u2, crashed. the disk from the other vms (hot-add) where mounted and the proxies died.....

[Njumaen]

Veeam will only be able to fully support 6.5U2 deployments starting from some newer buld (like 6.5U2a or something, as VMware usually names them).

A new build seems to be available!
https://kb.vmware.com/s/article/55913

[Gostev]

It's a patch that addresses the specific issue, which is clearly not the one in question.