vSphere 6.5U2 support

[m0ps]

[UPDATE] Jan 31st, 2019
vSphere 6.5 U2 patch level ESXi650-201811002 or later is officially supported with Veeam Backup & Replication 9.5 Update 4

Is there any official position of Veeam regarding just released vSphere 6.5U2? Is it safe to upgrade vCenter and ESXi hosts, or we need to wait for new Veeam B&R release?

[Gostev]

We have not had a chance to deploy and test it yet, as it was literally just released. Normally updates do not require any code changes from us, as typically they bring nothing but bug fixes and minor enhancements. Nevertheless, we recommend that you wait until we update our QC labs and run it through the paces. Thanks!

[m0ps]

Ok, and when you will update your QC lab? Is there any ETA, so we can plan to update our installation to 6.5U2?

[foggy]

We plan to update the lab next week. Cannot provide any ETA at the moment, just keep an eye on this thread - we need to let it roll for some time and will let you know if we see any issues.

[jamerson]

AS i understand now, we cannot update to the U2 6.5 neither to 6.5 until VEEAM tested this ?

[Gostev]

It's up to you. Some people choose to jump these minor updates the day they are released. For obvious reasons, there's some risk involved with upgrading to new platform versions before they've been thoroughly tested and declared as officially supported. Thanks!

[Gostev]

UPDATE: Our auto-tests have exhibited one potentials issue with 6.5 U2 which may reduce reliability of a number of Veeam Backup & Replication features dependent on the corresponding protocol. Having said that, the issue is only reproducible in our stress testing lab where we emulate a heavy environment load, meaning for those who already upgraded these issue will likely exhibit "random" behavior which is in reality tied up to peak load in the environment. We're opening a support case with VMware - and I will keep you posted on the progress. Thanks!

[Saad]

i am one of the people who jumped to that release .. and yes since i have upgraded vcenter and esxi hosts to 6.5u2 i've been having issues with replication jobs with veeam proxies not being able to hot add vmdks and its random and it falls back to network mode . proxies are a mix between 2008 R2 and 2012 R2 fully patched. and in this case i'm not sure if its related to the vcenter 6.2u2

[bmcdonald]

UPDATE: Our auto-tests have exhibited one potentials issue with 6.5 U2 which may reduce reliability of a number of Veeam Backup & Replication features dependent on the corresponding protocol. Having said that, the issue is only reproducible in our stress testing lab where we emulate a heavy environment load, meaning for those who already upgraded these issue will likely exhibit "random" behavior which is in reality tied up to peak load in the environment. We're opening a support case with VMware - and I will keep you posted on the progress. Thanks!

Is this only with the ESXi update or does it also happen with the vCenter update? We're looking at updating vCenter to 6.5U2 for now and holding off on completing the update to the ESXi hosts until Veeam gives an all clear but if issues have been seen with the vCenter update as well then we'll hold off.

[cdbarnes]

Hi all,

Unfortunately, we also jumped to 6.5U2 and are now encountering issues with random failures with our SQL transaction log backups. Currently working with Veeam Support but we haven't really pinned down a definitive answer to what is causing the job to fail.

[rgreen83]

This really needs to be a priority for Veeam to release a fix and not wait until whatever next update that supports 6.7, the 6.7 upgrade is a manual deliberate process, going to 6.5U2 isn't. If you simply use VUM to install security patches etc you will end up with 6.5U2 so Veeam is asking folks to forgo security patches to be supported on their product. I understand what VMware does is outside the control of Veeam and time is needed for testing and fixing if necessary, I'm just requesting that fixes not languish waiting for other things. The distance between patches for product support and fixes has grown far too long in the past couple years while Veeam holds things back waiting for new features to be ready. I do though greatly appreciate the communication from Gostev and others here so at least we aren't in the dark.

[niteb]

Also there is probably an issue with adding vcenter to veeamone. It shows credentials problems. The same cred on vcenter 6.5 u2 works fine.

[marcvanhoutum]

We've noticed some issues with application aware processing when a VM has the VMWare tools installed that come with vSphere 6.5u2 (version 10305). Veeam seems unable to deploy the Vix guest helper application inside the VM.

Downgrading VMWare tools seems to resolve the issue.

[andrie]

I can confirm problems with SQL / Exchange.

On Exchange servers our customer gets the following error message:
Unable to truncate Microsoft Exchange transaction logs. Details: Failed to call RPC function 'Vss.FinishSnapshot': Error code: 0x800401fd. Failed to invoke func [FinishSnapshot]: Object is not connected to server.

On SQL servers, the error message is:
Enumerating SQL Server databases
Failed to call RPC function 'Vss.ExploreInstances': Error code: 0x800401fd. Failed to invoke func [ExploreInstances]: Object is not connected to server.

Support case is opened, Case #03010946

[spiritie]

This really needs to be a priority for Veeam to release a fix and not wait until whatever next update that supports 6.7, the 6.7 upgrade is a manual deliberate process, going to 6.5U2 isn't. If you simply use VUM to install security patches etc you will end up with 6.5U2 so Veeam is asking folks to forgo security patches to be supported on their product. I understand what VMware does is outside the control of Veeam and time is needed for testing and fixing if necessary, I'm just requesting that fixes not languish waiting for other things. The distance between patches for product support and fixes has grown far too long in the past couple years while Veeam holds things back waiting for new features to be ready. I do though greatly appreciate the communication from Gostev and others here so at least we aren't in the dark.

The problem with 6.5U2 is that VMware has back-ported a few features from 6.7 into this update.
So this isn't just another "small" update, but actually quite big one.

[Gostev]

We would like to investigate all issues that customers who have already upgraded to 6.5 U2 are seeing, so please don't be shy to open support cases even if 6.5 U2 is not officially supported yet. This should help us to accelerate its support by identifying all issues that this update introduced sooner. Thank you!

[cdbarnes]

I can confirm problems with SQL / Exchange.

On Exchange servers our customer gets the following error message:
Unable to truncate Microsoft Exchange transaction logs. Details: Failed to call RPC function 'Vss.FinishSnapshot': Error code: 0x800401fd. Failed to invoke func [FinishSnapshot]: Object is not connected to server.

On SQL servers, the error message is:
Enumerating SQL Server databases
Failed to call RPC function 'Vss.ExploreInstances': Error code: 0x800401fd. Failed to invoke func [ExploreInstances]: Object is not connected to server.

Support case is opened, Case #03010946

This is one of the few errors we have seen in our environment during SQL transaction log runs. We would see a couple of successful runs then a failure or two, then another failure. This seems to have tapered off in the last few days with no change in the environment. Support case is closed, due to no recent failures, Case #02899195.

[nd39475]

Is this only with the ESXi update or does it also happen with the vCenter update? We're looking at updating vCenter to 6.5U2 for now and holding off on completing the update to the ESXi hosts until Veeam gives an all clear but if issues have been seen with the vCenter update as well then we'll hold off.

I would also like a definitive answer on this. I can report we are running VCSA 6.5.0.20000 307201 with ESXi 6.0.0 7967664 and we have not seen any backup failures; However, we have not had any restore opportunities or testing of the backups.

[mazcredi]

What is the latest release of vSphere that VEEAM recommends below 6.5U2?

[DGrinev]

Hi Mazen,

You can stick with any supported vSphere version before 6.5U2. Thanks!

[cambiumphil]

I would also like a definitive answer on this. I can report we are running VCSA 6.5.0.20000 307201 with ESXi 6.0.0 7967664 and we have not seen any backup failures; However, we have not had any restore opportunities or testing of the backups.

I don't think you'll have a problem with vCenter 6.5U2 as long as you aren't using ESXi 6.5U2. My (limited) understanding is Veeam B&R uses vCenter to figure out where in your vSphere cluster stuff is so that it can talk to ESXi and VMware Tools in your VMs. ESXi 6.5U2 and the associated VMware Tools is where a lot of people are having trouble.

[Henrik.Grevelund]

You can stick with any supported vSphere version before 6.5U2. Thanks!

Hi,

Note sure you you would link to a documentation page that doesn't mention anything about which updates are supported ?

[DGrinev]

Hi,

It is vSphere 6.5 any update or patch before vSphere 6.5 Update 2 that Veeam B&R supports. Thanks!

[ChuckS42]

Personally, folks really need to wait for Veeam (and any other third-party provider's software you use that integrates with vSphere/vCenter) to test before installing anything from VMware labelled as an "Update", as opposed to a "Patch". Patches are generally safe, Updates need testing. You're aiming a loaded gun at your foot every time you jump the gun. (How's that for a mixed metaphor?)

[cambiumphil]

Agreed! My only 6.5U2 environment (VC 6.5U2 / ESXi 6.0) is my test lab where if it doesn't work it's a "oh. I guess I'll wait for the patch" type situation

We will absolutely hold off on updating anything production until there's a Veeam B&R update that supports it!

[dgapinski]

We are at vSphere 6.5 5973321 and ESXi, 6.5.0, 6765664, and are experiencing issues with the bug that causes our esxi hosts to lose ability to manage VMs & can only be fixed by reboot. So we're anxious to upgrade to U2, and I'll keep watching this thread. Please post progress? (thank you!)

[Gostev]

No material response from VMware Support for 12 days now, so we keep digging the issue on our end and trying to find possible workarounds. It's quite unlikely we will be able to fix it reliably from our side though, as the issue appears to sit in the ESXi itself. So in case VMware confirms this regression, you will likely have to wait for them to ship ESXi 6.5 U2a (or something) before upgrading.

[dgapinski]

No material response from VMware Support for 12 days now, so we keep digging the issue on our end and trying to find possible workarounds. It's quite unlikely we will be able to fix it reliably from our side though, as the issue appears to sit in the ESXi itself. So in case VMware confirms this regression, you will likely have to wait for them to ship ESXi 6.5 U2a (or something) before upgrading.

Thanks for your update Gostev. So to be specific, U2 & Veeam only have problems with SQL/Exchange app-aware jobs? Regular jobs are ok? Given our pain with the current version, I'm wondering if we can update vSphere and ESX to U2 on all the hosts where we don't have Exchange and SQL running, and pin those VMs (with host affinity rules) to non-upgraded hosts. Would that work?

[Gostev]

No, you misunderstood the thread. The main issue is this one, and if you upgrade to vSphere 6.5 U2 now, almost all Veeam functionality will start behaving unreliably during periods of heavy environment load because of a regression in one of the cornerstone ESXi APIs that we're using.

There are no other big known issues at the moment that we know about, AFAIK. We've been investigating all reported issues, but so far all of them were either unrelated to the 6.5U2 upgrade, or caused by the API issue above.

[Sorenemig]

I upgraded to 6.5U2 by mistake. So far, I have not experienced errors (not what I realize). All backup jobs seem to work fine. All my jobs are simple backup jobs without application integration.

Unfortunately, it is not possible to remove an update. My option is to reinstall all my hosts!

Gostev:

If I understand correctly, it is only under heavy load the issue(s) manifest itself? Will it then be reasanble safe to stay on the current version?

Also, what do you mean by heavy load?