vSphere 6.7 U1 Support

[Gostev]

Folks, just a heads up!

We're seeing consistent backup failures after installing Update 1 in our first test lab. Judging on the error, it looks like VMware has possibly introduced some breaking API change which affects our transport selection logic, so this does not look like something minor - and will likely affect 100% of our users.

I will update after devs had a chance to investigate this issue, and QC tests the update in more labs - but for now, please do avoid installing this update just to be on a safe side.

Official support KB article > KB2784

Thanks!

[UPDATE] October 17th
The issue has been fully understood, so I am documenting it here until the KB article is updated.

Symptons
After updating vCenter to 6.7U1, processing of all VMs fails with "Object reference not set to an instance of an object"

Cause
The issue is triggered by an update to the vSphere API version in vCenter 6.7 U1.

Workaround
Create the following registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key on the backup server:

Code: Select all

Value: VMwareOverrideApiVersion
Type: Multi-String Value (REG_MULTI_SZ)
Data: 6.7.1 = 6.7

Important
This is a temporary workaround against the specific error, which will allow the jobs to complete successfully. Overriding VMware API version may potentially cause issues with other Veeam functionality, because we don't know all the specific API changes that made VMware increment the API version. We're working with VMware to obtain these details while continuing to test vSphere 6.7 U1 with the workaround applied.

The release vehicle for the official out-of-the-box support for vSphere 6.7 U1 will be our next release (Update 4). We'll be able to determine supportability of vSphere 6.7 U1 with the workaround applied after full regression testing cycle is done against the U1 GA code (usually takes 3-4 weeks).

[UPDATE] October 19th
All auto-tests of Update 3a with the workaround enabled have completed successfully, so all base Veeam Backup & Replication functionality is now confirmed to work with vSphere 6.7 U1. I will provide another update once the full regression testing cycle completes.

[ngreen]

Thanks for the warning. Would be useful to identify if it's ESXi or vCenter related (or both). Ideally we'd like to update to vCenter 6.7 U1 to get the fully featured HTML5 client, even if we left the hosts on ESXi 6.7 GA.

[mwvme]

I reproduced this issue with only vCenter updated and hosts still at 6.7. You need to wait, unfortunately.

Michael

[tka]

ESXi 6.7 U1 with "old" vCenter works - at least with SAN transport.

[Trelor]

How are you running a version of vCenter that is behind ESXI that is not even supported

[mwvme]

I upgraded my vCenter, which is in fact the correct order, and I did not update my hosts yet. So this is all legit.

Michael

[tka]

How are you running a version of vCenter that is behind ESXI that is not even supported

Yes, in this case it is supported to run vCenter 6.7 with ESXi 6.7 u1:
https://www.vmware.com/resources/compat ... matrix.php

[tka]

I did an Update via VCSA Installer wizard from 6.5 to 6.7 u1 today and veeam failed afterwards.

[stefanbrun]

I did an upgrade from VCenter 6.7 to 6.7 U1 via the VCSA web updater.
Updatet also the ESXi hosts from 6.7 to 6.7 U1.

Tested backup via VCenter and ESXi direct, on both tests same problem.

[Gostev]

Updated the original post with the workaround.

[stefanbrun]

Thx Gostev, workaround does the job.

[Gostev]

Super, glad to hear that

P.S. In case you're as fast as one of my colleagues, please note that the registry key location was off originally, but is corrected now.

[ddenson]

I can confirm that the workaround is working for us also. Thank you!

Workaround
Create the following registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key on the backup server:

Code: Select all

Value: VMwareOverrideApiVersion
Type: Multi-String Value (REG_MULTI_SZ)
Data: 6.7.1 = 6.7

[rendest]

We can confirm it works (upgrade both VCSA and ESX to 6.7u1 in lab)

[jorgedlcruz]

Tried on the Home Lab, working like a charm.

Thank you Anton!

[ochm]

Works perfectly (vCenter6.7U1, ESXi some older 6.7).
Failed job (before applied workaround) muste be RUN after registry setting, do not try RETRY, it leads to error.

[stanthewizzard]

Yes
workaround does the job

[Squuiid]

@Gostev Would the issue be present when using vCenter 6.7u1 but with 6.5 hosts?

[chris.childerhose]

@Squuiid - thanks for asking this as we have the same situation with VCSA at 6.7 but hosts remain on 6.5. I am curious to know if updating VCSA will have issues as getting the full HTML5 client is a must at some point.

UPDATE - received confirmation that if the hosts stay at 6.5 and the VCSA 6.7 is updated with U1 that the API goes at "base" 6.7 so no need for the registry setting if you use this configuration of VC & Hosts.

[Gostev]

We have not explicitly tested this scenario (vCenter 6.7U1 with ESXi 6.5 hosts).

[kaschenberg]

@Squuiid - thanks for asking this as we have the same situation with VCSA at 6.7 but hosts remain on 6.5. I am curious to know if updating VCSA will have issues as getting the full HTML5 client is a must at some point.

UPDATE - received confirmation that if the hosts stay at 6.5 and the VCSA 6.7 is updated with U1 that the API goes at "base" 6.7 so no need for the registry setting if you use this configuration of VC & Hosts.

This is not true. We use all 6.5 esx hosts and after updating to vcenter 6.7 U1, my backups were failing. After applying the registry fix they are running fine again.

[jlipari]

This is not true. We use all 6.5 esx hosts and after updating to vcenter 6.7 U1, my backups were failing. After applying the registry fix they are running fine again.

Same here - All backups were failing after upgrading from a previous build of vCenter 6.7 to 6.7U1, with all ESXi 6.5 hosts. The registry fix resolved the failures.

[ashleyw]

Thanks @Gostev and team, Registry fix works a charm on our side 6.7 update 1 vcentre appliance (build 6.7.0.20000), esxi hosts running 6.7 update 1 as well (build 10302608).
I was worried there for a second, but good news for us is that there are a few fixed issues around cert handling in VCSA 6.7 that appear to have been fixed in update 1 release, and to top it off we have backups again

[Gostev]

All auto-tests of Update 3a with the workaround enabled have completed successfully, so all base Veeam Backup & Replication functionality is now confirmed to work with vSphere 6.7 U1. I will provide another update once the full regression testing cycle completes.

[chris.childerhose]

Thanks everyone for the confirmation and I will be applying the fix to my Veeam server. I will also talk to my source about the misinformation.

[saltymb08]

Hey guys, does this registry adjustment need to be done on the proxy servers as well? I have done this on the backup server with ZERO luck on the effect. Also, does this require a certain patch level of 9.5?

[Gostev]

Only on the backup server, and you have to be on Update 3a.

[jamerson]

I can confirm the registry has done the job.
big thank you guys.

[zimeon]

Works in the lab as well, no problems so far!

[zadrian]

I do not think it is wise to ask customers not to update vCenter or ESXi to 6.7u1 as there are so many security patches nowadays.
I would think that Veeam should have some sort of announcement made to their customers via the Veeam GUI about such issues and workarounds.