Host-based backup of VMware vSphere VMs.
Locked
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

vSphere 6.7 U1 Support

Post by Gostev » 8 people like this post

Folks, just a heads up!

We're seeing consistent backup failures after installing Update 1 in our first test lab. Judging on the error, it looks like VMware has possibly introduced some breaking API change which affects our transport selection logic, so this does not look like something minor - and will likely affect 100% of our users.

I will update after devs had a chance to investigate this issue, and QC tests the update in more labs - but for now, please do avoid installing this update just to be on a safe side.

Official support KB article > KB2784

Thanks!

[UPDATE] October 17th
The issue has been fully understood, so I am documenting it here until the KB article is updated.

Symptons
After updating vCenter to 6.7U1, processing of all VMs fails with "Object reference not set to an instance of an object"

Cause
The issue is triggered by an update to the vSphere API version in vCenter 6.7 U1.

Workaround
Create the following registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key on the backup server:

Code: Select all

Value: VMwareOverrideApiVersion
Type: Multi-String Value (REG_MULTI_SZ)
Data: 6.7.1 = 6.7
Important
This is a temporary workaround against the specific error, which will allow the jobs to complete successfully. Overriding VMware API version may potentially cause issues with other Veeam functionality, because we don't know all the specific API changes that made VMware increment the API version. We're working with VMware to obtain these details while continuing to test vSphere 6.7 U1 with the workaround applied.

The release vehicle for the official out-of-the-box support for vSphere 6.7 U1 will be our next release (Update 4). We'll be able to determine supportability of vSphere 6.7 U1 with the workaround applied after full regression testing cycle is done against the U1 GA code (usually takes 3-4 weeks).

[UPDATE] October 19th
All auto-tests of Update 3a with the workaround enabled have completed successfully, so all base Veeam Backup & Replication functionality is now confirmed to work with vSphere 6.7 U1. I will provide another update once the full regression testing cycle completes.
ngreen
Novice
Posts: 7
Liked: 1 time
Joined: Jun 15, 2017 2:48 pm
Full Name: Neil Green
Contact:

Re: vSphere 6.7 U1 Support

Post by ngreen »

Thanks for the warning. Would be useful to identify if it's ESXi or vCenter related (or both). Ideally we'd like to update to vCenter 6.7 U1 to get the fully featured HTML5 client, even if we left the hosts on ESXi 6.7 GA.
mwvme
Expert
Posts: 163
Liked: 33 times
Joined: Dec 05, 2015 10:19 pm
Full Name: Michael White
Location: Calgary, Alberta Canada
Contact:

Re: vSphere 6.7 U1 Support

Post by mwvme » 1 person likes this post

I reproduced this issue with only vCenter updated and hosts still at 6.7. You need to wait, unfortunately.

Michael
Michael White
Field Product Manager
https://notesfrommwhite.net
@mwVme
tka
Service Provider
Posts: 6
Liked: 2 times
Joined: Apr 04, 2014 6:24 am
Full Name: Tim
Location: Cologne, Germany
Contact:

Re: vSphere 6.7 U1 Support

Post by tka » 1 person likes this post

ESXi 6.7 U1 with "old" vCenter works - at least with SAN transport.
Trelor
Enthusiast
Posts: 47
Liked: 15 times
Joined: Apr 27, 2015 6:02 pm
Contact:

Re: vSphere 6.7 U1 Support

Post by Trelor »

How are you running a version of vCenter that is behind ESXI that is not even supported
mwvme
Expert
Posts: 163
Liked: 33 times
Joined: Dec 05, 2015 10:19 pm
Full Name: Michael White
Location: Calgary, Alberta Canada
Contact:

Re: vSphere 6.7 U1 Support

Post by mwvme » 2 people like this post

I upgraded my vCenter, which is in fact the correct order, and I did not update my hosts yet. So this is all legit.

Michael
Michael White
Field Product Manager
https://notesfrommwhite.net
@mwVme
tka
Service Provider
Posts: 6
Liked: 2 times
Joined: Apr 04, 2014 6:24 am
Full Name: Tim
Location: Cologne, Germany
Contact:

Re: vSphere 6.7 U1 Support

Post by tka » 1 person likes this post

Trelor wrote: Oct 17, 2018 1:28 pmHow are you running a version of vCenter that is behind ESXI that is not even supported
Yes, in this case it is supported to run vCenter 6.7 with ESXi 6.7 u1:
https://www.vmware.com/resources/compat ... matrix.php
tka
Service Provider
Posts: 6
Liked: 2 times
Joined: Apr 04, 2014 6:24 am
Full Name: Tim
Location: Cologne, Germany
Contact:

Re: vSphere 6.7 U1 Support

Post by tka »

I did an Update via VCSA Installer wizard from 6.5 to 6.7 u1 today and veeam failed afterwards.
stefanbrun
Service Provider
Posts: 27
Liked: 5 times
Joined: Apr 26, 2011 7:36 am
Full Name: Stefan Brun | Streamline AG
Location: Switzerland
Contact:

Re: vSphere 6.7 U1 Support

Post by stefanbrun »

I did an upgrade from VCenter 6.7 to 6.7 U1 via the VCSA web updater.
Updatet also the ESXi hosts from 6.7 to 6.7 U1.

Tested backup via VCenter and ESXi direct, on both tests same problem.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.7 U1 Support

Post by Gostev » 1 person likes this post

Updated the original post with the workaround.
stefanbrun
Service Provider
Posts: 27
Liked: 5 times
Joined: Apr 26, 2011 7:36 am
Full Name: Stefan Brun | Streamline AG
Location: Switzerland
Contact:

Re: vSphere 6.7 U1 Support

Post by stefanbrun »

Thx Gostev, workaround does the job.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.7 U1 Support

Post by Gostev »

Super, glad to hear that :D

P.S. In case you're as fast as one of my colleagues, please note that the registry key location was off originally, but is corrected now.
ddenson
Lurker
Posts: 1
Liked: 1 time
Joined: Sep 10, 2013 4:12 pm
Full Name: Drew Denson
Location: Madison, Wisconsin
Contact:

Re: vSphere 6.7 U1 Support

Post by ddenson » 1 person likes this post

I can confirm that the workaround is working for us also. Thank you!
Gostev wrote: Oct 16, 2018 8:43 pmWorkaround
Create the following registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key on the backup server:

Code: Select all

Value: VMwareOverrideApiVersion
Type: Multi-String Value (REG_MULTI_SZ)
Data: 6.7.1 = 6.7
rendest
Influencer
Posts: 20
Liked: 6 times
Joined: Feb 01, 2017 8:36 pm
Full Name: Stef
Contact:

Re: vSphere 6.7 U1 Support

Post by rendest »

We can confirm it works (upgrade both VCSA and ESX to 6.7u1 in lab)
jorgedlcruz
Veeam Software
Posts: 1355
Liked: 613 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: vSphere 6.7 U1 Support

Post by jorgedlcruz » 1 person likes this post

Tried on the Home Lab, working like a charm.

Thank you Anton!
Jorge de la Cruz
Senior Product Manager | Veeam ONE @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2024 / InfluxAce / Grafana Champion
ochm
Service Provider
Posts: 3
Liked: 1 time
Joined: Jul 14, 2017 7:32 am
Full Name: Martin Och
Contact:

Re: vSphere 6.7 U1 Support

Post by ochm »

Works perfectly (vCenter6.7U1, ESXi some older 6.7).
Failed job (before applied workaround) muste be RUN after registry setting, do not try RETRY, it leads to error.
stanthewizzard
Influencer
Posts: 22
Liked: 4 times
Joined: Aug 09, 2012 8:16 am
Full Name: Stan
Contact:

Re: vSphere 6.7 U1 Support

Post by stanthewizzard »

Yes
workaround does the job
Squuiid
Lurker
Posts: 1
Liked: 1 time
Joined: Oct 18, 2018 12:08 pm
Contact:

Re: vSphere 6.7 U1 Support

Post by Squuiid » 1 person likes this post

@Gostev Would the issue be present when using vCenter 6.7u1 but with 6.5 hosts?
chris.childerhose
Veeam Vanguard
Posts: 572
Liked: 132 times
Joined: Aug 13, 2014 6:03 pm
Full Name: Chris Childerhose
Location: Toronto, ON
Contact:

Re: vSphere 6.7 U1 Support

Post by chris.childerhose » 1 person likes this post

@Squuiid - thanks for asking this as we have the same situation with VCSA at 6.7 but hosts remain on 6.5. I am curious to know if updating VCSA will have issues as getting the full HTML5 client is a must at some point.

UPDATE - received confirmation that if the hosts stay at 6.5 and the VCSA 6.7 is updated with U1 that the API goes at "base" 6.7 so no need for the registry setting if you use this configuration of VC & Hosts. 8)
-----------------------
Chris Childerhose
Veeam Vanguard / Veeam Legend / Veeam Ceritified Architect / VMCE
vExpert / VCAP-DCA / VCP8 / MCITP
Personal blog: https://just-virtualization.tech
Twitter: @cchilderhose
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.7 U1 Support

Post by Gostev »

We have not explicitly tested this scenario (vCenter 6.7U1 with ESXi 6.5 hosts).
kaschenberg
Novice
Posts: 3
Liked: 1 time
Joined: Oct 18, 2018 3:01 pm
Full Name: Kyle Aschenberg
Contact:

Re: vSphere 6.7 U1 Support

Post by kaschenberg » 1 person likes this post

chris.childerhose wrote: Oct 18, 2018 1:44 pm @Squuiid - thanks for asking this as we have the same situation with VCSA at 6.7 but hosts remain on 6.5. I am curious to know if updating VCSA will have issues as getting the full HTML5 client is a must at some point.

UPDATE - received confirmation that if the hosts stay at 6.5 and the VCSA 6.7 is updated with U1 that the API goes at "base" 6.7 so no need for the registry setting if you use this configuration of VC & Hosts. 8)
This is not true. We use all 6.5 esx hosts and after updating to vcenter 6.7 U1, my backups were failing. After applying the registry fix they are running fine again.
jlipari
Influencer
Posts: 10
Liked: never
Joined: Nov 15, 2012 9:46 pm
Contact:

Re: vSphere 6.7 U1 Support

Post by jlipari »

kaschenberg wrote: Oct 18, 2018 3:03 pm This is not true. We use all 6.5 esx hosts and after updating to vcenter 6.7 U1, my backups were failing. After applying the registry fix they are running fine again.
Same here - All backups were failing after upgrading from a previous build of vCenter 6.7 to 6.7U1, with all ESXi 6.5 hosts. The registry fix resolved the failures.
ashleyw
Service Provider
Posts: 181
Liked: 30 times
Joined: Oct 28, 2010 10:55 pm
Full Name: Ashley Watson
Contact:

Re: vSphere 6.7 U1 Support

Post by ashleyw » 1 person likes this post

Thanks @Gostev and team, Registry fix works a charm on our side 6.7 update 1 vcentre appliance (build 6.7.0.20000), esxi hosts running 6.7 update 1 as well (build 10302608).
I was worried there for a second, but good news for us is that there are a few fixed issues around cert handling in VCSA 6.7 that appear to have been fixed in update 1 release, and to top it off we have backups again ;-)
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.7 U1 Support

Post by Gostev »

All auto-tests of Update 3a with the workaround enabled have completed successfully, so all base Veeam Backup & Replication functionality is now confirmed to work with vSphere 6.7 U1. I will provide another update once the full regression testing cycle completes.
chris.childerhose
Veeam Vanguard
Posts: 572
Liked: 132 times
Joined: Aug 13, 2014 6:03 pm
Full Name: Chris Childerhose
Location: Toronto, ON
Contact:

Re: vSphere 6.7 U1 Support

Post by chris.childerhose »

Thanks everyone for the confirmation and I will be applying the fix to my Veeam server. I will also talk to my source about the misinformation. :)
-----------------------
Chris Childerhose
Veeam Vanguard / Veeam Legend / Veeam Ceritified Architect / VMCE
vExpert / VCAP-DCA / VCP8 / MCITP
Personal blog: https://just-virtualization.tech
Twitter: @cchilderhose
saltymb08
Lurker
Posts: 1
Liked: never
Joined: Oct 19, 2018 4:44 pm
Full Name: Adam Costello
Contact:

Re: vSphere 6.7 U1 Support

Post by saltymb08 »

Hey guys, does this registry adjustment need to be done on the proxy servers as well? I have done this on the backup server with ZERO luck on the effect. Also, does this require a certain patch level of 9.5?
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.7 U1 Support

Post by Gostev »

Only on the backup server, and you have to be on Update 3a.
jamerson
Veteran
Posts: 366
Liked: 24 times
Joined: May 01, 2013 9:54 pm
Full Name: Julien
Contact:

Re: vSphere 6.7 U1 Support

Post by jamerson »

I can confirm the registry has done the job.
big thank you guys.
zimeon
Novice
Posts: 4
Liked: 1 time
Joined: May 26, 2018 5:55 pm
Contact:

Re: vSphere 6.7 U1 Support

Post by zimeon »

Works in the lab as well, no problems so far!
zadrian
Expert
Posts: 133
Liked: 4 times
Joined: Jul 14, 2015 8:26 am
Contact:

Re: vSphere 6.7 U1 Support

Post by zadrian » 1 person likes this post

I do not think it is wise to ask customers not to update vCenter or ESXi to 6.7u1 as there are so many security patches nowadays.
I would think that Veeam should have some sort of announcement made to their customers via the Veeam GUI about such issues and workarounds.
Locked

Who is online

Users browsing this forum: tomo yasu and 92 guests