Discussions specific to the VMware vSphere hypervisor
Locked
Gostev
SVP, Product Management
Posts: 24295
Liked: 3330 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

vSphere 6.7 U1 Support

Post by Gostev » Oct 16, 2018 8:43 pm 8 people like this post

Folks, just a heads up!

We're seeing consistent backup failures after installing Update 1 in our first test lab. Judging on the error, it looks like VMware has possibly introduced some breaking API change which affects our transport selection logic, so this does not look like something minor - and will likely affect 100% of our users.

I will update after devs had a chance to investigate this issue, and QC tests the update in more labs - but for now, please do avoid installing this update just to be on a safe side.

Official support KB article > KB2784

Thanks!

[UPDATE] October 17th
The issue has been fully understood, so I am documenting it here until the KB article is updated.

Symptons
After updating vCenter to 6.7U1, processing of all VMs fails with "Object reference not set to an instance of an object"

Cause
The issue is triggered by an update to the vSphere API version in vCenter 6.7 U1.

Workaround
Create the following registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key on the backup server:

Code: Select all

Value: VMwareOverrideApiVersion
Type: Multi-String Value (REG_MULTI_SZ)
Data: 6.7.1 = 6.7
Important
This is a temporary workaround against the specific error, which will allow the jobs to complete successfully. Overriding VMware API version may potentially cause issues with other Veeam functionality, because we don't know all the specific API changes that made VMware increment the API version. We're working with VMware to obtain these details while continuing to test vSphere 6.7 U1 with the workaround applied.

The release vehicle for the official out-of-the-box support for vSphere 6.7 U1 will be our next release (Update 4). We'll be able to determine supportability of vSphere 6.7 U1 with the workaround applied after full regression testing cycle is done against the U1 GA code (usually takes 3-4 weeks).

[UPDATE] October 19th
All auto-tests of Update 3a with the workaround enabled have completed successfully, so all base Veeam Backup & Replication functionality is now confirmed to work with vSphere 6.7 U1. I will provide another update once the full regression testing cycle completes.

ngreen
Novice
Posts: 7
Liked: 1 time
Joined: Jun 15, 2017 2:48 pm
Full Name: Neil Green
Contact:

Re: vSphere 6.7 U1 Support

Post by ngreen » Oct 17, 2018 12:57 pm

Thanks for the warning. Would be useful to identify if it's ESXi or vCenter related (or both). Ideally we'd like to update to vCenter 6.7 U1 to get the fully featured HTML5 client, even if we left the hosts on ESXi 6.7 GA.

mwvme
Veeam Software
Posts: 161
Liked: 33 times
Joined: Dec 05, 2015 10:19 pm
Full Name: Michael White
Location: Calgary, Alberta Canada
Contact:

Re: vSphere 6.7 U1 Support

Post by mwvme » Oct 17, 2018 1:03 pm 1 person likes this post

I reproduced this issue with only vCenter updated and hosts still at 6.7. You need to wait, unfortunately.

Michael
Michael White
Field Product Manager
https://notesfrommwhite.net
@mwVme

tka
Novice
Posts: 5
Liked: 2 times
Joined: Apr 04, 2014 6:24 am
Full Name: Tim
Location: Cologne, Germany
Contact:

Re: vSphere 6.7 U1 Support

Post by tka » Oct 17, 2018 1:22 pm 1 person likes this post

ESXi 6.7 U1 with "old" vCenter works - at least with SAN transport.

Trelor
Influencer
Posts: 16
Liked: 3 times
Joined: Apr 27, 2015 6:02 pm
Contact:

Re: vSphere 6.7 U1 Support

Post by Trelor » Oct 17, 2018 1:28 pm

How are you running a version of vCenter that is behind ESXI that is not even supported

mwvme
Veeam Software
Posts: 161
Liked: 33 times
Joined: Dec 05, 2015 10:19 pm
Full Name: Michael White
Location: Calgary, Alberta Canada
Contact:

Re: vSphere 6.7 U1 Support

Post by mwvme » Oct 17, 2018 1:30 pm 2 people like this post

I upgraded my vCenter, which is in fact the correct order, and I did not update my hosts yet. So this is all legit.

Michael
Michael White
Field Product Manager
https://notesfrommwhite.net
@mwVme

tka
Novice
Posts: 5
Liked: 2 times
Joined: Apr 04, 2014 6:24 am
Full Name: Tim
Location: Cologne, Germany
Contact:

Re: vSphere 6.7 U1 Support

Post by tka » Oct 17, 2018 1:36 pm 1 person likes this post

Trelor wrote:
Oct 17, 2018 1:28 pm
How are you running a version of vCenter that is behind ESXI that is not even supported
Yes, in this case it is supported to run vCenter 6.7 with ESXi 6.7 u1:
https://www.vmware.com/resources/compat ... matrix.php

tka
Novice
Posts: 5
Liked: 2 times
Joined: Apr 04, 2014 6:24 am
Full Name: Tim
Location: Cologne, Germany
Contact:

Re: vSphere 6.7 U1 Support

Post by tka » Oct 17, 2018 2:07 pm

I did an Update via VCSA Installer wizard from 6.5 to 6.7 u1 today and veeam failed afterwards.

stefanbrun
Service Provider
Posts: 23
Liked: 3 times
Joined: Apr 26, 2011 7:36 am
Full Name: Stefan Brun | MSupport Networks AG
Location: Lengnau, Switzerland
Contact:

Re: vSphere 6.7 U1 Support

Post by stefanbrun » Oct 17, 2018 2:18 pm

I did an upgrade from VCenter 6.7 to 6.7 U1 via the VCSA web updater.
Updatet also the ESXi hosts from 6.7 to 6.7 U1.

Tested backup via VCenter and ESXi direct, on both tests same problem.

Gostev
SVP, Product Management
Posts: 24295
Liked: 3330 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.7 U1 Support

Post by Gostev » Oct 17, 2018 2:26 pm 1 person likes this post

Updated the original post with the workaround.

stefanbrun
Service Provider
Posts: 23
Liked: 3 times
Joined: Apr 26, 2011 7:36 am
Full Name: Stefan Brun | MSupport Networks AG
Location: Lengnau, Switzerland
Contact:

Re: vSphere 6.7 U1 Support

Post by stefanbrun » Oct 17, 2018 2:46 pm

Thx Gostev, workaround does the job.

Gostev
SVP, Product Management
Posts: 24295
Liked: 3330 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.7 U1 Support

Post by Gostev » Oct 17, 2018 2:47 pm

Super, glad to hear that :D

P.S. In case you're as fast as one of my colleagues, please note that the registry key location was off originally, but is corrected now.

ddenson
Lurker
Posts: 1
Liked: 1 time
Joined: Sep 10, 2013 4:12 pm
Full Name: Drew Denson
Location: Madison, Wisconsin
Contact:

Re: vSphere 6.7 U1 Support

Post by ddenson » Oct 17, 2018 3:39 pm 1 person likes this post

I can confirm that the workaround is working for us also. Thank you!
Gostev wrote:
Oct 16, 2018 8:43 pm
Workaround
Create the following registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key on the backup server:

Code: Select all

Value: VMwareOverrideApiVersion
Type: Multi-String Value (REG_MULTI_SZ)
Data: 6.7.1 = 6.7

rendest
Influencer
Posts: 19
Liked: 5 times
Joined: Feb 01, 2017 8:36 pm
Full Name: Stef
Contact:

Re: vSphere 6.7 U1 Support

Post by rendest » Oct 17, 2018 4:17 pm

We can confirm it works (upgrade both VCSA and ESX to 6.7u1 in lab)

jorgedlcruz
Veeam Software
Posts: 253
Liked: 109 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: vSphere 6.7 U1 Support

Post by jorgedlcruz » Oct 17, 2018 8:05 pm 1 person likes this post

Tried on the Home Lab, working like a charm.

Thank you Anton!
Jorge de la Cruz
Systems Engineer UKI @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2019 / Nutanix Technology Champion 2018-2019

ochm
Lurker
Posts: 2
Liked: never
Joined: Jul 14, 2017 7:32 am
Full Name: Martin Och
Contact:

Re: vSphere 6.7 U1 Support

Post by ochm » Oct 18, 2018 7:29 am

Works perfectly (vCenter6.7U1, ESXi some older 6.7).
Failed job (before applied workaround) muste be RUN after registry setting, do not try RETRY, it leads to error.

stanthewizzard
Influencer
Posts: 10
Liked: 1 time
Joined: Aug 09, 2012 8:16 am
Full Name: Stan
Contact:

Re: vSphere 6.7 U1 Support

Post by stanthewizzard » Oct 18, 2018 8:31 am

Yes
workaround does the job

Squuiid
Lurker
Posts: 1
Liked: 1 time
Joined: Oct 18, 2018 12:08 pm
Contact:

Re: vSphere 6.7 U1 Support

Post by Squuiid » Oct 18, 2018 12:12 pm 1 person likes this post

@Gostev Would the issue be present when using vCenter 6.7u1 but with 6.5 hosts?

chris.childerhose
Service Provider
Posts: 104
Liked: 12 times
Joined: Aug 13, 2014 6:03 pm
Full Name: Chris Childerhose
Location: Toronto, ON
Contact:

Re: vSphere 6.7 U1 Support

Post by chris.childerhose » Oct 18, 2018 1:44 pm 1 person likes this post

@Squuiid - thanks for asking this as we have the same situation with VCSA at 6.7 but hosts remain on 6.5. I am curious to know if updating VCSA will have issues as getting the full HTML5 client is a must at some point.

UPDATE - received confirmation that if the hosts stay at 6.5 and the VCSA 6.7 is updated with U1 that the API goes at "base" 6.7 so no need for the registry setting if you use this configuration of VC & Hosts. 8)
-----------------------
Chris Childerhose
Veeam Vanguard / VMCE v9
VCAP-DCA / VCP6 / MCITP
Personal blog: https://just-virtualization.tech
Twitter: @cchilderhose

Gostev
SVP, Product Management
Posts: 24295
Liked: 3330 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.7 U1 Support

Post by Gostev » Oct 18, 2018 2:34 pm

We have not explicitly tested this scenario (vCenter 6.7U1 with ESXi 6.5 hosts).

kaschenberg
Novice
Posts: 3
Liked: 1 time
Joined: Oct 18, 2018 3:01 pm
Full Name: Kyle Aschenberg
Contact:

Re: vSphere 6.7 U1 Support

Post by kaschenberg » Oct 18, 2018 3:03 pm 1 person likes this post

chris.childerhose wrote:
Oct 18, 2018 1:44 pm
@Squuiid - thanks for asking this as we have the same situation with VCSA at 6.7 but hosts remain on 6.5. I am curious to know if updating VCSA will have issues as getting the full HTML5 client is a must at some point.

UPDATE - received confirmation that if the hosts stay at 6.5 and the VCSA 6.7 is updated with U1 that the API goes at "base" 6.7 so no need for the registry setting if you use this configuration of VC & Hosts. 8)
This is not true. We use all 6.5 esx hosts and after updating to vcenter 6.7 U1, my backups were failing. After applying the registry fix they are running fine again.

jlipari
Lurker
Posts: 1
Liked: never
Joined: Nov 15, 2012 9:46 pm
Full Name: Jason Lipari
Contact:

Re: vSphere 6.7 U1 Support

Post by jlipari » Oct 18, 2018 6:29 pm

kaschenberg wrote:
Oct 18, 2018 3:03 pm
This is not true. We use all 6.5 esx hosts and after updating to vcenter 6.7 U1, my backups were failing. After applying the registry fix they are running fine again.
Same here - All backups were failing after upgrading from a previous build of vCenter 6.7 to 6.7U1, with all ESXi 6.5 hosts. The registry fix resolved the failures.

ashleyw
Service Provider
Posts: 154
Liked: 20 times
Joined: Oct 28, 2010 10:55 pm
Full Name: Ashley Watson
Contact:

Re: vSphere 6.7 U1 Support

Post by ashleyw » Oct 18, 2018 7:40 pm 1 person likes this post

Thanks @Gostev and team, Registry fix works a charm on our side 6.7 update 1 vcentre appliance (build 6.7.0.20000), esxi hosts running 6.7 update 1 as well (build 10302608).
I was worried there for a second, but good news for us is that there are a few fixed issues around cert handling in VCSA 6.7 that appear to have been fixed in update 1 release, and to top it off we have backups again ;-)

Gostev
SVP, Product Management
Posts: 24295
Liked: 3330 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.7 U1 Support

Post by Gostev » Oct 19, 2018 2:05 pm

All auto-tests of Update 3a with the workaround enabled have completed successfully, so all base Veeam Backup & Replication functionality is now confirmed to work with vSphere 6.7 U1. I will provide another update once the full regression testing cycle completes.

chris.childerhose
Service Provider
Posts: 104
Liked: 12 times
Joined: Aug 13, 2014 6:03 pm
Full Name: Chris Childerhose
Location: Toronto, ON
Contact:

Re: vSphere 6.7 U1 Support

Post by chris.childerhose » Oct 19, 2018 2:49 pm

Thanks everyone for the confirmation and I will be applying the fix to my Veeam server. I will also talk to my source about the misinformation. :)
-----------------------
Chris Childerhose
Veeam Vanguard / VMCE v9
VCAP-DCA / VCP6 / MCITP
Personal blog: https://just-virtualization.tech
Twitter: @cchilderhose

saltymb08
Lurker
Posts: 1
Liked: never
Joined: Oct 19, 2018 4:44 pm
Full Name: Adam Costello
Contact:

Re: vSphere 6.7 U1 Support

Post by saltymb08 » Oct 19, 2018 4:46 pm

Hey guys, does this registry adjustment need to be done on the proxy servers as well? I have done this on the backup server with ZERO luck on the effect. Also, does this require a certain patch level of 9.5?

Gostev
SVP, Product Management
Posts: 24295
Liked: 3330 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.7 U1 Support

Post by Gostev » Oct 19, 2018 7:46 pm

Only on the backup server, and you have to be on Update 3a.

jamerson
Expert
Posts: 311
Liked: 18 times
Joined: May 01, 2013 9:54 pm
Full Name: Julien
Contact:

Re: vSphere 6.7 U1 Support

Post by jamerson » Oct 20, 2018 9:24 pm

I can confirm the registry has done the job.
big thank you guys.

zimeon
Novice
Posts: 3
Liked: 1 time
Joined: May 26, 2018 5:55 pm
Contact:

Re: vSphere 6.7 U1 Support

Post by zimeon » Oct 22, 2018 4:22 am

Works in the lab as well, no problems so far!

zadrian
Enthusiast
Posts: 81
Liked: 4 times
Joined: Jul 14, 2015 8:26 am
Contact:

Re: vSphere 6.7 U1 Support

Post by zadrian » Oct 23, 2018 10:20 am 1 person likes this post

I do not think it is wise to ask customers not to update vCenter or ESXi to 6.7u1 as there are so many security patches nowadays.
I would think that Veeam should have some sort of announcement made to their customers via the Veeam GUI about such issues and workarounds.

Locked

Who is online

Users browsing this forum: tsightler and 30 guests