vSphere 6.7 U1 Support

[anakhla]

@Gostev

Just worked with our Sales Support Engineer on getting our vm's to backup. We applied Veeam Update 3a and the hotfix to our backup server. Hotfix was also applied to our proxy server. But it kept failing with the Object reference not set to an instance of an object. After realizing our vcenter, not esxi host, was on 6.7 Update 1 we applied this registry key to the backup server. However this did not fix the problem for us. Only after applying the registry key to the proxy were we able to successfully backup vm's.

So if you are on VMware 6.7 update 1 and have updated Veeam to 3a, applied the hotfix and done this registry key and your backups are still failing, try adding this registry key to your proxies as well.

[Did]

Just a funny note:
We have multiple Veaam enviroments connected to several vCenters. One of these vCenters was recently upgraded to 6.7U1 but I haven't had issues running backups using our first Veeam server. Today I connected another Veeam server to the 6.7U1 vCenter and I started getting the "Object reference not set"-error on that Veeam instance, but that was fixed by following by applying the hotfix to Veeam B&R and its proxy.

But hey, the first Veeam server was working the whole time?! With some more investigation I realised that it's running an older Veeam B&R 9.5.0.1536, which is running backups on 6.7U1 just fine. I haven't tried restoring though.

[whysyn]

Hello @Gostev. Where can I obtain a beta? There is one VMWare cluster we use internally that I'd be happy to test on. Thanks!

[Gostev]

Through your Veeam sales rep.

[GlenG]

Do we know if RVTools will need an update for 6.7U1?
Thanks,

[rbanchieri]

Hi,

Just after update VCSA to 6.7u1, and with the registry key, I have an error when I try to read objects by VMware Tags:

Code: Select all

QueryAllCategoryInfos failed.
Failed to initialize inventory service session
One or more errors occurred.
The remote server returned an error: (503) Server Unavailable.

Do you have too ?
Thanks.

[ochm]

@Gostev

Just worked with our Sales Support Engineer on getting our vm's to backup. We applied Veeam Update 3a and the hotfix to our backup server. Hotfix was also applied to our proxy server. But it kept failing with the Object reference not set to an instance of an object. After realizing our vcenter, not esxi host, was on 6.7 Update 1 we applied this registry key to the backup server. However this did not fix the problem for us. Only after applying the registry key to the proxy were we able to successfully backup vm's.

So if you are on VMware 6.7 update 1 and have updated Veeam to 3a, applied the hotfix and done this registry key and your backups are still failing, try adding this registry key to your proxies as well.

Did you restart the VBR server or Veeam services on VBR server?
The registry hotfix works well for me on 3 servers.

[sebastiaanvanduijn]

Is there already an expected date for the release? Seen the latest release notes of VMWare we are really keen to upgrade.

[wishr]

Hi Sebastiaan,

Welcome to Veeam Community Forums and thanks for posting.

Unfortunately, we are not able to comment on the release dates. It will be released when it's ready, our target to support new platform releases has always been 90 days. For now, please review this thread and reach out our support team if additional help is required.

BR,
Fedor

[redhorse]

Just to clarify: vSphere ESXi 6.7 Update 1 is not supported with the latest Veeam version? If so, may I install ESXi 6.7 Express Patch 5 (ESXi670-201811001)?

[Gostev]

Correct, official support for vSphere ESXi 6.7 Update 1 is coming in Veeam Backup & Replication 9.5 Update 4 only.

ESXi 6.7 GA is supported though, as well as its patched version. So, just make sure the patches you're installing are for ESXi 6.7 GA (and not for ESXi 6.7 Update 1).

[Nobody]

Hmm, where this patches can be found?

The Security Problem we need to fix is described in VMSA-2018-0027. VMware writes only about VMware ESXi 6.7, Patch Release ESXi670-201811001 to fix this issue.
If i take a look at this patch - it will upgrade to ESXi Build 10764712 which is higher than ESXi Build 10302608 from 6.7 u1.

From the Veeam Support side, you say ESXi 6.7 U1 is not supported - so you talk about >= Build 10302608?
I still think that i'm missing something - but what? Is VMware really providing these security updates only for 6.7u1 and not for 6.7 GA?

On the other hand i think that VMware really should involve it's partner more closely into testing new functions they are releasing.
How they can just change a API Version for example a lot of partners are using. We would have Support by Veeam for 6.7 U1 if they would have involved you guys a month earlier.

[wishr]

Hi Andreas,

Just to clarify, based on the VMSA details you have to apply the EP5 GA patch (ESXi 6.7 Express Patch 5, ESXi670-201811001, Released on 2018-11-09, Build 10764712) in order to fix the security issue. It's higher than U1, but it's a different "GA" branch.

You are right assuming that the official support for U1 Gostev mentioned in the post above relates to the build 10302608 that is not a part of the GA branch. I would recommend taking a look at the first page of this topic, though.

You can download the patches from the official MyVMware portal.

BR,
Fedor

[lazman]

It 's incredible that after spending € 10,000 for the license of this backup software, my employees and I have to be careful not to update esxi and remember to search in your forum at what point you are with the release of the update. I do not know the situation of the other vendors, but I find this absurd thing. (I understand it's not your fault, but you should get a preferential channel with vmware to get their updated version first, you're the market leader and even the most expensive backups on esxi). With the upgrade to ESXi 6.7, I had to go back to version 6.5 with 4 of my hosts. Everything worked perfectly, except to find out that your backup was not working anymore because of the update. I imagined having to wait a week at most, but I see that waiting times are at least a month and a half each time! I do not find it right to follow a forum to find out if and when I can update. Ok, it's not essential, but the worst thing is to remember not to do it and search the forum!

[Gostev]

Actually, forums is a very secondary source for this type of information. We always post these announcements over our web site, from the product download page to the customer portal. Did you not see this notification when you downloaded the product?

[lazman]

When I upgraded to 6.7, I upgraded directly to my vcenter and all went well, only to discover that my vm was no longer saved. So I had to install 6.5 on 4 hosts and reconfigure them. Now I know I do not update until the new version of B & R and I follow the forum to see the developments. Do you know when it will come out? Are you going to change the policy for the future and synchronize with the release of ESXi versions? Have you tried to find a solution with them or do you always wait 1 or 2 months? Thank you.

[Gostev]

Please note that 6.7 is fully supported already, it is only 6.7 U1 that we don't support out of the box due to the API version change. Out of the box support for 6.7 U1 will be included in Update 4, which should be shipped in a few weeks.

Our policy is to perform full regression and stability testing on the GA code of the new platform versions - which including bug fix requires 1 or 2 months (or longer in case of major reliability issues). So, in order for us to be able to synchronize with ESXi releases, VMware would have to provide us the RTM code in advance, instead of on the same day with everyone. This has been our request to them for a long time, but as of now their approach remains unchanged.

[lazman]

I know that 6.7 is fully supported, I was referring to a past fact. I said that it is unpleasant to have to remember not to update ESXi and every time having to wait (we do not know how much) the exit of the B & R update. You are the reference brand for backups with VMware, it is incredible that everything is resolved with a "This has been our request to them for a long time, but as of now their approach remains unchanged". You are not an ordinary customer.
I believe that what I said is not wrong, even if it is strongly said!
  We will wait, for every update if anything else can not be done! Thank you.

[wishr]

There are some know situations when VMware had to recall their patches (remember 6.5 U1e, 6.0 U3d, and some others), that's why I would think three times before blindly upgrading a VMware environment or any other critical business-driven workload right after a new version release.

Just my 2 cents.

Regards,
Fedor

[lazman]

You're right, I remember. But if after the summer break, I decide that today I have time to update my servers, I go to the vmware website and I see that the latest version was released on 10/16/2018 and today we are 03/12/2018, I imagine that the trial period is over and if there were big problems they would have been discovered after 15 days or 1 month at the most. The particular case exists, but I do not think it can happen that they recall an update after 2 months.

[mark.partin]

lazman, I've worked in IT for a long time and this is normal. Actually, Veeam does an excellent job with communication (particularly Gostev) of when to expect updates, issues that have been found, and where the "fault" lies in software breakdowns. It is wise to never upgrade systems until you have verified compatibility with your hardware and software systems. This is typically done in a testing environment after checking compatibility assurances with your hardware/software vendors.

I had to upgrade our UCS firmware/software in preparation for upgrade to ESXi 6.7. This required using the Cisco compatibility matrix, which is similar to Veeam's information, to find a compatible version for firmware, Ethernet, and Fiber drivers compatible with everything. I've also looked at compatibility with other software/hardware and have had extensive conversations with Veeam engineers about new cloud integrations. Upgrading our worldwide production systems is an important project and I don't take it lightly.

Doing due diligence research before upgrading critical systems such as Veeam should be standard practice. Upgrading on a whim whenever you have a free moment is a recipe for disaster, as you found. I'm anxiously awaiting the release of the update for Veeam. I hear it has some great functionality additions/improvements.

Good luck in future

[trippinnik]

6.7 U1 is a pretty important upgrade for us. The update 4 release states a few weeks away since October ... What's the current status? All the other integrations we have with vmware like disaster recovery and storage are already compatible. It's just veeam we are waiting on.

[lazman]

mark.partin, in theory, having time and money you could also have an exact copy of the server room, a pity that in reality it is not always the case. I have owned veeam for a few months, before I used VDP and I have never had problems with ESXi updates. We have too many software to have a compatibility matrix of everything, there is always something that can go wrong. I usually check the manufacturer's website (vmware) for specifications and when I get an idea I upgrade a first machine, in case I have backups. In the case of the last update everything went well, I realized only the day after VEEAM had problems, it had not occurred to me to check also its site / forum. Fortunately, it's not always like that with everything. Now that I know I'll come to the forum to see when Gost tells us that we can update. I am amazed that it is managed as a small community of people who are confronted and wait for the Ok to update. That's all.

[anpa]

I had to upgrade our UCS firmware/software in preparation for upgrade to ESXi 6.7. This required using the Cisco compatibility matrix, which is similar to Veeam's information, to find a compatible version for firmware, Ethernet, and Fiber drivers compatible with everything. I've also looked at compatibility with other software/hardware and have had extensive conversations with Veeam engineers about new cloud integrations. Upgrading our worldwide production systems is an important project and I don't take it lightly.

This is more important than people realize. VMware is doing a good job in later versions with the vSAN Health Check to remind people to use correct driver and firmware versions, and notify when there is a new version that has been released to fix issues etc.

As for our selfes, we're running the HPE CS700 compatibility matrix for our switches, blade chassis and 3PAR arrays, and thanks to designing the environment this way we have very few issues with firmware, software or even VMware issues. Yes, it puts us a couple of versions behind (i.e no 6.7 support yet, and 6.5 was ready for us after firmware upgrade this spring) but in terms of availability towards customers it is _so much_ worth it.

And generally speaking, we have been avoiding new major version of vSphere until U1 has been released, but since VMware now has changed their release schedule to include major feature upgrades even in update releases we may have to review our internal policy further down the road..

[Doug1974]

yep, got the old "Error: Object reference not set to an instance of an object." we only run a small server environment and usually wait a few months before applying the latest updates (to avoid issues) unless there is critical fixes, or trouble with existing environment (vCenter/chrome browser/flash drama). i got caught out this time, probably should have waited a few more months, but was having a lot of trouble with web GUI.

Fortunately the Registry key worked..

just needed to restart the server for the Reg Key to be applied.

[crackocain]

Hi

Could you test the replication and backup VM with Persistent Memory? Persistent Memory looks like great fit midrange customers want to quick speed up. Vsphere 6.7 feature.The data in the NVDIMM. Two battery unit protected the NVDIMMs but i think without replication Persistent Memory still risky.

Actually best solution i guess replicate the data in the NVDIMMs persistent memory to HDD or SSD in another server.

Thank you.

https://www.youtube.com/watch?v=-kXaMDhLqjA

[Gostev]

VMware does not support snapshotting VMs with virtual NVDIMM devices present, so you can only back them up with Veeam Agents.

[NightBird]

Waiting Waiting for vSphere 6.7U1 support => Veeam 9.5U4, but a think not before january, U4 RTM is always not available

[andreas2012]

Hi,

We have upgraded vCenter with the 7.6.1 update 1, and problem occured.
Then we tried to add the registry settings on the Veeam backup and replication server, rebooted, but same problem.
Tried rebooting the proxies also.
I noticed someone said to add it to the proxy also, but did you then create the complete path, since on the proxy there is no "Veeam" under HKLM\SOFTWARE\ ?

Thanks for reply.

[albertwt]

When will be the Update 4 release date?