vSphere 6.7 U1 Support

[Gostev]

Maybe it would be appropriate to write it in large letters on your site.

Well, to be fair, we say this in big splashes with large red letters all over the web site (product download area, support section, customer portal etc.) and they were there since day 1... this is our standard approach, we did this for every previous "breaking" vSphere release as well.

[rarens]

Hi,

look my last post ... can I get an answer please?

Thanks a lot

[Gostev]

Hi Reinhard, for answers on any technical issues you should contact support directly. Thanks!

[rarens]

Hi,

I only want to confirm, if the registry hack does not help, if I directly backup the server.
Manuelly started the job and same problem, but this night scheduled start worked for me.

Only for information if someone has the same problem.

Best regards

reinhard

[wishr]

Hi Reinhard,

As Gostev suggested, please reach out to our support team for comprehensive assistance if you still experience the issue and B&R server reboot did not help. The hotfix should work in this scenario.

Regards,
Fedor

[trippinnik]

Imho, 90 days support timeframe is reasonable for a quality product like Veeam. My experience with other vendor is terrible, there are one that claim support the day after but then simply don't work, and another one that don't support new updates for months.... Maybe could be positive to have a beta program where clients can register and be able to update before RTM, for sure not for production. Most vendor, even vmware, have something like that.

PS: I open support case asking if it's possible to access early RTM bits for our testing enviroment.

No, still the only vendor we are waiting on. Zerto, Nimble, nVidia all ready to go. We need that Vmotion with Grid support like 4 years ago and this is the only thing holding us back.

[indi]

So, we got an issue regarding replication. Starting multiple replications jobs, in our case atleast 5 jobs, kills the vpxd process on the vCenter appliance.

Running vCenter 6.7.0.20100 with external PSC. Veeam 9.5.01922 with the regfix for the 6.7.1 API.

Backups run fine, replication kills vCenter.

I have an open case with VMware but would like to know if this is a confirmed issue/limit somehow?

[indi]

Was likely fixed by disabling and re-enabling storage DRS on our pre-v6.5 created DSC's, which were used as targets for replication jobs. PodConfig.SpaceLoadBalanceConfig.MinSpaceUtilizationDifference gave an error while disabling SDRS but that was fixed.

Replication works again, yay!

[dustinn3]

No, still the only vendor we are waiting on. Zerto, Nimble, nVidia all ready to go. We need that Vmotion with Grid support like 4 years ago and this is the only thing holding us back.

That's my main reason for wanting to upgrade to 6.7 U1 as well. Funny, vmware told me a few years ago that it would be impossible to ever vmotion with Grid because it would be equivalent of removing and replacing the video card on a physical pc and Windows would crash. Still, I'm not rushing to install it because I'd rather let everyone else test it out first.

[adtww]

After updating to 9.5 u4, should we remove the registry key (6.7.1 = 6.7) ?

[Andreas Neufert]

@reinhard,
please contact support or check latest forum digest to get fully supported Update 4 RTM version.

[Andreas Neufert]

After updating to 9.5 u4, should we remove the registry key (6.7.1 = 6.7) ?

Yes, you can remove this setting and you should. Please restart the Backup Service afterwards. If the reg key will be present we will treat the new Update 1 like the normal 6.7 version which is compatible but I guess there are many optimizations done by VMware in their development kit and us for Update 1. So removing it would be a good idea.

[Novox]

I have come to appreciate Veeam’s commitment to quality, especially given the importance of an error free backup infrastructure. Imagine how my opinion would change were I to receive releases sooner, but which broke my backup infrastructure (and potentially prior backups).

Alternatively, you could start to dread Veeam’s “patch Tuesday.”

Microsoft recently had disastrous results releasing Windows 10 October Update (1809). That release deleted some user’s files in addition to little quirks like not reconnecting network drives. After it was finally re-released in November, issues like not reconnecting network drives still weren’t fixed…

Just last week, Microsoft released an “Important” (automatically installed using “recommended” settings) update for Windows 7, which in some cases, deactivated genuine windows activations and/or broke windows networking.

It’s worth noting that in the examples above, Microsoft broke their own products. They did not have the additional burden of integrating with third party software.

How many days to wait for a quality release is a matter of opinion, so I won’t try to convince anyone of anything. Personally, I consider ~90 days reasonable.

Thank you,
~Bill

[mcz]

I was new with your product and with the update to 6.5 I updated after 1 month about not knowing your timing. I found myself with the backup not working and I had to reinstall 4 esxi hosts. It's the first time in 20 years of work that such a thing happened to me.

Why didn't you check the official documentation or asked support if the version was supported? That should have been done first!

[Aurelio]

Well, to be fair, we say this in big splashes with large red letters all over the web site (product download area, support section, customer portal etc.) and they were there since day 1... this is our standard approach, we did this for every previous "breaking" vSphere release as well.

I ran into the same issue and I did check your website for information in advance, but to be fair I did not see any big splashes with large red letters all over the web site!!
To check compatibility you request that I check the download page of a product that I have already downloaded and installed and where I did know in advance that the last change and update was done in December 2017!
Why would I regularly check the download page if the software itself informs me if an update is available!
This is a serious issue and should be on the front page!

[Regnor]

In general I declare every new VMware release as unsupported before I find a statement which says that it works and/or is supported; at least all major releases. This goes for Veeam and Backup Exec which I both work with.
I can't risk the backup of our customers and VMware often introduces new incompatibilities.

On the other hand I've often requested a better support or compatibility matrix as find information on supported application is not always as easy with Veeam as with other Vendors. If I ever publish a blog I'll create such a matrix

[Gostev]

Release notes is the ultimate source for this information. Please check them out, and see if anything is missing that you think should be added (I usually personally work on them). Thanks!

[chris_lalala]

Is there a Veeam Release in sight witch supports vSphere 6.7 U1 without any Workaround?

I'm not willing to implement any Workaround witch causes later issues.

[Andreas Neufert]

There is no issue with the workaround later. You can remove the reg key later even after some day when you have upgraded to Update 4 which will support vSphere 6.7U1.

And yes, there is fully support Veeam Backup & Replication v9.5 Update 4 RTM available through Veeam support ticket or download method described in the Forum weekly digest. Release of update 4 is planned for Jan 22.

RTM Version can be updated to later Veeam versions without reinstall.

[ITP-Stan]

I suggest you subscribe to the weekly forum digest mail by Gostev.
That always contains information about major Microsoft or VMWare updates and compatibility.

[stvajnkf]

And yes, there is fully support Veeam Backup & Replication v9.5 Update 4 RTM available through Veeam support ticket or download method described in the Forum weekly digest. Release of update 4 is planned for Jan 22.

What if we don't subscribe to the weekly digests. Can someone share the secret handshake

[Novox]

Not really a secret, but:

[evilaedmin]

I have a slightly different complaint, which is not that Veeam isn't doing enough to support latest releases, but that VMware isn't doing enough to make sure their releases are supported by key partners in a timely fashion. Some examples:

• Gostev stated somewhere that there is no public or partner preview program for GA releases of vSphere which would allow partners to begin regression testing earlier. Sounds insane to me.
• Gostev stated somewhere that during critical bugs such as the VDDK hot-add slowness issues, a major partner like Veeam is at the mercy of VMware and there was no hotfix in sight. An argument can easily be formed that Veeam is VMware's largest backup partner and that their customers would be most dis proportionally affected.
• VMware for some reason doesn't seem to prioritize regressions such as the introduction of the crashing hotadd proxy in 6.5 Update 2, which affected more Veeam customers than any speculative execution attacks, which received many timely patches. Again, from a practical customer angle, this seems like a misallocation of resources. Yes, I am saying that for security and availability a crashing proxy had much more impact than speculative execution attacks.

So is there anything a customer of Veeam and VMware can do to help the situation?

[Novox]

Great work today Anton (Gostev)!

Where and how will RTM users be notified about the 0-day update for GA? I know it's a few days after GA, but will it be clearly listed on the download site? Will a download link be included in the next digest? Do I need to open a support ticket to request it?

Thank you,
~Bill

[Gostev]

Thank you!

From the official communication perspective, I assume we will likely just notify everyone who downloaded the RTM bits directly via email (unless they opted to unsubscribe from all Veeam communication). But of course, there will be a KB article, forum post, digest mention etc.

[pocjason]

Update 4: https://www.veeam.com/kb2878

After this update is installed, do we need to delete the registry key that was created for the workaround as per KB 2784 or does the installation of this update modify/delete that key?

[foggy]

Hi Jason, you can remove the key, however, this doesn't affect product operation.

[Andreas Neufert]

I have a slightly different complaint, which is not that Veeam isn't doing enough to support latest releases, but that VMware isn't doing enough to make sure their releases are supported by key partners in a timely fashion. Some examples:

• Gostev stated somewhere that there is no public or partner preview program for GA releases of vSphere which would allow partners to begin regression testing earlier. Sounds insane to me.
• Gostev stated somewhere that during critical bugs such as the VDDK hot-add slowness issues, a major partner like Veeam is at the mercy of VMware and there was no hotfix in sight. An argument can easily be formed that Veeam is VMware's largest backup partner and that their customers would be most dis proportionally affected.
• VMware for some reason doesn't seem to prioritize regressions such as the introduction of the crashing hotadd proxy in 6.5 Update 2, which affected more Veeam customers than any speculative execution attacks, which received many timely patches. Again, from a practical customer angle, this seems like a misallocation of resources. Yes, I am saying that for security and availability a crashing proxy had much more impact than speculative execution attacks.

So is there anything a customer of Veeam and VMware can do to help the situation?

Hi evilaedmin,

I think we fixed this on both sides and work now even more closely together than before. The details are NDA related but we are on a really good path now.

That said, we will still need to do regression testing with final code (VMware add bugfixes until last minute and we need to check to provide official support statement. Our work with VMware can provide at day 0 additional details and be able to react faster if something breaks. And this not only for release but as well for updates.

@All, please remember to delete the added reg key some time after update 4 implementation to enable Veeam to use the new VMware vSphere 6.7 U1 related API calls.

[someonesomewhere]

Could I just ask, does 9.5 U4 support the latest vCenter 6.7 U1b (6.7.0.21000)

Thanks

[Andreas Neufert]

Patches usually do not change the APIs and it just works. Yes