vCenter 8.0 Update 3 is out. Does this need QA approval?

[Gostev]

It's in the post right above mine

[karsten123]

it is mid Q3. where is v12.2?

[Mildur]

Hi Karsten

We never said ETA is mid-Q3
The targeted release window is Q3 2024. Which means the release date can be any date till end of September.

Best,
Fabian

[RubinCompServ]

it is mid Q3. where is v12.2?

If i had to guess, I'd say it's either in testing or in beta release.

I expect it to be supported with our next release v12.2 (Q3 2024).

Like @Gostev said, there was no promise of it hitting full release in mid-Q3.

[karsten123]

i dont know guys. then communicate it like it is. end of Q3, early Q4. THANKS!

[Gostev]

Nobody said it is "end of Q3, early Q4" either

It's ready When It's Ready™ so it's next to impossible to predict the exact week when it ships. We're only reasonably confident it will be this quarter.

[vfwdit]

We goofed up and upgraded our servers from 7.0 U3 (latest) to 8.0 U3 (latest). (vCenter and all hosts included.)

We now get CBRC errors, even though the hosts all have CBRC enabled, and the VMs all have CTK on, when backing up using VEEAM 12.1.2.172 (More specifically, "CBRC cannot be hot-enabled on a disk unless the VM was powered-on with digest disks or cbrc.cacheEnable was set.")

Backup seems to actually run, though, and back up data, though it takes longer than usual.

All of our datastores are vSAN data stores and we're using the VEEAM proxies as we did before.

Error immediately appeared during our hourly backups after the update.

I'm not complaining, this is our fault for forgetting to check VEEAM compatibility before the upgrade, but yeah... Hopefully this issue will go away once 12.2 comes out or whatever version officially supports 8.0 U3

[Gostev]

This looks like VMware Horizon specific configuration setting. I'm not sure if our QA has ever tested compatibility with this product, at least I don't see it in the list of officially supported VMware products (unlike VSAN or Cloud Director for example). But I will double check.

{UPDATE] I confirmed that our QA does not have any labs with VMware Horizon, so it's not something we validate or officially support protection of. Of course, since there are regular VMware VMs underneath of Horizon, backup "just works" (well - worked, in your case). The error you're seeing is likely simply passed by jobs from vSphere and is likely completely unrelated to Veeam, so I would suggest opening a support case with VMware on Horizon - perhaps they already saw something like this before and know how to fix/reset it.

Also, according to the Release Notes VMware did indeed touch some code around CBRC in U3:

PR 2931647: You cannot create snapshots of virtual machines due to an error in the Content Based Read Cache (CBRC) that a digest operation has failed
A rare race condition when assigning a content ID during the update of the CBRC digest file might cause a discrepancy between the content ID in the data disk and the digest disk. As a result, you cannot create virtual machine snapshots. You see an error such as An error occurred while saving the snapshot: A digest operation has failed in the backtrace. The snapshot creation task completes upon retry. This issue is resolved in this release.

[johannesk]

We are being pressed to go to vSphere 8 U3 because of CVE-2024-37087. I wonder what the risk is of going ahead with it and VBR still on 12.1.2.172?
Is it known what would potentialy break?
If we run into issues, will we be able to get hotfixes?

regards,
Jóhannes

[Gostev]

The risk is very low based on our testing results. We're getting ready to declare compatibility-level support with 12.1.2 officially.

[vfwdit]

This looks like VMware Horizon specific configuration setting. I'm not sure if our QA has ever tested compatibility with this product, at least I don't see it in the list of officially supported VMware products (unlike VSAN or Cloud Director for example). But I will double check.

{UPDATE] I confirmed that our QA does not have any labs with VMware Horizon, so it's not something we validate or officially support protection of. Of course, since there are regular VMware VMs underneath of Horizon, backup "just works" (well - worked, in your case). The error you're seeing is likely simply passed by jobs from vSphere and is likely completely unrelated to Veeam, so I would suggest opening a support case with VMware on Horizon - perhaps they already saw something like this before and know how to fix/reset it.

Also, according to the Release Notes VMware did indeed touch some code around CBRC in U3:

So, interesting thing about that... Our hourly backups don't back up anything Horizon related (well, except the View Connection Servers). They're backing up the core vSAN datastore which is running 8.0U3 with the vSAN disk version 20. Still getting the CBRC stuff there, so doubtful the VMware Horizon team can help out.

We only do Horizon backups at 4am and those are only our instant clone templates and a couple of static VMs.

[Gostev]

Right, but CBRC looks to be a host-wide feature which configuration parameters thus affect all VMs on the host, not just those few static ones. From the error it looks like the ESXi host is trying to enable caching for VMs that should not be using one, or something along these lines.

Plus QA does not see any issues with VSAN in their U3 labs (which do not have Horizon). So it does appear to be specific to environments with Horizon present, and the likely cause is a regression from the above-mentioned CBRC bug fix.

[vfwdit]

Yeah, I forgot to mention the CBRC error happens on the VEEAM proxies. Sorry, that was an important thing.

See picture here: https://imgur.com/a/EjfhQnM

In this picture, it happened when proxy 201 tried to attach some disks. Also, it's not just proxy 201, it randomly happens to all 4 proxies, that just what happened in this particular screenshot.

I just find it odd if it is Horizon based with vSAN, why is it happening on the core VMs which aren't linked into Horizon?

As I stated everything is typically backed up through the VEEAM proxies because everything is on vSAN. So the error appears to occur, but not always, when a disk is hotadded to the VEEAM proxy.

Anyhow! Thanks for your input. I didn't expect any sort of fix or workaround or anything! Just reported what I was seeing after upgrading. Sorry for derailing the thread.

We'll probably live with it, FOR NOW, because I'm sure if I open a ticket with Broadcom about it, it will take forever before anyone will get back to me and since this particular error only happens when VEEAM does a backup and when VEEAM is trying to attach a disk to the backup proxy, they'll probably point the finger at VEEAM instead of themselves. So, just not going to deal with it for now. Maybe when I have more time.

Need to finish up deployment of some things which take higher priority than this and giving myself a headache with Broadcom.

[Sturniolo]

Hey Everyone,

I wanted to share that Veeam now claims compatibility support of VMware vSphere 8.0.3 in the latest version of Veeam Backup & Replication V12.1.2.172.

Please be aware of a couple of important limitations:

Backup jobs including vCLS VMs: Due to changes in vSphere 8.0.3, backup jobs that include vCLS VMs (typically when using vSphere Cluster or ESXi Server scope) may fail. As a temporary solution, you can manually exclude these VMs from your backup jobs or utilize Global VM exclusions. Veeam is aware of this issue and plans to address it in the upcoming V12.2 update.

NSX-T 4.2.0 support: At this time, we have not been able to test and confirm support for NSX-T 4.2.0 due to Broadcom's current unavailability of the software for backup partners. We are actively working with Broadcom to resolve this and will provide updates as soon as possible.

For more details, please refer to the following KB Articles:

Veeam Backup & Replication support for VMware vSphere
https://www.veeam.com/kb2443

Veeam Backup & Replication 12.1.2 Compatibility with vSphere 8.0 U3
https://www.veeam.com/kb4641

[tyler.jurgens]

NSX-T 4.2.0 support: At this time, we have not been able to test and confirm support for NSX-T 4.2.0 due to Broadcom's current unavailability of the software for backup partners. We are actively working with Broadcom to resolve this and will provide updates as soon as possible.

My disappointment is immeasurable.

[vfwdit]

Right, but CBRC looks to be a host-wide feature which configuration parameters thus affect all VMs on the host, not just those few static ones. From the error it looks like the ESXi host is trying to enable caching for VMs that should not be using one, or something along these lines.

Plus QA does not see any issues with VSAN in their U3 labs (which do not have Horizon). So it does appear to be specific to environments with Horizon present, and the likely cause is a regression from the above-mentioned CBRC bug fix.

Hi, I just wanted to thank you for pointing me in the right direction.

The problem we were experiencing with 8.0U3 and VEEAM 12.1.2 was indeed due to the View Storage Accelerator option in VMware Horizon.

We have desktop pools that had the View Storage Accelerator turned on that were causing the issue.

The desktop pools in question weren't instant clone pools, they were full fat VM pools that had some individual users assigned, but we did back them up on the hourly backup.

You don't really need the VSA turned on for those pools and we certainly don't need it on with our storage speed anyway for those VMs, so we turned off the VSA on the pool and powered down the desktops. View would get rid of the digest disks on the VMs then automatically power them back on (due to power policy).

After this was done, the CBRC messages went away during VEEAM backup.

[Gostev]

Great, we really appreciate you coming back with the solution!

[JarnoA]

NSX-T 4.2.0 support: At this time, we have not been able to test and confirm support for NSX-T 4.2.0 due to Broadcom's current unavailability of the software for backup partners. We are actively working with Broadcom to resolve this and will provide updates as soon as possible.

I was under the impression that only official way to backup NSX-T was through its own filebackup method. If snapshot backups are supported, that'd be lovely for it takes an eternity via native restore.

[Vmware2024]

Just following up on this had this been release yet if not when?

When is Veeam B&R 12.2 expected to be released?

[tyler.jurgens]

I suspect they want to test NSX to ensure that port groups are able to be connected properly when using replication to/from different NSX enabled clusters, more than backing up NSX. Very disappointing that Broadcom is not distributing versions to their partners.

[mkretzer]

Will 12.2 be released when the NSX issue is solved? If so my worry is that it will take longer than Q3....

[Gostev]

No, we're not waiting for NSX simply because it is unclear how long the wait is going to be.

[mkretzer]

Very nice, thank you!

[tyler.jurgens]

Has there been any update on NSX 4.2 compatibility? We might be running into an issue where we are required to update to NSX 4.2 and Veeam is one vendor that I cannot find clarity on whether NSX 4.2 is supported.

[FrenchBlue]

Hello,

What's the issue exactly with NSX? It is not supported by VMware to backup NSX-T managers using any form of snapshots, so I don't get the point? Only the inbuilt configuration backup is officially supported.

Thanks.

[Gostev]

We need to test backup/restore of VMs which are using NSX-T networking. There were issues around this configuration in the past that didn't exist with regular vSphere networking.

[FrenchBlue]

Hmmm, interesting, ans this is specific to Vsphere 8? We're doing it frequently in 7.0u3, no issues so far.

[Gostev]

It's unclear what are you asking. What is specific to vSphere 8?

[FrenchBlue]

The issue you mention with these VMs using NSX networking.

[Gostev]

Well, it was quite a while ago already but from what I remember, that issue was affecting all vSphere versions.