vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

[skate88]

hallo

https://www.vmware.com/security/advisor ... -0023.html

where are we with the support for vcenter 8 u2 ?
this problem appears to be quite severe if you read the following .....
"While VMware does not mention end-of-life products in VMware Security Advisories, due to the critical severity of this vulnerability and lack of workaround VMware has made a patch generally available for vCenter Server 6.7U3, 6.5U3, and VCF 3.x. For the same reasons, VMware has made additional patches available for vCenter Server 8.0U1."

I would like to know the time frame for u2 support, or are you going to recommend we install the 8 u1 patch for now ? ( 8.0u1D)

Genuine question now, what is your guidance in the situation where vmware were to say , "critical vuln , you have to update to version x now, and version x is still not supported by veeam.

thanks in advance
neil

[mamosorre84]

Same thoughts this morning..

[Mildur]

Hi Skate88

Our upcoming version 12.1 will support vSphere 8.0 U2. We plan to release v12.1 before the end of this year.
If you want to keep your environment fully supported with Veeam, I recommend to install patches for vSphere 8.0 U1 for now.

what is your guidance in the situation where vmware were to say , "critical vuln , you have to update to version x now, and version x is still not supported by veeam.

I don't think, VmWare will ever force you to update to U2 or a new major version because of a high security vulnerability. VmWare provides minor patches for older versions as well. With support for vSphere 8.0 U1, we automatically support all minor patches such as a,b,c,d,e,f,...
If such situation would occur, we will consider releasing basic support faster than our general target window of 90 days.

Best,
Fabian

[skate88]

hi
thanks for the prompt response.
will install the update 1 d patch for now.
just wasnt sure if you were making 12.1 available with the launch event .....

thanks again
neil

[Andreas Neufert]

Patch releases with a,b,c,d,... from VMware versions will be automatically supported. Being merely a collection of existing hotfixes, they never broke our integrations.

[Gostev]

In fact, our QA has just finished the regression testing of version 12.0.0.1420 P20230718 (latest) against vSphere 8.0 U2 and we're ready to declare compatibility-level support. "Compatibility-level" means no support for new features like virtual hardware version 21 or new vSAN ESA features. In other words, you can upgrade to vSphere 8.0 U2 but you cannot leverage any U2-specific functionality at this time. It will be supported in 12.1 only.

The related support KB article should be updated shortly, I'm writing this a split second after receiving the announcement from QA and remembering seeing this thread earlier today

[MGT1981]

ARGH! I just had maintenance on my servers last night and I specifically didn't install update 2. I swear I have as much luck timing these things as I do the stock market

[Gostev]

Always a good idea not to be the first to jump significant new releases of any software at all.

[pmichelli]

Just install 7.0.3o (1700) or 8.0.1 (1400) . Why is everyone in such a panic to get to 8.2. Go take a look at the VMware communities and you'll change your mind. 8.0 U2 is horrifically bugged. They released 8.2a and it's no better. So many people having issues.

[Zew]

Always a good idea not to be the first to jump significant new releases of any software at all.

New Major releases, I can see, patches that fix CVE's of a score of 9.8. Patch ASAP.

[Gostev]

Sure, and you can patch this CVE without going to 8.0 U2

[Gostev]

Important information for those who have already adopted vSphere 8.0 U2. This issue does not seem to apply to ANY previous vSphere builds.

There's a strong possibility that the CBT corruption bug from 8 years ago was reintroduced in 8.0 U2 as we're able to reproduce it reliably in our labs at least in some configurations. We have opened a support case with VMware and will do a wider customer announcement if/when they confirm the issue and its scope from their side.

If you're on vSphere 8.0 U2 and want to act immediately, you can do the following:

1. Apparently we still have an atavism implemented to fight the original issue and QA confirmed that after all these years it still works in V12! Create the ResetCBTOnDiskResize (DWORD, 1) registry value to prevent this issue for any NEW disk size changes from that moment on. This value goes to the usual HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key on the backup server. You will want to remove it after VMware patches the issue, as this setting will increase your backup window each time a disk is resized.

2. You cannot "fix" your existing backups but you can ensure your future backups are good. For that you should reset CBT on all your vSphere VMs that have had their disk resized following the upgrade to vSphere 8.0 U2 (not before). There are a couple of approaches you can use:

a) You can use this VMware KB > https://kb.vmware.com/s/article/2139574 , or

b) You can instead perform an Active Full backup in Veeam, ensuring the Reset CBT on each Active Full backup automatically is selected (which is the default setting).

[pirx]

This only affects ESXi with 8.0.2 version? Is there any way to limit the regkey to those versions? We still have >90% ESXi 7.0.3 so the regkey would affect all others too.

[Gostev]

There's no way to limit this registry value to specific hosts, it's all or nothing.

[skate88]

hello


does this only relate to the esxi itself ?
the document from 8 years ago says esxi
if i have only upgraded vcenter so far is this still an issue ?
thanks

[EWMarco]

Can anybody tell me how this bug manifests? Are the backups based on corrupt CBT marked as failed or do we have hidden mines sprinkled across our backup history now? If so, how do we find out which VMs need an active full?

[FrancWest]

And if we use the CBT reset method, does this create an active full also, or does it only read the entire VM and create an incremental file with the correct blocks?

[FrancWest]

hello


does this only relate to the esxi itself ?
the document from 8 years ago says esxi
if i have only upgraded vcenter so far is this still an issue ?
thanks

Only ESXi. vCenter is not related to this.

[Gostev]

Correct.

And if we use the CBT reset method, does this create an active full also, or does it only read the entire VM and create an incremental file with the correct blocks?

It's the latter. Following CBT reset Veeam will perform an incremental backup created by the "full scan" method, so it will take much longer but produce a normal increment. And because this process will physically compare current production VMDKs state to what was stored in the last backup, it will bring over all non-matching blocks into the incremental backup, thus healing all accumulated inconsistencies.

[Gostev]

Can anybody tell me how this bug manifests? Are the backups based on corrupt CBT marked as failed or do we have hidden mines sprinkled across our backup history now? If so, how do we find out which VMs need an active full?

It's a "hidden mines" type of situation. VMs will even restore fine but some guest files might be incomplete or corrupted.

Try to find out through the change request history which VMs had their disks resized, as only these VMs would be affected. It should not be that common operation and vSphere 8.0 U2 is fairly recent too, so we're talking just the last few weeks.

[ITP-Stan]

@Gostev
So using VMware ESXi, 8.0.1, 21813344 is not an issue?

[Gostev]

Correct. From what we know, this looks to be only

vSphere 8.0 U2. This issue does not apply to ANY previous vSphere builds.

[cerberus]

Hi Gostev,

We just updated to v8u2 few weeks ago, have not had any vmdk disk resize actions since the upgrade as it's still very recent. Is my assumption correct, we are not affected by this *until* we re-size a VM under v8u2?

If we end up having to resize a VM, we will need to add the ResetCBTOnDiskResize registry key on the backup server and its an all-or-nothing setting affecting the backup of all VMsThe way around this until it is fixed is to either not re-size OR use the regkey?

We run in reverse incremental backup mode using change block tracking, are users in this mode affected?

[Gostev]

1. Your assumption is correct.

2. Correct. However do note CBT will be reset only once and only on the particular VM that had its disk resized. I'm writings this just in case if you're thinking we will go and reset CBT on all of your OTHER VMs too for no good reason

3. Backup mode or backup software you use don't matter. The issue is that CBT API returns wrong information for the disks that have been resized under vSphere 8.0 U2.

[FrancWest]

Is setting the ResetCBTOnDiskResize registry key effective immediately or must we restart the Veeam backup service?

[perjonsson1960]

Gostev,

1. Apparently we still have an atavism implemented to fight the original issue and QA confirmed that after all these years it still works in V12! Create the ResetCBTOnDiskResize (DWORD, 1) registry value to prevent this issue for any NEW disk size changes from that moment on. This value goes to the usual HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key on the backup server. You will want to remove it after VMware patches the issue, as this setting will increase your backup window each time a disk is resized.

Do I need to restart the Veeam Backup Service after adding the registry value?

PJ

[cerberus]

1. Your assumption is correct.

2. Correct. However do note CBT will be reset only once and only on the particular VM that had its disk resized. I'm writings this just in case if you're thinking we will go and reset CBT on all of your OTHER VMs too for no good reason

3. Backup mode or backup software you use don't matter. The issue is that CBT API returns wrong information for the disks that have been resized under vSphere 8.0 U2.

Thanks, Gostev for the details on #2, we've put a freeze on disk resize until this bug is fixed. If we end up having to re-size the disk, we will use the registry key workaround. Good to know that only the resized VMs will be affected.

Just to confirm, adding this key won't have a negative effect on Veeam replication? We also use Veeam to replicate from Site A to Site B, aside from a few extra changed blocks being picked up due to the CBT reset?

[Gostev]

Guys, I don't know when I can get QA to verify the registry value usage with and without service restart, so if you want to implement it right now then go ahead and restart just to be on a safe side. UPDATE: Opinion from the QA engineer who tested the registry value with V12 is that service restart is not required, because the value is checked each time when a disk resize is detected.

@cerberus excellent idea to just restrict disk resize. You need to make sure none happened following the vSphere 8.0 U2 upgrade though.
Please note that replication jobs do not support source disk resize in principle.

[cerberus]

Ah yes, I remember now, replication does some extra steps when there is a disk resize detected at source. Thanks, Gostev.

[Gostev]

Quick update: VMware was able to reproduce the issue. They are now working on identifying the root cause.