vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

[pmichelli]

Looks like the issue has been resolved in the latest ESXi 8.0.2b

PR 3316967: Changed Block Tracking (CBT) might not work as expected on a hot extended virtual disk

In vSphere 8.0 Update 2, to optimize the open and close process of virtual disks during hot extension, the disk remains open during hot extend operations. Due to this change, incremental backup of virtual disks with CBT enabled might be incomplete, because the CBT in-memory bitmap does not resize, and CBT cannot record the changes to the extended disk block. As a result, when you try to restore a VM from an incremental backup of virtual disks with CBT, the VM might fail to start.

This issue is resolved in this release.

https://docs.vmware.com/en/VMware-vSphe ... index.html

[ashleyw]

has anyone been brave enough to update vcentre and esxi to 8update2b and verified Veeam functionality? (If not I might slam it on one of our labs).

[FrancWest]

I updated vCenter in my lab. During the update to vcenter I received the following error 'Installation Failed: pre-install failed for vpxd:Expand' and the update failed. I had the option to resume the update. After that, the update was successful and vcenter is functional. However, I have my doubts that vcenter was fully updated due to the error. I opened a case for this with VMware support. Didn’t update ESXi yet.

[pmichelli]

has anyone been brave enough to update vcentre and esxi to 8update2b and verified Veeam functionality? (If not I might slam it on one of our labs).

Going to upgrade my DR site today. I'll post once I get some tests done

[pmichelli]

Successful upgrade of both vCenter and ESXi hosts to 8.0.2b. Veeam 12.1 latest build is working fine. I have run a number of test backups and restores, no issues. We'll give it a week or two before I pass final judgment

[sandsturm]

Successful upgrade of both vCenter and ESXi hosts to 8.0.2b. Veeam 12.1 latest build is working fine. I have run a number of test backups and restores, no issues. We'll give it a week or two before I pass final judgment

... that sounds great

[FrancWest]

I updated vCenter in my lab. During the update to vcenter I received the following error 'Installation Failed: pre-install failed for vpxd:Expand' and the update failed. I had the option to resume the update. After that, the update was successful and vcenter is functional. However, I have my doubts that vcenter was fully updated due to the error. I opened a case for this with VMware support. Didn’t update ESXi yet.

vmWare confirmed the update of vCenter was completed successfully even after getting the mentioned error message. Also updated the first ESXi hosts without issue.

[Novell2]

Hi all, Hi Pirx,
thanks, i am waiting until esx 8.02b from HPE will arrive.

Best regards
Novell2

[FrancWest]

No need to wait for the HPE iso, you can simply install the VMware patches. The HPE iso only contains drivers and HPE tools, the vSphere install is the same.

[skate88]

https://www.veeam.com/kb2443

looks like this was never updated in light of the cbt bug? as it was last modified in december 2023? it still the initial guidance for u2 that no new features were supported.
What is the official word now from veeam ? I see lots of people asking if anyone has updated yet. Ironically i see we are back to the original question i started this thread about
There are critical vulnerabilities right now. What is the official veeam guidance right now? Should we just use the security patches for u1. Or is u2b now fully supported? I guess you still need time to validate if the cbt problem is actually fixed or not ?

[william88]

Can we safely upgrade our vCenter Server to version 8.0 Update 2b(8.0.2.00200) and our ESXi to version 8.0 Update 2b(23305546) with Veeam 12.1.0.2131?

[d.artzen]

You can either update to 8U1d or to 8U2b. The CBT-Bug is officially listed as fixed in the release notes for 8U2b. Both versions have the fixes for the new vulnerabilities. The only thing you should not do if you are on 8U1c is update to 8U1d and then upgrade to 8U2 (without b) as that is not supported. But this is prominently displayed at the beginning of the release notes for 8U1d.

[william88]

Thanks Daniel for your help.
Has Veeam tested this 8U2b version and can confirm that backups are now fully functional without adding the registry key?

[Gostev]

Yes, our autotest shows that the bug was fixed. QA wanted to look at this closer next week before announcing (the required team was busy with the last 11a patch).

For now please assume that performing the second step is still required. It is yet unknown if this patch attempts to do anything about CBT map of VMs which have had their disks resized while the bug existed, like resetting it automatically.

[skate88]

hallo Gostev
any eta ? have the team been able to work on it this week ?
many thanks
neil

[Sturniolo]

Our QA teams are currently taking a closer look. We will share an update as soon as we have additional information available.

[Sturniolo]

During our investigation and various testing scenarios, we did not encounter any data corruption issues in this particular scenario with version 8.0.2b.

In cases where a VM's disk was previously resized, the CBT map may remain invalid. However, with the ESXi update to version 8.0.2b and subsequent VM migration to another host as part of the update process, the CBT will be automatically reset. Therefore, there is no need for manual CBT reset for VMs with invalid CBT maps if updating the infrastructure to 8.0.2b.

[tyler.jurgens]

Thanks Andy, that's great information. I guess there is still a scenario of a single host - where VMs are shut down to upgrade ESXi - the CBT map would not be reset in that scenario. Wouldn't expect that often, but would be a good case to point out.

[ITP-Stan]

Important addendum for setups where VM's are not migrated as part of the update process.

[colsztyn]

by Sturniolo » Tue Mar 19, 2024 6:56 am

However, with the ESXi update to version 8.0.2b and subsequent VM migration to another host as part of the update process, the CBT will be automatically reset. Therefore, there is no need for manual CBT reset for VMs with invalid CBT maps if updating the infrastructure to 8.0.2b.

Question, when you say "VM migration" are we talking a regular VMotion or an SVMotion to migrate to a new storage volume?

Thanks

[Sturniolo]

Some additional information....

Our QA team has tested this behavior thoroughly and can confirm that powering off, suspending or performing a vMotion of a VM helps to prevent corruption during backup or replication jobs. But at least one of the these actions will happen for each VM on the host when you update it to Update 2b regardless of your deployment scenario (standalone host or cluster).

The hint is in the Update 2b release notes:

In vSphere 8.0 Update 2, to optimize the open and close process of virtual disks during hot extension, the disk remains open during hot extend operations. Due to this change, incremental backup of virtual disks with CBT enabled might be incomplete, because the CBT in-memory bitmap does not resize, and CBT cannot record the changes to the extended disk block. As a result, when you try to restore a VM from an incremental backup of virtual disks with CBT, the VM might fail to start.

So the root cause of corruption was not CBT itself but only the invalid in-memory bitmap that gets purged by any of the above-mentioned actions.

[colsztyn]

Super helpful. Thank you!