vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

[pirx]

I think you need to ask VMware not Veeam. I just wanted to note that there seems to no update yet.

[Andreas Neufert]

Did not saw the release in public, but might be that the patch will be distributed first based on support cases to get customer feedback before a public release. So, I think opening a VMware support case might be the best way to address this.

New patch release ticker from VMware: https://kb.vmware.com/s/article/2143832

[jsprinkleisg]

I noticed VMware updated their KB article today. It previously included the patch release info (fixed in ESXi 8.0 U2 P03), but this appears to have been removed. I'm guessing this means the fix did not make it into this patch.

[pirx]

Yep, the usual, breaking things is much easier than fixing it.

[masonit]

If vcenter is running 8.0 U2 but the esxi hosts are running 8.0 U1. Are you then safe from this bug?

[Gostev]

Yes, you're safe. The bug is in ESXi.

[MichelD87]

Hi,

Question,

To get good reliable backups, what do we need to do?
Is it enough to create new backups ( Manual Active Full Backup) ?
Or do we need reset the CBT in Vmware first ?

[Gostev]

Hi, please refer to the big post on the first page of this topic, it answers exactly that and explains all available options. Thanks

[skarsin]

Hi All,

@Gostev

Did you hear/aware of update on vSphere 8.0 U2 ESXi on the CBT? I finished installed VCSA 8.0 U2a and in the plan to upgrade/install my ESXi with v8. Luckily found this forum.
If no updates I may go to ESXi 8.0 U1 latest first

[Gostev]

Hi, we did get an update last week that the patch for ESXi 8.0 U2 has been delayed. They did provide us the new planned RTM and GA dates but I can't share these without getting an explicit approval like last time, which usually takes a while to obtain (all dates are under NDA by default).

[FrancWest]

VMware updated the KB article again (https://kb.vmware.com/s/article/95940). Fix is planned for ESXi 8.0p03 and 8.0u3.

[Aschtra]

Hi,
Thank you for the update.
A bit confusing to me. ESXi 8.0p03 what does that mean? is that 8.0 Update 2? or 8.0 U1D. If the latter, I thought this bug wasn't present in U1?

[d.artzen]

Neither, the patch isn't released yet by VMware. If I understand the linked KB correctly, they plan to release the fix in an update for 8U2 (which they designate currently as 8.0p03 and will most likely be renamed on release to fall into their naming scheme) and in a new version 8U3.

[skarsin]

With 8.0 U3, I hope Veeam will consider to support them with Veeam 12.1

[d.artzen]

I think this will largely depend on what changes come with the U3. The official policy is normally 6 month after release, but there were times when the compatibility was released faster, as seen with 8U2, even if there were limitations in place. But personally I would stay away from new Main/Minor versions until at least the first patch.
VMware has not a good track record of not breaking things with new versions and I don't think this will get better now with Broadcom as owner. We are still on 8U1 with our systems.

[skarsin]

Thanks Daniel. As I go further installing 8.0 U2, I find it more annoying that 7.0 U2. I might go back and test 8U1.
I have feedback on the GUI of 8 but this is not the place. I'll used VMWare Forum and see if this is intended or wrongly coded in v8 or just 8.0 U2.
(e.g No Yes button instead of typical Yes No button placement)

[william88]

Important information for those who have already adopted vSphere 8.0 U2. This issue does not seem to apply to ANY previous vSphere builds.

There's a strong possibility that the CBT corruption bug from 8 years ago was reintroduced in 8.0 U2 as we're able to reproduce it reliably in our labs at least in some configurations. We have opened a support case with VMware and will do a wider customer announcement if/when they confirm the issue and its scope from their side.

If you're on vSphere 8.0 U2 and want to act immediately, you can do the following:

1. Apparently we still have an atavism implemented to fight the original issue and QA confirmed that after all these years it still works in V12! Create the ResetCBTOnDiskResize (DWORD, 1) registry value to prevent this issue for any NEW disk size changes from that moment on. This value goes to the usual HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key on the backup server. You will want to remove it after VMware patches the issue, as this setting will increase your backup window each time a disk is resized.

2. You cannot "fix" your existing backups but you can ensure your future backups are good. For that you should reset CBT on all your vSphere VMs that have had their disk resized following the upgrade to vSphere 8.0 U2 (not before). There are a couple of approaches you can use:

a) You can use this VMware KB > https://kb.vmware.com/s/article/2139574 , or

b) You can instead perform an Active Full backup in Veeam, ensuring the Reset CBT on each Active Full backup automatically is selected (which is the default setting).

Hi @Gostev,
If we know which VMs have had their disks resized, can we just do step 2b to start again with healthy backups?
Thanks

[Gostev]

Yes. It's actually a OR b. Not both.

[william88]

Thank you @Gostev but my question was more: is step 1 mandatory if we track which VMs have had their disks resized ?
If we make an active full backup every time we resize a disk, is that enough to get around the problem without step 1 ?

[Gostev]

What's your reason not to implement Step 1, when it will automatically and more reliably deliver the same effect (without you having to track anything) and in a much better way too, by not requiring all the extra disk space for a new active full backup and just creating a normal incremental backup file instead?

[william88]

Thanks for the advice. It was more to avoid having a registry key that we would forget to remove during the next update, knowing that we have a small infrastructure and that we increase the size of our disks very little.

If in a backup job where there are 10 VMs, we increase the disk size of only one of these VMs, is it possible to trigger an active full only on this VM? Do we need to do a separate job with just this VM to reset the CBT? Would a Veeam ZIP suffice?

Thanks

[Gostev]

Thing is, forgetting to remove the key will not cause any problems, especially in a small environment. Unlike forgetting to do an active full manually, which is too easy to do.

Starting from V12 you can in fact trigger active full only on specific VM, using right-click in the job's session.

[william88]

Thank you Gostev, I will follow your advice and set up the registry key.

Last question: if my resized VM is part of several jobs, is a single active full trigger on one job enough to restore the situation in the other jobs?

[Gostev]

Yes, it's enough.

[pirx]

Checkout new vCenter updates: 8.0.2.00200 Fix Feb 29, 2024 Yes Critical

This is 8.0 Update 2b, so ESXi version will probably be available soon too.

[d.artzen]

Sadly Vmware doesn't seem to see the need for such inconsequential things as release notes for this patch. So currently no one (aside maybe from any insiders) knows what they addressed in that patch. Also in the past we got vCenter updates without a new patch for ESXi. Which is the reason for the situation that the version numbering of both is not identical anymore.

[pmichelli]

This bug isn't a vCenter problem, it's an ESXi one. Updating to vCenter 8.0.2 patch 3 won't fix this. Hopefully they release a new ESXi as well.
My vCenter shows the new build for vCenter is available but I can't see it in the patch section of my portal or in the general downloads.
I'm sure the release notes are coming

[pirx]

That vCenter update will not fix it, is clear. But ESXi is coming too.

https://x.com/lamw/status/1763226124728082625?s=20

[tyler.jurgens]

Sadly Vmware doesn't seem to see the need for such inconsequential things as release notes for this patch. So currently no one (aside maybe from any insiders) knows what they addressed in that patch. Also in the past we got vCenter updates without a new patch for ESXi. Which is the reason for the situation that the version numbering of both is not identical anymore.

Release notes *always* lag behind certain notifications of new builds being available. Check back in a day or two and they will be there, or YOLO it and hope for the best. Its been like this for years now.

[pirx]

https://docs.vmware.com/en/VMware-vSphe ... index.html

PR 3316967: Changed Block Tracking (CBT) might not work as expected on a hot extended virtual disk

But my highlight and reason to update is this.

Reduction in the memory usage of the ps command: In ESXi 8.0 Update 2b, an optimization of memory usage for the ps command reduces memory required to run the command by up to 96%.