Host-based backup of VMware vSphere VMs.
skate88
Influencer
Posts: 19
Liked: never
Joined: Jan 25, 2011 8:05 am
Contact:

vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by skate88 »

hallo

https://www.vmware.com/security/advisor ... -0023.html

where are we with the support for vcenter 8 u2 ?
this problem appears to be quite severe if you read the following .....
"While VMware does not mention end-of-life products in VMware Security Advisories, due to the critical severity of this vulnerability and lack of workaround VMware has made a patch generally available for vCenter Server 6.7U3, 6.5U3, and VCF 3.x. For the same reasons, VMware has made additional patches available for vCenter Server 8.0U1."

I would like to know the time frame for u2 support, or are you going to recommend we install the 8 u1 patch for now ? ( 8.0u1D)

Genuine question now, what is your guidance in the situation where vmware were to say , "critical vuln , you have to update to version x now, and version x is still not supported by veeam.

thanks in advance
neil
mamosorre84
Veeam Legend
Posts: 340
Liked: 34 times
Joined: Oct 24, 2016 3:56 pm
Full Name: Marco Sorrentino
Location: Ancona - Italy
Contact:

Re: VMSA-2023-0023 CVSSv3 base score of 9.8 and update 2 support

Post by mamosorre84 »

Same thoughts this morning..
Mildur
Product Manager
Posts: 9165
Liked: 2409 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: VMSA-2023-0023 CVSSv3 base score of 9.8 and update 2 support

Post by Mildur » 2 people like this post

Hi Skate88

Our upcoming version 12.1 will support vSphere 8.0 U2. We plan to release v12.1 before the end of this year.
If you want to keep your environment fully supported with Veeam, I recommend to install patches for vSphere 8.0 U1 for now.
what is your guidance in the situation where vmware were to say , "critical vuln , you have to update to version x now, and version x is still not supported by veeam.
I don't think, VmWare will ever force you to update to U2 or a new major version because of a high security vulnerability. VmWare provides minor patches for older versions as well. With support for vSphere 8.0 U1, we automatically support all minor patches such as a,b,c,d,e,f,...
If such situation would occur, we will consider releasing basic support faster than our general target window of 90 days.

Best,
Fabian
Product Management Analyst @ Veeam Software
skate88
Influencer
Posts: 19
Liked: never
Joined: Jan 25, 2011 8:05 am
Contact:

Re: VMSA-2023-0023 CVSSv3 base score of 9.8 and update 2 support

Post by skate88 »

hi
thanks for the prompt response.
will install the update 1 d patch for now.
just wasnt sure if you were making 12.1 available with the launch event .....

thanks again
neil
Andreas Neufert
VP, Product Management
Posts: 6844
Liked: 1442 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: VMSA-2023-0023 CVSSv3 base score of 9.8 and update 2 support

Post by Andreas Neufert » 1 person likes this post

Patch releases with a,b,c,d,... from VMware versions will be automatically supported. Being merely a collection of existing hotfixes, they never broke our integrations.
Gostev
Chief Product Officer
Posts: 31747
Liked: 6917 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VMSA-2023-0023 CVSSv3 base score of 9.8 and update 2 support

Post by Gostev » 4 people like this post

In fact, our QA has just finished the regression testing of version 12.0.0.1420 P20230718 (latest) against vSphere 8.0 U2 and we're ready to declare compatibility-level support. "Compatibility-level" means no support for new features like virtual hardware version 21 or new vSAN ESA features. In other words, you can upgrade to vSphere 8.0 U2 but you cannot leverage any U2-specific functionality at this time. It will be supported in 12.1 only.

The related support KB article should be updated shortly, I'm writing this a split second after receiving the announcement from QA and remembering seeing this thread earlier today :)
MGT1981
Enthusiast
Posts: 39
Liked: 6 times
Joined: Nov 21, 2014 12:30 am
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by MGT1981 » 2 people like this post

ARGH! I just had maintenance on my servers last night and I specifically didn't install update 2. I swear I have as much luck timing these things as I do the stock market
Gostev
Chief Product Officer
Posts: 31747
Liked: 6917 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by Gostev »

Always a good idea not to be the first to jump significant new releases of any software at all.
pmichelli
Enthusiast
Posts: 87
Liked: 19 times
Joined: Mar 16, 2023 5:47 pm
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by pmichelli » 1 person likes this post

Just install 7.0.3o (1700) or 8.0.1 (1400) . Why is everyone in such a panic to get to 8.2. Go take a look at the VMware communities and you'll change your mind. 8.0 U2 is horrifically bugged. They released 8.2a and it's no better. So many people having issues.
Zew
Veteran
Posts: 365
Liked: 80 times
Joined: Mar 17, 2015 9:50 pm
Full Name: Aemilianus Kehler
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by Zew »

Gostev wrote: Oct 26, 2023 3:29 pm Always a good idea not to be the first to jump significant new releases of any software at all.
New Major releases, I can see, patches that fix CVE's of a score of 9.8. Patch ASAP.
Gostev
Chief Product Officer
Posts: 31747
Liked: 6917 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by Gostev » 1 person likes this post

Sure, and you can patch this CVE without going to 8.0 U2
Gostev
Chief Product Officer
Posts: 31747
Liked: 6917 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by Gostev » 4 people like this post

Important information for those who have already adopted vSphere 8.0 U2. This issue does not seem to apply to ANY previous vSphere builds.

There's a strong possibility that the CBT corruption bug from 8 years ago was reintroduced in 8.0 U2 as we're able to reproduce it reliably in our labs at least in some configurations. We have opened a support case with VMware and will do a wider customer announcement if/when they confirm the issue and its scope from their side.

If you're on vSphere 8.0 U2 and want to act immediately, you can do the following:

1. Apparently we still have an atavism implemented to fight the original issue and QA confirmed that after all these years it still works in V12! Create the ResetCBTOnDiskResize (DWORD, 1) registry value to prevent this issue for any NEW disk size changes from that moment on. This value goes to the usual HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key on the backup server. You will want to remove it after VMware patches the issue, as this setting will increase your backup window each time a disk is resized.

2. You cannot "fix" your existing backups but you can ensure your future backups are good. For that you should reset CBT on all your vSphere VMs that have had their disk resized following the upgrade to vSphere 8.0 U2 (not before). There are a couple of approaches you can use:

a) You can use this VMware KB > https://kb.vmware.com/s/article/2139574 , or

b) You can instead perform an Active Full backup in Veeam, ensuring the Reset CBT on each Active Full backup automatically is selected (which is the default setting).
pirx
Veteran
Posts: 587
Liked: 81 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by pirx »

This only affects ESXi with 8.0.2 version? Is there any way to limit the regkey to those versions? We still have >90% ESXi 7.0.3 so the regkey would affect all others too.
Gostev
Chief Product Officer
Posts: 31747
Liked: 6917 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by Gostev »

There's no way to limit this registry value to specific hosts, it's all or nothing.
skate88
Influencer
Posts: 19
Liked: never
Joined: Jan 25, 2011 8:05 am
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by skate88 »

hello


does this only relate to the esxi itself ?
the document from 8 years ago says esxi
if i have only upgraded vcenter so far is this still an issue ?
thanks
EWMarco
Service Provider
Posts: 44
Liked: 7 times
Joined: Feb 20, 2023 9:28 am
Full Name: Marco Glavas
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by EWMarco »

Can anybody tell me how this bug manifests? Are the backups based on corrupt CBT marked as failed or do we have hidden mines sprinkled across our backup history now? If so, how do we find out which VMs need an active full?
FrancWest
Veteran
Posts: 499
Liked: 98 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by FrancWest »

And if we use the CBT reset method, does this create an active full also, or does it only read the entire VM and create an incremental file with the correct blocks?
FrancWest
Veteran
Posts: 499
Liked: 98 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by FrancWest » 1 person likes this post

skate88 wrote: Dec 11, 2023 9:27 am hello


does this only relate to the esxi itself ?
the document from 8 years ago says esxi
if i have only upgraded vcenter so far is this still an issue ?
thanks
Only ESXi. vCenter is not related to this.
Gostev
Chief Product Officer
Posts: 31747
Liked: 6917 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by Gostev »

Correct.
FrancWest wrote: Dec 11, 2023 9:34 am And if we use the CBT reset method, does this create an active full also, or does it only read the entire VM and create an incremental file with the correct blocks?
It's the latter. Following CBT reset Veeam will perform an incremental backup created by the "full scan" method, so it will take much longer but produce a normal increment. And because this process will physically compare current production VMDKs state to what was stored in the last backup, it will bring over all non-matching blocks into the incremental backup, thus healing all accumulated inconsistencies.
Gostev
Chief Product Officer
Posts: 31747
Liked: 6917 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by Gostev »

EWMarco wrote: Dec 11, 2023 9:30 am Can anybody tell me how this bug manifests? Are the backups based on corrupt CBT marked as failed or do we have hidden mines sprinkled across our backup history now? If so, how do we find out which VMs need an active full?
It's a "hidden mines" type of situation. VMs will even restore fine but some guest files might be incomplete or corrupted.

Try to find out through the change request history which VMs had their disks resized, as only these VMs would be affected. It should not be that common operation and vSphere 8.0 U2 is fairly recent too, so we're talking just the last few weeks.
ITP-Stan
Expert
Posts: 203
Liked: 57 times
Joined: Feb 18, 2013 10:45 am
Full Name: Stan G
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by ITP-Stan »

@Gostev
So using VMware ESXi, 8.0.1, 21813344 is not an issue?
Gostev
Chief Product Officer
Posts: 31747
Liked: 6917 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by Gostev »

Correct. From what we know, this looks to be only
Gostev wrote: Dec 09, 2023 11:09 amvSphere 8.0 U2. This issue does not apply to ANY previous vSphere builds.
cerberus
Expert
Posts: 156
Liked: 15 times
Joined: Aug 28, 2015 2:45 pm
Full Name: Mirza
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by cerberus »

Hi Gostev,

We just updated to v8u2 few weeks ago, have not had any vmdk disk resize actions since the upgrade as it's still very recent. Is my assumption correct, we are not affected by this *until* we re-size a VM under v8u2?

If we end up having to resize a VM, we will need to add the ResetCBTOnDiskResize registry key on the backup server and its an all-or-nothing setting affecting the backup of all VMsThe way around this until it is fixed is to either not re-size OR use the regkey?

We run in reverse incremental backup mode using change block tracking, are users in this mode affected?
Gostev
Chief Product Officer
Posts: 31747
Liked: 6917 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by Gostev »

1. Your assumption is correct.

2. Correct. However do note CBT will be reset only once and only on the particular VM that had its disk resized. I'm writings this just in case if you're thinking we will go and reset CBT on all of your OTHER VMs too for no good reason :)

3. Backup mode or backup software you use don't matter. The issue is that CBT API returns wrong information for the disks that have been resized under vSphere 8.0 U2.
FrancWest
Veteran
Posts: 499
Liked: 98 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by FrancWest »

Is setting the ResetCBTOnDiskResize registry key effective immediately or must we restart the Veeam backup service?
perjonsson1960
Veteran
Posts: 497
Liked: 53 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by perjonsson1960 »

Gostev,
Gostev wrote: Dec 09, 2023 11:09 am 1. Apparently we still have an atavism implemented to fight the original issue and QA confirmed that after all these years it still works in V12! Create the ResetCBTOnDiskResize (DWORD, 1) registry value to prevent this issue for any NEW disk size changes from that moment on. This value goes to the usual HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key on the backup server. You will want to remove it after VMware patches the issue, as this setting will increase your backup window each time a disk is resized.
Do I need to restart the Veeam Backup Service after adding the registry value?

PJ
cerberus
Expert
Posts: 156
Liked: 15 times
Joined: Aug 28, 2015 2:45 pm
Full Name: Mirza
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by cerberus »

Gostev wrote: Dec 11, 2023 3:39 pm 1. Your assumption is correct.

2. Correct. However do note CBT will be reset only once and only on the particular VM that had its disk resized. I'm writings this just in case if you're thinking we will go and reset CBT on all of your OTHER VMs too for no good reason :)

3. Backup mode or backup software you use don't matter. The issue is that CBT API returns wrong information for the disks that have been resized under vSphere 8.0 U2.
Thanks, Gostev for the details on #2, we've put a freeze on disk resize until this bug is fixed. If we end up having to re-size the disk, we will use the registry key workaround. Good to know that only the resized VMs will be affected.

Just to confirm, adding this key won't have a negative effect on Veeam replication? We also use Veeam to replicate from Site A to Site B, aside from a few extra changed blocks being picked up due to the CBT reset?
Gostev
Chief Product Officer
Posts: 31747
Liked: 6917 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by Gostev »

Guys, I don't know when I can get QA to verify the registry value usage with and without service restart, so if you want to implement it right now then go ahead and restart just to be on a safe side. UPDATE: Opinion from the QA engineer who tested the registry value with V12 is that service restart is not required, because the value is checked each time when a disk resize is detected.

@cerberus excellent idea to just restrict disk resize. You need to make sure none happened following the vSphere 8.0 U2 upgrade though.
Please note that replication jobs do not support source disk resize in principle.
cerberus
Expert
Posts: 156
Liked: 15 times
Joined: Aug 28, 2015 2:45 pm
Full Name: Mirza
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by cerberus »

Ah yes, I remember now, replication does some extra steps when there is a disk resize detected at source. Thanks, Gostev.
Gostev
Chief Product Officer
Posts: 31747
Liked: 6917 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

Post by Gostev » 3 people like this post

Quick update: VMware was able to reproduce the issue. They are now working on identifying the root cause.
Post Reply

Who is online

Users browsing this forum: No registered users and 62 guests