vSphere 8.0 U2 support in light of VMSA-2023-0023 CVSSv3 base score of 9.8

[EWMarco]

It's a "hidden mines" type of situation. VMs will even restore fine but some guest files might be incomplete or corrupted.

Try to find out through the change request history which VMs had their disks resized, as only these VMs would be affected. It should not be that common operation and vSphere 8.0 U2 is fairly recent too, so we're talking just the last few weeks.

I see your humor is intact

Say, Veeam notices disk resizes, no? Shouldn't we be able to scan through all the logs and get a comprehensive list that way?

[Gostev]

Hmm, but why mess with and rely on unpredictable debug logs of some 3rd party app when you can just look through the vSphere events history instead?

[coolsport00]

Great thread and info @gostev and all others on v12.1 and VMW upgrade! I was thinking of upgrading my env to 8U2, but nope. Not yet
@cerberus - if you resize a VM disk, for Replication, all your restore points (snapshots) on the Replica get removed. FYI.

[kevlahau]

Can you confirm if this is also an issue in 8.0.2a?

[Gostev]

Hi Shane, for the avoidance of doubt the issue has nothing to deal with version 12.1... it does not matter which Veeam version you're using, or if it's even Veeam. Every VMware backup product will be impacted by this VMware CBT regression.

Hi Kevin, I don't know if we tested this particular build. But in any case, from this moment on

VMware was able to reproduce the issue. They are now working on identifying the root cause.

it is best we all wait for further details to come from VMware, as ultimately they have the source code and it is much easier for them to determine the scope of the issue once a root cause analysis is done.

Besides, Veeam really should not be making definitive statements about someone else's software and its diffident builds... it feels wrong. We just found a possible issue and gave everyone a heads up!

Needless to say, we will be sharing any significant updates from our VMware support case here, although they may hold most details like the list of affected builds until a public KB.

[EWMarco]

Hmm, but why mess with and rely on unpredictable debug logs of some 3rd party app when you can just look through the vSphere events history instead?

Because Events go back only 8 days here and as far as I can tell a resize is just logged as a plain VM reconfiguration. Your software explicitly checks for changes in disk geometry, no?

[Gostev]

Correct, but this may happen only when the above-mentioned registry hack is used for example. Nobody knows!

This is exactly why I characterized this as "messing with an unpredictable": unlike vSphere events, our debug logs were never meant to be used as audit logs to start with. So we would need to research all scenarios and settings configuration where certain record is created before you can use our debug logs as an "ersatz audit log" and rely on their content to confidently conclude whether or not the disk resize did take place for certain VMs.

[EWMarco]

Yeah, well better than nothing. Certainly less prone to oversights than wading through the ticketing system...

I am using this bit of code to go through all the task logs and spit out time of noticed size change, VM name and path to log file, if anyone is interested. Obviously as I don't hold Veeam accountable for this all being correct and complete, neither should you expect to hold me accountable for the code

Code: Select all

$path = read-host "Enter the path for the log files "  
$Searchstring = "capacity changed"
$Logs = Get-ChildItem -path $path -recurse -include "Task*.log"  -exclude "*Transform*"
foreach($Log in $Logs)  
{  
    $StringExist = Select-String -Path $log.fullname -pattern $Searchstring  
    if($StringExist)  
    {  
		$null = $StringExist -match '\d\d\.\d\d\.\d\d\d\d\ \d\d\:\d\d\:\d\d' 
	
		
        write-host "$($matches[0]),$(($log.name).Split(".")[1]),$($Log.fullname)" -fore green  
    }  

}

Full disclosure, my code is based on this: https://www.c-sharpcorner.com/UploadFil ... e-log-fil/

[coolsport00]

"Hi Shane, for the avoidance of doubt the issue has nothing to deal with version 12.1... it does not matter which Veeam version you're using, or if it's even Veeam. Every VMware backup product will be impacted by this VMware CBT regression."
Yep...good add and thanks Anton. I didn't intend to assume this is a Veeam issue.

[BertM]

TL:DR - If I put a VM in a new job and run an Active Full, will this also fix it's backup in it's original job?

So, we made the early hop to 8.0.2 on some of our clusters (thankfully not all), and I found one VM that was expanded. Looking at VMware's KB for resetting CBT, I don't feel comfortable. Although it looks simple, and seems low risk, if anything goes wrong, my rollback plan is to restore from backup... which may be corrupt. I don't feel super comfortable with that plan, but if I'm being overly cautious, let me know.

The catch is that the VM is a part of a Veeam job with 118 VMs, so I'd rather not run an Active Full. If I put that VM in a second job and run a full on just that, will it fix CBT for the original job also? I'm pretty sure it will, but I want to be 100% sure.

[Gostev]

Yes, you can do that. But keep in mind that V12 allows you to trigger Active Full on individual VMs in a job.

[BertM]

I totally forgot that was a feature. :facepalm:
Thanks!

[Gostev]

VMware has just released a public KB article for the identified issue.

[DonZoomik]

For finding resized VMs from vCenter perspective, I threw together something in PowerCLI in 10 minutes. It's really ugly but seems to get the job done. It finds disk extension events by some heuristics (didn't go too deep into data model) and resets CBT.

Code: Select all

#get events for all vms that includes config changes, since 8.0u2 release
$events=get-vm|? name -notmatch 'veeamproxy'|% {
    write-host $_.name
    $match=$_|get-vievent -Start ([DateTime]"2023-09-20") -Finish (get-date) -Types Info|? configchanges
    if ($match){write-host bingo;$match}
}

$problemvms = $events|%{
    #didnt dive too deep into data model, this heuristic looks good enough
    if ($_.configspec.devicechange.operation -eq 'edit' -and $_.fullformattedmessage -match 'capacityInKB'){$_}
}|select -expandproperty vm|%{get-vm -name $_.name}

$problemvms
pause

#copypaste from vmware's cbt script
foreach ($vm in $problemvms) {
    $snaps = get-snapshot -vm $vm
    if (!$snaps) {
    $spec = New-Object VMware.Vim.VirtualMachineConfigSpec
    $spec.ChangeTrackingEnabled = $false 
    $vm.ExtensionData.ReconfigVM($spec) 

    #Take/Remove Snapshot to reconfigure VM State
    $SnapName = New-Snapshot -vm $vm -Quiesce -Name "CBT-Rest-Snapshot"
    $SnapRemove = Remove-Snapshot -Snapshot $SnapName -Confirm:$false 

    #Enable CBT 
    #Write-Host "Enabling CBT for" $vm
    $spec = New-Object VMware.Vim.VirtualMachineConfigSpec
    $spec.ChangeTrackingEnabled = $true 
    $vm.ExtensionData.ReconfigVM($spec) 

    #Take/Remove Snapshot to reconfigure VM State
    $SnapName1 = New-Snapshot -vm $vm -Quiesce -Name "CBT-Verify-Snapshot"
    $SnapRemove1 = Remove-Snapshot -Snapshot $SnapName1 -Confirm:$false
    } else {write-host ($vm.name, ' snaps present!')}
}

[DonZoomik]

Also @Gostev, you might want the original https://www.veeam.com/kb1940 article revised, it states that ResetCBTOnDiskResize should be 0.

[Gostev]

Wow, best to take down completely.

[cerberus]

VMware has just released a public KB article for the identified issue.

Interesting, one workaround is to extend disk offline (VM turned off, extend disk). Has any tests been done at Veeam Labs to confirm this workaround works for backups?

[Gostev]

No, we didn't test that as far as I know.

[stevea123]

Firstly, thank you Gostev for making companies like mine aware of this issue. so pleased I signed up to the Veeam Digest.
As a small company we rely on VMware/Veeam - having no dedicated IT.

I remain confused over the vCenter 8.0.2 version Vs ESXi host 8.0.2 version.
My understanding is that the CVE-2023-34048 vulnerability is a vCenter not an ESXi issue?
I'm running:
ESXi Hosts: Version: 8.0.1 - Build: 22088125
vCenter : Version: 8.0.2 - Build: 22617221
I realise this is not a Veeam issue but would appreciate if someone could confirm - an 'as is' opinion - that such a scenario is unaffected by this bug.

[Origin 2000]

@stevea23,
yes the CVE is about vCenter. Please look at https://www.vmware.com/security/advisor ... -0023.html

Than this thread goes into a different direction and the CBT issue is ESXi related.

Regards,
Joerg

[stevea123]

@Joerg
Thank you

[mdwophil]

Hi Skate88

Our upcoming version 12.1 will support vSphere 8.0 U2. We plan to release v12.1 before the end of this year.
If you want to keep your environment fully supported with Veeam, I recommend to install patches for vSphere 8.0 U1 for now.


I don't think, VmWare will ever force you to update to U2 or a new major version because of a high security vulnerability. VmWare provides minor patches for older versions as well. With support for vSphere 8.0 U1, we automatically support all minor patches such as a,b,c,d,e,f,...
If such situation would occur, we will consider releasing basic support faster than our general target window of 90 days.

Best,
Fabian

Yes, 100%. People forget the fine art of staying with a supported version that has a patch for the issue. Don't just blindly jump to the latest.

Most enterprise vendors patch security vulnerabilities in multiple release trains at the same time. In VMware's case, that's vCenter 7.0 Update 3, vCenter 8.0 Update 1, and vCenter 8.0 Update 2. They even did a patch for 6.7 Update 3.

[widmerkarl]

Hello folks,

Hopefully, I'm not digging out dead bodies. But it seems that VMware released a KB article regarding the recently (newly) discovered CBT issues:

https://kb.vmware.com/s/article/95965

They will update the KB article as soon as more information is available.

[coolsport00]

Thanks for sharing Karl!

[Gostev]

By way of further update, VMware PM team has informed us that we can communicate to our customers the upcoming resolution for the issue:
"The issue will be fixed in ESXi 8.0 U2 P03 and 8.0 U3. The RTM date is 1/18/2024."

[coolsport00]

Nice..great news. Thanks for the update Anton!

[widmerkarl]

Thanks for sharing Karl!

You're welcome

[widmerkarl]

By way of further update, VMware PM team has informed us that we can communicate to our customers the upcoming resolution for the issue:
"The issue will be fixed in ESXi 8.0 U2 P03 and 8.0 U3. The RTM date is 1/18/2024."

Great news Anton!

[pirx]

By way of further update, VMware PM team has informed us that we can communicate to our customers the upcoming resolution for the issue:
"The issue will be fixed in ESXi 8.0 U2 P03 and 8.0 U3. The RTM date is 1/18/2024."

No update available yet - at least not for me.

[sakthisaravananp]

IS this released and fixed please confirm