Impact of Microsoft KB5025885 Secure Boot Patch for CVE-2023-24932 on Veeam Recovery Media

[jpennin1]

After reading through the scope of Microsoft KB5025885 on bootable media, I immediately wondered what I need to do in regard to Veeam recovery media? After applying the patch to Windows, do I just need to re-create the Veeam recovery media on the patched systems? Will the recovery media utility use the updated Windows files so that recovery will be possible again?

Did I miss any Veeam bulletins or news on this issue? This seems like a significant impact that should be warned about to all Veeam agent users especially those that want to do bare metal recoveries.

[BFost]

I have this question too. That Microsoft article that you linked to is quite a confusing read.

[Mildur]

Hi Guys

I‘m checking it already internally.
Creating a new recovery media to be on the safe side is not a bad idea. Will update this topic when we have more information.

Best,
Fabian

[azpets]

More or less... 100 boot media to recreate. I'm eager to know that I don't need to.
But I will need a KB article to make me budget time for do it, if necessary. The KB topic from Microsoft tells me that "could be needed". Veeam must confirm that for all media created.
Fortunately they are only files...

[ILU-ITDE]

Hi,

maybe this Powershell script is a good starting point to automate the Recovery Media update:
https://ccmexec.com/2023/05/ps-script- ... 023-24932/

Igor

[andym]

For the same reasons I'm also interested in Veeam's official response to this.

[Dima P.]

QA team is investigating this patch and once ready we will publish a KB article. Meanwhile it looks like after installing the MS patch you must recreate machines recovery media, because old RE wont work. Thanks!

[Brunok]

What about backups created before May 9th? Can these backups be restored once the patch has been installed and activated?

[BackupBytesTim]

I don't fully understand the ins and outs of Microsoft's Secure Boot process for Windows, but I imagine if existing Windows RE and PE images won't boot, similarly full Windows images may not boot either. So my guess is that if you attempted a recovery of the entire computer, specifically whatever volume contains the booting Windows image, then it could also fail. Which would be very important to be aware of, so now I'm quite curious also about this, I don't have time to test that myself right now so I'm hoping the Veeam team can get an answer soon.

[JPMS]

The update requires manual changes to fully implement.

My understanding is that you won't need to recreate boot media until you also implement those manual changes. That said, I haven't tested any of this and will wait for Veeam's official response.

MS have also promised further changes in July but won't be making the new patches mandatary until next year. This one is going to run and run...

[jpennin1]

What about backups created before May 9th? Can these backups be restored once the patch has been installed and activated?

This is a great point. I was mostly thinking about the recovery media aspect, but I hadn't considered that backed-up images may no longer be bootable after this update. This could mean every backup prior to the revocations being applied would be unusable for boot drive recovery. Now I'm even more interested in Veeam's official response after testing the various scenarios.

Edit...

maybe this Powershell script is a good starting point to automate the Recovery Media update:
https://ccmexec.com/2023/05/ps-script- ... 023-24932/

I read through that page and according to the author...
"Note that before implementing the manual steps to mitigate the vulnerability make sure you have all required lifecycle inplace as deploying an unpatched image even if the boot images are updated will fail."
This implies that all our pre-patch backups will no longer be something we can restore the boot drive with. Yikes! Big impact!

[Robvil]

"Note that before implementing the manual steps to mitigate the vulnerability make sure you have all required lifecycle inplace as deploying an unpatched image even if the boot images are updated will fail."

This will only be a big issue, if you have a big retension. For short retension times the old unpatched backups will be gone.

[Brunok]

We need to keep some backups several month/sometimes years (by law). What about the backup retention times on NAS-boxes, WORMs etc ? Useless ?? This is really critical.

[Robvil]

Do this not only affect the Veeam agent backups and not Veeam VMWare/Hyper V?

[Brunok]

This is my main concern - but until now, nobody answered this question. Will a (restored) VM also have problems after applying the patch ?

[Mildur]

Hey guys

Our QA team is still checking all the possible scenarios.
But Guest OS file or application item restore should always work, even for older restore points. There is no boot or recovery media involved in such recovery scenarios.
This topic will be updated with an official statement as soon the investigation has finished.

Best,
Fabian

[BackupBytesTim]

Good to know the Veeam team is checking on it. Just to clarify though to make sure there's no confusion,

There is no boot or recovery media involved in such recovery scenarios.

The assumption by the other users here was that the recovery media was not the only issue, but that even if the recovery media was updated and patched properly, that a full computer or boot volume recovery would "complete" but then fail to boot up if the restore point was created before the update patch was installed on the source computer because the resulting recovered Windows image would be unpatched at that point and fail to pass the Secure Boot verification process.

But yes, if only restoring files or applications without any modifications to the OS itself then there shouldn't be an issue that I can think of.

[Dima P.]

Hello folks,

We've confirmed that after installing the mentioned KB you will have the following issues:

1. Veeam RE stops to work. MS KB does not include needed components to patch MS Recovery Environment, but there is a manual workaround for that and we will publish it later.
2. Once the KB is installed, and you've succeeded in booting the recovery media, say with the manually added components, you will have issues booting machine after Bare Metal Restore was performed from the restore point created from 'pre-patched' operating system. There is a workaround for that too - you can disable Secure Boot in the machines BIOS whenever possible (but that kills the concept of this KB and should be used only as a last resort).

What to do next?

A. You can install the KB today, manually path the recovery media (KB to follow). I'd strongly recommend to start a new backup chain with and AF full backup too.
B. Wait till MS updates the KB to include components to the RE, install the KB and then start a new backup chain.

Hope it helps, I'll publish the workaround on how to manually patch the RE as soon as it's ready!

UPDATE: Here is the KB article from support team Support Statement regarding Microsoft CVE-2023-24932

[Saskatoon Kevin]

If the only issue is the MSRE, would this have any impact on using a Windows agent and then restoring it using one of the methods to convert to vm like instant recovery or disk export? We've a large number of VMs running with agents due to microsoft clustering.

[Gostev]

If the issue is only with the MSRE then this will not impact either instant recovery to a VM or direct restore to cloud because neither leverage a recovery environment.

[azpets]

What to do next?

A. You can install the KB today, manually path the recovery media (KB to follow). I'd strongly recommend to start a new backup chain with and AF full backup too.
B. Wait till MS updates the KB to include components to the RE, install the KB and then start a new backup chain.

Hope it helps, I'll publish the workaround on how to manually patch the RE as soon as it's ready!

Updating now seems to:

• Invalidate RE Media
• Invalidate Bare Metal restore
• Invalidate backup chain

So "wait" is the only option.

• Wait for MS to publish updated KB
• Wait for install the patch
• Wait for Veeam's KB for patch RE media...

Am I missing something or this is the only option list?

[BackupBytesTim]

Hypothetically you can update now, manually apply the update to RE media, start a completely new backup chain, and you'd be good.
There is presently no updated RE from Microsoft so Veeam doesn't have an updated Veeam Recovery Media either. Unsure if Veeam will attempt to update the software that creates the RE media to include the "manual" update process, but I don't expect it to be likely since to my knowledge Veeam can't actually distribute the updated Windows RE itself though I could be wrong, however they may release a script or something to update existing and newly created media.

For standard Windows RE or PE media you should be able to update it manually following the steps here: https://learn.microsoft.com/en-us/windo ... mic-update

As far as I know the same should work for Veeam media, but I've not tested it.

Regardless of modifications to bootable media your old restore points will be unbootable, unless you disable the UEFI Secure Boot option in your computer's UEFI settings, so technically you could probably still boot an old backup, but I would definitely still make new ones, both Veeam and Microsoft recommend new backups after the update is applied.

From Dima P:

What to do next?

A. You can install the KB today, manually path the recovery media (KB to follow). I'd strongly recommend to start a new backup chain with and AF full backup too.
B. Wait till MS updates the KB to include components to the RE, install the KB and then start a new backup chain.

From Microsoft at KB5025885:

If you use backup software to save the contents of your Windows installation into a recovery image, be sure to run a complete backup after installing the Windows updates released on or after May 9, 2023. Be sure to backup the EFI disk partition in addition to the Windows operating system partition. Clearly identify backups made before the May 9, 2023 updates versus those made after May 9, 2023 updates.

Currently the actual blocking of pre-update Windows from being able to boot does appear to be dependent on manually applying revocation policies, however Microsoft presently states:
See the timing section for more information.

First Quarter of 2024 - Enforcement Phase

When updates are released for the enforcement phase, they will add the following:

The revocations (Code Integrity Boot policy and Secure Boot disallow list) will be programmatically enforced after installing updates for Windows to all affected systems with no option to be disabled.

We’re looking for opportunities to accelerate this schedule, if possible and will announce any updates here.

Which would suggest that the current plan is to enforce the changes, rendering previous backups unbootable, early next year and that they may be moving up that timeline.

[Dima P.]

Hypothetically you can update now, manually apply the update to RE media, start a completely new backup chain, and you'd be good.

Correct.

Veeam will attempt to update the software that creates the RE media to include the "manual" update process

We would love to, but we cant. Needed components must be set to Recovery Environment by Microsoft.

Here is the official KB with the workarounds: Veeam Statement regarding CVE-2023-24932.

[azpets]

Summary steps:

• download from https://support.microsoft.com/en-us/top ... 3eb56fb589 the correct script file
• download from https://www.catalog.update.microsoft.co ... =Safe%20OS the correct .cab file for the OS
• run the script (optionally with the path manually provided via command and not via input
• checking the output, the WinRE.wim file should be patched
• run the new media creation with update WinRE.wim file inside
• test the media
• then run a full backup, just in case of

In any case, until microsoft do not provide better... mitigation, the download of script, .cab file, and WinRe.wim patching seems... mandatory? At least for safety purposes only...

Disabling SecureBoot when it was showed down the throat of system and environment admins seems a quite hefty "no no" for me...

[BackupBytesTim]

That looks correct, though I've not actually done it yet myself.

As for Microsoft's update, it does appear as though the plan is for a release on or after July 11th, which will include the WinRE update, after that update is installed you shouldn't need to follow the manual process to create an updated bootable Veeam recovery media, as Veeam should just acquire the already updated WinRE image from the computer when creating new media, if my understanding is correct.

See the timing section for more information.

I agree, probably shouldn't disabled Secure Boot unless absolutely necessary, but updating your WinRE and recovery media will only enable you to boot from your Veeam media, if you recover a pre-update Windows image it won't boot up, assuming the revocations have been applied, which isn't yet automated.

The only good solution is really to perform new full backups after the update, which will then be bootable after a recovery. That is what is recommended by both Veeam and Microsoft at this time.

You should also be able to reinstall Windows and recover files and applications from the backup, so long as you leave your Windows image unmodified, but at that point you're not performing an entire disk recovery.

[azpets]

FWIW, i added at the end of the Microsoft script a little addon

Code: Select all

cmd.exe /c "C:\Program Files\Veeam\Endpoint Backup\Veeam.EndPoint.Manager.exe" /createrecoverymediaiso /f:C:\Install\VeeamRecoveryMedia_%ComputerName%_602P.iso

Which fit my current conventions and defaults, creating the ISO file on the correct folder and with the computername, informing me that is the 6.0.2 version of veeam and it is a Patched iso with the updated winRE.wim file.

Few test systems have received this job, during next days I will check if the updated ISO media will be bootable.

[Brunok]

Thanks for the informations. I still have some questions.

For example:
I have a Hyper-V Host, running some virutal servers. One virtual server is a Veeam-Backup Server, backing up all other VMs on that host since one year (with external backups to USB). Lets say, we do not backup the physical host.

I install the patch now (13 june 23) on all VMs and the physical host.

Next week, my servers are victim of a ransomware attack and encrypted and cannot be restored.

So, i build a new Hyper-V Host, install one VM with the latest Veeam version, import the backups from the USB Disk and like to restore the other virtual servers.

Will this work
-> when i restore the latest backup (after patching)
-> or when i restore a VM from the 1st april ?

sorry for the confusion - but this is a possible scenario (building a new hyper-v host, create a Veeam-VM and restore the other VMs from the backups). So it is important to know..

Thanks
Bruno

[HannesK]

@Brunok: I'm not sure what the question has to do with this topic. This is only about Veeam Agent for Windows and Secure Boot.

[Brunok]

@HannesK: the question is, if that this secure boot problem also will be there, if i restore a VM that was backupped befor the patch was installed on this VM. Will the VM boot into the OS ?
Or does the problem only exist on PHYSICAL machines ?

[BackupBytesTim]

The problem would also be present on Hyper-V VMs with the corresponding Secure Boot features enabled.

If you did the process you mentioned right now, it should be okay without any new backups or modifications EXCEPT if you manually apply the revocations. Per the Timing Section the revocations that actually block pre-update Windows from booting are not yet in place automatically. The plan is for those revocations to be enforced next year, but there is a note that they may move that date up to an earlier point in time.

Note this is just based on my understanding and our limited internal tests to verify no modifications to Veeam media are required at this time when the revocations are NOT manually applied. I, of course, will not guarantee my advice to be 100% accurate to your environment, if you have time and need to know for certain, I would perform a test recovery in an isolated virtual environment on the host hardware you would be recovering to in the event of an actual recovery.

Note if you are actually attempting to protect your environment from the bootkit vulnerability, you DO need to actually apply the revocations manually at this time, in which case you will need updated full backups, as well as updated recovery media.