Veeam backed up DC boots into Active Directory repair mode

[cjm1903]

Hello all,

We have a virtual domain controller (Windows Server 2012) that is backed up by Veeam every evening. Occasionally we see the error "ErrorL VSSControl: Failed to prepare guest for freeze", and the only solution to this is to reboot the domain controller and run the backup again - after which the backup of the domain controller completes successfully.

When rebooting the domain controller we have to be careful to check the boot options in msconfig, because we often find that the Server is set to Safe boot in Active Directory repair mode. Unchecking Safe boot and changing the startup selection from selective startup to normal startup does not remove the safe boot option and we have to run bcdedit /deletevalue safeboot to remove the flag. Once this is done we can safely reboot the Server and it boots up as normal.

We can't figure out however why it occasionally does this, and we are wondering if it is something to do with the Veeam backup. I realise that the Server will boot into DSRM mode if a restore is performed, however we have not performed a restore of the domain controller and indeed never have to date, so I'm unsure why this keeps happening. It doesn't appear to happen after every reboot, but something is clearly putting into DSRM mode.

Any ideas?

Thanks,
Craig

[foggy]

Craig, I don't think this is Veeam B&R-related, however, you can contact technical support to investigate occasional VSSControl issues, probably they have similar nature.

[pbaideme]

Craig, we are experiencing the same issue, with the safe boot Active Directory Repair Mode being randomly checked, after being backed up by Veeam. Did you ever find a solution to this issue?

[foggy]

Do you see similar errors during backups prior to that?

[pbaideme]

No, We started to notice this randomly when the DC's do their weekly reboots, after we installed Veeam Endpoint Backup ver 1.5.0.306. This has only happened on our physical DC's (2008 R2) so far. This seems to be very sporadic, 1 or 2 times per DC over the past 3 months since we have bee using the VEB. The VM DC's being backed up via Veeam have been fine. We have had 3 different Windows engineer searching the Windows logs but have found nothing yet.

[foggy]

Veeam B&R indeed enables DSRM inside DC's guest OS during backup, but disables it back after the backup. There was an issue on MS side that might result in DSRM not being disabled that was worked around in the latest version of Veeam B&R. I'm not sure about Endpoint though. You can contact support for a closer look.

[StephenWagner7]

Veeam B&R indeed enables DSRM inside DC's guest OS during backup, but disables it back after the backup. There was an issue on MS side that might result in DSRM not being disabled that was worked around in the latest version of Veeam B&R. I'm not sure about Endpoint though. You can contact support for a closer look.

Thank you for this info!

I'm having the same issues as the posters above. No issues until I loaded up Veeam trial and this occured on one of my Server 2016 "Server Core" DCs. Didn't even realize what was going on for hours. Took it out of DSRepair, but it's nice to see this explanation as it can now put my mind at ease!

[mortenmadsen]

I just ran into this issue as well on Windows Server 2019 domain controllers. Am I the only one that is surprised that Veeam alters production systems in order make an backup?

Can this behavior be turned off and still run application aware processing? and are other changes made as well?

If I knew this in advance before we made the purchase, I think Veeam would have been removed for consideration as a backup tool in our environment.

I do not have an case ID at the moment as our AD team has asked to hold off with creating one for now.

[Gostev]

Veeam does not alter production system permanently, but only for the duration of snapshots. These changes are required to ensure successful full VM restore, so disabling it is not a good idea. However, this functionality was in place for 12 years now, and has been used in (currently) over 600000 active Veeam installations, so in general it's very reliable. As you can see, even this topic is 4 years old.

If you're running into some issues, most likely this is caused by a conflict with some 3rd party software. Please open a support case to that we can investigate the guest processing logs, and see what went wrong in your case.

Thanks!

[ejenner]

We've had this problem on 1 of our DCs quite a lot. 2012. On two more we have it occasionally. One of the less frequent (least frequent of all, only happened 2 or 3 times in total) is our PDC!

Not seen any solution above even though it's been going on for years?

I've tried doing things like changing the timing of the backup. It seems to happen on days when security patches are being deployed. Could there be some incompatibility between what patch deployment tools do and what Veeam does which causes this problem to occur?

Just thinking aloud but maybe both activities set the same flag and when they both do it one cancels the other and the server ends up set to boot in safe mode?

Some other interesting questions would be why this only happens on some DCs and only occasionally. Ours are agent backups as all our DCs are physical but I see above this happens on VMs as well.

[patricknh]

Actually looks like we had this problem last night. Our physical 2012r2 Dc which got patched last night after it had been backed up, would only come up in active directory restore mode until we modified the config.

[ejenner]

Update on this. I disabled the backup of our DC before patching and it still came up in directory restore mode after patching. That pretty firmly decouples the backing up session and the patching session if you don't even run the backup.

[MikeRickert]

Just as a general information regarding B&R modifies DSRM during AppAware backups of DC's. Yesterday we had a lot of failed DC backups on all platforms using PaloAlto Cortex XDR. PaloAlto implemented a new BTP rule with CU 650. This rule detects modifing DSRM as a Behavioral threat, which makes sense because nobody want's some processe tampering with the boot options of a OS, especially not on a DC.
We opend a support case with PaloAlto but me guess is the only solution is to create a general override for all DC's.

[Andreas Neufert]

We are following Microsoft Best practices for backup here. And as you shared it is expected. Let us know if PaloAlto wants to discuss that topic with us. Maybe they can modify their rules so that the backup processing is expected and not flagged while other modifications are flagged.

[MikeRickert]

Hi Andreas,

just got a responce from PaloAlto. PaloAlto engineering is aware of this issue and working on a solution. I forwarded your gentle offer to dicuss this topic with them. I will keep you updated.

[MikeRickert]

Issue with the DC AppAware backup is fixed in PaloAlto Cortex XDR Content version 660-12209.

[Andreas Neufert]

Hi Michael, thanks for sharing!

[Marc_Punte]

Hi all,
My two cents on this, it happened to me also.
In our case the VSS snaphot failed because patches were installed at the same time.

Breakdown:
Veeam-job set the SaveBoot flag > VSS failed due to OS-updates > job failed > SafeBoot flag NOT unset.
OS Patches installed > DC reboot into SafeMode.

Hope this helps....