Comprehensive data protection for all workloads
Post Reply
cjm1903
Lurker
Posts: 1
Liked: never
Joined: Jul 22, 2016 1:43 pm
Full Name: Craig Melvin
Contact:

Veeam backed up DC boots into Active Directory repair mode

Post by cjm1903 »

Hello all,

We have a virtual domain controller (Windows Server 2012) that is backed up by Veeam every evening. Occasionally we see the error "ErrorL VSSControl: Failed to prepare guest for freeze", and the only solution to this is to reboot the domain controller and run the backup again - after which the backup of the domain controller completes successfully.

When rebooting the domain controller we have to be careful to check the boot options in msconfig, because we often find that the Server is set to Safe boot in Active Directory repair mode. Unchecking Safe boot and changing the startup selection from selective startup to normal startup does not remove the safe boot option and we have to run bcdedit /deletevalue safeboot to remove the flag. Once this is done we can safely reboot the Server and it boots up as normal.

We can't figure out however why it occasionally does this, and we are wondering if it is something to do with the Veeam backup. I realise that the Server will boot into DSRM mode if a restore is performed, however we have not performed a restore of the domain controller and indeed never have to date, so I'm unsure why this keeps happening. It doesn't appear to happen after every reboot, but something is clearly putting into DSRM mode.

Any ideas?

Thanks,
Craig
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam backed up DC boots into Active Directory repair mo

Post by foggy »

Craig, I don't think this is Veeam B&R-related, however, you can contact technical support to investigate occasional VSSControl issues, probably they have similar nature.
pbaideme
Lurker
Posts: 2
Liked: never
Joined: Mar 04, 2017 4:58 pm
Full Name: Philip Baideme
Contact:

Re: Veeam backed up DC boots into Active Directory repair mo

Post by pbaideme »

Craig, we are experiencing the same issue, with the safe boot Active Directory Repair Mode being randomly checked, after being backed up by Veeam. Did you ever find a solution to this issue?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam backed up DC boots into Active Directory repair mo

Post by foggy »

Do you see similar errors during backups prior to that?
pbaideme
Lurker
Posts: 2
Liked: never
Joined: Mar 04, 2017 4:58 pm
Full Name: Philip Baideme
Contact:

Re: Veeam backed up DC boots into Active Directory repair mo

Post by pbaideme »

No, We started to notice this randomly when the DC's do their weekly reboots, after we installed Veeam Endpoint Backup ver 1.5.0.306. This has only happened on our physical DC's (2008 R2) so far. This seems to be very sporadic, 1 or 2 times per DC over the past 3 months since we have bee using the VEB. The VM DC's being backed up via Veeam have been fine. We have had 3 different Windows engineer searching the Windows logs but have found nothing yet.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam backed up DC boots into Active Directory repair mo

Post by foggy »

Veeam B&R indeed enables DSRM inside DC's guest OS during backup, but disables it back after the backup. There was an issue on MS side that might result in DSRM not being disabled that was worked around in the latest version of Veeam B&R. I'm not sure about Endpoint though. You can contact support for a closer look.
StephenWagner7
Lurker
Posts: 1
Liked: 1 time
Joined: Apr 29, 2018 4:03 pm
Full Name: Stephen Wagner
Contact:

Re: Veeam backed up DC boots into Active Directory repair mo

Post by StephenWagner7 » 1 person likes this post

foggy wrote:Veeam B&R indeed enables DSRM inside DC's guest OS during backup, but disables it back after the backup. There was an issue on MS side that might result in DSRM not being disabled that was worked around in the latest version of Veeam B&R. I'm not sure about Endpoint though. You can contact support for a closer look.
Thank you for this info!

I'm having the same issues as the posters above. No issues until I loaded up Veeam trial and this occured on one of my Server 2016 "Server Core" DCs. Didn't even realize what was going on for hours. Took it out of DSRepair, but it's nice to see this explanation as it can now put my mind at ease! :)
mortenmadsen
Novice
Posts: 8
Liked: never
Joined: Feb 28, 2018 9:28 am
Full Name: Morten Madsen
Contact:

Re: Veeam backed up DC boots into Active Directory repair mode

Post by mortenmadsen »

I just ran into this issue as well on Windows Server 2019 domain controllers. Am I the only one that is surprised that Veeam alters production systems in order make an backup?

Can this behavior be turned off and still run application aware processing? and are other changes made as well?

If I knew this in advance before we made the purchase, I think Veeam would have been removed for consideration as a backup tool in our environment.

I do not have an case ID at the moment as our AD team has asked to hold off with creating one for now.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam backed up DC boots into Active Directory repair mode

Post by Gostev »

Veeam does not alter production system permanently, but only for the duration of snapshots. These changes are required to ensure successful full VM restore, so disabling it is not a good idea. However, this functionality was in place for 12 years now, and has been used in (currently) over 600000 active Veeam installations, so in general it's very reliable. As you can see, even this topic is 4 years old.

If you're running into some issues, most likely this is caused by a conflict with some 3rd party software. Please open a support case to that we can investigate the guest processing logs, and see what went wrong in your case.

Thanks!
ejenner
Veteran
Posts: 636
Liked: 100 times
Joined: Mar 23, 2018 4:43 pm
Full Name: EJ
Location: London
Contact:

Re: Veeam backed up DC boots into Active Directory repair mode

Post by ejenner » 1 person likes this post

We've had this problem on 1 of our DCs quite a lot. 2012. On two more we have it occasionally. One of the less frequent (least frequent of all, only happened 2 or 3 times in total) is our PDC!

Not seen any solution above even though it's been going on for years?

I've tried doing things like changing the timing of the backup. It seems to happen on days when security patches are being deployed. Could there be some incompatibility between what patch deployment tools do and what Veeam does which causes this problem to occur?

Just thinking aloud but maybe both activities set the same flag and when they both do it one cancels the other and the server ends up set to boot in safe mode?

Some other interesting questions would be why this only happens on some DCs and only occasionally. Ours are agent backups as all our DCs are physical but I see above this happens on VMs as well.
patricknh
Influencer
Posts: 14
Liked: 1 time
Joined: Dec 21, 2020 4:17 pm
Full Name: Patrick holt
Contact:

Re: Veeam backed up DC boots into Active Directory repair mode

Post by patricknh »

Actually looks like we had this problem last night. Our physical 2012r2 Dc which got patched last night after it had been backed up, would only come up in active directory restore mode until we modified the config.
ejenner
Veteran
Posts: 636
Liked: 100 times
Joined: Mar 23, 2018 4:43 pm
Full Name: EJ
Location: London
Contact:

Re: Veeam backed up DC boots into Active Directory repair mode

Post by ejenner » 1 person likes this post

Update on this. I disabled the backup of our DC before patching and it still came up in directory restore mode after patching. That pretty firmly decouples the backing up session and the patching session if you don't even run the backup.
MikeRickert
Service Provider
Posts: 3
Liked: never
Joined: Jan 28, 2018 4:38 pm
Full Name: Michael Rickert

Re: Veeam backed up DC boots into Active Directory repair mode

Post by MikeRickert »

Just as a general information regarding B&R modifies DSRM during AppAware backups of DC's. Yesterday we had a lot of failed DC backups on all platforms using PaloAlto Cortex XDR. PaloAlto implemented a new BTP rule with CU 650. This rule detects modifing DSRM as a Behavioral threat, which makes sense because nobody want's some processe tampering with the boot options of a OS, especially not on a DC.
We opend a support case with PaloAlto but me guess is the only solution is to create a general override for all DC's.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Veeam backed up DC boots into Active Directory repair mode

Post by Andreas Neufert »

We are following Microsoft Best practices for backup here. And as you shared it is expected. Let us know if PaloAlto wants to discuss that topic with us. Maybe they can modify their rules so that the backup processing is expected and not flagged while other modifications are flagged.
MikeRickert
Service Provider
Posts: 3
Liked: never
Joined: Jan 28, 2018 4:38 pm
Full Name: Michael Rickert

Re: Veeam backed up DC boots into Active Directory repair mode

Post by MikeRickert »

Hi Andreas,

just got a responce from PaloAlto. PaloAlto engineering is aware of this issue and working on a solution. I forwarded your gentle offer to dicuss this topic with them. I will keep you updated.
MikeRickert
Service Provider
Posts: 3
Liked: never
Joined: Jan 28, 2018 4:38 pm
Full Name: Michael Rickert

Re: Veeam backed up DC boots into Active Directory repair mode

Post by MikeRickert »

Issue with the DC AppAware backup is fixed in PaloAlto Cortex XDR Content version 660-12209.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Veeam backed up DC boots into Active Directory repair mode

Post by Andreas Neufert »

Hi Michael, thanks for sharing!
Marc_Punte
Novice
Posts: 5
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Veeam backed up DC boots into Active Directory repair mode

Post by Marc_Punte »

Hi all,
My two cents on this, it happened to me also.
In our case the VSS snaphot failed because patches were installed at the same time.

Breakdown:
Veeam-job set the SaveBoot flag > VSS failed due to OS-updates > job failed > SafeBoot flag NOT unset.
OS Patches installed > DC reboot into SafeMode.

Hope this helps....
Post Reply

Who is online

Users browsing this forum: No registered users and 273 guests