Blue screen after Domain Controller restore

[DVelez]

Hi,
One of my customer is having blue screens when restoring a Domain Controller. The blue screen has the error 0xc00002e2. There are times where the restore works without issues. We tried to disable CBT, but the issue still present. Application-aware processing is enable and there are no errors in backup or restore process. The host is a Windows Server 2022 with Hyper-V and the guest is Windows Server 2022. I ran the Active Directory tests, and all of them passed.

We also noticed that a Remote Desktop Server restored without errors, but when I logged in, the licensing database was corrupted.

We talked to tech support, but didn't find any issues. The customer have Veeam BR 11 with all updates installed. Does someone have a similar experience? Any suggestion to fix this issue?

[chris.childerhose]

I would suggest continuing with support on this one since some restores work and others don't.

[DVelez]

I could try to open a new case. Still, the last time the didn't find anything, so we had to close the case. Now I'm hoping to find a solution in the forums.

[PetrM]

Hi Daniel,

The error 0xc00002e2 means that most probably NTDS.DIT is corrupted. I would suggest to review Windows event logs (in particular System and Application) on DC and look for errors occurred during backup. One more idea is to collect vsstrace during backup and examine it, but this test should be carried out by your support engineer. Also, do you see the same issue when you restore from full point or does it persist when you take an incremental point? In which mode does incremental chain work? For example, are there some synthetic operations or this is forward incremental chain?

Basically, you need to work with our support team until you get a clear answer and technical explanation of the issue to know what to do next, for instance to involve 3rd-party support team etc. I bet the issue does not come from our code but obviously just this hypothesis is not enough to close the case. Please share a support case ID with us so that we can keep an eye on it.

Thanks!

[Magion]

Most articles explaining error 0xc00002e2 are about corrupted NTDS files or database, however it might also have to do with age of the backup.

How old is the backup you are trying to restore? I ran into this error after restoring an older backup of a DC, and found out that if a backup is older than the tombstonelifetime in AD (60 or 180 days default depending on Windows version) you will end up with a blue screen as well. Quick fix is to boot to DSRM, adjust time to just after the backup date and reboot (allthough DFS-R might still be complaining after this).

[Anders]

99% of the time when, restoring an AD server from backup causes a bluescreen of death, its because the restore point is older than the tombstone timer in AD, in my experience.
The tombstone timer deletes all AD objects in the database if the restore point is older than the timer!

Default value on the timer depends on what windows version the AD was installed on. Often its 60 days. - MS best practices is to change this to 180 days right now. Setting it to a year or more is not an issue in most environments in my experience.

You can find a Microsoft description of "the problem" here https://docs.microsoft.com/da-DK/troubl ... -backup-ad - where they describe it as:
"Specifically, the useful life of a backup is the same as the "tombstone lifetime" setting for the enterprise. The default value for the tombstone lifetime entry is 60 days. "

/Anders

[AlexHeylin]

One other option here. Have you confirmed that the NTFS on the DC C: is completely valid and optimal, and that all the system files are valid and healthy?

The ONLY authoritative way to do that is CHKDSK /F c: and reboot. Running it online will NOT do the same thing and often misses errors which the offline scan finds and fixes. With that done SFC /SCANNOW then DISM /ONLINE /CLEANUP-IMAGE /RESTOREHEALTH should be your moves. Those ensure that the system files aren't corrupted - because damaged system files is just as (probably more) likely to cause BSOD than damaged data.

You might like to upvote cloud-connect-backup-f43/vbr-vaw-val-to ... 80878.html which would allow VBR / VAW to check the filesystem and ensure the filesystem in the backup is fully healthy and at least alert if it's not. Damaged system files usually (IME) arises from a damaged file system.

I've run into this BSOD myself, and in our case it turned out that while the backups had always backed up OK - they weren't using VSS (not Veeam backup) so the DC was not restorable (and no-one had ever done a test restore). This is one of those cases where the only real solution to dealing with this is good preparation combined with test restores and booting the restored OS. Otherwise, by the time you find you've got the problem it's far too late to do anything about fixing it.

[bhagen]

99% of the time when, restoring an AD server from backup causes a bluescreen of death, its because the restore point is older than the tombstone timer in AD, in my experience.
The tombstone timer deletes all AD objects in the database if the restore point is older than the timer!
/Anders

Thank you so much for this @Anders! Our AD environment got wiped out and our backup repos were formatted, so all we had was 3 month old tapes to recover from. All our DCs were getting the BSOD.

So I recovered another one yesterday, and before it automatically rebooted I logged in as administrator and manually changed the date to 5 months ago. When it rebooted, it did not BSOD, and I was able to log in!!

After confirming the operations, I set the time back to auto, and the dc is still running just fine.

Whew!!

[DVelez]

@PetrM, looks like the NTDS.NIT is corrupted in the restored server. We had to rebuild the database, and now it's running. I checked the production AD, and has no issues and I'm not seeing errors in the event viewer. Would it help if we try disabling the application-aware processing for the DC?

@Magion / @Anders / @bhagen, the backup was just one day old. The customer is a small one, with a single DC. Does the issue you mentioned applies to a single DC?

@AlexHeylin, I ran check disk and didn't find any errors.

Thanks everyone for your responses.

[PetrM]

Hi Daniel,

Would it help if we try disabling the application-aware processing for the DC?

I don't see any reason why would it help, moreover we cannot guarantee application data consistency without AAIP in action. Anyway, I strongly recommend to continue troubleshooting with our support team and share a support case ID over here so that we can make sure it's on right track.

Thanks!

[DVelez]

I don't see any reason why would it help, moreover we cannot guarantee application data consistency without AAIP in action. Anyway, I strongly recommend to continue troubleshooting with our support team and share a support case ID over here so that we can make sure it's on right track.

We are doing some additional tests before opening the support ticket. I'll post it as soon as I have it.

[DVelez]

I don't see any reason why would it help, moreover we cannot guarantee application data consistency without AAIP in action. Anyway, I strongly recommend to continue troubleshooting with our support team and share a support case ID over here so that we can make sure it's on right track.

We did some additional tests. One of them was to create a new server and migrate the AD roles to it. It has been working and restoring without issues.

The old server was demoted, and now is a file / print and DHCP server. Now we noticed that the DCHP database is corrupted after the restore. We also are seeing database corruption in the Remote Desktop Licensing Server. At the same time, we have a SQL Server, and all databases are operational, and no errors are being reported. It looks like the issue is only with JET databases.

We open a support ticket, case #05589279. The previous case related to the AD blue screen was #02262774.

[PetrM]

Hi Daniel,

Many thanks for information, much appreciated! Let's see what our support engineers can find out.

Thanks!