How to restore old DC VM with expired AD db in offline mode on Hyper-V?

[hpadm]

I want to restore a replica of a domain controller VM to inspect its configuration. It is running on server 2019's Hyper-V services. It has to start in offline mode, and cannot (and must not) synchronize. Trying to start it up results in a 0xc00002e2 bluescreen. I tried some ideas from similar threads, but couldn't get it to work. I would appreciate if you could give me exact instructions how to do this. The issues are:

• VBR application-aware processing for full archival snapshots changes the DC to non-authoritative. Documentation has steps how to fix this using DSRM. (I have not tested if these are needed in my case.)
• Active Directory entries expire after 60 days (tombstoneLifetime). The only workaround I know is to change the system clock before the AD services start for the first time.
• Microsoft Hyper-V services don't seem to save the VM's system clock state between reboots.

[HannesK]

Hello,
just to clarify... are we talking about a Veeam replica?. About how old are we talking here?

I never heard of "offline mode" for Active directory, but disconnecting the virtual network cable would stop any synchronization (or connect to a new separated VLAN etc.)

What type of configuration do you want to inspect? I ask myself, whether using the Veeam Explorer for Active Directory could be an alternative to starting the whole domain controller.

Best regards,
Hannes

[hpadm]

Sorry for the bad phrasing.
Replica as in a copy of an existing VM that is running in production and doing its job just fine.
Old as in 6 months old, from one of the monthly or yearly periodic full backups.
Offline as in the VM runs permanently in cable-disconnected or private host-only virtual LAN. It will not be allowed to talk to the running AD servers to synchronize, because the whole point is to inspect its state from 6 months ago.
The point of interest was Group Policy. I am aware that I could have just recovered the SYSVOL files and guessed the meaning of their contents. I also thought of Veeam's Data Explorers AD functionality, but I do not have it installed and I do not know if it's available for VBR Community Edition (it might have been and I skipped it on the initial installer screen).

I have done copy restores of this VM several times in the past, it is easy and just takes a few minutes. I did not realize that a problem existed, until I tried booting an older backup. When it started (in DC recovery mode), it refused to log me in using the domain admin password, and when I logged in via local admin DC recovery password, it started in safe mode, with AD services turned off and unable to start. I tried for a while, but couldn't get the OS to boot in normal mode.

I learned that VBR does application-aware processing that changes the AD DB from authoritative to non-authoritative and forces the DC to sync. I cannot tell if this affects anything, or if this is a primary cause or a contributing factor to my issue.

I learned that according to Microsoft docs, AD system state backups have a 'shelf life'. If they're too old, the contents are automatically expired when the OS starts up. The maximum age can be configured, but it has to be done before the backup is made (or somehow patched into ntds.dit before first startup?). When searching the forums for '0xc00002e2' I found one case where this problem was worked around by changing the system clock, but when I tried that in Hyper-V manager, I discovered that the clock value is not retained between reboots, and is likely obtained from the host at every VM boot. I did not attempt to change the clock on the production hosting server.

I now suspect that this is less a Veeam issue and more a Microsoft AD issue, however it is closely related to backup & restore, and if there is a more proper recovery procedure for such a scenario (no available DC to sync from, all backups are past expiration date).

[HannesK]

hmm okay... with 6 months, that could be beyond the 180 days timeout, which might lead to some side-effects. I'm not deep enough into Active Directory... but sounds like an explanation.

Veeam Explorer for Explorer for Active Directory is installed automatically. It can open NTDS.dit directly and show group policies

With community edition (=standard edition), you cannot restore directly to production AD. But you could do export to LDIF format and check whether that helps.

[hpadm]

Ah! My mistake was looking for the explorers UI items inside the main VBR UI or under individual file restore, when it has its own application items restore UI. Or it can be launched standalone via a start menu shortcut, and pointed to an AD database file, perhaps on a FLR mount.

The problem I ran into with the MS AD explorer is that it seems to only interact with group policies on the AD database level. That is, it reads, compares, exports and recovers the policy metadata, like GUID and version number, but doesn't actually do anything with the files in SYSVOL that hold the actual policy rules. Or at least, the compare function and the export function doesn't.

I think however, that this could be combined with a FLR operation on the entire SYSVOL directory, to achieve the desired result. The restore operation also allows choosing a different target AD host, which could be the test VM. Then again, this requires the test VM to be online, at least enough for the VBR server to interact with it. This could be done, but only through some sort of elaborate network isolation method.

I did test the recovery of individual policy objects back into production, and it successfully reverted it on the server. So the product limitation you mentioned doesn't seem to exist anymore, or, applies to something else. Maybe a full AD database restore, or entries in the Users and Computers tree?

I now think that a more straightforward approach would be to create a new placeholder policy object in production, and overwrite its contents with the .pol file from the real policy file from a backup. These .pol files don't seem to be tied to a specific GUID (the .xml files are, but that can be edited), so it should work. Then all that's needed is to view the policy in gpmc.

PS: The AD expiration is 180 days on older server versions, but has been reduced to 60 days in recent releases. That is not a lot. I can imagine a disaster scenario where a small business is forced to shut down its servers for an extended period, for example covid lockdowns, seizure by the police, area evacuation, bankrupcy proceedings, etc. Imagine bringing the servers back up only to find out that the Active Directory is irreversibly lost and the backups don't help. Here, VBR or some other third-party ntds editor could solve it by patching the database right after restoring it, setting tombstoneLifetime to 99999 or something.