Veeam B&R recovery of a domain controller

[peeky1323]

Hi everyone.

I have a strange issue here. We have 3 domain controllers (2x2012 R2 and 1 2008 R2). I am trying to restore them into a test setup and I have restored one and followed the guide here:

https://www.veeam.com/blog/how-to-recov ... ction.html

for performing an authoritative restore. I then restored the other 2. My AD is broken and cannot start AD users and computers, sites and services etc, replication will not work, and can't login to any member servers I have restored into this test setup. After much troubleshooting I tracked down the regitry key sysvolready was set to 0 on all servers. After changing this to 1 things are improving and cna now login and start the AD tools (I haven't looked beyond this yet to see if there are still issues). Is there a set procedure for restoring multiple DCs? Do you perform the authoritative restore steps on one that holds a certain role or anyone?

Many thank,

Matt

[foggy]

Hi Matt, please review this KB describing all the nuances of domain controller recovery, including your scenario. Just to confirm, you have application-aware image processing enabled on the corresponding backup job, right?

[SteeleTek]

Greetings,

One of the most common issues I experience when using Veeam to restore a backup or failover a replica of Windows Server 2008 R2 Active Directory Domain Controllers is being unable to create new objects in AD (user/computer/group). We receive the error "Windows cannot create the object OBJECTNAME because: The directory service was unable to allocate a relative identifier."

The DC we restore is the PDC master, Infrastructure master, and RID master.

This article from Microsoft doesn't seem applicable since the DC we restored already has all the necessary master roles: https://support.microsoft.com/en-us/hel ... irectory-s

Anyone else experience this issue when restoring Veeam backups of a DC or performing a failover to a replica for a DC?

Thank you!

[SteeleTek]

It seems like we mainly experience this when other Domain Controllers in the domain (such as the secondary one) is not also restored/available. Obviously, this can occur if the RID master is not available but in our case we were restoring the RID master.

Lesson learned: always restore ALL the domain controllers, even the secondary ones!

[final]

Restoring AD is a pain if you have multiple DCs (which you should). Each AD-Object has a version number attached to it. During DC-Sync, it's "higher version wins" - so your newly restored DC will get all the newer AD objects from the other DCs as soon as it has a connection to them. If you restore your DC because you actually want to restore your AD to an older state, then you have two options:
1. restore all DCs to a state before the "catastrophic event"
2. restore one DC, boot it up in "restore mode" (F8 during boot) and without a network connection, and use ntdsutil to perform an "authoritative restore". I believe this does nothing else than increase the version number of all AD objects on the specific DC by a large number, therefore ensuring that the DC now has the highest version number on all AD objects. After that, you can give the DC back its network connection, and all other DCs will replicate the old (now new) state from the restored DC.

Fortunately, the last and only AD restore I had to do is about 8 years ago. Maybe the restoring process is more sophisticated these days than it was back in the day, so my info may be a bit outdated.

[foggy]

Here's a KB article describing domain controller recovery peculiarities in detail, should answer your concerns.

[Fwiler]

Planning our disaster recovery solution.

If our AD servers are down will Veeam work on restoring these? The reason I ask is I've provided domain admin credentials within Veeam for backups and replication.
Should I be using different credentials, or does Veeam use cached credentials once you've entered them?

If the Veeam server itself goes out and I have to reinstall Veeam on a different server, would that also require a working AD server?

Thanks.

[DGrinev]

Hi Dustin,

You're good to go with the domain admin credentials. Also, check the KB about the DC restore process.
You can restore Veeam Backup server without working AD.
Please review this thread for additional information. Thanks!

[BIGNOOKIE]

I have seen some other posts on this forum regarding AD restores but I seem to be going nowhere rapidly.
The jobs are setup to be application aware etc.
The AD server is backed up over CloudConnect and I have asked our provider to restore the server for DR testing.
I boot the server and login using DSRM and then reboot normally.
However I have no DNS and no AD.
At first I thought it was because it was a member AD server (no FSMO roles) (2012R2) but seizing didn't work.
So I removed that server and backed up the server hosting all the FSMO roles (2008R2).
Had it restored again and essentially the same result.
What am I missing here? Google isn't been that helpful
I thought Veeam was all about availability but if you can't even stand up your AD, there's not much point bothering with your other servers....

[foggy]

Hi Nick, please review the domain controller recovery peculiarities for better understanding.

[interimcptech]

Hi there. New member here asking for some guidance. To make a long story short, we are completely virtualized on VMware. Last weekend I installed Windows Updates on (3) Server 2008 R2 domain controllers. The updates completed fine, and the OS booted up and everything looked like it was working fine and I deleted the snapshot once finished. Then I came in Monday morning to find that replication ceased functioning and there seems to be O/S corruption and it has been a worst week of my life. I've tried everything but nothing works. Am currently working with Microsoft business support.

We still have the Veeam restore points for the Friday just before I installed those updates but we've never actually used Veeam to restore before let alone restoring multiple domain controller. I think we will have to restore all 3 of them but its a very scary situation. So my question is:

Is restoring domain controllers from Veeam destructive? Once I restored will computers/users be able to authenticate?
What are the risks associated with restoring 3 domain controllers?
What about connectivity to an Exchange 2010 server?
If we need to do an authoritative restore, do we need that preconfigured username & password for directory services restore mode or will our enterprise admin credentials work fine? I think there's a password that you set when you promote a DC.

I have already read the article on restoring domain controllers non-authoritatively per this article which seems straight forward:

https://www.veeam.com/blog/how-to-recov ... tion.html

For some reason my associates believe that if we restore from Veeam, the active directory objects' properties/attributes will change and we will lose connectivity to our Exchange server or user & devices won't be able to authenticate. This is our last resort.

I appreciate any help. It really really helps. Thank you.

[wishr]

Hi Chris,

Welcome to Veeam Community Forums and thanks for posting!

Sad to hear you got into this situation. I'm merging your topic to an existing discussion - please take a brief look and note the post by Foggy above mentioning a relevant KB article.

This is a common use-case scenario. If done properly there should not be any authentication issues.

Thanks

[Andreas Neufert]

That do not sound good. Sorry to hear.

Regarding the directory object properties. I do not know exactly where this comes from, but please let me describe what is needed a bit more.

1) Shutdown all of your corrupt AD server.
2) Restore one AD Server and manually place them into Authorative Restore mode. There are good explanations in the internet how to do this and what password is needed.
3) Restore the other AD Server with network connection (!!!). They will be placed autoamtically in non-authorative restore mode then.

You AD should then work again. There is one thing that could have happened. Kerberos will update the machine passwords frequently and there could be some servers that got from your AD a new Kerberos password after the Friday backup. The result will be that these servers or PCs can not login into domain. There are 2 fixes. 1) rejoin the server into the domain. 2) restore with Veeam the Active Directory computer object (Veeam Explorer for Microsoft Active Directory).
My guess is that this is as well what the objects property from above want to tell you.

Overall I would recommend to check the above and overall procedure with a Active Directory specialist.
You can let them know the following:
1) Veeam do image level backup on the server with application awareness (consistency)
2) If you restore server with AD controlers on it and network enabled, we set them automatically in non authorative restore mode.
3) Our Veeam Explorer for AD can restore AD objects without harming the AD. If tompstones are still present we can restore as well deleted objects to the original without to change any object ID.

[interimcptech]

Sad to hear you got into this situation. I'm merging your topic to an existing discussion - please take a brief look and note the post by Foggy above mentioning a relevant KB article.

This is a common use-case scenario. If done properly there should not be any authentication issues.

Thank you wisher. I have been reading a lot over the weekend on the forums and my plan of action was to restore all DCs back to last Friday. Everything was looking good and I double checked the restore stpes, checked that all (3) DCs wee running FRS, checked the DSRM logon password. However now I am mortified because when I was checking the application-awareness of the backups it turns out that while CHILDDC2 at SITE B does have it checked, the backup job that stores CHILDDC2 and ROOTDC does NOT have application-aware designation. This backup job was created before I got here and didn't know the feature even existed until this all happened. This is where I stand now. It seems at this point, the only restore we can do is the non-authoritative restore of CHILDDC2 at the other site. Any insight onto the ramifications of this?

[interimcptech]

Regarding the directory object properties. I do not know exactly where this comes from, but please let me describe what is needed a bit more.

Hi Andreas. That was my original plan and I had everything worked out until I came to find out the my ROOTDC and CHILDDC1 were part of a backup job that does not have application-aware checked. So now I think I'm dead in the water. Right now the plan is to do a non-authoritative restore of CHILDDC2 only and hope that replication comes back. Prior to this though my coworker found a patch that might address the replication issue and thinks it might have to do with VMware virtual NIC card on the VM.

[interimcptech]

Is it safe to do an authoritative restore on the primary DC if I don't have application-aware processing checked?

[Andreas Neufert]

Usually this is not an issue and you can continue. You have to select anyway the authoritative restore manually. But anyway this is more of a question for an Active Directory expert.

Btw. you can try the restore in our Virtual Lab to play with it and check if you can restore everything the right way.

[wishr]

Hi Chris,

I'm back with additional suggestions.

While there is no 100% guarantee that a DC restored from a job with AAIP disabled will boot with no issues, but in most cases, it should be fine. However, In certain rare cases, the AD database may become corrupted - this can be easily identified by having BSODs during the initial DC boots after restore.

As far as I got (please, confirm) you have a single domain with three DCs. Since you have three DCs and at least one of them has been backed up with AAIP enabled it should be possible to perform the recovery.

You may try to restore CHILDDC2 located at the site B (that has AAIP enabled within the job), and, as Andreas suggested above, manually place it into Authorative Restore mode (check for guides on the Internet). The rest two DCs located at the site A (those that have AAIP disabled within the job) can be restored as well but should be placed into Non-Authoritative Restore mode to replicate AD data from the restored DC located at the site B.

We strongly to encourage you to run the recovery in the test lab first of all! You have two options here:

• You may use SureBackup Virtual Lab, as per suggestion by Andreas. Please note that it is only available in Enterprise and Enterprise Plus B&R editions.

• You may try to restore all the DCs into an isolated non-production network and play with them a little to make sure the things will go smoothly.

Hope it helps! Please let us know how it goes.

[Valle1975]

Good morning to everyone.
Reading a lot of docs and posts about Veeam B&R and Microsoft Active Directory, I'm approach to migrate our Clients from workgroup to Active Directory (Windows Server 2016) on my Vmware Infrastructure, where I have already prepared and working n.2 VM Windows Server 2016 as DC (main and secondary already in Sync), that supports VCSA anche some other Servers.
So I use Veeam B&R both for Replica (every 2 hours) and Backup (one incremental per day, with full in weekend) for all my infrastructure, but I want to understand the best practice for backupping this two DC, and most important the Right procedure for the restore from replica and from backups.
Thanks.

[foggy]

Hi Valerio, I'm merging your post to existing discussion regarding DC recovery, please review hints and links mentioned there, specifically this KB. Thanks!