Veeam B&R recovery of a domain controller

[zoltank]

[merged]

In a hypothetical DR situation where you have more than one domain controller you're restoring, after restoring the first domain controller do you need to perform an authoritative restore of the AD database before restoring addition domain controllers?

I've read through the domain controller restore thread, but it didn't seem like there was a solid answer provided.

For reference the authoritative restore procedure:
- Open a command prompt and type ntdsutil and then press ENTER.
- At the ntdsutil: prompt, type authoritative restore and then press ENTER.
- At the ntdsutil authoritative restore: prompt, type restore database and press ENTER.
- At the Authoritative Restore Confirmation dialog box, click OK.
- Type quit and press ENTER until you have exited Ntdsutil.exe.

[Vitaliy S.]

Actually there is a solid answer

This is not required in the case of a single DC recovery. Though you do need to perform authoritative SYSVOL restore on the first DC in case of restoring the whole Active Directory. Here are more good topics on that: Multiple Domain Controllers - How to Backup? and Active Directory and DR Site.

[ejleipold]

I test recover both my DC's (both 2008 R2) into my dev environment every month, all I do is a normal recovery, boot them up, wait 15 minutes, reboot them both, wait another 15 minutes and they sync and off they go, happy as larry.

[Unison]

I test recover both my DC's (both 2008 R2) into my dev environment every month, all I do is a normal recovery, boot them up, wait 15 minutes, reboot them both, wait another 15 minutes and they sync and off they go, happy as larry.

Sry for long post guys, but anyone trying DC recovery may be able to relate...

When you say 'normal recovery' - what exactly do you mean? Because if you mean "Veeam Instant Recovery" then dont say that to anyone at Veeam because they will tell you thats impossible. If your starting both your DCs via instant recovery and just loading them to an isolated network - both DCs should boot to non authoritative mode....which will basically break your AD/domain. Apparently.

Ive just been working with veeam for the last few weeks on a case all about DC recovery. I personally have found the process of recovering a DC with veeam to be....inconsistent. Seems that is fairly normal among other veeam users too. Maybe DC recovery will be more robust and handled a bit better in future releases.
I say "inconsistent" because sometimes the recovery of a DC works, sometimes it doesnt, sometimes you have to do 'this' as an extra step, sometimes 'that' etc and then almost always you have the issue that this post is mainly talking about - authoritative vs non authoritative restore and the fact that veeam doesnt allow you to choose which to do. Instead we are just told that 99% of the time all you will need to do is a non-auth restore so dont worry about the fact that veeam doesnt let you choose....or just follow xyz if you want an auth restore...where xyz doesnt really work and is not very 'clean'.....not exactly the perfect recovery of your DC/domain that you would expect anyway. And none of that helps if you just want to recover your DCs to a test setup for any purpose you require.

However, there is a method to get veeam to to boot your DC into authoritative mode.....its called Surebackup. If you put one of your multiple DCs into a surebackup job, veeam will do its magic and make that one DC boot into auth mode so your DC comes up working perfect, your domain is good and AD is good....sometimes. Ive seen it work and seen it fail...all with the same backup file...just running the same job over and over to check consistency of recovery. So surebackup will boot your DC into auth mode (if you set it in the app group to be a DC and GC server - doing this is what makes veeam run certain scripts and prep work on the DC, veeam support tells me) - if you want to test your DCs in an isolated setup.

What i find the most inconsistent is the recovery of a DC using Veeam Instant Recovery. Its mentioned all over this forum and call up veeam and ask them about this.....recovery of just ONE or ALL of your DCs to an isolated environment using Veeam Instant Recovery WILL FAIL - IT CANT WORK......but again, sometimes it does
I asked veeam about restoring just one of my DCs to a test setup as i was wanting to try something with software in an isolated environment - so i needed my DC, DNS and the other server. I told them that it wasnt working with Instant Recovery. They said "no, it wont work with instant recovery because IR will boot your DC into non-auth mode so your DC wont act like a DC. Instead, use Surebackup to setup the test environment"......and that was the basis for my support case which lasted weeks.
During that time we were using instant recovery and surebackup to try and achieve this setup - with the same backup files. Sometimes surebackup would work, sometimes it wouldnt. Sometimes instant recovery would work, sometimes it wouldnt....inconsistent.

When i first reported to veeam that "i got my one DC and one app server working together using INSTANT RECOVERY in an isolated environment with no use of burflags or ntdsutil etc", they didnt believe me.....so i showed them....and they couldnt explain it.....how can a DC successfully recover if its booting in non-auth mode and has no contact with other DCs???? Simple, it cant....veeam must be in control and doing something to make this happen....so why wont it do this all the time? I can recover just ONE of my multiple DCs to an isolated environment using instant recovery, veeam boots it first time aound, does its magic then boots it again into normal mode where i can login with a domain account - i can then see AD is working, netlogon/sysvol shares are working, no errors in event log (like there are when DC recovery fails...you all know those!).....the only exception is that i have to seize the RID master role from my other DC depending on which one i recover so that i can create new objects in AD....thats it. My domain, DC, AD is all working in a test setup!

When i have recovered a DC and it fails...i have tried solutions offered in this post - use ntdsutil to force the DC to boot into auth mode, stop the replication service, edit burflags reg key etc etc but even those solutions have not worked. Only way i have seen one of my DCs boot successfully was when i never made ANY modification during or after the recovery.....i just let instant recovery or surebackup do its thing....didnt change a thing, didnt run any commands after recovery....it just either works on its own or it fails and i run the same job again until it works. No manual process as per posts in here or MS articles has ever got one of my DCs working right again in a test lab....the only thing that has done it has been VEEAM successfully....both in instant recovery and surebackup. I just wish this was way more reliable.

Ive also heard the argument about 3rd party interference impacting recovery. If the recovery doesn't go well, its not veeam or the recovery process....it must be something else interfering with the recovery like your AV or some other product. Wonder if that has ever been proven, veeam staff couldnt find anything here that would do it and if something like that was interfering, then it would interfere every time, not just sometimes cause a problem then other times not.

So now im going through this thread, after my support case with veeam trying to find out if im doing something wrong or if there is a true tried and tested way of backing up and recovering your DCs........and i still feel way uneasy about recovering my DCs. Hopefully i never need to do it for real because it seems very inconsistent and a bit of a lucky dip when it comes to success/failure. Is there any other methods that have not already been mentioned? There must be other shops out there wanting to recover their DCs to a test setup - how are you doing it? what works for you? We shouldnt really need more doco on this because in my network, DCs have only ever restored if veeam has done it 100% on its own...ive never got a test DC working properly with manual tweaks/commands after the recovery.....so it seems that Veeam can do it but perhaps someone has ideas about why it doesnt do it every single time? what problems can cause it? how we can detect/prevent those etc etc?

Im pretty confident that veeam would work if say one of my DCs died then i did a recovery with veeam to the production network....veeam would recover the DC as non-auth and that recovered DC would then replicate with the working production DC and everything would work and go back to normal......but i cant really test that without risking everything. Because if testing that caused major problems, its not like i could just easily recover both my DCs from a backup before the test and get everything back to normal......i should be able to, but my testing shows this to not be the case....as does others who have bothered to test DC recovery in this post....its not very reliable.

Thanks for reading if you got this far Hopefully no one takes this as me 'veeam bashing' because i in no way intend that and hope i dont come across like that. It is frustrating that i cant predictably/reliably recover my DCs easily and every time...but thats why im here talking to you guys, perhaps there is something i dont know, perhaps its something im doing wrong or something that can be improved....or perhaps veeam know about this stuff and really are working on improving the consistency of DC recovery???

Thanks guys..

[ejleipold]

I hate to say it, but I recover them using Instant Recovery and they work everytime <_< >_> Like I said I usually have to reboot them a couple times and wait 15 minutes or so, but they get there and I'm able to get into all the AD utils (users and computers etc) and add/remove things from the domain and operate as normal (All in my isolated dev environment of course). I'm not using anything fancy like SureBackup or anything, just Application Aware Image processing.

[Unison]

And your recoveries work 100% every single time....even if you have to do a few more reboots yourself after recovery? Have you ever recovered, rebooted 10 times and things still were not right so you kill the running instant recovery session and start it again with the same backup file and then bingo it works this time - or it just works for you after reboots?
Veeam will complete 2 reboots of the VM for you when it recovers.....how many reboots after that are you usually having to complete? is it the same number of reboots for all your DCs or some are worse than others?
Each time your DC reboots....what do you look at to make sure things are running correctly? Do you do anything in between reboots....or when it boots, you check its working right....if its not right, you just reboot straight away.....and continue that cycle until it finally boots and all is good?


Surebackup does complicate things which is why i prefer to use instant recovery....but veeam will tell you that instant recovery wont work...but surebackup will. they lie

Instant recovery works for me too sometimes....but i just wish it was all the time....i must admit though, i have rebooted the VM several times after recovery too when the DC was not working properly but that was probably only 2-3 reboots at most before i started the whole instant recovery again....perhaps i need to just reboot a few more times...i will wait for your responses on the above queries.
Also....my dcs are both 2003...so maybe 2008 recovers better????

Thanks again for posting

[ejleipold]

No worries, I'll try rebooting it a bunch of times next testing DRP testing day and see how it fares. It's pretty rock solid though, it usually requires a little bit of patience though, like I said I generally bring both DC's up (instant recovery) wait for about 15-20 minutes, then reboot them once and wait for another 15 minutes or so. If I still cant open AD users & computers I'll reboot them again and wait another 10 mins or so. I havent had it not work yet and I generally test recover them every couple of months as part of my DRP testing. When I recover my other servers into that environment I sometimes have to remove them from the domain and add them again due to trust issues, I guess that might be an issue in a real DRP situation (having to re-add all my desktops might be a bit of a pain in the ass). Oh yeah, I've also successfully recovered my Exchange 2010 server into that dev environment and had it all working too!

[Unison]

Ok great thanks - yes please i would really like to hear from you after your next DR test - pls post here or PM me!! Any detail you can get on the above queries would be great!!

I am going to try that this afternoon....i will recover BOTH my DCs together and keep rebooting them until things start working or i get to 10 reboots!! ....because as you know the reboots can take for ages!!

Yes i have experienced those trust issues that you mention.....sometimes if i am recovering other servers (non DCs) into the same test area as the recovered DCs then i have to remove those servers from the domain (delete them from AD, then move other servers to workgroup before connecting them to the test setup) and then all works fine....wonder why that happens or if there is a process the veeam recovery could complete to avoid that.

[Unison]

ejleipold! Thanks mate!
Ive just spent the afternoon testing what you said....how you recover and simply reboot several times then everything comes good. haha it actually works!

So surprised that veeam dont actually know this or wont openly tell us about it?!?@?!?

This afternoon i tried your approach.

# - Using Veeam Instant Recovery - i started the recovery of DC1 and DC2 from their most current backup files (for DC1 that was a few hrs ago, but DC2 that was about 12hrs ago....so the backups of each DC were several hrs apart).
# - Both DCs were attached to the same isolated TEST network.
# - I then booted both DCs at the same time for the first time after veeam mounted them.
# - They both booted to safe mode...did their thing, then rebooted....all on their own.
# - Both DCs then came back to the ctrl alt del screen and i was able to login to both of them with normal dom admin account.
# - DC1 (the main DC with all FSMO roles) was not working at this point (its sysvol/netlogon shares were non existent, AD wasnt working and event log shows that this DC is looking for other DCs)
# - DC2 (secondary DC) at this point, this DC is working perfectly. Its netlogon/sysvol shares are present, the login script ran, AD is launching, no bad errors in event log.
# - So at this point DC1 is not working, DC2 is working.
# - Then all of a sudden DC1 (which is not working properly) reboots on its own, right as im reading through events. So i assume this is a good sign.
# - DC1 comes back up and i login....everything is still broken and events still show that this DC1 doesnt think its a DC anymore and is looking for other DCs....it has been able to ping DC2 all along.
# - almost an hr passes....and DC1/DC2 have done nothing since, no more auto reboots and DC1 is still broken, DC2 is perfect.
# - So i decide to reboot DC1 manually for the first time - this is my fist intervention in the whole process.
# DC1 comes back online, i login and bam, the login script runs - i check and yep the sysvol and netlogon shares are now present, no errors in the log....and i check that i can add things to AD from both servers and the changes replicate....all is good.

So DC1 went through 3 full reboots (2 were automatic, 1 was me rebooting manually) and DC2 just had the one reboot that is done automatically by veeam during recovery.

After all this, i have my DCs and domain perfect, up and running in a test setup!

Thanks to ejleipold for your input man! Im going to test this a few more times with different recovery points and see if i can claim consistency with the recovery of DCs via this method using instant recovery!

[ejleipold]

Haha nice one! I'm glad I could help!

[zoltank]

Actually there is a solid answer

I saw that, but I'm asking about doing an authoritative restore on the database, not the sysvol.

[Unison]

# - Using Veeam Instant Recovery - i started the recovery of DC1 and DC2 from their most current backup files (for DC1 that was a few hrs ago, but DC2 that was about 12hrs ago....so the backups of each DC were several hrs apart).
# - Both DCs were attached to the same isolated TEST network.
# - I then booted both DCs at the same time for the first time after veeam mounted them.
# - They both booted to safe mode...did their thing, then rebooted....all on their own.
# - Both DCs then came back to the ctrl alt del screen and i was able to login to both of them with normal dom admin account.
# - DC1 (the main DC with all FSMO roles) was not working at this point (its sysvol/netlogon shares were non existent, AD wasnt working and event log shows that this DC is looking for other DCs)
# - DC2 (secondary DC) at this point, this DC is working perfectly. Its netlogon/sysvol shares are present, the login script ran, AD is launching, no bad errors in event log.
# - So at this point DC1 is not working, DC2 is working.
# - Then all of a sudden DC1 (which is not working properly) reboots on its own, right as im reading through events. So i assume this is a good sign.
# - DC1 comes back up and i login....everything is still broken and events still show that this DC1 doesnt think its a DC anymore and is looking for other DCs....it has been able to ping DC2 all along.
# - almost an hr passes....and DC1/DC2 have done nothing since, no more auto reboots and DC1 is still broken, DC2 is perfect.
# - So i decide to reboot DC1 manually for the first time - this is my fist intervention in the whole process.
# DC1 comes back online, i login and bam, the login script runs - i check and yep the sysvol and netlogon shares are now present, no errors in the log....and i check that i can add things to AD from both servers and the changes replicate....all is good.

Just to let you all know....i have tested this now 4 times with completely different recovery points each time and recovery has been successful 100% of the time!!!! So this process works if you are really trying to recover your DC or if you need your DCs running in a test lab setup. Basically, recover ALL your DCs together with instant recovery, sit back and wait for them to play nice.....in some of my tests both DCs auto rebooted twice and were perfect all on their own - i never even had to do a manual reboot for 2 of the tests so far. Other 2 tests i just had to reboot 1 DC just once manually and then all was perfect.

Thanks to ejleipold for chiming in with your experience. I had pretty much accepted that DC recovery with veeam was just not really reliably possible and had all but given up. Now i can repeat it successfully over and over and will conduct regular tests to ensure it is always possible.

Now im going to setup replication using veeam to a spare host - this should also work fine....if i even need to bring the replica set online, i will just start with the two DCs - power them up together first and wait for them to work together before powering up other servers.

[ejleipold]

Patience is a virtue Glad I could help!

[mpierlot]

In reply to the post "2008 R2 DC backed up and run surebackup, DS Repair mode?", I have experienced the exact same thing. I have a Windows 2008 R2 domain controller in a test environment that I have backed up. I then created a Virtual lab and ran a SureBackup job, and when the domain controller boots up, it boots into Directory Services Restore mode, and you can't log into it or do anything with it. Since this is a test environment, I then took down the running domain controller and performed an Instant Recovery of the domain controller and it boots up fine and into normal running mode. Not sure what's happening here, but without this, I can't perform a restore of Exchange items.

[okrehan]

Hi,

I've installed a new Windows Server 2012 Domain Controller in our vSphere 5.1 environment and back it up with Veeam B&R 6.5 patch3. Application aware processing is enabled and backup job runs without any errors.
When I create a Sure Backup job to test the DC or to restore AD items with the AIR wizard, the SureBackup job is unsuccessful because it neither gets heartbeat nor network(ping) access to the VM.
I checked the VM during boot up and found out that no matter what you do the DC starts to DSrepair mode where neither VMtools nor network is available. I was unable to force the DC to start normally, it always boots directly to DSrepair mode.
Did a search on the internet and found that you should be able to remove booting into DSrepair by running bcdedit /deletevalue safeboot inside the SureBackup VM but this command doesn't complete successfully.
By the way, I think Sure Backup jobs should run without user intervention or they are quite useless

So anyone can help with this issue? Haven't found any information that Server 2012 domain controllers aren't supported with Veeam 6.5...

Regards,
Oliver

[nefes]

Oliver, have you selected Domain Controller role in SureBackup job for this VM?
Also, is that DC the only in the domain or there are several others in it?

[okrehan]

Hi,

sorry for the late reply, the forum didn't sent me an email as requested when someone sends an answer....
So, this is the only DC in the environment (it is only a test env.) and I have set the role to DC, Global Catalog and DNS server.

Regards,
Oliver

[foggy]

Oliver, did you get a chance to try recover this DC using Instant Recovery? Have you contacted support with that?

[james575]

I spent quite a bit of time trying to successfully test my replicated domain controllers at my DR site. I read through this whole thread, among others. I finally succeeded. As it seems like many people have had difficulty with this, I thought I would share my experience in case it can help others.

Two replica DC's trying to just power them up on their own vSwitch at the DR site and test them with a domain workstation over there. The problem was that active directory would not work. I did the following:

Powered each one up, logged in as local administrator. They said they had an abnormal shutdown. They both booted up in safe mode, so I went into msconfig and turned that off. They wanted to restart so I let them.

Then...
Command line: net stop ntfrs
Regedit: hklm\SYSTEM\CurrentControlSet\Services\NtFrs\Parameters\Backup/Restore\Process at Startup - Set the burflags to d4
Command line: net start ntfrs

After a minute or two, I saw my sysvol and netlogon being shared and all was fine! That was all I had to do. Of course it took a long time to get that process together from all the information in here.

[Unison]

You should DEFINITELY NOT have to use the burflags key to get your DCs and environment back up and running.
If i had to do that in a real disaster and got my environment back up and running with that key set, i wouldnt trust the AD recovery/environment and would fear that it would eventually fail in some horrible way.

I have tested my DC recoveries many times over and have got several comments in this thread - ive tried the burflags solution too and a million other things but in the end, i have found and given in to the fact that you should not have to do anything at all when you recover DCs - difficult for me to believe at first, but regular testing of DC recovery has proven to me that this is true. When you recover your DCs with Veeam (and probably other packages do this too) - you shouldn't have to do anything - the veeam recovery process prepares the process and your server OS's will do the rest.

When i moved to a full 2008R2 DC setup, i ran into problems again with this. My VMs were using the VMXNET3 NICs and this was causing massive problems with DC recovery - this issue only impacts 08R2 that are using VMXNET3 NICs. What was happening was that after restoring 08R2 DCs, their NIC details were all cleared so they lost their static details - so when i brought DCs online in the recovery environment, the could not reach each other until i manually went in and set their NIC details to the correct values....but by the time i could do this, the internal recovery process of AD appeared to be 'buggered' - DC recovery was unreliable. Then i found out that this is a known issue with 08R2 and MS have released a fix.....which i applied.....and now my 08DCs recover perfectly every single time without me having to do a thing (so long as i recover 2 of my DCs together - if you try recover just one DC on its own then you get in to the territory of needing to set reg keys, stop services etc). MS fix... http://support.microsoft.com/kb/2550978

If you want to recover your DCs for real or for a test - kick off the recovery of both your DCs at the same time, so that they come up online at the same time together on the same isolated network.....they will talk to each other and sort themselves out.
I have even tested this with recovering DCs at the same time but with a time difference between their recovery points and this also works successfully every time (now that i have applied the MS fix).

If anyone takes anything away from this now entire/massive thread its that you should not have to intervene in the DC recovery process at all - that's the way its designed.

[tsightler]

Powered each one up, logged in as local administrator. They said they had an abnormal shutdown. They both booted up in safe mode, so I went into msconfig and turned that off. They wanted to restart so I let them.

Correct, DC's are supposed boot up in safe mode (technically Directory Services Restore Mode), and Veeam should handle everything automatically at that point, typically after a delay of anywhere from 5-15 minutes based on the size of the environment. The DC should then automatically reboot without you having to do anything and come up in normal mode. If that's not happening then you should probably contact support as having to manually change thing with msconfig and burflags should not be required.

[peteclem]

Hey Unison,

This post caught my eye. I've recently done a DR test on a handful of servers at a head office site. The difficulty I had was with AD replication. I don't want to spin this topic off-course, but I'd be really keen to know if you or anyone has had the same issue or advice how to deal with it better than I did.

In total there were about 15 DCs for the remote sites. I restored a single DC from the head office site in an isolated DR test, and it took about 2 hours to come online (for AD and DNS services to come online). From what I could see the services weren't starting as they were trying to sync with all the remote sites before timing out (after 2 hours) and going online. The way I got around this was to disable inbound and outbound replication (using repadmin). This is fine for a single DC DR recovery test, but doesn't scale if I needed to restore 2 or more of the 15 DCs for the isolated DR test.

Your post mentioned services/reg keys that could be used in the DR test scenario. I appreciate that in a real DR situation, all the other DCs will be available and things should be much simpler. My problem seemed to occur because of the fact the other DCs weren't contactable and that caused delays in getting services back online.

Anyone seen/experienced something similar? So much material online assumes that all other DCs are available as if it were a real DR, and not an isolated test.

Pete

[Jonathan]

after restoring 08R2 DCs, their NIC details were all cleared so they lost their static details

I went through the same experience building an lab setup. Funny thing is, the DCs could still reach eachother through ipv6, just not ipv4.
This issue didn't come up with 2012 DCs (either upgraded or fresh installs).

[Unison]

haha - yes, thats right about ipv6. For a second there when i first noticed the DCs could talk even though there were no NIC (ipv4) details set - i thought, "this is weird" .
Good to note this fact here because some people might be wondering why they can still ping the DCs by name and still have these problems.

[Andreas Neufert]

This good old known lost IP address bug at restore hit all Win2008R2 on VMware Users hard at restore.
Because the Microsoft patch for this isn´t in the normal autoinstall patch rotation, VMware changed back the default NIC for new Win2008R2 VMs from VMXnet3 (vSphere4.x) to PRO1000 (vSphere5.x).

There are 2 patches available from Microsoft:
Win2008R2 without SP1
http://support.microsoft.com/kb/2344941
Win2008R2 with SP1
http://support.microsoft.com/kb/2550978

This bug is a really great example how SureBackup can help to identify problems at restore.
Even today 4 out of 5 of my customers who speak with me at the first Veeam live demo meetings, didn´t know about that patch and we can show them at POC that Veeam are able to detect that Microsoft bug and protect them for restore problems.

[Jonathan]

Because the Microsoft patch for this isn´t in the normal autoinstall patch rotation, VMware changed back the default NIC for new Win2008R2 VMs from VMXnet3 (vSphere4.x) to PRO1000 (vSphere5.x).

Yeah, I noticed that. It's just that I ALWAYS have to use the latest bleeding edge features in production VMs. VMXNET 3, paravirtual, vmx-09, WS 2012... until something goes completely haywire.

This bug is a really great example how SureBackup can help to identify problems at restore.

+1

[ori]

Correct, DC's are supposed boot up in safe mode (technically Directory Services Restore Mode), and Veeam should handle everything automatically at that point, typically after a delay of anywhere from 5-15 minutes based on the size of the environment. The DC should then automatically reboot without you having to do anything and come up in normal mode. If that's not happening then you should probably contact support as having to manually change thing with msconfig and burflags should not be required.

What you describe is just not happening for us (and also for others as it seems for this thread). If we don't set the burflags key, the ntfrs removes the sysvol and netlogon shares after the file replication process does not succeed. It turns that loading the domain controllers is a major time consumer in our DR drills.

[Andreas Neufert]

In General the follwing things needed for AD Backup/REstore of a whole VM:
The AD VM you want to backup needs to be a Global Catalog
In the Veeam Job, use Veeam VSS Processing including an Domain\(Administrator Account) and passoword. (VMware Tools quiescense or Crash consistent will not fit)
At Restore or in the Virtual Lab connect the Network (without the Server maybe not jump in restore mode)
Do not login to the VM or Focus the mouse on the VM. This can Interrupt the recover process.
As Tom said it took some time till the Server is back. Try to start the VM in our Virtual Lab including appliation Check for GC/DNS/AD. The time it took here, the Server likely Need in production at restore till the AD Service are back.
The Server jumps into non authorative restore mode.

I will test the mentioned sysvol replication thing later this week an come back here.

[james575]

Unison, I saw people in this thread talking about leaving everything alone and Veeam would take care of the DC/AD restores, but it did not at our site. I did however always power on one DC, wait a little while, and then power on the second DC. For a final test, I just powered them both on at the same time, will leave them running all night, and check them in the morning when I arrive.

For the record, I opened a case with Veeam support and they told me to change the burflags.

[Unison]

Hi James,
Originally, this is how i used to try DC recovery - exactly as you have mentioned. I would fire up one DC then wait a good 10 minutes before firing up the second DC recovery. I have since discovered that this is not the best way to do it.
What i believe is the correct way and the method that works for me every time is to recover at least 2 of your DCs together at the same time. For example.....

# recover both DCs using instant recovery but choose not to power them on
# When you see both DCs appear in vmware - change their networking details so that they are on an isolated network - but both are connected to the same isolated network!
# power them both on together as quickly as you can - i usually power them both up within a couple of seconds of each other
# they will both boot up in directory restore mode - during this time, they will both try to contact another DC on the network and because you have recovered 2 DCs at the same time, this process will succeed.
# they will both reboot on their own and after this point you should be able to login to them normally and AD should be up and running.
# If your two recovered DCs are not completely online and happy within 30-60mins from the start of your recovery then it is my experience that they will never come back to a working state without serious intervention like editing AD, setting reg keys etc etc.........so after 60mins, if your DCs are not perfect, give up on that recovery test because it failed.....start again.

I have now just got into the habit of recovering my 2 DCs every time i want to recover any of my app servers for what ever reason - be it just a test or to trial some new software install or an update. For example, if i want to test a new update on AppServer1 - i will first recover both my DCs to the isolated network first, and about 30mins later after they are happy and talking, then i will recover AppServer1 into that environment - then i will test the update. This way DNS, AD, everything is working as it does in the production setup.

The DC recovery process is very smooth and 'easy'. Some of my biggest problems were happening because i was not recovering 2 DCs together at the same time, like you i was recovering one, then waiting some time, then recovering the other - this was resulting in unreliable recovery (sometimes it would work other times not). My other biggest issue was the VMXNET3 NIC issue with 08R2. Once i applied the MS update i mentioned above and started recovering 2 DCs together at the same time - i get perfect DC recovery every time.

Are any of your DCs 08R2? If so, are they using VMXNET3 NICs?