Comprehensive data protection for all workloads
Post Reply
zoltank
Expert
Posts: 221
Liked: 33 times
Joined: Feb 18, 2011 5:01 pm
Contact:

DR: Restoring first domain controller

Post by zoltank » Sep 11, 2012 7:04 pm

[merged]

In a hypothetical DR situation where you have more than one domain controller you're restoring, after restoring the first domain controller do you need to perform an authoritative restore of the AD database before restoring addition domain controllers?

I've read through the domain controller restore thread, but it didn't seem like there was a solid answer provided.

For reference the authoritative restore procedure:
- Open a command prompt and type ntdsutil and then press ENTER.
- At the ntdsutil: prompt, type authoritative restore and then press ENTER.
- At the ntdsutil authoritative restore: prompt, type restore database and press ENTER.
- At the Authoritative Restore Confirmation dialog box, click OK.
- Type quit and press ENTER until you have exited Ntdsutil.exe.

Vitaliy S.
Product Manager
Posts: 22311
Liked: 1415 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by Vitaliy S. » Sep 11, 2012 9:41 pm

Actually there is a solid answer ;)
foggy wrote:This is not required in the case of a single DC recovery. Though you do need to perform authoritative SYSVOL restore on the first DC in case of restoring the whole Active Directory. Here are more good topics on that: Multiple Domain Controllers - How to Backup? and Active Directory and DR Site.

ejleipold
Enthusiast
Posts: 62
Liked: 9 times
Joined: Oct 19, 2011 6:14 am
Full Name: Evan Leipold
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by ejleipold » Sep 12, 2012 4:13 pm 2 people like this post

I test recover both my DC's (both 2008 R2) into my dev environment every month, all I do is a normal recovery, boot them up, wait 15 minutes, reboot them both, wait another 15 minutes and they sync and off they go, happy as larry.

Unison
Enthusiast
Posts: 87
Liked: 16 times
Joined: Feb 17, 2012 6:02 am
Full Name: Gav
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by Unison » Sep 13, 2012 3:31 am

ejleipold wrote:I test recover both my DC's (both 2008 R2) into my dev environment every month, all I do is a normal recovery, boot them up, wait 15 minutes, reboot them both, wait another 15 minutes and they sync and off they go, happy as larry.
Sry for long post guys, but anyone trying DC recovery may be able to relate...

When you say 'normal recovery' - what exactly do you mean? Because if you mean "Veeam Instant Recovery" then dont say that to anyone at Veeam because they will tell you thats impossible. If your starting both your DCs via instant recovery and just loading them to an isolated network - both DCs should boot to non authoritative mode....which will basically break your AD/domain. Apparently.

Ive just been working with veeam for the last few weeks on a case all about DC recovery. I personally have found the process of recovering a DC with veeam to be....inconsistent. Seems that is fairly normal among other veeam users too. Maybe DC recovery will be more robust and handled a bit better in future releases.
I say "inconsistent" because sometimes the recovery of a DC works, sometimes it doesnt, sometimes you have to do 'this' as an extra step, sometimes 'that' etc and then almost always you have the issue that this post is mainly talking about - authoritative vs non authoritative restore and the fact that veeam doesnt allow you to choose which to do. Instead we are just told that 99% of the time all you will need to do is a non-auth restore so dont worry about the fact that veeam doesnt let you choose....or just follow xyz if you want an auth restore...where xyz doesnt really work and is not very 'clean'.....not exactly the perfect recovery of your DC/domain that you would expect anyway. And none of that helps if you just want to recover your DCs to a test setup for any purpose you require.

However, there is a method to get veeam to to boot your DC into authoritative mode.....its called Surebackup. If you put one of your multiple DCs into a surebackup job, veeam will do its magic and make that one DC boot into auth mode so your DC comes up working perfect, your domain is good and AD is good....sometimes. Ive seen it work and seen it fail...all with the same backup file...just running the same job over and over to check consistency of recovery. So surebackup will boot your DC into auth mode (if you set it in the app group to be a DC and GC server - doing this is what makes veeam run certain scripts and prep work on the DC, veeam support tells me) - if you want to test your DCs in an isolated setup.

What i find the most inconsistent is the recovery of a DC using Veeam Instant Recovery. Its mentioned all over this forum and call up veeam and ask them about this.....recovery of just ONE or ALL of your DCs to an isolated environment using Veeam Instant Recovery WILL FAIL - IT CANT WORK......but again, sometimes it does :)
I asked veeam about restoring just one of my DCs to a test setup as i was wanting to try something with software in an isolated environment - so i needed my DC, DNS and the other server. I told them that it wasnt working with Instant Recovery. They said "no, it wont work with instant recovery because IR will boot your DC into non-auth mode so your DC wont act like a DC. Instead, use Surebackup to setup the test environment"......and that was the basis for my support case which lasted weeks.
During that time we were using instant recovery and surebackup to try and achieve this setup - with the same backup files. Sometimes surebackup would work, sometimes it wouldnt. Sometimes instant recovery would work, sometimes it wouldnt....inconsistent.

When i first reported to veeam that "i got my one DC and one app server working together using INSTANT RECOVERY in an isolated environment with no use of burflags or ntdsutil etc", they didnt believe me.....so i showed them....and they couldnt explain it.....how can a DC successfully recover if its booting in non-auth mode and has no contact with other DCs???? Simple, it cant....veeam must be in control and doing something to make this happen....so why wont it do this all the time? I can recover just ONE of my multiple DCs to an isolated environment using instant recovery, veeam boots it first time aound, does its magic then boots it again into normal mode where i can login with a domain account - i can then see AD is working, netlogon/sysvol shares are working, no errors in event log (like there are when DC recovery fails...you all know those!).....the only exception is that i have to seize the RID master role from my other DC depending on which one i recover so that i can create new objects in AD....thats it. My domain, DC, AD is all working in a test setup!

When i have recovered a DC and it fails...i have tried solutions offered in this post - use ntdsutil to force the DC to boot into auth mode, stop the replication service, edit burflags reg key etc etc but even those solutions have not worked. Only way i have seen one of my DCs boot successfully was when i never made ANY modification during or after the recovery.....i just let instant recovery or surebackup do its thing....didnt change a thing, didnt run any commands after recovery....it just either works on its own or it fails and i run the same job again until it works. No manual process as per posts in here or MS articles has ever got one of my DCs working right again in a test lab....the only thing that has done it has been VEEAM successfully....both in instant recovery and surebackup. I just wish this was way more reliable.

Ive also heard the argument about 3rd party interference impacting recovery. If the recovery doesn't go well, its not veeam or the recovery process....it must be something else interfering with the recovery like your AV or some other product. Wonder if that has ever been proven, veeam staff couldnt find anything here that would do it and if something like that was interfering, then it would interfere every time, not just sometimes cause a problem then other times not.

So now im going through this thread, after my support case with veeam trying to find out if im doing something wrong or if there is a true tried and tested way of backing up and recovering your DCs........and i still feel way uneasy about recovering my DCs. Hopefully i never need to do it for real because it seems very inconsistent and a bit of a lucky dip when it comes to success/failure. Is there any other methods that have not already been mentioned? There must be other shops out there wanting to recover their DCs to a test setup - how are you doing it? what works for you? We shouldnt really need more doco on this because in my network, DCs have only ever restored if veeam has done it 100% on its own...ive never got a test DC working properly with manual tweaks/commands after the recovery.....so it seems that Veeam can do it but perhaps someone has ideas about why it doesnt do it every single time? what problems can cause it? how we can detect/prevent those etc etc?

Im pretty confident that veeam would work if say one of my DCs died then i did a recovery with veeam to the production network....veeam would recover the DC as non-auth and that recovered DC would then replicate with the working production DC and everything would work and go back to normal......but i cant really test that without risking everything. Because if testing that caused major problems, its not like i could just easily recover both my DCs from a backup before the test and get everything back to normal......i should be able to, but my testing shows this to not be the case....as does others who have bothered to test DC recovery in this post....its not very reliable.

Thanks for reading if you got this far :) Hopefully no one takes this as me 'veeam bashing' because i in no way intend that and hope i dont come across like that. It is frustrating that i cant predictably/reliably recover my DCs easily and every time...but thats why im here talking to you guys, perhaps there is something i dont know, perhaps its something im doing wrong or something that can be improved....or perhaps veeam know about this stuff and really are working on improving the consistency of DC recovery???

Thanks guys..

ejleipold
Enthusiast
Posts: 62
Liked: 9 times
Joined: Oct 19, 2011 6:14 am
Full Name: Evan Leipold
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by ejleipold » Sep 13, 2012 3:39 am

I hate to say it, but I recover them using Instant Recovery and they work everytime <_< >_> Like I said I usually have to reboot them a couple times and wait 15 minutes or so, but they get there and I'm able to get into all the AD utils (users and computers etc) and add/remove things from the domain and operate as normal (All in my isolated dev environment of course). I'm not using anything fancy like SureBackup or anything, just Application Aware Image processing.

Unison
Enthusiast
Posts: 87
Liked: 16 times
Joined: Feb 17, 2012 6:02 am
Full Name: Gav
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by Unison » Sep 13, 2012 3:52 am

And your recoveries work 100% every single time....even if you have to do a few more reboots yourself after recovery? Have you ever recovered, rebooted 10 times and things still were not right so you kill the running instant recovery session and start it again with the same backup file and then bingo it works this time - or it just works for you after reboots?
Veeam will complete 2 reboots of the VM for you when it recovers.....how many reboots after that are you usually having to complete? is it the same number of reboots for all your DCs or some are worse than others?
Each time your DC reboots....what do you look at to make sure things are running correctly? Do you do anything in between reboots....or when it boots, you check its working right....if its not right, you just reboot straight away.....and continue that cycle until it finally boots and all is good?


Surebackup does complicate things which is why i prefer to use instant recovery....but veeam will tell you that instant recovery wont work...but surebackup will. they lie :)

Instant recovery works for me too sometimes....but i just wish it was all the time....i must admit though, i have rebooted the VM several times after recovery too when the DC was not working properly but that was probably only 2-3 reboots at most before i started the whole instant recovery again....perhaps i need to just reboot a few more times...i will wait for your responses on the above queries.
Also....my dcs are both 2003...so maybe 2008 recovers better????

Thanks again for posting :)

ejleipold
Enthusiast
Posts: 62
Liked: 9 times
Joined: Oct 19, 2011 6:14 am
Full Name: Evan Leipold
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by ejleipold » Sep 13, 2012 4:02 am

No worries, I'll try rebooting it a bunch of times next testing DRP testing day and see how it fares. It's pretty rock solid though, it usually requires a little bit of patience though, like I said I generally bring both DC's up (instant recovery) wait for about 15-20 minutes, then reboot them once and wait for another 15 minutes or so. If I still cant open AD users & computers I'll reboot them again and wait another 10 mins or so. I havent had it not work yet and I generally test recover them every couple of months as part of my DRP testing. When I recover my other servers into that environment I sometimes have to remove them from the domain and add them again due to trust issues, I guess that might be an issue in a real DRP situation (having to re-add all my desktops might be a bit of a pain in the ass). Oh yeah, I've also successfully recovered my Exchange 2010 server into that dev environment and had it all working too!

Unison
Enthusiast
Posts: 87
Liked: 16 times
Joined: Feb 17, 2012 6:02 am
Full Name: Gav
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by Unison » Sep 13, 2012 4:24 am

Ok great thanks - yes please i would really like to hear from you after your next DR test - pls post here or PM me!! :) Any detail you can get on the above queries would be great!!

I am going to try that this afternoon....i will recover BOTH my DCs together and keep rebooting them until things start working or i get to 10 reboots!! :)....because as you know the reboots can take for ages!!

Yes i have experienced those trust issues that you mention.....sometimes if i am recovering other servers (non DCs) into the same test area as the recovered DCs then i have to remove those servers from the domain (delete them from AD, then move other servers to workgroup before connecting them to the test setup) and then all works fine....wonder why that happens or if there is a process the veeam recovery could complete to avoid that.

Unison
Enthusiast
Posts: 87
Liked: 16 times
Joined: Feb 17, 2012 6:02 am
Full Name: Gav
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by Unison » Sep 13, 2012 6:10 am

ejleipold! :) Thanks mate!
Ive just spent the afternoon testing what you said....how you recover and simply reboot several times then everything comes good. haha it actually works! :)

So surprised that veeam dont actually know this or wont openly tell us about it?!?@?!?

This afternoon i tried your approach.

# - Using Veeam Instant Recovery - i started the recovery of DC1 and DC2 from their most current backup files (for DC1 that was a few hrs ago, but DC2 that was about 12hrs ago....so the backups of each DC were several hrs apart).
# - Both DCs were attached to the same isolated TEST network.
# - I then booted both DCs at the same time for the first time after veeam mounted them.
# - They both booted to safe mode...did their thing, then rebooted....all on their own.
# - Both DCs then came back to the ctrl alt del screen and i was able to login to both of them with normal dom admin account.
# - DC1 (the main DC with all FSMO roles) was not working at this point (its sysvol/netlogon shares were non existent, AD wasnt working and event log shows that this DC is looking for other DCs)
# - DC2 (secondary DC) at this point, this DC is working perfectly. Its netlogon/sysvol shares are present, the login script ran, AD is launching, no bad errors in event log.
# - So at this point DC1 is not working, DC2 is working.
# - Then all of a sudden DC1 (which is not working properly) reboots on its own, right as im reading through events. So i assume this is a good sign.
# - DC1 comes back up and i login....everything is still broken and events still show that this DC1 doesnt think its a DC anymore and is looking for other DCs....it has been able to ping DC2 all along.
# - almost an hr passes....and DC1/DC2 have done nothing since, no more auto reboots and DC1 is still broken, DC2 is perfect.
# - So i decide to reboot DC1 manually for the first time - this is my fist intervention in the whole process.
# DC1 comes back online, i login and bam, the login script runs - i check and yep the sysvol and netlogon shares are now present, no errors in the log....and i check that i can add things to AD from both servers and the changes replicate....all is good.

So DC1 went through 3 full reboots (2 were automatic, 1 was me rebooting manually) and DC2 just had the one reboot that is done automatically by veeam during recovery.

After all this, i have my DCs and domain perfect, up and running in a test setup!

Thanks to ejleipold for your input man! Im going to test this a few more times with different recovery points and see if i can claim consistency with the recovery of DCs via this method using instant recovery!

ejleipold
Enthusiast
Posts: 62
Liked: 9 times
Joined: Oct 19, 2011 6:14 am
Full Name: Evan Leipold
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by ejleipold » Sep 13, 2012 6:13 am

Haha nice one! I'm glad I could help!

zoltank
Expert
Posts: 221
Liked: 33 times
Joined: Feb 18, 2011 5:01 pm
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by zoltank » Sep 13, 2012 8:30 pm

Vitaliy S. wrote:Actually there is a solid answer ;)
I saw that, but I'm asking about doing an authoritative restore on the database, not the sysvol.

Unison
Enthusiast
Posts: 87
Liked: 16 times
Joined: Feb 17, 2012 6:02 am
Full Name: Gav
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by Unison » Sep 14, 2012 1:37 am 1 person likes this post

Unison wrote: # - Using Veeam Instant Recovery - i started the recovery of DC1 and DC2 from their most current backup files (for DC1 that was a few hrs ago, but DC2 that was about 12hrs ago....so the backups of each DC were several hrs apart).
# - Both DCs were attached to the same isolated TEST network.
# - I then booted both DCs at the same time for the first time after veeam mounted them.
# - They both booted to safe mode...did their thing, then rebooted....all on their own.
# - Both DCs then came back to the ctrl alt del screen and i was able to login to both of them with normal dom admin account.
# - DC1 (the main DC with all FSMO roles) was not working at this point (its sysvol/netlogon shares were non existent, AD wasnt working and event log shows that this DC is looking for other DCs)
# - DC2 (secondary DC) at this point, this DC is working perfectly. Its netlogon/sysvol shares are present, the login script ran, AD is launching, no bad errors in event log.
# - So at this point DC1 is not working, DC2 is working.
# - Then all of a sudden DC1 (which is not working properly) reboots on its own, right as im reading through events. So i assume this is a good sign.
# - DC1 comes back up and i login....everything is still broken and events still show that this DC1 doesnt think its a DC anymore and is looking for other DCs....it has been able to ping DC2 all along.
# - almost an hr passes....and DC1/DC2 have done nothing since, no more auto reboots and DC1 is still broken, DC2 is perfect.
# - So i decide to reboot DC1 manually for the first time - this is my fist intervention in the whole process.
# DC1 comes back online, i login and bam, the login script runs - i check and yep the sysvol and netlogon shares are now present, no errors in the log....and i check that i can add things to AD from both servers and the changes replicate....all is good.
Just to let you all know....i have tested this now 4 times with completely different recovery points each time and recovery has been successful 100% of the time!!!! So this process works if you are really trying to recover your DC or if you need your DCs running in a test lab setup. Basically, recover ALL your DCs together with instant recovery, sit back and wait for them to play nice.....in some of my tests both DCs auto rebooted twice and were perfect all on their own - i never even had to do a manual reboot for 2 of the tests so far. Other 2 tests i just had to reboot 1 DC just once manually and then all was perfect.

Thanks to ejleipold for chiming in with your experience. I had pretty much accepted that DC recovery with veeam was just not really reliably possible and had all but given up. Now i can repeat it successfully over and over and will conduct regular tests to ensure it is always possible.

Now im going to setup replication using veeam to a spare host - this should also work fine....if i even need to bring the replica set online, i will just start with the two DCs - power them up together first and wait for them to work together before powering up other servers.

ejleipold
Enthusiast
Posts: 62
Liked: 9 times
Joined: Oct 19, 2011 6:14 am
Full Name: Evan Leipold
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by ejleipold » Sep 14, 2012 1:43 am

Patience is a virtue :) Glad I could help!

mpierlot
Lurker
Posts: 1
Liked: never
Joined: Sep 24, 2012 2:25 pm
Full Name: Mike Pierlot
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by mpierlot » Sep 24, 2012 2:35 pm

In reply to the post "2008 R2 DC backed up and run surebackup, DS Repair mode?", I have experienced the exact same thing. I have a Windows 2008 R2 domain controller in a test environment that I have backed up. I then created a Virtual lab and ran a SureBackup job, and when the domain controller boots up, it boots into Directory Services Restore mode, and you can't log into it or do anything with it. Since this is a test environment, I then took down the running domain controller and performed an Instant Recovery of the domain controller and it boots up fine and into normal running mode. Not sure what's happening here, but without this, I can't perform a restore of Exchange items.

okrehan
Service Provider
Posts: 13
Liked: never
Joined: Mar 17, 2011 10:53 am
Full Name: Oliver Krehan
Contact:

[MERGED] SureBackup of Windows 2012 Domain Controller

Post by okrehan » May 23, 2013 12:41 pm

Hi,

I've installed a new Windows Server 2012 Domain Controller in our vSphere 5.1 environment and back it up with Veeam B&R 6.5 patch3. Application aware processing is enabled and backup job runs without any errors.
When I create a Sure Backup job to test the DC or to restore AD items with the AIR wizard, the SureBackup job is unsuccessful because it neither gets heartbeat nor network(ping) access to the VM.
I checked the VM during boot up and found out that no matter what you do the DC starts to DSrepair mode where neither VMtools nor network is available. I was unable to force the DC to start normally, it always boots directly to DSrepair mode.
Did a search on the internet and found that you should be able to remove booting into DSrepair by running bcdedit /deletevalue safeboot inside the SureBackup VM but this command doesn't complete successfully.
By the way, I think Sure Backup jobs should run without user intervention or they are quite useless :-)

So anyone can help with this issue? Haven't found any information that Server 2012 domain controllers aren't supported with Veeam 6.5...

Regards,
Oliver

Post Reply

Who is online

Users browsing this forum: Cullan, Karinne and 14 guests