Veeam B&R recovery of a domain controller

[Jfmoots]

My VSS enabled restores of domain controllers boot to DSRM, then reboot, and are good to go.

[rawtaz]

And you just did a regular full restore (including registration)?

I wonder if it could be caused by having restored to a new installation of ESXi on the same host (so I had to select a new datastore and I also set a new name for the VMs). Seems like something that shouldn't matter though.

Alright, so assuming this hasn't worked like it should, what would be the way to have it dealt with to get the DC running? Should I boot it up in DSRM myself and do whatever one does in there, and then it should be fine after a reboot? I guess this would be the sensible way to deal with it, aside from having support look at the issue of course (which I don't fancy at the moment as I would like to get going with this DC ASAP). On the other hand I have read at least once that in 2008 the DSRM is no longer needed except for in some special situations.

[foggy]

Reading through this topic, DC recovery is fully automated and does not require any user interaction. The restored VM should first boot in safe mode and then reboot automatically immediately to boot up next time normally.

[Jfmoots]

Alright, so assuming this hasn't worked like it should, what would be the way to have it dealt with to get the DC running? Should I boot it up in DSRM myself and do whatever one does in there, and then it should be fine after a reboot? I guess this would be the sensible way to deal with it, aside from having support look at the issue of course (which I don't fancy at the moment as I would like to get going with this DC ASAP). On the other hand I have read at least once that in 2008 the DSRM is no longer needed except for in some special situations.

You're on the right track here. Booting in to DSRM will "check-in" with other running AD Controllers and update the DB. After that, it's my understanding, that your DC is good to go.

[rawtaz]

Ok, status update (for the lazy one: problem solved, Veeam does what it should do, but documentation/instructions for this Automatic Process should be updated/provided):

- Veeam seems to do the job it is claiming to do, i.e. automatically takes care of the final pieces of restoring a 2008 R2 Domain Controller so that you can just power it on after restore and then let it be and once Veeam is finished with it you can login and it is good to go.

- There was some confusion due to the facts that Veeam documentation on this topic is lacking (in my and others opinion) and that Veeam representatives have (in this thread) claimed that the process should automatically first boot the system in *safe* mode before doing the rest of its work, and this does not seem to be accurate. On first boot after restore, it counts down to booting *normal* mode, not safe mode. When booting the machine and seeing it counting down for normal boot, I stopped the process and started investigating how this should really be done, and ended up with my post in this thread. Had there been an outline of the expected process in the user manual things would've been clearer, even if it is all automatic in the end.

- After not having seen the automatic process outlined by Veeam (I wasn't expecting the machine to boot to normal mode, as this was a DC that I thought needed special restoring) I started trying to reach DSRM myself, and at one point noticed the VM restarted itself (the first time this was at a login prompt and second time it was half a minute after the desktop had loaded when i logged in under safe mode). This made me think that there was still a chance that things would progress as intended, so I reverted to the preFirstBootAfterRestore snapshot, booted it and just let it be. Windows then counted down and started in normal mode, then at the login prompt which I didn't touch it rebooted itself once, after about 5-10 minutes. I assume this is after it had done some magic to polish up the DC/AD to be functional again. This time it booted normally (no "unexpected shutdown" screen displayed), and sat at the normal login prompt. I waited for to see if it would do something else, but after 20 minutes of no further activity I logged in and AD seems operational. I tried logging in from an RDS server (also restored with Veeam) that was joined to the domain of this DC and it works too.

- I couldn't agree more with daryanx in this reply where he requests that Veeam provide a step-by-step instruction or process outline for restoring a Windows DC. I too have spent several hours on this now, including booting and not seeing the expected (nor officially mentioned) result and thereby not knowing if things are still the way they should be or if I have a deviation from what is normal, and whether I should do something manually. Not to mention writing this summary up so that others might not need to do the same process over again.

- Again, we are not asking Veeam to supply technical details, all we ask is that Veeam provide a step-by-step instruction on what to expect when looking at the screen of a restore for a DC. Quote: "For example, is the dirty power off screen expected?, should you login to the machine?, how many times will the machine reboot?, do I need to take any additional steps to restore a domain controller?, what do I do if the restore fails or appears to fail?" - Things like that. It should be in the user manual for Backup & Recovery! Heck, if you guys don't take it on you to write one, I will. For the sake of the community, your users.

- In the end all things are well. Personally I did not encounter the issue that some people in this thread have reported, where the machine keeps booting into DSRM even after successful restoration of the VM and AD/DC.

Thanks for everyone's input!

PS: I changed the topic of this post, as the thread really applied to B&R v6 just as v5. Might want to update the initial post to reflect this or something, so the thread is more version neutral.

[aparker]

I'm running a trial of Veeam and so far very pleased with the results. This thread is of interest as I'm about to test recovery of our DCs and like others wondered how Veeam handled this but ended up here due to no guide. But I have some questions I feel haven't been addressed so far.

Do I need to be concerned with the following KBs when recovery DCs either in production or DR?

VMware KB 1000171

http://kb.vmware.com/selfservice/micros ... Id=1000171

Microsoft KB 888794

http://support.microsoft.com/kb/888794

My DR process is currently based on recovering everything from scratch (we do not have replicated DCs at our DR site). We have multiple DCs in our production environment.

Do I need to be aware of any issues with restoring multiple DCs in this scenario? I.e. I have 2 root DCs and 2 user DCs - can I restore them all without issue (bearing in mind the backups will be taken at different times - usually minutes apart but if a backup should fail then could be longer). Or should I only restore 1 root and 1 user, then build out new secondary root and user DCs?

Thanks for any advice on this.
Adrian

[Gostev]

If your backup was done with application-aware processing enabled in the backup job settings, then you just restore the DC VM normally and Veeam will take care of the rest. Thanks!

[Loki]

Hello there.


Just to add more experience into the discussion, here's our case.

One of our customers asked us a recovery test of vm backup that are daily made.
What we wanted to to was to restore a vm to test the functionality of the backups and in how much time it would take
to recover from a disaster.
We tried to perform a vm backup of our single microsoft SBS 2008 (our only DC).
The server is hosted on Vmware ESXi 4.1 and we used Veeam Backup 4.1.
Backup went good, but when we tested the saved vm we stumbled upon the (in)famous missing reboot.

We tried to log in as domain administrator but an error message told us that no server was available for logging.
Waited for the reboot that never came.
Then we logged in as local administrator, loaded msconfig and unchecked the box regarding the Active Directory Recovery, then we rebooted the server.

Domain is ok, dns and dhcp services are ok but there's nothing to do with the Exchange.

It appears the exchange database is in dirty state and a long series of Eseutils are necessary to bring things back on.

We're actually repairing the database and wondering if this is how things are supposed to go in the aftermath of a hypotetical disaster that would force us
to restore from a replica.

@Gostev: we've already contacted the support, exchanged a couple of email and send the veeam logs of the backup server.

Any hint, suggestion or impression is welcome!

Thank you.

Alessio

[velowulf]

Reading through this topic, DC recovery is fully automated and does not require any user interaction. The restored VM should first boot in safe mode and then reboot automatically immediately to boot up next time normally.

Point to note here, to avoid confusion of others who need to read this thread to reassure themselves of their DC restore. The machine boots to the "dirty shutdown" screen but DOESN'T then boot to safe mode - it actually boots normally. In other words the default option at the dirty shutdown screen is something like 'Load Windows Normally' and if the machine is not touched (as it shouldn't be) this is the boot option chosen. This is as per Veeam design although is not documented anywhere

[foggy]

We tried to log in as domain administrator but an error message told us that no server was available for logging.
Waited for the reboot that never came.

Alessio, you do not even need to log in, just let the VM to reboot automatically. As was stated in this topic many more times, the process of the DC restore is FULLY automated and you do NOT need to do ANYTHING (even try to log in until the VM is booted the second time normally). Logging in after the first boot brakes the whole restore process.

Point to note here, to avoid confusion of others who need to read this thread to reassure themselves of their DC restore. The machine boots to the "dirty shutdown" screen but DOESN'T then boot to safe mode - it actually boots normally. In other words the default option at the dirty shutdown screen is something like 'Load Windows Normally' and if the machine is not touched (as it shouldn't be) this is the boot option chosen. This is as per Veeam design although is not documented anywhere

Although the default choice is 'Load Windows Normally', Veeam B&R tweaks the startup mode to boot the VM in safe mode (for the first time).

[Loki]

Foggy wrote:
Alessio, you do not even need to log in, just let the VM to reboot automatically. As was stated in this topic many more times, the process of the DC restore is FULLY automated and you do NOT need to do ANYTHING (even try to log in until the VM is booted the second time normally). Logging in after the first boot brakes the whole restore process.

Hello Foggy,

My fault, I didn't make myself clear.
After the restore we let the VM alone for aprox 30 minutes. It didn't reboot, just stood there doing anything (we were checking CPU and disk usage on the Vmware Console).
Then we tried to log in using DC administrator credential and got the message that no server was available for logging.
After several research i stumbled upon this post where i've found the tip of loggin' as local administrator and uncheck the ADR box.
(Which seems to me the same solution veeam support team suggests in this KB: http://www.veeam.com/kb_articles.html/KB1277 even if we used Gui instead of CLI, am i wrong?)
We rebooted the server and services were back to normal. All except the Exchange database which was in Dirty State and needed check after check to be back to normal use.
Hope this would shed a little bit of light on our situation!

Thanks again

Alessio

[foggy]

After the restore we let the VM alone for aprox 30 minutes. It didn't reboot, just stood there doing anything (we were checking CPU and disk usage on the Vmware Console).

Then this is the reason to continue working with support on this case.

[davidb1234]

[merged]

I backed up a domain controller and used application aware image processing.

I created a sure backup job and ran it and the DC boots in to safe mode directory services restore mode. Is that correct?

[mwant]

[merged]

I have been testing DR of an Active Directory 2003 domain using Failover to replica. One of the things I have come across is the need to mark SYSVOL as authoritative (BurFlags) after the domain controllers have gone through the DSRM process. The process you need to go through to do this manually is complex and adds lots of time to the recovery.

In Backup Exec/Windows Backup you can mark SYSVOL as authoritative and it sorts it out for you. My question is can Veeam provide the same functionality and thus make the whole recovery process a lot easier and smoother?

[velowulf]

[merged]

I have been testing DR of an Active Directory 2003 domain using Failover to replica. One of the things I have come across is the need to mark SYSVOL as authoritative (BurFlags) after the domain controllers have gone through the DSRM process. The process you need to go through to do this manually is complex and adds lots of time to the recovery.

In Backup Exec/Windows Backup you can mark SYSVOL as authoritative and it sorts it out for you. My question is can Veeam provide the same functionality and thus make the whole recovery process a lot easier and smoother?

If this post is factual then I would also be very interested in hearing from Veeam on the answer to the question...

[foggy]

This is not required in the case of a single DC recovery. Though you do need to perform authoritative SYSVOL restore on the first DC in case of restoring the whole Active Directory. Here are more good topics on that: Multiple Domain Controllers - How to Backup? and Active Directory and DR Site.

[mwant]

I am refering to restore of the whole domain so yes sysvol restore is required and is frankly a bit of a pain to do manually so it would be very useful to be able to mark sysvol as authoritative as you could do in BE or windows backup as stated. This would be a good funtional addition to Veeam.

I have restored a domain in a live DR situation before and didn't have to mess around with SYSVOL as I used BE so was a bit confused initially when I uncovered the need for it.

I need some backup here from other Active Directory users....

[ori]

mwant, I'm with you on this one. Currently we are not using vss on our DCs in replication jobs because if we do, they become non-functional few minutes after we start them in our DR site. Adding this simple feature would be a great enhancement for us, and I'm sure that for many more.

[SoloIT]

[merged]

I'm trying to restore a single DC from a multi DC environment to a test server. I've restored the VM, let Veeam do the auto-reboot to do the non-authoritative restore, but the DC and AD are not functioning correctly. The sysvol is not being shared, PC's cannot join the domain, etc. I'm running 2008 R2. What is the proper method?

Thanks,
George

[tsightler]

I'm assuming your test servers is isolated from the other environment? How long have you waited? It can take about 15-30 minutes for the sysvol to share out while the system attempts to communicate with other replica partners. Veeam performs some "magic" to overcome this when we boot the DC in a vLab.

[hannisch]

I've one question regarding restoring a dc from Backup or starting a Replika. What is about USN rollback. Does Veeam sets the required registry Key in both situations, automaticly, or will I run into an USN rollback in a multi dc environment, after restore?

Sven

[Gostev]

You will not run into USN rollback, since Veeam performs DC backup and restore according to Microsoft requirements (using VSS).

[SoloIT]

I've let it sit all weekend to ensure the "magic" happened. However, it's still not sharing out the sysvol. I'm not sure if there is something wrong with the backup, or I'm doing something wrong. Guess I'll be calling support.

[SoloIT]

I've exchanged a few emails with tech support, and I'm getting nowhere. I know someone out there is smarter than me and has this figured out. I'm guessing my issue is due to coming from a multiple DC environment to the single DC test world.

[tsightler]

I've let it sit all weekend to ensure the "magic" happened. However, it's still not sharing out the sysvol. I'm not sure if there is something wrong with the backup, or I'm doing something wrong. Guess I'll be calling support.

This "magic" is only performed in a vLab, it doesn't sound like you are using a vLab. Are you backing up with Application Aware Processing enabled?

[dellock6]

The single DC you are trying to boot has all the FSMO roles in it, or are they on another server? Maybe some missing roles are causing the restored DC to hang somewhere. Also, is this DC also an authoritative DNS server for the active directory zone?

Luca.

[ori]

I've let it sit all weekend to ensure the "magic" happened. However, it's still not sharing out the sysvol. I'm not sure if there is something wrong with the backup, or I'm doing something wrong. Guess I'll be calling support.

The first time you start the replica, when the SYSVOL stops to be shared, if you have in the SYSVOL\sysvol\yourdomainname folder a folder named Ntfrs_PreExisting, try this:

- net stop ntfrs
- on the SYSVOL folder, move the content of the PreExisting folder to the root of the SYSVOL\sysvol\yourdomainname folder.
- set the "BurFlags" value in 'HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\NtFrs\Parameters\Backup/Restore\Process at Startup' key to "D4"
- net start ntfrs
- wait to see if SYSVOL is shared.

[SoloIT]

Thanks for all the info. Short version..I had some corruption in my Veeam backup. After doing a new full image, things are working better. However, I have documented my process to maybe help save others down the road. I may have a few extra or unnecessary steps.

1. Restore from Veeam.
2. Allow Veeam to auto-reboot machine. (this can take 30-45 minutes)
3. Copy %systemroot%\sysvol\domain (just in case you need them)
4. Seize all FSMO roles
run ntdsutil
roles
connections
connect to server [servername]
q
seize pdc
seize naming master
seize infrastructure master
seize rid master
seize schema master
q
q
5. Remove other DC refs
run ntdsutil
metadata cleanup
connections
connect to server [servername]
q
select operation target
list domains
select domain [domain number]
list sites
select site [site number]
list servers in site
select server [server number to remove]
q
remove selected server

repeate as necessary to remove other servers
6. Edit network setting to remove other DNS servers
7. Remove old servers from DNS server
including _msdcs
_ldap._tcp.[site].DomainDnsZones.[Domain]
_ldap._tcp.DomainDnsZones.[Domain]
_ldap._tcp.[site].ForestDnsZones.[Domain]
_ldap._tcp.ForestDnsZones.[domain]
8. Stop ntfrs server (net stop ntfrs)
9. Edit registry \HLM\SYSTEM\CurrentControlSet\services\NtFrs\Parameters\Backup/Restore\Process at Startup
BurFlags set to D4 Hex
10. Start ntfrs server (net start ntfrs)
11. Restart server and run dcdiag to ensure clean DC

[rawtaz]

I don't know all of that stuff above, but big kudos for taking the time to jot it down for others!

[zoltank]

Thanks for all the info. Short version..I had some corruption in my Veeam backup.

Do yo know what kind of corruption? What caused it?

What kind of backups were you doing? How long had it been since an Active Full backup?

Do you use SureBackup?