Restoring Windows 2008 Domain Controllers

[khughes]

We switched over to Veeam a wihle ago and have had nothing but good things with the software, the stability of the backups and restoring. This past weekend we had a D/R drill which was the first time since we made the switch to Veeam backing up our VMware environment.

While everything restored fine to the hosts, our domain was corrupted. It's being backed up via application aware so I would've thought that doing it that way would've prevented any sort of corruption issues. Should we be doing anything differently backing up the domain or should I have done anything differently restoring the domain?

I'm kind of at a loss of what went wrong as I've never had a failed D/R drill in the past 5 years until now. My initial thoughts are that the backup was just bad. Any other input would be gladly accepted.

[Gostev]

What specifically do you mean by "corrupted"? Did you restore all domain controllers during your DR test, or just one of them?

One thing for sure - if the restore went fine, then backup was good (exactly identical to what was obtained from the source VM). If the backup file was corrupted, restore would fail - since part of the restore process includes verifying checksums for each restored block (any corruption would make the restore process to fail).

[khughes]

We restored both (2 of 2) domain controllers during the DR test. What I meant by corrupted was the actual domain inside the servers was not in a functioning order. Once we restored the virtual machines, powered them on, seemed like they would talk for a few minutes then everything just basically stopped working within the domain. We weren't able to login to certain systems that didn't have our credentials cached, and weren't able to add/join systems to the domain. Tried shutting everything down, and restoring just one of them to see if we could get it up and running but that didn't work either.

Also just want to throw it out that I'm not blaming Veeam for whatever happened, just trying to get a better understanding of what may have gone wrong / why things didn't seem to work. I brought the tapes with our backup files on them back to reload those backups into my test lab to try and replicate the problem but will take a few days to get that up and running.

[vMO]

Veeam normaly restores domain controller 'non authoritative' if you start from scratch, you have to restore first domain controller 'authoritative'
Veeam B&R v5 recovery of a domain controller

[lobo519]

Veeam normaly restores domain controller 'non authoritative' if you start from scratch, you have to restore first domain controller 'authoritative'
Veeam B&R v5 recovery of a domain controller

Instead of reading though the pages and pages on the forums - Perhaps Veeam could write a KB article on DC recovery for their knowledge base?? I think it would certainly eliminate a lot of questions here..

[tsightler]

Perhaps this would be a good idea, but my guess is it would be a really short paper as, in general, the process is fully automated. If you read that thread, you can see that the final outcome was pretty much "restore the DC's...wait for them to boot, reboot, sync". The final outcome was that the user didn't have to do anything except restore the VMs. This is as it should be as the process is fully automated, and any manual intervention that is require means something has gone wrong with the process. Of course, as with any automated process, things can certainly go wrong, but it's very difficult for a paper to cover all of the various things that could happen. That being said, I think having a paper for DC recovery is worth considering since it does come up on the forums on a regular basis.

For the OP, at this point it's pretty much impossible to know what might have been wrong, I can guess, but it would be just that, a guess. It appears you assumed they were "corrupt" but I think that is unlikely. My guess is that the replication services failed to start for some reason, perhaps they were not restored at the same time. Did you make any attempt to troubleshoot what was wrong with the DCs? Were you able to logon to the DCs and look at the messages in the event log? That should have given some clue. How many DCs do you have in your environment? Did you restore them all? For the ones that you restored, were they on the same subnet? Were name services working correctly in the DR environment?

[lobo519]

Perhaps this would be a good idea, but my guess is it would be a really short paper as, in general, the process is fully automated. If you read that thread, you can see that the final outcome was pretty much "restore the DC's...wait for them to boot, reboot, sync". The final outcome was that the user didn't have to do anything except restore the VMs. This is as it should be as the process is fully automated, and any manual intervention that is require means something has gone wrong with the process. Of course, as with any automated process, things can certainly go wrong, but it's very difficult for a paper to cover all of the various things that could happen. That being said, I think having a paper for DC recovery is worth considering since it does come up on the forums on a regular basis.

For the OP, at this point it's pretty much impossible to know what might have been wrong, I can guess, but it would be just that, a guess. It appears you assumed they were "corrupt" but I think that is unlikely. My guess is that the replication services failed to start for some reason, perhaps they were not restored at the same time. Did you make any attempt to troubleshoot what was wrong with the DCs? Were you able to logon to the DCs and look at the messages in the event log? That should have given some clue. How many DCs do you have in your environment? Did you restore them all? For the ones that you restored, were they on the same subnet? Were name services working correctly in the DR environment?

I agree - I would expect it to not be much more than explaining what automatically happens when doing a restore, the normal steps that would occur, explain the fact that it is a non-authoritative restore by default and how to do an authoritative restore if needed (at at least a link to a MS KB)

[tsightler]

OK, I'll think about how something like this might be put together and perhaps talk with out support team about the common issues they see with DC recovery.

[Gostev]

My guess is that the replication services failed to start for some reason, perhaps they were not restored at the same time.

NETLOGON service stops after some time if the DC is unable to connect to any replication partner (other DC), so this very well might be the case!

[kurbycar32]

I would still love to see an article that says "Yes you can backup a domain controller, here is how to do it". Popular belief is that you don't want to pull a snapshot of any type of replicated database because the restore would result in a mismatch of its partner system. Obviously Veeam makes this work by booting into restore mode but i need a document that explains this. Also setting up the job for the first time: do i want to enable application aware image processing? Should i use full backups? Please spell it out in a KB

[Yuki]

I second this - while the steps and procedures may be obvious to some, it should be documented so people can find the information easier and faster. I find that searching forums can be frustrating on occasion due to certain word being used in many discussions.

[kurbycar32]

it should be documented so people can find the information easier and faster.

And present said information to less technical people such as management

[Vitaliy S.]

Though it's not a whitepaper describing how to backup a DC VM, but this webinar should still answer all basic questions our new users might have > http://www.veeam.com/videos/advanced-ac ... s-107.html

[kurbycar32]

That's a start but management isn't going to want to watch an hour long YouTube video. Just a simple KB article stating whats supported and how its supposed to work is all we need.

[jlyle]

I think a KB would be a good idea! I am configuring surebackups for our DC's which pass the DC test scripts, however I configured the lab environment to remain until stopped manually as I wanted to do some manual checks (call me paranoid). Logging in and trying the AD users & computers results in domain cannot be found/contacted error message, turns out that a reboot corrects the issue but is this not going to cause an issue when I get only Exchange and the exchange services cannot start because AD needs a reboot???

The point is I agree that a KB for recommended best practices would be useful and confidence boosting for new customers, especially if common issues can be highlighted.

[Yuki]

I'm not sure why this topic is taken lightly.

We have some clients on Veeam - they are public medical companies and are regulated by SOX and FDA. Many of them REQUIRE disaster recovery procedures to be fully written out in documentation so anyone can pick up the documentation package and follow the procedures to restore all systems. We don't get to debate this policy, as we don't get to debate proper user creation and termination procedures and others. So how does a KB help us with this? Simple - we can cite the source of the information when we put it together. If there are any issues with the restore by people other than the original staff who implemented the system - they can always refer to the source of the steps/procedure instead of registering on forums and searching.

[lobo519]

I couldn't agree more.

[jlyle]

It does look like DC backup and restore is supposed to be handled automatically, perhaps we are the minority having difficulty. I have just been escalated to teir 2 support, and am having quite a pleasant experience so far. I will post the solution to my problem when I have it on this http://forums.veeam.com/viewtopic.php?f ... 457#p70457 thread.

[DamMayhem]

Restoring Windows 2008 Domain Controllers is not recommended due to domain replication issues that will most certainly come up. However, if you are running VMware 5.1 and you upgrade your DCs to Server 2012 they will be 'VM aware' because of the new attribute 'msds-generationid'. After that, restoring (as well as cloning) them is not a problem.

Found this: http://www.sole.dk/virtualizing-your-do ... ing-fired/

[technerd]

Very interesting. More curious about 2012 now. Thank you for sharing!

Very new to Veeam here, and had to comment here before moving to another thread / starting a new one.

I haven't tried a full restore of a (2008) DC yet, but can say I did run into trouble with the time service on one, which consequently caused our (Cisco) phone system to have some issues. I've since stopped using Veeam to backup our two virtual DCs, as I read elsewhere that snapshotting them can cause issues exactly like this.

[Vitaliy S.]

I did run into trouble with the time service on one, which consequently caused our (Cisco) phone system to have some issues. I've since stopped using Veeam to backup our two virtual DCs, as I read elsewhere that snapshotting them can cause issues exactly like this.

Issues with the time service can be caused by wrong time on the ESXi host that provides resources to a DC VM, configuring NTP service on the host does fix this issue.

[Gostev]

Restoring Windows 2008 Domain Controllers is not recommended due to domain replication issues that will most certainly come up.

While this might most certainly come up with other backup solutions, Veeam takes care of USN rollback issue since version 2.0. Our application-aware processing will detect if processed VM is a domain controller, and will perform the required steps during both backup and restore, resulting in correct restore with no impact on AD replication.

[DamMayhem]

That's excellent information, Gostev. I wasn't aware of that. One more reason for me to love Veeam!