Veeam B&R recovery of a domain controller

[Unison]

I understand that virtualizing all DC would make life a lot easier, but what if one of Veeam's customers has a need for a physical DC? And did you know that even VMWare has one Physical DC in their environment?

source: http://www.ballblog.net/2010/12/do-you- ... omain.html

"Once again, there is no right or wrong answer to virtualizing your domain controllers though you may find it interesting, as I did, that VMware has a 48 DCs, 47 are virtual and 1 is physical. The physical domain controller is the Forest PDC emulator. "

This is a pretty old page - 3yrs. A lot has probably change at VMWare in that time + they have a HUGE/complicated environment which is evident from the fact they have a need for 48 DCs lol

If your not prepared (for what ever reason) to VM your DC1, then i would agree too that putting a VM DC (3rd DC) into your DR environment will also greatly help with your recovery....but its not going to solve all your problems.
In a disaster where your main site is taken out 100%, you will still lose DC1 and never get it back. When you fail over to your DR site, your AD will still be messed up and require manual cleaning/clearing of DC1 to get everything right again. Not impossible to do, but it is a mess all the same....something you could avoid completely/easily.

Creating a new VM DC and leaving a DC physical is going to make things more complicated in your environment than just taking the one single action of VMing DC1. Your adding a 3rd DC unnecessarily and your strapping a DC to physical hardware for no real reason which results in that DC losing DR ability.
The chicken/egg/dns issues apply only to exceptional setups - i am pretty sure you could identify if you would have any of those issues....rule them out. When all of my VMs are down including the DCs (say for a planned total power down event) - when i bring the hosts back online, its true, i have no DNS.....i rely on IP addresses to get to the hosts to bring vcentre and the DCs back online....but once they are up - all is back to normal and i can use host/dns names.
My DCs VMDKs live on the same shared storage (raid 10, dual controllers etc all the redundant bits)....and thanks to veeam, i replicate all my VMs to a totally separate host with totally separate storage so can bring all that online in minutes if my primary hosts/storage fails spectacularly together......veeam also backs everything up to another local totally separate store.....AND veeam also backs everything up over the WAN to a location 100 miles away. You dont NEED to have your DC VMDKs on separate storage if your primary storage is reliable/good + you leverage veeam properly to protect your environment from the very rare major disasters we are talking about here.

Sry for commenting on this again (just trying to help) - its completely your choice of course but i do believe you could greatly improve your DR ability by taking a different action than the one your planning to. VM DC1 - all your problems are solved (except for the alien thing )

[push3r]

Thanks Unison. I really appreciate your insightful input. Great stuff!

With regards to the DR site, if I have DC3 there, then DC2 (holding all FSMO roles) will work as normal as it can replicate with DC3; and my AD infrastructure will work as usual. It's true that DC1 will need to be manually cleaned but at least I don't have to deal with that until the dust settle.

As for the 100% virtual DC, the SAN is still a single point of failure (one enclosure) even though it's very reliable, but I would not feel comfortable putting everything in one basket. I definitely don't want to roll out a half ass solution by putting the second DC on Local Storage or complicating it further. I see that you also replicate locally to a separate domain or hardware in the case that your SAN and Host both die. This is great as an extra layer of protection and I didn't think about this. I have extra reliable hardware laying around that can be used for this purpose.

I will reconsider my DR plan and test out all possible scenario, especially the "chicken-egg-dns" issue.

Thanks again!

[pcrebe]

I definitely don't want to roll out a half ass solution by putting the second DC on Local Storage or complicating it further

Why? If your DC1 in your san dies you have a dc2 for veeam DC restore. If your DC2 in your local dies vice versa.
I put the veeam server and DC2 on the local storage of the DR esxi full of VM replicas then i replica the DC2 on a local storage of one production esxi and the veeam server in the SAN in case of the DR esxi fails.

Bye

[Unison]

Thanks Unison. I really appreciate your insightful input. Great stuff!

With regards to the DR site, if I have DC3 there, then DC2 (holding all FSMO roles) will work as normal as it can replicate with DC3; and my AD infrastructure will work as usual. It's true that DC1 will need to be manually cleaned but at least I don't have to deal with that until the dust settle.

As for the 100% virtual DC, the SAN is still a single point of failure (one enclosure) even though it's very reliable, but I would not feel comfortable putting everything in one basket. I definitely don't want to roll out a half ass solution by putting the second DC on Local Storage or complicating it further. I see that you also replicate locally to a separate domain or hardware in the case that your SAN and Host both die. This is great as an extra layer of protection and I didn't think about this. I have extra reliable hardware laying around that can be used for this purpose.

I will reconsider my DR plan and test out all possible scenario, especially the "chicken-egg-dns" issue.

Thanks again!

No worries mate!
Yep, if you have the spare hardware around - you can mitigate the 'everything in one basket' issue for storage by replicating locally

Either way....your DR setup/position is going to get better

[lukeup]

Hello,

What is the best practice for backup\replicating a DC?

When using backup\restore does Veeam restore the DC in non-authoritative mode, therefore restoring with no USN issues or any other issues?

Is it good practice to replicate and failover a DC, does this use the same techniques as a backup restore using non-authoritative mode or will it just boot up\failover and create USN issues?

Thanks.

[Fiskepudding]

Excellent questions Luke.

We don’t replicate the DC's so I cannot comment on that.
But for some setups it seems that is best to back the DC's up in sequence.
So regarding replication; i guess replication times should also not overlap.....its a guess

Yes, as far as I can tell Veeam restores the DC in non-authorative mode ALWAYS. You can not change this manually in the Veeam restore process.
If all DC's are gone, non-authorative might not work very well. But I guess Veeam assumes there still is an available/running DC.

Have no clue how Veeam handles a failover on a DC, but I interesting question.

[foggy]

During failover, DC is also restored in non-authoritative mode and should be synced up with the other DCs automatically. There's also another existing thread specifically on DC replication, worth reviewing.

[plataformacarnica]

We're implementing a disaster recovery system with veeam backup , and we're making tests to assure that in disaster recovery fail everything works ok.

Our schema is the next, we have a main server who is a windows 2008 domain controler over a esx 5.1 server we replicate this server to other machine.

The idea is that in case of hardware failure we can start the replica an continue working

The test is the next :

1- Stop the computer hardware
2-Go to veeam backup and make a Failover on the replica
3- Make some test on the functionality of the failover machine
4- Failback the machine or make a permanent failover.

We have test this procedure with several machines without problem but when we tried with the domain server it failed with the next error.

Cannot complete login due to an incorrect user name or password

we haven't changed any password or username.

I guess it's failing cause as we don't have domain controler (is the one we are simulating the fail) the veeam backup is trying to authenthicate to the vcenter (who's is linked to the domain) and cannot do it.

But what we have to do in that case? can we specify the password manually without domain?
Or in the case of a domain controler, we have to make an special procedure for it , I mean starting the replica manually , but it that case:

- How we can make the failback to the production one when this is arranged?

- How can we make a "permanent failover" avoiding that the replicate machine runs from replica?

- In case we want to start in a other restore point (not the last one replicated) how can we do this without veeam backup (who's is not working cause password problem)

Any idea?


Thanks

[foggy]

Please review this thread, and specifically this post, for better understanding of how Veeam B&R DC recovery works. Basically, Veeam B&R performs a non-authoritative restore of the DC assuming there are other functional DCs in the environment it can replicate with. In cases with a single DC, authoritative restore is required.

Also, you may consider utilizing SureReplica functionality available in v7 to test your DC replicas.

[Fiskepudding]

I guess there is no document yet.

I know this have been suggested before in this thread, but no official answer if this is technically possible, not an god idea at all.. or whatever.
So here it goes again

Considder this...

For SureBackup Veeam can do an authoritative restore:

... Veeam makes some changes to the registry of the DC prior to powering it on the the Suerbackup lab. These change force an authoritative restore of the DC, and specifically of SYSVOL. You can see the exact changes being made by looking in the Surebackup job log in the Veeam log directory. Open up the logs and search for "PrepareDC" and you'll quickly find the section where this "magic" is performed during Surebackup, including all of the gory details of the registry entries, but in summary it's the following.....

Seems like all the coding/loigic/magic is already in place.

But it cannot do this in a real restore. You have to do that manually, just like Tom explains here:

.
So if I were doing a complete disaster restore, I'd restore two of the original DCs, power them on, wait for their reboot, and force one to become authoritative for SYSVOL, then restore the other DCs and they should recover automatically.

My hope is that, since Veeam already know how to do an authoritative restore, there sholud be a "Domain Controller restore option/wizard", where you could choose if you wanted authoritative restore or not.
So instead of powering on multiple DCs and wait... who knows for how long, then force one of them to be authoritative, the first one you restore is already authoritative when it boots.....then the rest can follow as non-authoritative.

[Frosty]

I've done a fair bit of reading but unfortunately am not much the wiser. Am looking for some definitive information on whether I do or do NOT need to perform an Authoritative Restore when I restore my DCs in a DR scenario.

My understanding of an Authoritative Restore is that it is used when you want to return AD to a previous/earlier state. But in my DR scenario, I just want to restore my two (2) DCs as at whatever the date/time was when they were backed up. With that in mind, I don't see that I need to perform an authoritative restore, provided that I am careful to restore the DCs in the right sequence.

At the moment I back them up in this sequence: DC1, then DC2 about 5 minutes later. But I think I should probably reverse that order: DC2 first, then DC1 (which holds the FSMO roles) about 5 minutes later.

When I restore them, I would restore DC1 first, then DC2 about 15-20 minutes later. Since DC1 was backed up last, it would have the "most up to date" version of AD. Since DC2 was backed up first, it would be well "out of date" when restored and presumably can replicate if it needs to catch up with DC1.

Is this too risky a strategy? Would I be best to forcibly boot DC1 into DSRM? In earlier versions of Veeam this used to happen automatically when restoring DC1, and I had to go into the Boot Options in Windows and manually turn off Safe Mode booting. But when I tested a restore yesterday with VBR 7.0R3 this didn't seem to be necessary any longer.

So now I am a tad confused...

[foggy]

Stephen, I believe this thread (I know, a huge one!) and, particularly, this post, contain answers to all of your questions. However, if you still need any clarification after reviewing it, feel free to ask here.

[Frosty]

I did read that thread, or at least, much of it. Read the post you highlighted in your last. The recommendation there is to restore two (2) DCs simultaneously and "force one of them to be authoritative". So it looks like my strategy should be:

1. backup DC1 and DC2 (in reverse order) in the same backup job
2. restore both DC1 and DC2 to my DR servers and boot them simultaneously
3. force DC1 to be Authoritative

So its that last step I'm still not sure about. Most documentation I have seen for Authoritative restores refer to having separate System State backups and so on. That's not the case with Veeam; I just have my VM images.

I still think there needs to be a one-stop-shop document for this, as the information is around somewhere, but its just not clear to people like myself. I found these articles:

http://windowsitpro.com/windows-server- ... eplication

http://support.microsoft.com/kb/2218556

Does it really come down to that: using ADSIEDIT and running all those manual steps as per the Microsoft instructions?

[Frosty]

So I did some more testing at my temporary DR site this morning. My DC1 and DC2 restored VMs seemed remarkably happy:

* dcdiag /test:replications ... came up clean with no errors

* repadmin /showrepl * ... came up clean with no errors

* created some changes in AD on DC1, manually triggered a replication to DC2, then checked that those changes replicated ... they did

Authentication generally seemed just fine. Event Logs were clean with no errors apparent.

So now I am wondering whether the whole "must do an authoritative restore" is a red herring. What am I missing?

[tsightler]

Hi Frosty! I think you might have misinterpreted my post regarding an authoritative restore during a recovery. I don't believe I mentioned anything about needing to perform an authoritative restore of AD, but I did mention the possibility of needing to force an authoritative restore of SYSVOL to get replication running. Replication of AD and replication of SYSVOL are completely different things. Obviouslt AD objects are stored in the AD database and replicated to other domain controllers, but SYSVOL contains things like login scripts and group policy settings that are stored on the filesystem and made available via the SYSVOL share on all DCs.

If everything works correctly SYSVOL replication should come online automatically as well, although it can take up to 30 minutes, but based on a multitude of factors there are cases where the conflict resolution can't determine which replica should "win", so the only way to force replication of SYSVOL to start is to manually designate one of the copies to be "authoritative". The process for doing this varies based on whether you are using FRS and DFS-R for SYSVOL replication. As long as SYSVOL replication comes online then you should be in good shape.

[aacable]

I have one Domain Controller on Windows 2008 (VM) and two ADC (also hosted in VM)
I am taking backup using VEEAM , my question is if somehow my Primary domain Controller corrupts , and I want to restore using veeam backup,
How I can do that? If i do simple restore will it start replicating with other ADC's automatically ?
any point of concerns ?

[foggy]

Syed, please review this topic regarding domain controller restore. Basically, if your primary domain controller was backed up with application-aware image processing enabled, everything should be performed automatically.

[aacable]

Dear Foggy, Thank you,
and Yes I am taking backup with 'Application aware' option enabled.
Hopefully it will be OK. I guess I have to test it in isolated vmware , just to be sure

[foggy]

Sure, you can perform initial testing to see how it goes in your environment.

[zoltank]

Sorry to dredge this thread back up.

When performing a complete network recovery (restoring all domain controllers from scratch), what is the final verdict on having to do an authoritative restore on the first DC to come up? Is it required or not?

By "authoritative restore" I mean the AD database itself, not the SYSVOL.

[zoltank]

So, the answer I found was no, don't ever do an authoritative restore on the AD database.

[ejleipold]

Sorry for the late response But no, I've never had to. I just restore them normally (like any other VM), start them up, reboot once then wait 15 - 20 minutes for them to sort themselves out. I sometimes need to reboot them again a couple times, but eventually they come good. I test restore mine into the dev environment once ever couple months and have never had an issue getting them back up and running properly.

[veremin]

By "authoritative restore" I mean the AD database itself, not the SYSVOL.

Authoritative restore AD is not needed. There might be occasional cases when authoritative restore of SYSVOL is needed (replica are struggling for master role). Thanks.

[renilde]

We use veeam8 and from a full backup the domain controller does not work properly.
In a disaster recovery we want to test the backup of our crititcal servers. In the production environment we have two domain controllers (all win2012r2),a virtual one who has all the FSMO roles and a fysical dc.
In our DR we only want to restore the dc with the fsmo roles, we can boot the server, but active directory won't start, naming information cannot be located because the domain either does not exist or could not te contacted. No sysvol shares or netlogin shares.
I have found a solution for win2008r2 controllers where you have to do something with the burflags, but no solution for 2012.
I made a case for support yesterday evening.

Has this to do something with the backup that has to be done in a different way or is this typical microsoft windows who wants to make our lives not to simple?

Thanks for your help.
Renilde

[stevericks]

We restore DC's when we do DR testing. The only thing we do is restore the full VM, switch on and test. All works perfectly.

[piedpiper]

Hi Guys,

To all the AD Admins out there, does Veeam deal with this issue well when using Application aware backup and restore?

I have been reading up on it, as I like to be aware of the possible issues when restoring a DC. I am having trouble getting my head round how USN rollback works exactly.

I have tested it out in a test environment, by backing up DC2 using application aware backup , binning the actual VM and then restoring it using the backup.

What is the maximum age of the backup that you should use when recovering a domain controller?

I let it restore and set it to auto boot, when I came back to it, it was at the login screen and it was in Safe Mode (as expected) I had an issue where the 100mb system partition wasn't mounted, I mounted that and ran the commands as per the KB article http://www.veeam.com/kb1277

Replication seems to be working fine, across my Domain controllers DC,DC1,DC2 (recovered). DC is the original main controller.

I have run the repadmin /showutdvec command on DC2 (the recovered Domain Controller)

DC2 @USN 345605 @ Time 2015-02-24 21:37
DC1 @USN 334552 @ Time 2015-02-24- 21:30
DC2 (retired) @USN 341361 @ Time 2015-02-24 15:59
DC @USN 300711 @ Time 2015-02-24 21:37

I have run repadmin /showutdvec on DC1

DC2 @USN 345280 @Time 2015-02-24 20:59:30
DC1 @USN 334621 @Time 215-02-24 21:38:41
DC2 (retired) @USN 341361 @Time 2015-02-24 15:59:31
DC @USN 300716 @Time 2015-02-24 21:38:24

on DC2 the USN is higher then the value held by DC1, does this mean I have a rollback issue?

Cheers,

Bilal

[foggy]

Bilal, actually you should not take any manual steps upon DC recovery, since Veeam B&R handles everything automatically taking the appropriate steps during its application-aware processing to ensure that a USN rollback issue does not happen. Basically, you should wait until DC reboots normally.

[piedpiper]

Bilal, actually you should not take any manual steps upon DC recovery, since Veeam B&R handles everything automatically taking the appropriate steps during its application-aware processing to ensure that a USN rollback issue does not happen. Basically, you should wait until DC reboots normally.

Hi foggy!

I didn't watch it go through its process, I just came back a couple of hours later and it was at the Windows login screen.When I logged in it was in Safe Mode, I restarted it a few times but each time it booted in safe mode. So I had to follow the KB article to get it to boot normally. After that its been fine.

I was just curious about USN rollback, and there wasn't much written by Veeam on how/if it dealt with it. I was guessing this was because its a proprietary system which is fair enough, I was just looking for 'yes Veeam application aware backup/restore makes sure it doesn't happen'

From reading the forum and other threads, I could use a Veeam application aware backup from 30 days ago, and restore it and then the DC would sync with the others and replicate the missing data.

I have to say I am very new to Veeam but find this totally amazing lol

FYI
Ok after further investigation, it looks like I am in the clear rollback wise:

If the direct replication partners have a higher USN number for the domain controller than the domain controller has for itself, and the repadmin /showreps command does not report replication errors between direct replication partners, you have compelling evidence of a USN rollback.

My 3 Domain Controllers :

DC2 has a higher number for itself than the other DCs do. The same goes for the other DCs when compared to their replication partners.

repadmin /showreps - shows replication is running and shows inbound neighbors
HKLM\System\CurrentControlSet\Services\NTDS\Parameters\DSA not Writable - This does not exist on DC2
Directory Services Events – Look for the following events in the Directory Services log: 2095, 1113, 1115. - Don't exist

I have done gpupdate /force a few times and it all appears to be good.

So it looks like my restore went well.

[jb1095]

Hey all,

I am having trouble finding any official information regarding the best practices for backing up a virtualized domain controller using Veeam 8. I spoke with Shawn from support earlier today and he stated that if I am only doing a backup, all I need to do is to make sure that I check the "Enable Application Aware Processing" processing box. In my experience, there is always a best practice for any product when backing up a DC. Is it really as simple of just checking that box or do any of you know of any official documentation that will point me in the right direction.

Also, if there is not document out there(which may very well be true), are any of you Veeam 8 users backing up your VDCs using Veeam 8 and have you ever had to do a restore..Specifically a restore back to your production environment where you have both a physical as well as a virtual DC?

Thanks

[veremin]

Is it really as simple of just checking that box

Yes.

Specifically a restore back to your production environment where you have both a physical as well as a virtual DC?

More information regarding restoring DC can be found in the posts above.

Thanks.