Veeam B&R recovery of a domain controller

[Fiskepudding]

I have a second reboot, every time. (the one at the CTR-ALT-DEL screen.

I have changed the DNS settings and created a seperate job for one DC, so i did not have to back up the entire host.
Backup is currently running. Wil post back as soon is i have the chance to do the restore (and the wait for reboots ).

[veremin]

Might sound like a stupid question, but, is there any other way to know if my 2008R2 servers use DFSR or FRS? Both our DCs are 2008R2 and domain functional level is also 2008R2.

As far as I know, DFSR should be enabled by default on WS2008R2 DC. Though, you might want to check it.

In order to determine whether DFSR or FRS is being utilized on a Windows Server 2008 DC you can check the value of the following regkey :

Code: Select all

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\DFSR\Parameters\SysVols\Migrating Sysvols\LocalState 

Providing this registry subkey can be found there and is set to 3, it means that DFSR is being leveraged. Otherwise, it’s FRS that is being used.

More informative description can be found here.

Thanks.

[Fiskepudding]

Thanx for the info, the key is there. But the value of Local State is 0 (on both servers)

(Using the DNS settings Gav suggested)
DC1 DNS
Primary = DC1
Secondary = DC2

DC2 DNS
Primary = DC2
Secondary = DC1

I can confirm that it made THE difference. I have a working domain after reboot.
(It might not be that important if i were running DFSR... witch i appearingly am not...)
Gav what is your reg setting on DFSR?...
I guess you also must be using FRS (since you have a old w2k3 on your domain?)

Now, after a restore both DC's do 1 auto reboot, (shows no logon screen).
Then when the logon screen shows, one DC reboots after a few minutes. The other never has a second auto reboot.
I waited about 20 minutes for the second reboot on the last DC... never happened.

Even tried a few manualy reboots.
Everything seems to be in order. Sysvol replicates, DC tools works perfect.

I tested this procedure two times.

Its looknig good, it came down to the DNS settings it seemd.
No wonder there are meny different results out there.

I am very curious if DFSR would have handeled this better.

[veremin]

I am very curious if DFSR would have handeled this better.

In order to check this, you can restore previously backed up DCs into isolated virtual environment (Virtual Lab), follow FRS -> DFSR migration steps. After migration process is finished, backup newly-updated servers, then, try to restore them and see whether the restore process is successfully handled by DFSR or not.

Thanks.

[Fiskepudding]

Yea, but since it seems like DFSR is recomended by EVERYONE, i migrated in production.
I will run new backups and test differnet DNS settings when restoring. Thanx.

[Fiskepudding]

I have done some testing, and i have found that:

When SYSVOL uses FRS, it seems to be important to have DNS settings like this:
DC1 DNS
Primary = DC1
Secondary = DC2

DC2 DNS
Primary = DC2
Secondary = DC1

If they are set like above I was able to recover even when both (all) of my DCs was down.
If DNS settings were opposite, I never made successful restore of both DC's using FRS.
I think we need to repeat that this is only neccacery/important if there are NO other DC's running in the domain (which could very well happen…).
Restoring only 1 DC shuold work regardless.

------------------------------ World of DFSR -----------------------------
Now, for me this is history, I have now migrated SYSVOL to DFSR.

I can now restore BOTH DC's regardless of DNS settings, even if I power both on at the same time or one at a time.
I have documented the behaviour of the servers during the powerup, using different settings.

Using DNS settings as mentioned above. (That worked with FRS):
Booting both at the same time:

Booting 1 at a time.


Change back to my original DNS settings (my prefered settings):
DC1 DNS
Primary = DC2
Secondary = DC1

DC2 DNS
Primary = DC1
Secondary = DC2

- Rebooted the servers
- New backup and restore process:

Booting both servers at once

Booting 1 at a time

I have repeated this process many times, and it works every time. Also tried instant restore and full VM restore.

To summarise, it seems like if you want to restore DC’s when there are no other DCs running.
(Total disaster). You would either need to use DNS settings marked in blue above.
OR migrate to DFSR….. Or fix all the AD errors manually, after a restore...


This might be specific for my environment, but might be helpful to others…

I am really glad I found this thread and tested this, and not relied 100% on my successfull SureBackup results

[Unison]

Looks like we are getting even more information together for Tom's official document/guide :

DNS order is important for FRS - DNS order IS NOT important for DFSR!

I had a look on my servers for that DFSR reg key - and i dont see it on any of my 08R2 DCs - so even though i see DFSR events in my logs, it seems as though its not actually functioning. No reg key so DFSR cant be working....but DFSR events are logged as if it was working - got to love that.

Having your DNS settings in the order suggested (have DC list its self as primary) must be what is making everything work when FRS is in use for the environment. Then, as shown in the above tests, when your using DFSR - it doesnt matter what order you have your DNS settings.....DFSR is truly 'self healing' and bringing the DCs back online!

Because it looks like i am using FRS too - i am going to change my DNS settings around to be the 'wrong' way (i.e. have the DC list its self as the SECONDARY and another DC as its PRIMARY) and see if my recoveries FAIL.

[tsightler]

2008R2 will always have DFSR, however, that doesn't mean that SYSVOL has been migrated to it. Any domain that was upgraded from 2003 and earlier will still be using FRS for SYSVOL unless the manual migration process has been completed. I'd say about 75% of the 2008 domains I see are still using FRS because people don't complete the migration.

[Fiskepudding]

DNS order is important for FRS - DNS order IS NOT important for DFSR!

True.

When using DFSR, the restore process i TRUELY automatic (Like Gostev claims )

It would be very interesting to know if you also have problems with the domain, after changnig DNS and restore.
Some times the AD tools actually worked, but after some time, and / or, reboots they were not. Sysvol was not replicating either.

Another good way to see if you use DFSR for replication would be to look in ADUC, under the domain, -> System -> DFSR-GlobalSettings
The "DFSR-GlobalSettings" folder/OU is created during migration. (I guess it is there if you have a new domian with 2008 + server as well.)

[Gostev]

Just to update this topic, we've reached out to some customers reporting issues and identified a problem that could result in SYSVOL replication issues with fully automated Domain Controller recovery when DFS-R is leveraged (FRS is unaffected, surprisingly). We were able to include the required fixes into the version 7.0.

[Unison]

Thanks for the update Gostev!
Can you share details on what the issue was that Veeam identified? The conditions and triggers? Would be interesting to know and others might recognise the problem when they come here - but good to hear v7 will fix it!

[Unison]

True.

When using DFSR, the restore process i TRUELY automatic (Like Gostev claims )

It would be very interesting to know if you also have problems with the domain, after changnig DNS and restore.
Some times the AD tools actually worked, but after some time, and / or, reboots they were not. Sysvol was not replicating either.

Another good way to see if you use DFSR for replication would be to look in ADUC, under the domain, -> System -> DFSR-GlobalSettings
The "DFSR-GlobalSettings" folder/OU is created during migration. (I guess it is there if you have a new domian with 2008 + server as well.)

I have just completed a test of this as mentioned above in bold at my site and can confirm the FRS behaviour with DNS settings.

I changed the DNS settings of my produtciont DCs (08R2) to:
PrimaryDNS = Some other DNS
SecondaryDNS = its self

Then took new backups with Veeam.
Now i started the recovery process with these new backups that have DNS settings as above.

The DCs went through the same recovery process and reboot steps as i have documented above. Everything looked like it would be normal.
When it finally came time for me to login to both DCs......i discovered that the recovery was not successful.
DNS was working as i could ping and access each others shares via name (and ipv4) - but SYSVOL was non existent on both, errors in the logs, AD wasn't working and there was no replication happening. Recovery failed.

If you are running FRS in your environment - get your DCs DNS settings in the format noted in earlier posts - and again here...

For FRS environments - make your DC DNS settings:
PrimaryDNS = its self
SecondaryDNS = some other DNS

With DNS like this and FRS in your environment, you will be able to recover your DCs and AD successfully without any manual intervention!

[Fiskepudding]

Thanx for the update Anton.

And "Good news" Gav.
At least we know the reason behind the different behaviour when restoring DC's (when there are no other DC's running).

But if there are no pre 2008 servers running, I would say migrate to DFSR. It seems more robust and is a generally recommended.
AND....DNS order seems NOT to matter when restoring DC's running DFSR. It works either way.

If you were to migrate to DFRS Gav, I for one would love to know the results. In my envoronment restore have never failed.

[tsightler]

I have a question about the DNS settings you guys are using, why are so many of you using "flipped" primary DNS? I'm pretty sure that Microsoft best practice for DCs since at least Windows 2003 has been for the primary to always point to itself. In Windows 2000 there were some known potential problems when pointing a DC at itself that could cause a Forest root DC to become an "island", but that was a very long time ago.

[tsightler]

Well, to answer my own question, after some research, it appears there's a huge amount of debate on the issue of proper DNS configuration for a DC, even among Microsoft DS professionals. In some cases there are even dueling blog entries from MS employees each pointing to different Technet or KB articles.

[Fiskepudding]

I guess the answer is... old habits die hard?...
After reading this thread I search for best practice on DNS settings. And found that, just like you Tom, there are no consensus.

Regardless, lets say 50% use "flipped" primary DNS.. and 50% of them still use FRS. Then 25% of (windows) veeam users will not be able to recover the domain if all domain controllers are gone.
So it is quite an important point. Provided that mine and Gav's test apply to other setups..

[Unison]

Well, to answer my own question, after some research, it appears there's a huge amount of debate on the issue of proper DNS configuration for a DC, even among Microsoft DS professionals. In some cases there are even dueling blog entries from MS employees each pointing to different Technet or KB articles.

Thats right - this is what i have found also when researching DNS configs on DCs.
Does not seem to be any clear cut right or wrong DNS config for DNS on DCs - well as to what should be the primary.

BUT - at least here, with DC Recovery and FRS - we know for certain that if you dont have your DCs pointing to themselves then your DC restores will fail!

So while there is argument going on about what should be the primary DNS on your DCs.....at least here, we can say that if your using FRS in your domain and you want to be able to recover your DCs automatically.....then....

PRIMARY DNS MUST = Its self!

[push3r]

I apologize if there is already a solution for my scenario which I couldn't find in this thread. This is with regards to FSMO validation issue.

My Production Site:
- ESXi 5.0, All DC's on Win2008R2
- DC-1 (physical) and DC-2 (VM)
- DC-2 holds all FSMO roles
- I'm still using FRS

My DR Site: (in case alien mother ship beams up my data center)
- Failover to DC-2 ONLY
- Steps need to be taken to successfully restore DC-2, ACCORDING to this thread.
1) Let Veeam does its things and boot DC-2. (it's doing non-authoritative restore here)
2) Do a SysVol Authoritative restore. (Set Burflags = D4)
3) Disable replication to other DC. (Set NTDS\Repl Perform Initial Synchronizations = 0)
4) Done?

In Reality:
BUT, in my brief failover testing to my isolated DR Site, the above is not the case. I didn't have to do anything, and my Sysvol is shared out and I can login just fine.

I get 2 errors with regards to Active Directory on DC-2
1) Event 2092: about the RID Master not being valid because it couldn't replicate with the other DC even though it holds the RID Master role. And it's telling me to seize this role with NTDSUtil. This is understandable because the RID Pool ID needs to be in sync with other DC's. I could NOT create a User object until after I manually seized the RID Master on DC-2.
2) Event 13508: FRS complains kept complaining about not able to replicate with DC-1. This is fine as DC-1 is supposedly beamed up by the alien mother ship. This can be ignored.

So, has anyone run into this FSMO validation issue?
Also, what is the best way to handle restore/failover the Virtual DC only in the DR site while production has a physical and virtual dc?

EDIT: In fact, I also see two other AD errors (event 2092) about the other FSMO roles not being valid, 1) infrastructure 2)the FQDN domain, which sounds like schema.

TIA!

[Unison]

Hey Push3r,
Can you confirm if DC2 is pointing to its self for primary DNS on the NIC?
As we have discovered in this thread (recently) - if you are running FRS and have your DCs pointing to themselves for primary DNS then the DC should recover perfectly fine on its own without having to manually force the DC into authoritative mode with burflags.

But....i have to ask the question.....Why is DC1 still physical? Why not VM it too, replicate it to your alien proof DR site and then you can be far more confident in your DR plan because you will have your DC partner pair in the primary site and in the DR site. I would not consider it a good DR position/site to cut off DC1 when the aliens come ......if you cant ever get DC1 back into the fold then you are going to have to manually 'clean up' AD and that is messy.
Dont think there are really any good reasons these days to need a physical DC....think the only good reason i have ever heard is if your SAN Storage Infrastructure requires AD to come online (but i guess if you have enough $$$ to be running a SAN like this, you could probably get one of your VM DCs to run off separate VMFS stores).

[push3r]

After further research, I think I found part of the answer to my issue at this link.

https://standalonelabs.wordpress.com/20 ... ntrollers/

"when a DC holding a FSMO role starts, the initial synchronization will be performed with that DC’s known replication partners. If those DCs are offline, then the DC owning the FSMO role will not take ownership of the role until the KCC rearranges the replication topology and replication is able to take place."

I then have 3 options to remove the physical DC-1 from AD.
1) perform metadata cleanup
2) delete the incoming replication links from the FSMO owner
3) seize the roles again

Now, the dilemma.
1) If the production site is completely gone, then I will be doing the Permanent Failover and won't be Failing Back. Thus, deleting the physical DC-1 is fine.
2) If the production site is temporarily gone and then come back online again, then I better make sure the Physical DC is turned off and never come back online again because in my DR Site, I have deleted all records of it in Active Directory.
3) Or I can deploy a third DC at the DR site which has it's disadvantages of 1)additional Windows licensing (maybe) 2)extra resources 3)another vm to maintain.

Are my assumption correct?

This is getting complicated.

[push3r]

Hey Push3r,
Can you confirm if DC2 is pointing to its self for primary DNS on the NIC?
As we have discovered in this thread (recently) - if you are running FRS and have your DCs pointing to themselves for primary DNS then the DC should recover perfectly fine on its own without having to manually force the DC into authoritative mode with burflags.

But....i have to ask the question.....Why is DC1 still physical? Why not VM it too, replicate it to your alien proof DR site and then you can be far more confident in your DR plan because you will have your DC partner pair in the primary site and in the DR site. I would not consider it a good DR position/site to cut off DC1 when the aliens come ......if you cant ever get DC1 back into the fold then you are going to have to manually 'clean up' AD and that is messy.
Dont think there are really any good reasons these days to need a physical DC....think the only good reason i have ever heard is if your SAN Storage Infrastructure requires AD to come online (but i guess if you have enough $$$ to be running a SAN like this, you could probably get one of your VM DCs to run off separate VMFS stores).

Thanks Unison for your reply.

The DC's are pointing to themselves as primary for DNS and to the other DC's for secondary.

And yes, when I first failover, I did NOT have to do anything (i.e. set burflags, etc) and my DC2 booted up a couple of time and I can login, see Sysvol, etc. Except that I get those FSMO validation errors which is due to the physical DC not being in DR site. As I replied to my post, the only solution for this is to manually remove the physical DC1 records from AD which is very messy. On top of that, failing back would be a complicated as well.

Believe me, I am all for 100% virtual. The big cheese, my boss, is old school who believed in the consultant (who setup the virtual infrastructure) is also somewhat old school. So, they believe in having a physical DC. Now this is making DR plan complicated. I will convince the big cheese to go 100% virtual and see. Do you have any official document from VMWare and/or Microsoft showing that 100% virtual DC is acceptable and is supported?

[Unison]

Thats right in my response to you, i touched on the fact that it will get messy/complicated......
you can make all this go away though with a small change to your production environment and that change will also fix the problems in your DR site should you ever need to fail to it.

VM your DC1 - then you wont have to worry about any of this and you will have confidence in being able to reliably recover to your DR site - it will take a lot less time and messing around too! (+ u can regularly test the operation....easily).

[Unison]

Thanks Unison for your reply.

The DC's are pointing to themselves as primary for DNS and to the other DC's for secondary.

And yes, when I first failover, I did NOT have to do anything (i.e. set burflags, etc) and my DC2 booted up a couple of time and I can login, see Sysvol, etc. Except that I get those FSMO validation errors which is due to the physical DC not being in DR site. As I replied to my post, the only solution for this is to manually remove the physical DC1 records from AD which is very messy. On top of that, failing back would be a complicated as well.

Believe me, I am all for 100% virtual. The big cheese, my boss, is old school who believed in the consultant (who setup the virtual infrastructure) is also somewhat old school. So, they believe in having a physical DC. Now this is making DR plan complicated. I will convince the big cheese to go 100% virtual and see. Do you have any official document from VMWare and/or Microsoft showing that 100% virtual DC is acceptable and is supported?

we replied to each other at the same time for that last one sry.
That is definitely an old school way of thinking....but at least now you have hard evidence/facts as to why you SHOULD VM both the DCs.....because it WILL cause you issues if you ever had to fail to the DR site. Can they give you a list of problems that would result from VMing both your DCs......i think not.

I could find a tonne of material on the internet about it - believe i have read discussions on it here, spiceworks, technet, vmware etc. A quick google on the topic is going to land you with hrs of reading and arguments for your boss. But you are seeing a real world impact in your own environment. You have a problem that is resulting from not having your DCs both virtual.......try to find just one reason why one should be PHYSICAL in your environment (beyond old school fear and baseless comments like "its just good to keep one DC physical"!).
You ARE going to improve your environment and DR position by VMing both DCs......your boss cannot deny that....and surely he will be happy with your findings on this and why you should VM it

[tsightler]

I'm a little curious why you are so concerned about the FSMO roles. They should eventually sort themselves out once the KCC runs (every 15 minutes), and who really cares about replication not running since the replication partner (DC-1) is unreachable. It should not cause any actual issues in the mean time and eventually KCC will run and life will be good. Perhaps you can just force KCC to run with the following command:

Code: Select all

repadmin /kcc

[Fiskepudding]

Damn, ailians...
Tom has a valid point here. Have you waited long enough to see if it actually sorts itself out?

I would also like to lobby a bit for DFSR, don’t know if that would help you in this situation, but all my DC restore tests works flawlessly after the migration.
It costs you MAX 30 min, and is an extra insurance in my book.

On with the tin foil hat

[Unison]

Vote for DFSR + Making DC1 virtual!
Perfect solution.

[push3r]

I'm a little curious why you are so concerned about the FSMO roles. They should eventually sort themselves out once the KCC runs (every 15 minutes), and who really cares about replication not running since the replication partner (DC-1) is unreachable. It should not cause any actual issues in the mean time and eventually KCC will run and life will be good. Perhaps you can just force KCC to run with the following command:

Code: Select all

repadmin /kcc

I am concerned because of several event 2092 where all of my FSMO function were not working until I had to manually step in and do one of the 3 things (1.metadata cleanup to remove DC1, 2.delete incoming replication links with DC1, or 3.seize all the roles again) I mentioned above to remove the unreachable DC1 from AD in my DR site. And I tested this. I could not add another AD object (i.e. user account) until i seized the the RID Master role again even though DC2 has all the FSMO role.

What is KCC going to do for me if DC1 is unreachable? Replication topology can re-arrange all day and it will never find DC1. DC2 is looking to replicate with somebody before it will say that it is the master for all the FSMO and thus allow those functions.

And yes, I waited several hours and event 2092 kept returning.

I think someone should test this scenario where you have at least two DC's and on your DR site, you can only restore 1 of them. You will probably see the behavior similar to mine where you can login, see sysvol share, etc...but the FSMO functions are dead until you manually intervene.

I understand that virtualizing all DC would make life a lot easier, but what if one of Veeam's customers has a need for a physical DC? And did you know that even VMWare has one Physical DC in their environment?

source: http://www.ballblog.net/2010/12/do-you- ... omain.html

"Once again, there is no right or wrong answer to virtualizing your domain controllers though you may find it interesting, as I did, that VMware has a 48 DCs, 47 are virtual and 1 is physical. The physical domain controller is the Forest PDC emulator. "

[tsightler]

OK, fair enough, I thought you were saying these errors occurred during startup, which wouldn't be unexpected, if they continued for hours that's a little more concerning. The reason KCC should help is because after several attempts to establish replication with DC-1 the KCC should determine that this DC is no longer responding, issue a 1308 error to the event log, and recalculate the topology to allow replication to continue (in your case, replication to nowhere). The link you posted earlier specifically mentioned the fact that KCC would eventually rearrange the topology. Otherwise, anytime a DC became unreachable replication would stop, which wouldn't be very robust.

I'm not really sure if this is something Veeam can address since it's mostly an issue with the behavior of AD in the absence of it's replication partner. Once Veeam restores the DC, the behavior of that domain controller after it's restored is mostly up to Microsoft and the admin must choose what to do, if DC-1 is never coming back, the cleaning it up is probably the correct response. The scenario you describe above would seem likely to occur even without Veeam. For example, if you had an unexpected power loss that left your physical server inoperable, but your virtual DC was still OK, it would boot and not find it's partner.

That being said, I will attempt to reproduce this issue, although my environment uses DFS-R so it might not be the same.

[push3r]

Thank you for the verification! I was curious whether my thinking is inline with how things work.

At this point, I think I will deploy a virtual DC at the DR site instead of going 100% virtual. I read all the gotchas about egg-chicken issue and aware of best practice to keep both virtual DC's on separate hosts, but then storage comes into the picture. One should also keep the DC's vmdk on separate storage, so that means keeping one on the SAN and the other on Local Disk of one of the Esxi hosts. Then you have to keep track of all of this information of where things are. If I were to do all that, then might as well just leave the damn DC as physical and keep things simple. Putting another virtual DC in the DR Site will solve my problem of just failing over to DC2 (or I don't even have to if I don't want to) which would then have the DR DC sitting there to replicate. This solution would also make my AD even more robust in the case that the alien beam up the data center. Additionally, by having a DR DC, I don't have to mess around with host file on the Veeam Backup Server VM as I would have a DNS server sitting there too.

[tsightler]

Having a live DC in the DR site is best practice when possible so I agree with that approach. Having to recover your AD as a first step to recovery is one of the more complex steps so this simplifies the process of recovery significantly.