Comprehensive data protection for all workloads
Post Reply
nefes
Veeam Software
Posts: 616
Liked: 153 times
Joined: Dec 10, 2012 8:44 am
Full Name: Nikita Efes
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by nefes » May 24, 2013 9:28 am

Oliver, have you selected Domain Controller role in SureBackup job for this VM?
Also, is that DC the only in the domain or there are several others in it?

okrehan
Service Provider
Posts: 13
Liked: never
Joined: Mar 17, 2011 10:53 am
Full Name: Oliver Krehan
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by okrehan » Jun 03, 2013 12:23 pm

Hi,

sorry for the late reply, the forum didn't sent me an email as requested when someone sends an answer....
So, this is the only DC in the environment (it is only a test env.) and I have set the role to DC, Global Catalog and DNS server.

Regards,
Oliver

foggy
Veeam Software
Posts: 17712
Liked: 1481 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by foggy » Jun 03, 2013 3:56 pm

Oliver, did you get a chance to try recover this DC using Instant Recovery? Have you contacted support with that?

james575
Novice
Posts: 8
Liked: never
Joined: Jun 18, 2013 4:07 pm
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by james575 » Jul 18, 2013 8:44 pm

I spent quite a bit of time trying to successfully test my replicated domain controllers at my DR site. I read through this whole thread, among others. I finally succeeded. As it seems like many people have had difficulty with this, I thought I would share my experience in case it can help others.

Two replica DC's trying to just power them up on their own vSwitch at the DR site and test them with a domain workstation over there. The problem was that active directory would not work. I did the following:

Powered each one up, logged in as local administrator. They said they had an abnormal shutdown. They both booted up in safe mode, so I went into msconfig and turned that off. They wanted to restart so I let them.

Then...
Command line: net stop ntfrs
Regedit: hklm\SYSTEM\CurrentControlSet\Services\NtFrs\Parameters\Backup/Restore\Process at Startup - Set the burflags to d4
Command line: net start ntfrs

After a minute or two, I saw my sysvol and netlogon being shared and all was fine! That was all I had to do. Of course it took a long time to get that process together from all the information in here.

Unison
Enthusiast
Posts: 87
Liked: 16 times
Joined: Feb 17, 2012 6:02 am
Full Name: Gav
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by Unison » Jul 18, 2013 11:13 pm 4 people like this post

You should DEFINITELY NOT have to use the burflags key to get your DCs and environment back up and running.
If i had to do that in a real disaster and got my environment back up and running with that key set, i wouldnt trust the AD recovery/environment and would fear that it would eventually fail in some horrible way.

I have tested my DC recoveries many times over and have got several comments in this thread - ive tried the burflags solution too and a million other things but in the end, i have found and given in to the fact that you should not have to do anything at all when you recover DCs - difficult for me to believe at first, but regular testing of DC recovery has proven to me that this is true. When you recover your DCs with Veeam (and probably other packages do this too) - you shouldn't have to do anything - the veeam recovery process prepares the process and your server OS's will do the rest.

When i moved to a full 2008R2 DC setup, i ran into problems again with this. My VMs were using the VMXNET3 NICs and this was causing massive problems with DC recovery - this issue only impacts 08R2 that are using VMXNET3 NICs. What was happening was that after restoring 08R2 DCs, their NIC details were all cleared so they lost their static details - so when i brought DCs online in the recovery environment, the could not reach each other until i manually went in and set their NIC details to the correct values....but by the time i could do this, the internal recovery process of AD appeared to be 'buggered' - DC recovery was unreliable. Then i found out that this is a known issue with 08R2 and MS have released a fix.....which i applied.....and now my 08DCs recover perfectly every single time without me having to do a thing (so long as i recover 2 of my DCs together - if you try recover just one DC on its own then you get in to the territory of needing to set reg keys, stop services etc). MS fix... http://support.microsoft.com/kb/2550978

If you want to recover your DCs for real or for a test - kick off the recovery of both your DCs at the same time, so that they come up online at the same time together on the same isolated network.....they will talk to each other and sort themselves out.
I have even tested this with recovering DCs at the same time but with a time difference between their recovery points and this also works successfully every time (now that i have applied the MS fix).

If anyone takes anything away from this now entire/massive thread its that you should not have to intervene in the DC recovery process at all - that's the way its designed.

tsightler
VP, Product Management
Posts: 5294
Liked: 2146 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by tsightler » Jul 19, 2013 2:23 am

james575 wrote:Powered each one up, logged in as local administrator. They said they had an abnormal shutdown. They both booted up in safe mode, so I went into msconfig and turned that off. They wanted to restart so I let them.
Correct, DC's are supposed boot up in safe mode (technically Directory Services Restore Mode), and Veeam should handle everything automatically at that point, typically after a delay of anywhere from 5-15 minutes based on the size of the environment. The DC should then automatically reboot without you having to do anything and come up in normal mode. If that's not happening then you should probably contact support as having to manually change thing with msconfig and burflags should not be required.

peteclem
Lurker
Posts: 1
Liked: never
Joined: Jul 02, 2013 11:30 am
Full Name: Pete Clements
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by peteclem » Jul 21, 2013 9:33 pm

Hey Unison,

This post caught my eye. I've recently done a DR test on a handful of servers at a head office site. The difficulty I had was with AD replication. I don't want to spin this topic off-course, but I'd be really keen to know if you or anyone has had the same issue or advice how to deal with it better than I did.

In total there were about 15 DCs for the remote sites. I restored a single DC from the head office site in an isolated DR test, and it took about 2 hours to come online (for AD and DNS services to come online). From what I could see the services weren't starting as they were trying to sync with all the remote sites before timing out (after 2 hours) and going online. The way I got around this was to disable inbound and outbound replication (using repadmin). This is fine for a single DC DR recovery test, but doesn't scale if I needed to restore 2 or more of the 15 DCs for the isolated DR test.

Your post mentioned services/reg keys that could be used in the DR test scenario. I appreciate that in a real DR situation, all the other DCs will be available and things should be much simpler. My problem seemed to occur because of the fact the other DCs weren't contactable and that caused delays in getting services back online.

Anyone seen/experienced something similar? So much material online assumes that all other DCs are available as if it were a real DR, and not an isolated test.

Pete

Jonathan
Enthusiast
Posts: 62
Liked: 1 time
Joined: Oct 15, 2009 8:52 am
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by Jonathan » Jul 22, 2013 6:50 am

Unison wrote:after restoring 08R2 DCs, their NIC details were all cleared so they lost their static details
I went through the same experience building an lab setup. Funny thing is, the DCs could still reach eachother through ipv6, just not ipv4.
This issue didn't come up with 2012 DCs (either upgraded or fresh installs).

Unison
Enthusiast
Posts: 87
Liked: 16 times
Joined: Feb 17, 2012 6:02 am
Full Name: Gav
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by Unison » Jul 22, 2013 7:02 am

haha - yes, thats right about ipv6. For a second there when i first noticed the DCs could talk even though there were no NIC (ipv4) details set - i thought, "this is weird" :).
Good to note this fact here because some people might be wondering why they can still ping the DCs by name and still have these problems.

Andreas Neufert
Veeam Software
Posts: 3392
Liked: 596 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by Andreas Neufert » Jul 22, 2013 7:28 am

This good old known lost IP address bug at restore hit all Win2008R2 on VMware Users hard at restore.
Because the Microsoft patch for this isn´t in the normal autoinstall patch rotation, VMware changed back the default NIC for new Win2008R2 VMs from VMXnet3 (vSphere4.x) to PRO1000 (vSphere5.x).

There are 2 patches available from Microsoft:
Win2008R2 without SP1
http://support.microsoft.com/kb/2344941
Win2008R2 with SP1
http://support.microsoft.com/kb/2550978

This bug is a really great example how SureBackup can help to identify problems at restore.
Even today 4 out of 5 of my customers who speak with me at the first Veeam live demo meetings, didn´t know about that patch and we can show them at POC that Veeam are able to detect that Microsoft bug and protect them for restore problems.

Jonathan
Enthusiast
Posts: 62
Liked: 1 time
Joined: Oct 15, 2009 8:52 am
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by Jonathan » Jul 22, 2013 7:36 am

Andreas Neufert wrote:Because the Microsoft patch for this isn´t in the normal autoinstall patch rotation, VMware changed back the default NIC for new Win2008R2 VMs from VMXnet3 (vSphere4.x) to PRO1000 (vSphere5.x).
Yeah, I noticed that. It's just that I ALWAYS have to use the latest bleeding edge features in production VMs. VMXNET 3, paravirtual, vmx-09, WS 2012... until something goes completely haywire.
This bug is a really great example how SureBackup can help to identify problems at restore.
+1

ori
Enthusiast
Posts: 65
Liked: 1 time
Joined: Apr 28, 2012 9:51 pm
Full Name: Ori Besser
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by ori » Jul 22, 2013 2:31 pm

tsightler wrote: Correct, DC's are supposed boot up in safe mode (technically Directory Services Restore Mode), and Veeam should handle everything automatically at that point, typically after a delay of anywhere from 5-15 minutes based on the size of the environment. The DC should then automatically reboot without you having to do anything and come up in normal mode. If that's not happening then you should probably contact support as having to manually change thing with msconfig and burflags should not be required.
What you describe is just not happening for us (and also for others as it seems for this thread). If we don't set the burflags key, the ntfrs removes the sysvol and netlogon shares after the file replication process does not succeed. It turns that loading the domain controllers is a major time consumer in our DR drills.

Andreas Neufert
Veeam Software
Posts: 3392
Liked: 596 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by Andreas Neufert » Jul 22, 2013 3:05 pm

In General the follwing things needed for AD Backup/REstore of a whole VM:
The AD VM you want to backup needs to be a Global Catalog
In the Veeam Job, use Veeam VSS Processing including an Domain\(Administrator Account) and passoword. (VMware Tools quiescense or Crash consistent will not fit)
At Restore or in the Virtual Lab connect the Network (without the Server maybe not jump in restore mode)
Do not login to the VM or Focus the mouse on the VM. This can Interrupt the recover process.
As Tom said it took some time till the Server is back. Try to start the VM in our Virtual Lab including appliation Check for GC/DNS/AD. The time it took here, the Server likely Need in production at restore till the AD Service are back.
The Server jumps into non authorative restore mode.

I will test the mentioned sysvol replication thing later this week an come back here.

james575
Novice
Posts: 8
Liked: never
Joined: Jun 18, 2013 4:07 pm
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by james575 » Jul 22, 2013 9:13 pm

Unison, I saw people in this thread talking about leaving everything alone and Veeam would take care of the DC/AD restores, but it did not at our site. I did however always power on one DC, wait a little while, and then power on the second DC. For a final test, I just powered them both on at the same time, will leave them running all night, and check them in the morning when I arrive.

For the record, I opened a case with Veeam support and they told me to change the burflags.

Unison
Enthusiast
Posts: 87
Liked: 16 times
Joined: Feb 17, 2012 6:02 am
Full Name: Gav
Contact:

Re: Veeam B&R v5 recovery of a domain controller

Post by Unison » Jul 22, 2013 10:51 pm

Hi James,
Originally, this is how i used to try DC recovery - exactly as you have mentioned. I would fire up one DC then wait a good 10 minutes before firing up the second DC recovery. I have since discovered that this is not the best way to do it.
What i believe is the correct way and the method that works for me every time is to recover at least 2 of your DCs together at the same time. For example.....

# recover both DCs using instant recovery but choose not to power them on
# When you see both DCs appear in vmware - change their networking details so that they are on an isolated network - but both are connected to the same isolated network!
# power them both on together as quickly as you can - i usually power them both up within a couple of seconds of each other
# they will both boot up in directory restore mode - during this time, they will both try to contact another DC on the network and because you have recovered 2 DCs at the same time, this process will succeed.
# they will both reboot on their own and after this point you should be able to login to them normally and AD should be up and running.
# If your two recovered DCs are not completely online and happy within 30-60mins from the start of your recovery then it is my experience that they will never come back to a working state without serious intervention like editing AD, setting reg keys etc etc.........so after 60mins, if your DCs are not perfect, give up on that recovery test because it failed.....start again.

I have now just got into the habit of recovering my 2 DCs every time i want to recover any of my app servers for what ever reason - be it just a test or to trial some new software install or an update. For example, if i want to test a new update on AppServer1 - i will first recover both my DCs to the isolated network first, and about 30mins later after they are happy and talking, then i will recover AppServer1 into that environment - then i will test the update. This way DNS, AD, everything is working as it does in the production setup.

The DC recovery process is very smooth and 'easy'. Some of my biggest problems were happening because i was not recovering 2 DCs together at the same time, like you i was recovering one, then waiting some time, then recovering the other - this was resulting in unreliable recovery (sometimes it would work other times not). My other biggest issue was the VMXNET3 NIC issue with 08R2. Once i applied the MS update i mentioned above and started recovering 2 DCs together at the same time - i get perfect DC recovery every time.

Are any of your DCs 08R2? If so, are they using VMXNET3 NICs?

Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot], Karinne and 19 guests