Veeam 9.5 - Replica Domain Controllers non-operational

[sacd]

Hello,

Below refers to Veeam case 02054866, huge thanks to Veeam support for assistance here.

Wanted to share a really interestig and weird issue with our backed up and replica Domain Controllers since upgrading to Veeam 9.5 Update 1. If you have upgraded to 9.5 and have not performed a full test restore of your backed up and replica Domain Controllers, I suggest you do.

We've been using Veeam since v6. All vm's are backed up with 'Application aware processing'. I perform a weekly DR test where I bring up all the replica vm's on an isolated network and do a full test of the environment ie, exchange mail, intranet websites, hosted custom .net and SQL applications. The DC's bootup, do the non-authorative restore, get to the login screen and the reboot, as per normal.
The 2 domain controllers primary DNS is set to each other, and the secondary is set to themselves, as per best practice.

Symptoms:
NLA reports that the servers are on 'unidentified network' - should be on 'Domain Network' - so all servers firewall profiles change.
Unable to login to any servers - No domain
No sysvol share
No netlogon share (no global catalog)
Exch DC can ping each other by IP, hostname and FQDN, even telnet on DNS and AD ports are successful.
NSLOOKUP to hostname works but to FQDN fails - request times out
So it appears that whilst there is full name resolution, AD DNS resquests to finds resource records is failing
Directory Service Log - Event ID 2088, source: ActiveDirectory_DomainService Level: warning

Fix:
Veeam support pointed me here (Thank you!!) https://kb.vmware.com/selfservice/micro ... Id=1020078
Installed hotfix on the replica VM's and rebooted. Removed and reconfigured nic as per above article
Shutdown File Replication Service on both DC's
Restored C:\windows\sysvol to DC holding the FSMO roles from a backup
Performed a authorative restore (burflags d4) of sysvol on the DC holding the FSMO roles
Performed a non-authorative restore of sysvol (burflags d2) on the other server.

Once this was completed, I rebooted the DC's, they came up perfectly, on the Domain Network. Then rebooted the other member VM's and all was ok.

Hope this helps someone else.

Some helpful Links
https://www.veeam.com/blog/how-to-recov ... ction.html
https://groups.google.com/forum/#!msg/m ... kT394rNgEJ
http://doitfixit.com/blog/2013/04/17/re ... gon-share/
http://kpytko.pl/active-directory-domai ... store-frs/
http://www.extremesanity.com/blog/?p=165

[bryanmeche]

During our last DR test this one was a head scratcher for a bit. We ended up getting it going following a very similar fix, but I'm still wondering what changed in Veeam where we have to do this in the first place. Our previous DR tests and ad-hoc tests we never had to do this, we just booted up the DCs and they worked without having to even login. Now we notice the DR boots in Domain Restore mode (safe mode), where it sits for about 3-5 minutes then reboots normally. However like you we have to set the BURFLAGs in the registry for them to finally come up.

I liked it better when they just came up normally. Instead of a one-click launch for lab or DR tests, we have to access every DC and make the changes for them to work.

[Gostev]

Yes, this is a well known issue with legacy Windows versions and VMware. It is not specific to Domain Controllers and impacts a number of Veeam features (restore, replication, quick migration, sandbox). Normally, users catch this issue with SureBackup jobs, so you can find a few pages worth of posts about this issue if you search forums for "SureBackup VMXNET3".

[sacd]

Thanks - but this has never been a problem. Not in the 5+ years we've been using Veeam.