Long Term Labs, DC restore and suggested Veeam changes.

[notalabexpert]

Wondering if anyone can back me up/provide input/or define if i am going about it wrong, to help in our case example below.

Please see Case # 01821816 notes below. It was suggested by veeam support to post this to the forums to make it more visible to the devs.

Initial notes

Long story. Environment. 2 domain controllers 2012 r2 running frs not dsf-r. in 2012 funtional level domain and forest. Veeam 9. All other servers 2003 or higher. Most 2008r2. Problem- We use veeam surebackup and the restore process of domain controllers usually works great and everything is good to go. But veeam surebackup is not a perfect solution for us to do real lab work due to the possibility of having to reboot the veeam server and it tearing down the lab. We have also found the the sharepoint does not play nice with the labs in long term standups. So we normally (not sure if supported by veeam) stand up a long term lab by creating a lab!! switch using sureback switch creation process with static routes. This creates the lab switch in vmware. We then manually power on the switch with vmware. Once this is created we restore the latest backups of our DC’s into this lab switch. I have been testing working this since veeam 7 and with our old 2008 domain controllers basically with the same results. Our domain controllers are backed up with application aware process but I also tried it with no application aware enabled. The domain controllers either A: power on synchronize in 30 minutes or so and all is well (this is rare). B: Come in a unknown state and maybe synchronize after a few hrs and work ( rarer) or C: Come up in unknown state and never synchronize and stay that way until I go through the registry repl perform initial sync hack and restore one of the dcs in burflag d4 authoritative. But this process usually drops the sysvol netlogin scripts and policies folders so have to manually create these. Need to understand what possible changes we can make to make this process smoother or if veeam can create a process to do all this behind the scenes or just give up and deal with it.

Response from Veeam support

You have done an impressive amount of customization with this. It's a very creative solution.
While we can't support this, your use case is a perfect question for the forums at Veeam.com, and there's a good chance our developers may chime in with some ideas if you post the question there.

Response to Veeam support

After many (read 1000s) of google search’s and tests I have found that its never the same result twice. Let me detail a bit more.

We have the requirement to set up long term labs for sharepoint, citrix, adfs and Sql. We currently have 9 labs and one has been running for over a year now. We do single vm refreshs into the labs and also full lab refresh’s based on testing requirements.

Perfect world……………
I have been actively using this process for a couple of years now. After all the google foo I have done and testing I think that veeam would be interested on expanding the product based on the below and finding an active solution. Both as a restore testing process and as a lab buildout process. As to the fact that the surebackup process seems to work correctly with the domain controllers I think it could be ported to a long term lab solution.

I found 100s of users in the veeam forums and elsewhere that used labs heavily. All seem to have the same issues with the domain controllers. I really have tested all the possible fixes and none seem to work every time. The requirement to copy a portion of the production network seems to be a standard want or need.

What would be a solution would be in my thoughts.

1. Capability to create a long term lab switch either single host or across multiple hosts. I do understand that this is part vmare config also. Clean up the existing capabilities of veeam connecting to this virtual switch (I have lost the capability many times to connect to an existing switch from veeam to vmware due to config changes, having to reboot a host etc.. Also create an export import process for static routes (hint, hint#1) or allow pasting of a list of static routes. For example see below one of our existing long term labs. You would be amazed how many times I have had to recreate these size of labs static routing tables. I would love to be able to just cut and paste as opposed to add an individual line and click next and click add again and so on.

192.168.3.81 192.168.3.248 FISADC30
192.168.3.40 192.168.3.249 FISADC31
192.168.1.25 192.168.3.250 FISSQL04
192.168.3.47 192.168.3.251 FISSQL10
192.168.1.13 192.168.3.252 FISWEB03
192.168.3.45 192.168.3.253 FISCTX10
192.168.3.200 192.168.3.195 FISADFS18
192.168.3.203 192.168.3.196 FISAPPS18
192.168.3.204 192.168.3.197 FISBICR18
192.168.3.205 192.168.3.198 FISSQL18
192.168.3.206 192.168.3.199 FISWEB18
192.168.3.135 192.168.3.243 FISSQL16
192.168.3.136 192.168.3.244 FISADFS16
192.168.3.137 192.168.3.245 FISWEB16
192.168.3.139 192.168.3.246 FISCRWL16
192.168.3.140 192.168.3.247 FISBI16
192.168.3.184 192.168.3.212 TVCTX20
192.168.3.202 192.168.3.254 TVSQl20
192.168.0.82 192.168.3.213 DMCTX20
192.168.0.76 192.168.3.214 DMSQL20
192.168.3.130 192.168.3.209 C04CTX20
192.168.3.120 192.168.3.210 C04SQL20
192.168.3.208 192.168.3.207 FISLIC20
192.168.3.55 192.168.3.194 FISCTX09


2. Basically create a restore process for restoring one or two existing domain controllers into that lab switch (hint, hint#2) from backup and have them work (same as surebackup labs). Preferably one would be the perfect solution as it requires less vm resources but I understand that the solution may require two domain controllers. I have made it work with one in the past but it requires many reg key changes. I have found a way that works by creating the lab switch and cloning in vmware my two production domain controllers into that switch network ( amazed me that it works every time)
3. Create a process for just refreshing machines in the labs (hint,hint#3). Sometimes my labs have been running for months and I have to restore a new copy of for example a sql database machine. As the domain controllers have been running in the lab for long term the new restore of the sql vm does not allow login as the active directory on the old dcs does not recognize the machine. Disjoining the machine and rejoining fixs this problem easily. So creating a restore process that asks do you want to disjoin and rejoin this machine to the domain after restore and have that complete during first boot and then reboot.

From what I have found I would think that veeam could see the functionality of all the above with as many users I have found out there trying to do something similar. I understand that this may require possible have two diff process. Surebackup as it exists, plus the capability of the copy paste static routes above, for what veeam designed it to do. And a Surebackup Full Lab process including all the above. Any smart IT group has labs. We just expand on it a bit in our environment. Having sharepoint in diff stages of dev testing is mainly what our requirement is. I cannot believe that veeam doesn’t see this as a greate capability. I am not a developer programmer just know enough about the whole process to think this is doable and sellable. Wish I could write it myself and sell it.


Response from veeam support

Switch configuration should be permanent, but routing stops and starts with the virtual lab. With regard to persistent switches for the lab that potentially span multiple hosts, I think you may be describing a use-case for a virtual lab configured in Advanced Single host mode with Distributed Virtual Switches, or Advanced Multi-Host mode similarly (if VMs in the SureBackup job will exist on several hosts at once and must connect to each-other on the same private layer 2 network.)
https://helpcenter.veeam.com/backup/vsp ... tions.html

https://helpcenter.veeam.com/backup/vsp ... _vlab.html

https://helpcenter.veeam.com/backup/vsp ... ihost.html

As far as Domain Controller behavior, there is a complex relationship between our agent processes and runtimes, and the patches that Microsoft releases for Windows Server. I have noticed that outcomes become more stable with newer versions of Veeam Backup & Replication on the backup server, and Windows Server on the DCs. When Guest Processing (Application Aware Image Processing) is used to back up an ADDC, (as per our requirements) quiescence is done, and a small runtime is injected to bring the DC into an authoritative mode on its next startup. SureBackup uses the DC role (when assigned) to trigger some pre-boot modification of the DC's registry on disk to set BURflags and preconfigure it for a successful boot. This works when the DC is a DNS server and holds the global catalog, although it's not clearly documented which FSMO roles must be held to ensure successful testing every time, and boot times are a factor when the DC wants to synchronize with the rest of the domain or forest before finally timing out on sync and automatically assuming authoritative status.
In the virtual lab, as long as an authoritative DC is the first item in the Application Group, and the internal IP address of the vLab being presented to the VMs being tested is the same as the default gateway they expect to see when they power on, authentication and network location services should successfully set their network profiles to the domain profile and allow all the functionality and connectivity they expose in production.