Veeam Newbie, a few questions

[HendersonD]

I have read through the FAQs and a lot of documentation and had a few questions. Our setup:
Production
4 ESXi 6 hosts
45 VMs, mostly Windows servers but a few Linux boxes
5TB of data sitting on a new Nimble all flash array using 10 gig iSCSI
3TB of this data is files/folders since we are a K-12 school district. We have two virtualized Windows Files servers that house this data

DR site
2 ESXi 6 hosts
New Nimble CS300 array using 10 gig iSCSI

Production and DR are connected across a 10 gig link since we are a single campus
We have an Exagrid box located at our ISP 20 miles away
We will be using Veeam to backup from Production to DR. We will be using a backup copy job to send data off campus to the Exagrid
We will also be using Veeam to replicate between the two Nimble arrays

Questions

1. We have two domain controllers in Production and one in DR. My thought was to never replicate the DCs in production and if we have a disaster just promote the one DC in our DR site. Is that sound?
2. Can Direct SAN mode be used when the proxy server is virtualized? I do have one unused physical server that I could make the proxy server for Direct SAN but it seems like virtualizing the proxy is the way to go. I am also considering Hot Add mode so any thoughts on which mode to choose would be appreciated. We currently run Commvault and the Mediaagent (equivalent to Veeam's proxy) cannot be virtualized.
3. In our DR site we only run two VMs. A domain controller and a Ruckus wireless controller. The other Ruckus wireless controller runs in Production. In order to protect the Ruckus controller in our DR site I am thinking about doing replication both ways, can that be done?
4. Are there any other design considerations that you can pass on given our situation?

[DaveWatkins]

First a question, are you planning to backup to the DR Nimble, and presumably that is the same array that's used for the production VM's in your DR site? In your case it won't matter too much since you have 2 VM's there but if you start putting other stuff there and you lose that array then you lose the VM's and the local backups for them, which isn't ideal

1. it would work, but it will cause you some issues as you'll have to seize the roles form the now down servers and then do a meta data cleanup on the AD. That's doable but not something I'd look at doing in a disaster scenario. Assuming those DC's are only DC's and do nothing else, I'd just back them up, it'll only be 30GB or so each and that'll compress down to 15GB
2. For iSCSI technically yes, but why would you? The point of Direct SAN mode is to take network load off your ESX hosts and if you did this you wouldn't actually be achieving that purpose. If you're 10Gb everywhere it may not matter but given the extra complexity of Direct SAN I wouldn't bother since it won't buy you anything. It's unlikely to buy you anymore speed over HotAdd.
3. I don't see why you couldn't as long as the host OS is supported for Re-IP. Do you need too though? Can you just spin up a new VM, load the controller software and point it at the running one in Production and it'll sync all the config details over? Or are they an Active/Active pair and you actually get downtime when one fails? When you say replication both ways, you mean replicate the production controller to DR, and the DR controller to production? If so then that's fine, again as long as the OS is supported for Re-IP and your AP's can be made aware of an IP address change of one of their controllers
4. You don't mention the link speed to that Exagrid box or what Veeam license you're looking at but if the link speed is low you could look at using the WAN Accelerator if you're getting/got Ent+ Veeam Licensing. You'd need a VM or host running in the same place as the Exagrid to do that. Also, Have your Veeam Management server do only that, and replicate it between sites. That way if you lose a site you still have a working Veeam server to restore backups from

[HendersonD]

The Nimble in our DR site will be the backup repository for the Production site
The same Nimble array in our DR site is also running the two VMs, a domain controller and a Ruckus wireless controller

I just looked carefully at the Ruckus documentation. The two controllers load balance the access points but if one goes down, the other controller takes up the load. You are correct, the easiest way is to just spin up a new Ruckus controller from the OVA file. In this case, there is no reason to replicate from the DR site to Production. Our only replication will be from Production to DR.

From everything I have read, domain controllers should never be restored from a backup so having DC in Production as well as the DR site is the way to go. If either side goes down, there are still domain controllers available to run environment so I do think I will even backup my DCs unless there is a good reason to do so.

Thanks for the information about just using Hot Add with 10 gig connections, seems like a sound way to go. I have a 500Mbps connection to my ISP and we did purchase the enterprise version of Veeam so I am thinking we should be OK. The initial backup copy job will happen with the Exagrid on campus at 10 gig lan speeds and then we will rack it at our ISP. My understanding is every job after the initial will be an incremental so our speed to our ISP should be sufficient.

Should the Veeam management server also be the proxy server or am I better of with a separate proxy server? I have read that with two separate servers the management server should be at the DR site and the proxy server at the Production site

[DaveWatkins]

You need proxies wherever you're reading or writing data, so you'll need them at both sites anyway (you may not need one at your ISP for the Exagrid though, but you'll want to confirm that). I tend to make my management server only a management server and remove all the other roles that install by default to simplify the deployment.

Domain Controllers aren't interchangeable quite as easily as a lot of people think. There are a number of roles that run on what is basically a PDC. Without that DC things will go badly until it is rectified and if that DC is in your main building then problems will start cropping up if it's down. In saying that I'm not sure what happens to those roles if you do a non-authoritive restore (which Veeam does) or you failover to a replica. I woudl assume a replica would continue to host them without issue but a restore from backup.. not sure.

To an Exagrid you probably actually want to look at doing full backups to make the most of the dedup, although assuming you're using DDBoost you should be able to do synthetic fulls which will have the same data transfer rate as an incremental. Something else to think about is the distance to your ISP's DC, a decent natural disaster can span 20 miles easily

[HendersonD]

So I will need a Veeam proxy server running under ESXi in my Production site AND my DR site? I kept thinking I only need one proxy server to Hot Add the VMs and provide a path to the backup repository.

I agree on the domain controllers. If Production goes down, I should be able to take the one DC in my DR site and promote it to PDC. My DCs also run DNS and DHCP to the one in my DR site will just handle those services if the DCs in production get wiped out.

You are correct that certain disasters can cover a huge area. We are located right outside Rochester NY so this type of disaster has very low odds. We are not affected much by hurricanes, tornadoes, wildfires, earthquakes, etc.

[DaveWatkins]

Yup, proxies at each site, possibly more than one depending on how much throughput you want.

I think you're missing my point about domain controllers, you can't "promote" a DC to pickup the FSMO roles. If your existing DC that holds those roles goes away, you have to seize them and cleanup AD https://support.microsoft.com/en-us/kb/255504

It's not an entirely pleasant process and not something I'd want to deal with when in a disaster situation but is possible to do, it's just not seamless and firing up a replica is likely much easier.

Wasn't there a good storm a few years back that cut power to some fairly major areas for a decent period of time

[HendersonD]

Yes, a bunch of years back we had an ice storm and large swatches of greater Rochester lost power. In this case, there is no data loss just loss of service until power was restored. IF we had an event (fire, flood, nuclear war, etc) that could wipe out my data center and my ISP 20 miles away, I think I have bigger problems

I know that Active Directory is very time based. I thought I read that if a domain controller is restored from a backup, even a recent one, that it causes all kinds of havoc since events in the database have time/data stamps that are not found in the remaining DCs that are running. For that reason, I thought that seizing FSMO roles was the better route to go

[tdewin]

It can actually. That is why Application Aware Image Processing reconfigures the VM to start up in DSRM mode, and restart the server automatically at recovery time. The VM is than aware that it was restored from backup and will correctly resync (non-authoritative) with surviving peers. Recently we added this great kb article here:
https://www.veeam.com/kb2119

Notice that 2012 domain controllers + a supported hypervisor have a device called a VM generation ID. When this ID changes (on major VM event like clone or snapshots), AD will be aware that it might have gone "back in to time"