Suggestions for Backup Infrastructure

[LakersFan]

I was looking for any suggestions or tips on how I should setup my backup infrastructure as best as possible. Forgive my ignorance as I'm going through a learning curve and I've never had to do this before.

I have 3 identical servers (Poweredge R510, 2 x Xeon X5670 CPUs, 64GB RAM, 32TB of storage but will end up with 27TB using RAID6)

The base OS will be Windows 2012 Datacenter with Hyper-V as we might spin up a couple of VMs to use for monitoring. The OS is installed on the 500GB built-in drives (RAID1) C Volume and the rest of the drives are configured as RAID6 to get maximum space so will end up with 27TB of storage as D Volume. This D Volume is also configured as Thin Provisioning and the built-in Windows 2012 Data Deduplication enabled.

The plan is to install 1 server on-site at the datacenter in Phoenix, another server at a remote office up in Vancouver, and the 3rd server at a colo facility. I will use Veeam to backup the VMs (Esx 4.1 managed by vCenter) locally at the DC in Phoenix and the server room in Vancouver, then ship/replicate the backup to the colo facility (via rsync or a similar tool, any recommendations?). There is about 6TB of data in Vancouver (about 5 servers) and 16TB of data in Phoenix (about 100 servers).

As mentioned, Besides the physical server built-in redundancy on the servers (RAID, Dual PS, etc....), the data will be replicated using rsync or a similar tool to the colo facility. I would really appreciate your input whether or not this sounds like a good plan.

I hope I had covered everything and I really appreciate your input. Again, I'm learning all this so forgive my ignorance.

-TC

[veremin]

It might be worth reviewing the existing topic which is primarily concerned with the idea of having both: a local and off-site backup copy.

Additionally, please be aware that previous naming convention should be enabled in case of utilizing rsync and reversed incremental backup mode. Thus, only block level changes will be transferred by rsync and not the backup file as a whole. It can be done by the means of the regkey mentioned in the said topic.

Hope this helps.
Thanks.

[Yuki]

1) you need a big pipe if you are working with that much data and that many VMs.
2) server 2012 (at least in our case) has unexplained and unexpected large number of changed blocks on VMDKs causing backups to be very large (still investigating)
3) you may need to use registry change to force stable file names for RSYNC
4) you could use Veeam CloudBackup instead of RSYNC to move the files
5) you want to make sure your TARGET storage is fast. Doing reverse incremental or synthetic backups on certain RAID5/6 systems is very slow and target storage may become your bottleneck. On our Dell R720xd in RAID5 on 3TB NL SAS we see veeam reverse incremental jobs top out at about 35MB/s (of course active fulls or first run will go at about 90-110MB/s).
6) you want to have at least one VM in each site dedicated as your proxy to compress/decompress the data between site transfers.
7) deduplication may only give you good results if you don't use reverse incremental as those files will not get a chance to be duduplicated (last backup is always largest and "full")
8) if your VMs are very large - you may want to adjust compression/deduplication as otherwise Veeam may run into trouble with very small blocks.
9) you would probably benefit from WAN optimization devices like riverbed.
10) I would not use vCenter and instead would add the vSphere hosts to Veeam separately. If your vCenter goes down or DC is not available - you may run into issues with restoration process
11) you don't need to backup Veeam VM itself, but make sure to setup Veeam configuration backup to a remote site and then replicate those to the third location (files are pretty small compared to VM backups).
12) on RAID 5/6 and very large VMs - removing helper snapshots may take a lot of time (it looks like you are running on SATA/NL SAS drives - 7200RPM).

Working with IO Datacenter in Phoenix is a pain, especially if you are only setting your stuff up.

[veremin]

4) You could use Veeam CloudBackup instead of RSYNC to move the files.

While VeeamCloud Backup can be used in case of you moving files to Windows share and utilizing forward incremental mode, I wouldn’t recommend doing it in case of reversed incremental mode, since VB&R CE isn’t capable of catching changes on the block level; therefore, each time the whole .vbk file will be transferred.

Hope this helps.
Thanks.

[LakersFan]

Thank Eremin and Yuki. From the reading on that post, it looks like rsync may not be the best solution for this as it sounds like it has to read the VM on both ends to determine the diff.

Yuki,
- I like the idea of the Cloud version. How would I setup my servers at the colo to be cloud storage in that case? OpenStack or something similar?
- You are right, the drives are SAS drives 7200.
- I will certainly connect directly to the host instead of going through vCenter.
- I'm guessing what you're saying is setup the servers in Vancouver and IO to act as proxies and the main one at the colo?

Thanks again you guys, I think I should setup a one on one with a Veeam architect and see what he would suggest.

[veremin]

I like the idea of the Cloud version.

As I’ve mentioned already, cloud edition in an option only if you’re using forward incremental mode.

Otherwise, if it’s reversed incremental mode that is being used and you’re willing to constantly (after each job execution) update .vbk-file on remote location with the changes that have happened since the last job run, you should put into use something like RSync, that is built for block-level file synchronization.

How would I setup my servers at the colo to be cloud storage in that case? OpenStack or something similar?

For more information about Veeam Cloud Edition and OpenStack configuration please see this topic.

- I'm guessing what you're saying is setup the servers in Vancouver and IO to act as proxies and the main one at the colo?

If you’re using remote location as backup repository there is no need to deploy additional proxy. Veeam B&R is using two-agent based architecture and all of features like throttling rules, compression, etc, are implemented between them. In case of backup, these features would be implemented between backup proxy in your production site and backup repository.

Hope this helps.
Thanks.

[Yuki]

- I like the idea of the Cloud version. How would I setup my servers at the colo to be cloud storage in that case? OpenStack or something similar?
- You are right, the drives are SAS drives 7200.
- I will certainly connect directly to the host instead of going through vCenter.
- I'm guessing what you're saying is setup the servers in Vancouver and IO to act as proxies and the main one at the colo?

Thanks again you guys, I think I should setup a one on one with a Veeam architect and see what he would suggest.

As Eremin pointed out - only use Cloud for forward incremental, otherwise R-SYNC is still a good choice if you implement file name registry change. You don't need to use OpenStack, but you can if you want to. A VPN over internet between sites or layer 2 link (Metro Ethernet, MPLS, etc) will work just fine if you have enough bandwidth.

Well, if your target repositories are Windows hosts - then you don't need to deploy the proxy role "per se". If you are going to backup anything in the remote sites ( Backups from A and B going to C) then both A and B need a proxy. One of them may be the veeam server itself if that's how you want to structure it. Then C will have Windows repository role/agent installed. As far as vCenter goes - there are benefits and downsides to both ways of doing this. Veeam is able to track and protect VMs moved through vCenter by adding them to jobs on pool level i believe, but that won't be the case if you configure veeam to connect to individual vSphere hosts. On the other hand if your main site goes down - you can still perform a lot of tasks that would otherwise would rely on vCenter and domain controller. It just depends on how big your organization is and how you want to structure it. In large deployments you have to use vCenter simply because there are too many systems to manage otherwise and in order to utilize single sign-on. In small installs ( a few servers) you can manage systems manually without vCenter.

I highly recommend reading the user guide as it covers different scenarios pretty well.

[wirthon]

I have imported , ran a restore, ran the ovf tool. and have stood up a VM in a offsite as a Disaster Recovery . Now as this new DR vm runs how do I reverse to have any changes made in the DR back to the production VM ?

[foggy]

If I'm understanding you right, you want to replicate the VM back to the production site?