Backup and Disaster Recovery scenario

[johneijg]

Hi I'm looking for some advice in the design of a Backup and Disaster recovery scenario using the Veeam B&R 3.1 software.

Scenario: Mid-Size company with 2 remote offices.

HQ: 5TB of active Data on a HP EVA 4400 SAN (FC, 6TB), 6 ESX hosts, with about 32 VM's with various roles (Citrix, SQL, File & Print, Exchange). Both remote offices host a File & Print server backuped to Tape locally and also replicated to HQ using DFS (1TB in total, included in the 5TB).

Requirements: Backup and Disaster Recovery scenario in which the following requirements are met:
- Offsite backup of all data.
- 24 Hour Recovery of critical systems with about 25% or more of the performance at the core location.
- Maximum data loss 48 hours.
- Instant restore of data of at least one month.

In the current situation we are running Backup Exec 11d with VCB and specific agent for all VM's and Applications. VM's are directly written to LTO3 using an HP Autoloader, application data such as Exchange and SQL are written to disk and duplicated to tape. The tapes are stored offsite.
It works for backups but restores take a lot of time and disaster recovery isn't straight forward.

The management agrees and wants to invest in improving the situation.
We are in the lucky position to have a parent with a remote data center which we can use as a disaster recovery location.

With the above requirements I came up with the following scenario:
- Extend SAN capacity with 12TB FATA nearline storage.
- Purchase Server with sufficient storage for disaster recovery at remote datacenter.
- Implement Ethernet WAN connection to remote datacenter (required bandwith??).
- Daily Backup VM’s using Veeam Backup to the 12TB SAN partition.
- Daily Backup applications (SQL and Exchange) to 12TB SAN partition using Backup Exec 12.5.
- Duplicate above data to TAPE and store offsite using Backup Exec 12.5.
- Plan is to perform an initial replication of the VM’s locally at the HQ and then move the loaded server to the remote datacenter. The daily replication should then only require to replicate the delta’s.
- Replicate VM’s to ESX server at remote location using Veeam Replication via WAN connection.

Now in theory the above scenario should work, but what kind of performance can I expect from Veeam?
How will it perform over a WAN connection?
Am I correct to think that the replication is a separate process from the backup job and that the data in the backup and replication can differ?
Is there an impact in ESX performance running the replication during working hours?

Quite a lengthy post but I hope to get some of your feedback.

[Gostev]

Hello John,

You can get idea on some backup performance from this post, for example.

Replication will perform fine as long as WAN connection has decent speed and latency is not too high, since each WAN link is different, you should simply replicate the test VM to test this in your actual conditions. One problem with replication is that current Veeam Backup version does not support replica seeding, so initial sync will have to go over WAN. However, this has been already addressed in the next release.

Yes, replication is done in separate job from backup, so the data can differ. If you do not want that, they you can sync your backup files to DR site instead (but obviously restore from backup is much slower than just starting up replica).

When replication (or backup) job is running, the only notable impact on production ESX is snaphost presence. Snapshot is created when job is started, and deleted as soon as backup or replication cycle completes. Snapshot presence during backup or replication job cycle has the following drawbacks:
- vMotion/DRS/DPM is disabled for the corresponding VM
- all writes go to snapshot files (no notable performance overhead, but extra storage required)
- snapshot deletion, although does not take long, reduces disk I/O performance for the corresponding VM when performed

We have addressed all these drawbacks in Veeam Backup 4.0 by providing near-CDP replication capabilities (replication cycle takes less than a minute, so all drawbacks above are no longer noticeable). However, this functionality requires ESX 4.0 hosts and will not work on ESX 3.5 hosts (it leverages some new vSphere capabilites). Also, while this already works in our lab, Veeam Backup 4.0 is not released yet (we are aiming for late Q3).

Hope this helps!

[johneijg]

Thanks for the feedback.

We are currently looking at a WAN link between 10 and 50Mbit with an average 8ms latency.
The size of the link will mostly depend on the exact amount of data which is required to replicate daily.
Are there any best practices to calculate the data which will need to be replicated with Veeam?

Does Veeam use ssh to transfer the replica? Are you aware of any network optimizations or shapings which could improve the througput?

Isn't is so that if we would setup the destination server locally, on a network segment with same the IP configuration as the remote network, we then would be able to do the initial sync locally?
We can then switch over the network config to the remote site not changing anything to the Veeam setup or the destination server config.
The server would then be moved to the remote location.

[Gostev]

We are currently looking at a WAN link between 10 and 50Mbit with an average 8ms latency. The size of the link will mostly depend on the exact amount of data which is required to replicate daily.

Link like this would be simply great.

Are there any best practices to calculate the data which will need to be replicated with Veeam?

We do block-level incremental sync, where block size is 1024KB. We will transfer all blocks that got "dirty" since last sync, so amount depends heavily on:
- raw volume of changes (obviously)
- operating system (Windows likes to spam disk much more than Linux, even if user "does nothing" )
- state of disk (very fragmented disk result in minor changes scatter all across the disk)

To get a good grasp on expected volumes, you can simply test this with local replication.

Does Veeam use ssh to transfer the replica? Are you aware of any network optimizations or shapings which could improve the througput

No, we do not use SSH, so you can use things like Expand Networks Accelerator box and it will do great.
BTW, you may find this information useful: http://www.veeam.com/forums/viewtopic.php?f=2&t=908

Isn't is so that if we would setup the destination server locally, on a network segment with same the IP configuration as the remote network, we then would be able to do the initial sync locally?
We can then switch over the network config to the remote site not changing anything to the Veeam setup or the destination server config.
The server would then be moved to the remote location.

Yes, this is exactly how some of our customers are doing this today to overcome the absence of replica seeding.