DR Replication Advice Needed

[MoifMurphy]

This is our scenario. We have vCenter 5 on our main site and on our DR site. The two sites are on a 100mb WAN on the same IP range. We currently run in excess of 40 servers with 5 of those being at least 300GB. Even with proxies in place and the concurrent jobs set to more than we are comfortable with we are finding that the replication is taking far too long. In some cases we are looking at over 20 hours. Obviously this is causing a bottleneck and we aren't happy with what we are seeing. We need to have servers replicated withing a 24 hour period for failover.

We've broken down our servers into daily and weekly replication jobs to try and alleviate some of the stress but even in this situation there is a massive overlap. To copy a 300GB VM over a 100mb line is going to take 6hrs easily. The other thing we need to consider is that we are taking VMDK level backups with Backup Exec as well so we have to schedule the jobs so that they don't conflict.

Is Veeam going to copy over the entire 300GB on a daily basis? Or once the initial replication is done will it only scan for changes? Perhaps replication isn't what we need?

Any advice much appreciated.

[foggy]

Replication is similar to forward incremental backup meaning that during the first run the whole VM data is copied and all subsequent replication job runs are incremental (only blocks changed since the last replication cycle are copied). Btw, this is described in the product user guide detail.

Assuming you have proxies set up in both source and target locations, do you have also the source-side backup repository specified to store replication metadata?

[MoifMurphy]

Yes, we are using Backup Proxies as per the User Guide and our Backup Repository is the server that Veeam 6 is installed on. I'm having trouble finding where in the user guide where it stipulates how much space is required on the Repository to store the metadata. We have 11GB free, my instinct tells me this isn't enough.

Please advise.

[foggy]

More than enough. Approximate disk space required is about 40MB for every 2TB of data replicated (that is, with the default block size).

What does the bottleneck analysis say and what transport mode is used by both source and target proxies?

[MoifMurphy]

Transport mode at the moment is set to Automatic, I did consider changing this to Network.

Out of the jobs that have successfully completed the bottleneck varies between 'Source' and 'Target'. We've disabled all jobs at the moment other than our two largest servers. Server A is 360GB and is being sent down for the first time (started at 09:44 this morning and is 37% through) and Server B is 420GB and already has a replica in place (started at 13:10, is 7% through but has stalled).

I am assuming Server A has saturated the bandwidth only allowing Server B to 'burst'? The 100mb line we have is dedicated between the two sites, am I looking at setting up throttling here?

[foggy]

What is the transport mode effectively selected for each proxy? You can check this by opening Realtime Statistics and selecting the VM to the left.

[MoifMurphy]

The only information I can see relating to Proxy is the following:

Code: Select all

14/03/2012 13:18:35 :: Using source proxy VMware Backup Proxy HQ [hotadd;nbd]
14/03/2012 13:18:40 :: Using target proxy VMware Backup Proxy DR [hotadd;nbd]

The bottleneck on this one is 'target'. This is Server B by the way, 420GB.

[dellock6]

In optimal conditions, 100 mbits WAN allows you to do replicas at 12.5 MB/s. The 420 GB VM so will be fully transferred in 9.56 hours "at best". Subtract tcp and other network overhead and (if the link is used ONLY for replicas) you can expect speed between 10-12 MBs.
Can you check on the replica job statistics if the speed you are seeing is near to this limit or not?

[foggy]

Agree on the initial job run, but we are not talking about transferring the whole VM data each time, are we? Or does this VM produce this amount of changes between replication cycles? Is CBT enabled on this job?

[MoifMurphy]

In optimal conditions, 100 mbits WAN allows you to do replicas at 12.5 MB/s. The 420 GB VM so will be fully transferred in 9.56 hours "at best". Subtract tcp and other network overhead and (if the link is used ONLY for replicas) you can expect speed between 10-12 MBs.
Can you check on the replica job statistics if the speed you are seeing is near to this limit or not?

Just as we expected then. I was wondering if we had missed some vital component within Veeam that would help things along but it's pretty much down to data versus bandwidth.

Are we not able to allocate bandwidth per job? That would be ideal.

Agree on the initial job run, but we are not talking about transferring the whole VM data each time, are we? Or does this VM produce this amount of changes between replication cycles?

These are pretty busy servers. One being an Exchange server and the other an inhouse application server. Disk activity is constant.

We're also wondering if it matters if the disks are Thick or Thin provisioned and does the replica take into account total capacity of disks or only used space on subsequent jobs? Unfortunately we don't have uniformity when it comes to server creation, for example the 430GB server is 9 x Thin Provisioned disks with used space totalling 170GB and the 300GB Exchange server is 3 x Thick Provisoned disks comprising of 2 x 30GB and 1 x 300GB.

[foggy]

Are we not able to allocate bandwidth per job? That would be ideal.

Not per job but you can throttle traffic between proxies (look under Tools > Traffic Throttling from the main menu).

We're also wondering if it matters if the disks are Thick or Thin provisioned and does the replica take into account total capacity of disks or only used space on subsequent jobs?

As was mentioned above, only blocks that have changed since the last replication cycle are copied, all zero blocks are excluded from transfer (although unused space not necessarily consists of zero blocks).

[MoifMurphy]

Thanks for the help and advice. It's definitely given us food for thought.

[Gostev]

I was wondering if we had missed some vital component within Veeam that would help things along but it's pretty much down to data versus bandwidth.

The vital component you seem to be missing is replica seeding. Typically, you want to do full sync over removable storage versus WAN. After you seed your replicas, the replication jobs will only be transferring deltas between current and latest state, so it is much less data to send over WAN (up to 50x less comparing to initial full sync).