Replication Traffic.

[thuizenga]

I am working on setting up replication over our 20mbps wan connection and feel that the replication times are to long.

For instance i replicated a windows xp machine which transfered 3.37 GB of data and it took 4.5 hours. By my calculation transfereing just the raw data should take less that 30 min. When the server i am replicating to was in our data ceter (or connected via 1000mbps) delta backups only took 5-10 min.

I am guessing that i have somethign configured wrong.

I am using Vmware 7.0 hardware and am on vmware 4.0.1. I am using VMware vStorage API in SAN only mode. My production servers are running ESX and my DR server is a Paid ESXi.

[Gostev]

I would say that something is wrong with WAN connection settings, if the same job performs fine locally. Try monitoring your connection while the job is running, see what is the load and so on.

[thuizenga]

How much network traffic is required besidesd replication. Also does this have anythign to do with using esxi without a service console?

[Gostev]

Not sure I understand your first question. As for the second question, full ESX would have been better on target comparing to ESXi, as with full ESX we can use service console agent to do traffic compression.

[Bunce]

I think he means that the statistics shown in the VBR jobs don't reflect actual figures - ie the backup rate is misleading as it includes the entrie disk size, and ignores blank blocks, replication, CBT etc..

We found the size of VBR files changed at desitination was only about 50% of what was actually transferred.
Refer statistics here: http://www.veeam.com/forums/viewtopic.p ... 607#p15480

Hoping there will be improved statistics in V5..

[thuizenga]

"We found the size of VBR files changed at desitination was only about 50% of what was actually transferred.
Refer statistics here: viewtopic.php?f=2&t=2607#p15480"

This is kind of what i am leading at.

For instance when i start a backup it spends almost 20 min just creating the initial file that it starts to replicate over and about another 10-15 min before it even starts to replicate data. When the server was local this happened really quick (8min).

Here are some more stats on a specific replicated VM

hrserver
Total VM size: 45.00 GB
Processed size: 45.00 GB
Processing rate: 8 MB/s
Backup mode: SAN with changed block tracking
Start time: 6/28/2010 8:16:25 AM
End time: 6/28/2010 9:48:22 AM
Duration: 1:31:57

Size of newest delta file: 1gb (at 20mbps which i did verify was available at the time should take only 7 min to transfer). I verified that my full 20mbps was available to this location at the time of testing and the network was running fine (no collusions, errors or even a signal dropped packet according to the cisco ios stats). I was watchign my bandwith chart at the betting of my backup and it was bouncing between 10 to 20 mbps. Once the vmdk replication started i was only seeing about 10mbps average of traffic thought at that time i got pulled away from my desk and didn't see the rest.

[thuizenga]

Not sure I understand your first question. As for the second question, full ESX would have been better on target comparing to ESXi, as with full ESX we can use service console agent to do traffic compression.

I will try this after i come back from vacation. I have a fealing that my issues are with using ESXi vs ESX

[sjutras]

I did alot of testing and investigating on a similar issue, even had a few chat with Veeam support, as i had same problems trying to replicate a rather small set of VMs over a slow 1.5mbit wan link. The amount of traffic sent is about twice as much as the resulting .vrb and sometime even 4 and 6 times the size of the resulting .vrb. So if your .vrb is 2GB, Veeam perhaps sent 6Gb of data. It all depends really on what you are replicating, ie, a MS SQL VMs will generate alot of traffic but will remain a rather small .vrb size.

So your stats makes sens, you said your .vrb file was 1gb, yet it used between 10 and 20mbit for one hour and a half, which actually gives approximatly 6GB of traffic.

The trick, is to host your Veeam Server on the source site instead of the target site (yes, despite the downside of being unable to failover to previous version in case of main site failure), that along with ESX and agent console like Gostev said, you get to have network compression, but again, even with that, my tests concluded that traffic generated can be still 2 or 3 times as much as the .vrb size.

To make the replication works ok over the 1.5mbit link i had to tweak the VMs to minimize the block changes and also look for the use of Wan accelerator device or virtual appliance. This made the replication works nicely in my case, making my 1.5mbit link becomes nearly 7mbit.

p.s If you use Windows Volume shadow copy on your windows servers drives, move the contents of the shadow copy to an unreplicated vmdk. Shadow copy generate tons of block changes somehow and doesnt compress well at all.

Seb

[thuizenga]

Thanks sjutras for the input. I have replaced my esxi server with a esx server. Prior to posting this i had already tried to limit the block changes. I still need to move some VM's around to drives with small vmfs block sizes to further help this issue. Also my veeam installation is at my production site (i plan on replication the DB).

Whats the best mode for WAN replication? Do i choose SAN or Network under VMware vstorage API. Or select the "Network Replication" bullet?

[Gostev]

Travis, you definitely want to use any vStorage API processing mode, in order to be able to leverage changed block tracking (for fast incrementals). Specific mode will not matter match for WAN speed bound replication (they only make difference for local backup).

[thuizenga]

Thanks everyone for the help. I feel like i am finally getting the performance that i was expecting. I ran the backup on the lan @ 1gb to seed and then did an incremental in 16 min on 6 vm's 300gb thin (600gb thick). I changed my port to 10mpbs which is half of my wan speed and did the same incremental backup in just under 1 hour.

[spartridge]

sjutras,

What did you end up using for your WAN accelerator?