Replication across a WAN

[jayok]

Hi Folks,

I am trying to replication across a WAN link and I've noticed that every replication job that runs seems to replicate the entire VM each time. At 10Mb/s this means that it doesn't complete within the time window. My understanding is that only the delta are transferred after the inital sync, but this isn't what I am seeing. To test I move the replica ESX server back to the primary site and kicked off replication on the 1Gb/s link. All is well and 20GB is transferred fine, however there is ALWAYS 20GB/s transferred! If I kick a job off a 12:00pm and it completes at say 12:10pm, I kick another one off at 12:11pm it won't complete until 12:21pm (still 10 minutes). The logs indicate that the whole VM is being transferred - can anyone explain why?

Thanks.

[Gostev]

Hi Jason, yes we only replicate changes during subsequent passes (after the initial sync). Incremental replication pass may take about the same time because with ESX 3.x hosts whole VM needs to be scanned to determine changes, however then only changed blocks are transferred over WAN to the destination ESX.

What exactly did you find in the logs that made you think that the whole VM is transferred?

Also, what is your source server (ESX or ESXi), and what replication mode are you using (VCB or Network mode)?

[jayok]

Hi Gostev,

When I look at the logs, I see that it's processing a total of 20GB, and it counts up the bytes in each step to a total of 20GB. Maybe I am interpreting the logs incorrectly?

What transactions take place and where are the delta's stored? This is my understanding:

1. An initial sync of the VM takes place and gets transferred to the replica
2. A replicated copy takes place and the output is a .vrb file on the replica
3. This .vrb then becomes the incremental restore for the VM

My question is, what generates the delta, does the calculation happen on the source VM? If so what files are pulled from the destination VM?

I am using network mode from ESX 3.5 to ESX 3.5 same update.

Thanls.

[Gostev]

Jason, what you are seeing is the source VM image being scanned for changes since last sync. In case of "full" ESX and Network replication mode, this is done by the service console agent on the source ESX host. As soon as changes are determined, the changed blocks are sent over WAN to the destination ESX. The amount of data sent is about equal to the produced VRB file size (VRB file does not contain the actual delta data, but instead data substituted in the replica VM files by the most recect incremental pass).

[jayok]

So the logging from a processed bytes is the scanning? The VRB file on the destination is the only delta data transferred? I am just trying to understand what traffic is going down the WAN link to get to the bottom of the performance problem. If I check the logging then I see the following general process being followed:


1. Connection to Source server, agent is pushed
2. Connection to Replica server, agent is pushed
3. Commands issued to source server
4. Data transfer begins from ESX to ESX (either initial sync or replica)
5. Clean-up occurs

Is this VRB the on the replica the incremental data? That is does the .vmdk hold the most recent data and the VRB hold the the replica rollback?

Thanks.

[Gostev]

Check out this blog post on more information about rollback files:
Veeam synthetic backup explained

[jayok]

Thanks for that.

[jbhifi]

I've been experiencing the same issues. After reading this thread can I assume that what I am seeing is Veeam searching the source VM for changes? Even though this will take quite a few hours for a 400 GB VM am I best to run this replication once per day? I had hoped I could run hourly incrementals on the VM.

Thanks.

[Gostev]

Jon, you are correct. With 400GB VM, you should be able to run hourly incrementals, but only if you have fast shared storage and are using VCB SAN mode for replication.

However, you will be able to run incrementals even every few minutes with Veeam Backup version 4.0 (no matter of storage speed or the size of your VM). But, this functionality requires that you have vSphere ESX 4. Please review the sticky Veeam Backup 4.0 announcement for more information on this new near-realtime replication feature.

Hope this helps!

[jbhifi]

So will this improved functionality apply to Veeam backup AND replication or just the backup side of things?

[Gostev]

Jon, both backup and replication. Have you had a chance to view the Veeam Backup 4.0 video? It talks about this feature as it applies to replication in more details.