VEEAM (v5) Replication (and Backup) explained

[1-0-1]

Please correct me if I am wrong from here-on. I am trying to make sense and thought I understood VEEAM's replication mechanics. I always assumed that it works excactly like the VEEAM backup method except (main differences) it cannot do a synthatic full backup, one cannot specify when to do a full backup and does not support reverse incrementals.

I based these understandings from the v5 User Guide (from page 11). I will have to test this as well tonight but to my understanding the initial full replication does not get touched. Subsequent incremental replications are being stored seperate from the initial full and only when we need to do a restore it will apply all the incremental chains to the full? What I do not completely understand here is when one of the incremental fail will it then force the next incremental as a full replication as it cannot see anything from it last full/incremental backup as it is corrupt due to whatever failure?

My example I am working is having a VM that needs to be replicated across a pretty slow WAN connection. Initial replication one can deal with by doing it to a removable media and importing it at the other end. After that one basically just runs incremental but what happens if one of the incremental fails for whatever reason? Will it just do another incremental from the last successfull incremental or actually go and do a full replication of the VM?

Also how does the restore point fall into his? According to the User Guide, if this number is exceeded, the earliest restore point will be deleted (page 69). Does this mean it will only delete the oldest incremental and do a full sync next time or does it just delete the oldest incremental and leave the initial replication alone as this one will be used if the VM need to be brought up together with the incremental chain?

Is there any more in depth articles on this published somewhere? Please help ....

[Gostev]

Yes, you understanding is not correct. Replication does use reversed incremental approach.
See here, it talks about replication towards the end: Veeam Synthetic Backup Explained

Full replication is only done once (first pass), no matter what.

[1-0-1]

Thanks so much. I would also assume that if one of the incremental backups are interrupted while it started it does not mean the next incremental backup will iniate a new full run (provided it is run as soon as possible)?

I will try to extract the information from the User Guide and the above mentioned link as reference.

Would appreciate if there would be any other information regarding this from VEEAM.

[Heiko]

Hi

Veeam Backup and Replication version 5 Standard Edition

I am disagreeing with the incremental process when a REPLICATION job fails.
Done the test as follows.
Initial full sync took about 30 mins then all incremental are fast
Specified on a Test VM (Windows 2008) to replicate and keep 6 rollbacks. If all jobs are successful until the 6th, the 7th job is as well incremental.
The next test was at the 3 replication job I stop the job through the Veeam Console. If I looked at the replica the vbk file show corrupted and the incremental all ok. Could fall back on rollback VM only (Makes only sense ). Then I started the job again and it did a full resync of the VM and took again about 30 min. If it would only do a incremental it should be fast as normal?

[1-0-1]

Okay - so then as part of my new understanding I could explain that the long incremental backup after a corrupted incremental to the following reason based on the following:

"VEEAM replication uses a reverse incremental method. Except that the changes get “injected” into the .VMDK file on the target, there is no .VBK file. There are .VRB files and this is what allows for replication roll-back."

The time difference comes in then because it has to the rebuild the .VMDK file from the last successfull .VRB (reversed incremental changes file) and the original .VBK (the full recovery file)? Does this then happen after or before the next successfull incremental replication?

[Gostev]

Heiko, we do not have functionality that does full resync after failed incremental in our product.

When the job is started after failed incremental, it first repairs replica file to last known good state, rolling back modifications from incomplete incremental run. Once this is done, regular incremental run is performed. Thanks!

[Gostev]

1-0-1, I do not understand your question at all.

a) Replica VMDK is not being rebuild from VRB. As per article linked above, VRB contains blocks replaced in VMDK file by corresponding incremental run.
b) Replication does not use VBK at all, VBK is for backup only. Full recovery file for replica is VMDK itself.
c) Not sure what time difference you are talking about.

[1-0-1]

1-0-1, I do not understand your question at all.
a) Replica VMDK is not being rebuild from VRB. As per article linked above, VRB contains blocks replaced in VMDK file by corresponding incremental run.
b) Replication does not use VBK at all, VBK is for backup only. Full recovery file for replica is VMDK itself.

Okay - so in a replication the closest match for a VBK is the actuall VMDK. It uses then VRBs to rebuild the actuall VMDK. In case of failed incremental - does it 'corrupt' VMDK or is there a buld in mechanism in VEEAM to only commit a incremental replication once it went through successfully?

c) Not sure what time difference you are talking about.

I was refering to the increase time of a incremental backup, after a failed backup as mentioned by Heiko.

[Gostev]

No, it does not "corrupt" VMDK, it just makes it state inconsistent, so you cannot failover to it (but you can failover to previous restore points). And next incremental run will clean up this inconsistency, as per my explanation to Heico.

Yes, next incremental replication pass after previously failed incremental replication takes longer because of rolling back VMDK changes from failed incremental run to restore VMDK to last known good state.

[1-0-1]

Yes, next incremental replication pass after previously failed incremental replication takes longer because of rolling back VMDK changes from failed incremental run to restore VMDK to last known good state.

I would assume delay during the rebuild from a failed incremental replication is then depend on purely the disk subsystem the replication files are stored on?

The faster the disk subsystem the faster the rebuild process will complete (e.g. rebuild on 15k RPM disk will be faster than 10k RPM disk, rebuild on RAID10 array will be faster than RAID5 array etc)

[Gostev]

Yes with ESX target. Additionally, network connection to target (in case of ESXi, since there is no Veeam agent on target)...
Sounds like your are designing replication architecture assuming every second incremental run will fail?

[Heiko]

Heiko, we do not have functionality that does full resync after failed incremental in our product.

When the job is started after failed incremental, it first repairs replica file to last known good state, rolling back modifications from incomplete incremental run. Once this is done, regular incremental run is performed. Thanks!

I will rerun the test to confirm. As this is confusing because if we rerun the job it looks like it does a resync/new replication into the vk file. Because it does process about 25MB/s for 30 min. Its the same time and process as if we do the initial first replication. This is a huge problem over a WAN link when it has to restart a job due to what ever reason the job failed.

[1-0-1]

Yes with ESX target. Additionally, network connection to target (in case of ESXi, since there is no Veeam agent on target)...
Sounds like your are designing replication architecture assuming every second incremental run will fail?

Generally speaking our data connections are not of good quality and would like to prepare for worst case scenarios. There might a possibility we will have a lot of replication failure at random invervals and we need to be 100%, when going into testing how the VEEAM mechanism works.
Realisticly we will not be able to do a full replication over the WAN links and as such the question came up weather VEEAM does a full replication if a previous replication has failed.

Gostev - is there any more information from VEEAM how to tune for replication jobs (beyond the basics in the User Guide) or where to at least start looking for in regards to VMWARE ESXi, iSCSI, slow WAN (5mbps), recommended designs, etc?

[Gostev]

There couple of things which really matter when replicating over slow WAN link.

1. Replication target. With the current version, you want to replicate to full ESX as it provides more (3x) traffic efficient target due to the fact that we are able to place local agent on ESX. We are working to improve ESXi replication traffic in the next release.

2. Block size. You want to select "WAN target" optimization in advanced job settings, this will further reduce incremental replication traffic to about 2x compared to default settings.

[1-0-1]

Excellent - thanks again for the valuable insight!

We have already figured out that the Target option need to be changed accordingly but the change between ESXi and ESX is very interesting.

[st0623]

1-0-1,
I'm with the HyperIP team at NetEx and we partnered with Veeam to optimize replication over slow WAN links. HyperIP mitigates the effects of the WAN on Veeam Replication. Also, we compress the data over the WAN so not only do the incremental jobs complete faster, but if there is a network incident that causes the incrementals to restart or resync, then HyperIP enables these to complete faster. Also, in testing or actually recovering data from the remote replica, HyperIP's optimization is bi-directional allowing you to restore the replica into production quicker. Check out our product info at http://www.netex.com/products/hyperip. reach out of you have any questions: steve.thompson@netex.com