Replication Speed durin failback

[xefil]

Hello,

I would like to know if it's a way to calculate the speed of a failback operation over a 30MBit/s WAN.
AFAIK, the first time Replication starts, it copies the whole data from source to destination. Then, the next times, it uses VMware CBT to compare only the blocks signed as modified on the source against the destination. This for the replica.
During failover, CBT is not used. The VMs on DR-Site are simply powered on and used. From this point we can choose to undo the failback or commit it. In case of UNDO, we simple power off the replica and the source VM's are untouched. In case of failback, CBT is unusable too. The whole VM on the destination is compared, block by block using hash algorithms with the VM present in the source and the modified blocks are sent to the source. If all works well we can commit the failback and all starts from the beginning.
Now some questions:
Is the above scenario correct or I'm misunderstanding something?
In the failback operation, the incremental is much, much slower compared to the normal replica 'cause it's not using CBT, right?
I't possible to calculate how fast the failback works compared to normal CBT replica?
After a failback commit, the first Veeam's replica needs tho recreate the CBT index? Is this done automatically (like on first replica) or do we need to proceed with some further steps?

I've opened (and closed) a case as well (#5189242). These questions are an additional confirm if I've undersood all well and here I've asked some questions about the speed.

Thank's for the help!

Simon

[foggy]

In the failback operation, the incremental is much, much slower compared to the normal replica 'cause it's not using CBT, right?
I't possible to calculate how fast the failback works compared to normal CBT replica?

Yes, during replica failback to the original VM Veeam B&R needs to calculate and synchronize the differences between the original VM and the replica VM. The time required to scan the VM image depends on its size and is comparable to the full backup time of this VM.

After a failback commit, the first Veeam's replica needs tho recreate the CBT index? Is this done automatically (like on first replica) or do we need to proceed with some further steps?

CBT is enabled automatically on the first job run after failback commit (this is a full job run).

[xefil]

Yes, during replica failback to the original VM Veeam B&R needs to calculate and synchronize the differences between the original VM and the replica VM. The time required to scan the VM image depends on its size and is comparable to the full backup time of this VM.

Does it mean in fact that a failback of a 30GB VM on a 30MBit/s WAN takes more or less the time I need to transfer the whole 30GB over the WANlink? (should be more or less 136minutes).

CBT is enabled automatically on the first job run after failback commit (this is a full job run).

Ok, first time full backup/copy, then CBT incremental.

Thank you for the answer!

Simon

[Vitaliy S.]

Does it mean in fact that a failback of a 30GB VM on a 30MBit/s WAN takes more or less the time I need to transfer the whole 30GB over the WANlink?

No, it's not because such transfer never happens, but of course you will need more time for the failback operation comparing to VM failover process. Here is a similar thread with the same discussion, please look it through: Replication Failover and Failback

[xefil]

Thank you for the link. The only thing I've not tested was to select the proxies manually. I'll try this out too.

Bye, Simon

[xefil]

Hi!

Now I'm testing a failback.
I've executed a failover on DR-Site and the launched failback operation selecting manually both DR-Site proxy and Production Proxy.
I'm noticing a high traffic between Production Proxy and Production ESX. I would expect traffic between the proxies but for now the traffic is only from Production ESX to Production Proxy. In detail the process VeeamAgent.exe is receiving a lot of data from ESX.

Why this? Is it calculating the hash from source disk to be compared then with destination disk at the end?

Thank's

simon

[Gostev]

Correct. And it looks like the backup proxy is using the network processing mode (NBD).

[xefil]

Correct. And it looks like the backup proxy is using the network processing mode (NBD).

Yes, this is correct.
I'm noticing on the source proxy, the mode is always set in ndb, and seems it slows down the operations a lot:

Code: Select all

[ProxyDetector] Trying to detect source proxy modes for VM [MY-VM]
[ProxyDetector] Detecting storage access level for proxy [MY-PROXY]
[ProxyDetector] Detecting hotadd access level
[ProxyDetector] Wasn't able to find proxy vm but can failover to network
[ProxyDetector] Detected hotadd storage access level for proxy [MY-PROXY] - [Network]
[ProxyDetector] Detected mode [nbd] for proxy [MY-PROXY]

In which cases it uses ndb instead of hotadd? I would expect it always uses hotadd, cause the proxy is a virtual machine on the source ESX. The only difference to the destination proxy, which can access the datastore via hotadd, is that the source structure is managed trought every single ESX because due a bug (lock) in version 4.0 of VMware I cannot manage the source DR-Site trought the VCENTER.
The source structure has 3 ESX hosts. I've noticed, looking into log, veeam can use hotadd mode only with the VMs located on the same ESX of the proxy. All other VMs that are located on the other ESX are handled in Network mode. Is there a way to force it in hotadd?

Thank's!

Simon

[foggy]

The main requirement for hotadd is that the host running backup proxy server VM must have all datastores where protected VMs' disks reside connected to it. Does your backup proxy has access to the datastore, where VMs running on other host's reside?

Btw, there's a KB article listing all the limitations of using hotadd mode.

Thanks.

[xefil]

The main requirement for hotadd is that the host running backup proxy server VM must have all datastores where protected VMs' disks reside connected to it. Does your backup proxy has access to the datastore, where VMs running on other host's reside?

Backup proxy could have access to any datastore. All ESX on source infrastructure can access every datastore.

Btw, there's a KB article listing all the limitations of using hotadd mode.

Thank you for the link. I've found the main issue:

• In case of standalone host connection (no vCenter added to the console), you can only hot add disks of VMs which are located on the same host running Veeam Backup VM.

Due this VMware Issue I cannot add directly the vCenter on Veeam and have to manage all three ESX as standalone. In this way only the VMs that resides on the same ESX of the proxy are managed via hotadd. Meanwhile I've opened a Veeam Case as well #5193047 to know if it's possible to force the hotadd mode when the datastore are shared. In addiction into logs I've found this:

CheckHotAddLicense: We are connected to a server and not a Virtual Center
The host ha-host is not licensed for HotAdd

So now I've no idea if it use hotadd or not even if on the same ESX.

Simon

[foggy]

So now I've no idea if it use hotadd or not even if on the same ESX.

You can see the mode effectively being used in the Realtime Statistics window if you select the particular VM to the left.

[xefil]

You can see the mode effectively being used in the Realtime Statistics window if you select the particular VM to the left.

I've never clicked on it These informations are very interesting!
I can see the source and proxy methods:

Code: Select all

Using source proxy... [hotadd, ndb]
Using target proxy... [hotadd, ndb]

It's enough to read both has hotadd as first choise to be sure they will use both hotadd? On following lines this information is not written. Contrariwise I can see how the HARD DISK mounted on the selected VM is read at 5MB/s, and it's the speed of a traffic shape applied to the vlan. Seems like it uses ndb mode. Other VMs with only ndb uses the same read speed....

At the end of every job I can receive an email with the result. It's possible to include these informations too?

And, about these logs, it's possible to have realtime statistics and mail on failback operations as well?

Thank's again!

Simon

[foggy]

It's enough to read both has hotadd as first choise to be sure they will use both hotadd? On following lines this information is not written.

If hotadd is listed for the selected proxy and you do not see the failover to network warning, then hotadd is used.

At the end of every job I can receive an email with the result. It's possible to include these informations too?

There's no such option in the GUI but I believe this is achievable using PowerShell.

And, about these logs, it's possible to have realtime statistics and mail on failback operations as well?

Realtime statistics is available for all continuous tasks performed within Veeam B&R console (including failback), while e-mail notifications are available for backup and replication jobs only.