Parallel disk processing in replication jobs

[ori]

Hi,
I have just installed a fresh Veeam 7 R2 on our DR site for replication purposes only (I am replacing our current version 6.5 by separating the backup and replication roles to two new servers).
I am trying to use the parallel processing feature, I see that indeed multiple VMs are processed in parallel but these VMs also have 2 virtual disks each and the disks are processed sequentially. What can explain this behavior? I was really waiting for this parallel disks feature in order to shorten the time a VM is on snapshot.

[foggy]

Ori, are you using hotadd transport mode to read/write VM data during replication? We had to specifically disable parallel processing for hotadd proxies due to a VDDK limitation. You can add additional proxy servers at both ends to improve processing performance.

[ori]

The source read is in SAN mode and the destination is a virtual proxy server at the DR site to use hot-add mode. I am not sure I completely understood because I see that the proxy is hot-adding multiple disks when multiple VMs are processed so why can't it process multiple disk from the same VM?
What do you mean by additional proxies, will parallel disk processing work if I'll add additional hot-add proxy at the destination side?

[foggy]

I am not sure I completely understood because I see that the proxy is hot-adding multiple disks when multiple VMs are processed so why can't it process multiple disk from the same VM?

This is caused by peculiarities of VMware hotadd implementation.

What do you mean by additional proxies, will parallel disk processing work if I'll add additional hot-add proxy at the destination side?

Yes, different VM disks will be processed by different proxy servers.

[tsightler]

The other option is to force the target proxies to use Network (NBD) mode. This generally works well for replication target, especially if the target system has 10GbE.

[Gostev]

100% agree with Tom.

[skydok]

Gostev,
Do you know maybe if the limitation i spoke about comes from vCenter? I have heard that vCenter has Maximum of 100 concurrent operations,therefore not more than 100 VMs can be backed up parallel.

True?

[Vitaliy S.]

Do you know maybe if the limitation i spoke about comes from vCenter? I have heard that vCenter has Maximum of 100 concurrent operations,therefore not more than 100 VMs can be backed up parallel.

HotAdd limitation is not related to this.

[ori]

Just want to update that adding additional hot-add proxy at the destination did not help. Still VMs are processed in parallel, now by two proxies, but multiple disks within a VM are processed serially.

[tsightler]

Yes that is true and a known limitation with hotadd on target proxies. If the target proxies use hotadd then Veeam will only ever process a single disk concurrently even if there are multiple proxies. That's why I suggested the workaround simply using NBD mode on the target proxy. Is there some specific reason you do not want to follow that advice, perhaps only a 1Gb management network or something, because that issue also had a potential workaround?

[foggy]

Correct, sorry for confusion. This applies to the source side proxy servers only. However, even on target single proxy can process different VMs in parallel so this limitation does not significantly affect total job duration time (though, yes, each VM that has several disks runs longer on snapshot).

[ori]

Thanks tsightler.
I don't have 10gig for all of the hosts there yet, what is the potential workaround you wrote about?

[emachabert]

Hi,

I've just made a little diagram to help you understand the ways you can reach target storage through network mode.
Hope this helps.

[ori]

Thanks emachabert, very clear diagram. So basically I should put a proxy per management subnet for direct communication, preferably with a dedicated NIC in the management subnet and force it to NBD mode?

[emachabert]

Yes thats it, using a proxy on the host allows you to carry more data between source proxy and target proxy, thus reducing the time needed to replicate.


Another thing you should know about using hot-add in replication jobs. There is a random bug in the VDDK about writing to the target datastore through hot-add. For an unknown reason it overloads the target storage with thousand of I/Os resulting in very poor performance. See that post: http://forums.veeam.com/vmware-vsphere- ... 14183.html