v7 Parallel Virtual Disk Processing

[alex.conduit]

I have v7 installed at our production & DR sites, both sites have 'Enable Parallel VM & Virtual Disk Processing' enabled.

In Production I run backups & can see multiple disks of the same VM being processed in parallel.
In DR I run replication from Production to DR - I see VM's processed in parallel, but only ever one virtual disk at a time.

Is this by design, or indicative of some problem that needs further investigation?

Thanks!

[Vitaliy S.]

Hello Alex,

Can you please confirm that you have available slots on your proxy servers? Do you share proxy servers between your two backup servers?

Thank you!

[alex.conduit]

Hi Vitaliy,

Yes, I share the source proxy servers between both backup servers, but no backups are running when replication jobs are active.
Yes, there are available slots on the proxy servers.

[foggy]

Probably logs investigation is required to verify the proxy selection logic in your particular case, have you opened a case for that?

[alex.conduit]

I haven't yet - wanted to check if this was expected behaviour or not. If not, I'll open a support case.

[veremin]

Hi, Alex, assuming that you have at least one proxy CPU core available per data processing task (where the task is equal to VM disk) and still experience the said behavior, kindly, open a ticket with our support team and let them investigate it directly. Thanks.

[ashemenev]

Hi, Alex!
In which transport mode works target proxy in case of replication?
Is it hotadd? You can check this in the session log available in the backup console. Search for something like this - 'Using target proxy VMware Backup Proxy for disk Hard disk 1 [mode]'

[Fiskepudding]

you have checked proxy task number, but have you checked repository task number, that can also be a limiting factor.

[alex.conduit]

Support Call has been logged: Case # 00435450

Replication Infrastructure as follows:

Parallel Processing = Enabled
Proxy - Source1 - Virtual Appliance, 8 Tasks
Proxy - Source2 - Direct SAN, 4 Tasks
Proxy - Target1 - Virtual Appliance, 8 Tasks
Proxy - Target2 - Virtual Appliance, 8 Tasks
Proxy - Target3 - Direct SAN, 8 Tasks

Repository - Source - Windows Server, no task limit, no data ingestion limit
Repository - Target - Windows Server, no task limit, no data ingestion limit

Replication Job

1 Virtual Machine - 7 Disks, across 6 Datastores
Source Proxy - Source1, Source2
Target Proxy - Target1, Target2, Target3
Repository - Source
Compression - Optimal
Storage - WAN
CBT Enabled
Seeding - Mapped to Target Replica
Application Aware Processing = True

Replication job run with no other Veeam tasks running either at source or target - disks were processed sequentially, not in parallel

[alex.conduit]

Further testing carried out:

Job changed so that only HotAdd proxies were used (Source1, Target1) - disks were processed sequentially, not in parallel
Proxies altered to use only Network Transport mode (job proxy configuration restored to original settings) - disks were processed in parallel.

[foggy]

Job changed so that only HotAdd proxies were used (Source1, Target1) - disks were processed sequentially, not in parallel

This is the reason. There is a known issue that VM disks are not processed in parallel during replication via hotadd.

[Cokovic]

I can see this in my Testlab also on a normal backup job that disks are not processed in parallel when hotadd is used. So this is going to be addressed in one of the next patches?

[davidb1234]

This is the reason. There is a known issue that VM disks are not processed in parallel during replication via hotadd.

Will this be fixed in an upcoming patch?

[foggy]

Not in the nearest patch, maybe in the next minor version update, but not sure.

[alex.conduit]

Thanks for the update Foggy

[Gostev]

Will this be fixed in an upcoming patch?

No, as this is hot add limitation.

[Bunce]

Imposed by.. VMware?

[Gostev]

Yes. We had to specifically disable parallel processing at target for hot add proxies based on our v7 testing.

[davidb1234]

will we eventually be able to use direct SAN access for the DR proxy so that we can use parallel processing with replicas? I must have missed this and I am sad to hear this feature won't work with replicas.

[Vitaliy S.]

Hi David, If you're referring to target proxy processing mode, then it can only be either network or hotadd. If you want to leverage parallel processing, then you should be using network mode for the DR proxy. Thanks!

[alex.conduit]

From my testing: direct SAN access at source, network at target works well.

[davidb1234]

I'll do some testing. We always assumed network mode was slower than hotadd and not worth pursuing unless you had no choice.

[Gostev]

Correct for 1Gb Ethernet, not so for 10 Gb Ethernet.

[davidb1234]

Correct for 1Gb Ethernet, not so for 10 Gb Ethernet.

So network mode using a proxy that is a VM with VMXNET3 driver with 10GB connection speed is what you mean? Or are you talking 10GB iscsi to a proxy?
I have a physical server with 8 cores that can be a proxy but it only has 2 1GB nics in a team. My other choice is using a VM as a proxy with VMXNET3 nic driver.

[Gostev]

10Gb Ethernet from backup proxy to ESXi management interface.

[tsightler]

Even if you don't happen to have a 10GbE connection on the target ESXi management interfaces you can be creative by deploying a proxy on every target hosts and having each one have a vNIC on the same virtual network with the ESXi management interface (you can add a second management network and is IP/DNS for force the proxy to use that interface if required). Then you create replication jobs with different hosts for each target, and manually select the proxy that's on that host. This way the "Network mode" traffic on the target actually never leaves the virtual network on the host, which is effectively "unlimited". If you use a secondary management interface you don't even have to connect it to a physical network. Sure, it's a little more trouble to setup, but the replication performance is great.

[Gostev]

+ force proxy to use network processing mode