Comprehensive data protection for all workloads
Post Reply
alex.conduit
Novice
Posts: 7
Liked: never
Joined: Aug 21, 2013 8:16 pm
Full Name: Alex Conduit
Location: Auckland, New Zealand
Contact:

v7 Parallel Virtual Disk Processing

Post by alex.conduit »

I have v7 installed at our production & DR sites, both sites have 'Enable Parallel VM & Virtual Disk Processing' enabled.

In Production I run backups & can see multiple disks of the same VM being processed in parallel.
In DR I run replication from Production to DR - I see VM's processed in parallel, but only ever one virtual disk at a time.

Is this by design, or indicative of some problem that needs further investigation?

Thanks!
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by Vitaliy S. »

Hello Alex,

Can you please confirm that you have available slots on your proxy servers? Do you share proxy servers between your two backup servers?

Thank you!
alex.conduit
Novice
Posts: 7
Liked: never
Joined: Aug 21, 2013 8:16 pm
Full Name: Alex Conduit
Location: Auckland, New Zealand
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by alex.conduit »

Hi Vitaliy,

Yes, I share the source proxy servers between both backup servers, but no backups are running when replication jobs are active.
Yes, there are available slots on the proxy servers.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by foggy »

Probably logs investigation is required to verify the proxy selection logic in your particular case, have you opened a case for that?
alex.conduit
Novice
Posts: 7
Liked: never
Joined: Aug 21, 2013 8:16 pm
Full Name: Alex Conduit
Location: Auckland, New Zealand
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by alex.conduit »

I haven't yet - wanted to check if this was expected behaviour or not. If not, I'll open a support case.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by veremin »

Hi, Alex, assuming that you have at least one proxy CPU core available per data processing task (where the task is equal to VM disk) and still experience the said behavior, kindly, open a ticket with our support team and let them investigate it directly. Thanks.
ashemenev
Veeam Software
Posts: 9
Liked: 6 times
Joined: Nov 30, 2011 7:22 am
Full Name: AlexanderSh
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by ashemenev »

Hi, Alex!
In which transport mode works target proxy in case of replication?
Is it hotadd? You can check this in the session log available in the backup console. Search for something like this - 'Using target proxy VMware Backup Proxy for disk Hard disk 1 [mode]'
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by Fiskepudding »

you have checked proxy task number, but have you checked repository task number, that can also be a limiting factor.
alex.conduit
Novice
Posts: 7
Liked: never
Joined: Aug 21, 2013 8:16 pm
Full Name: Alex Conduit
Location: Auckland, New Zealand
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by alex.conduit »

Support Call has been logged: Case # 00435450

Replication Infrastructure as follows:

Parallel Processing = Enabled
Proxy - Source1 - Virtual Appliance, 8 Tasks
Proxy - Source2 - Direct SAN, 4 Tasks
Proxy - Target1 - Virtual Appliance, 8 Tasks
Proxy - Target2 - Virtual Appliance, 8 Tasks
Proxy - Target3 - Direct SAN, 8 Tasks

Repository - Source - Windows Server, no task limit, no data ingestion limit
Repository - Target - Windows Server, no task limit, no data ingestion limit

Replication Job

1 Virtual Machine - 7 Disks, across 6 Datastores
Source Proxy - Source1, Source2
Target Proxy - Target1, Target2, Target3
Repository - Source
Compression - Optimal
Storage - WAN
CBT Enabled
Seeding - Mapped to Target Replica
Application Aware Processing = True

Replication job run with no other Veeam tasks running either at source or target - disks were processed sequentially, not in parallel
alex.conduit
Novice
Posts: 7
Liked: never
Joined: Aug 21, 2013 8:16 pm
Full Name: Alex Conduit
Location: Auckland, New Zealand
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by alex.conduit »

Further testing carried out:

Job changed so that only HotAdd proxies were used (Source1, Target1) - disks were processed sequentially, not in parallel
Proxies altered to use only Network Transport mode (job proxy configuration restored to original settings) - disks were processed in parallel.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by foggy »

alex.conduit wrote:Job changed so that only HotAdd proxies were used (Source1, Target1) - disks were processed sequentially, not in parallel
This is the reason. There is a known issue that VM disks are not processed in parallel during replication via hotadd.
Cokovic
Veteran
Posts: 295
Liked: 59 times
Joined: Sep 06, 2011 8:45 am
Full Name: Haris Cokovic
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by Cokovic »

I can see this in my Testlab also on a normal backup job that disks are not processed in parallel when hotadd is used. So this is going to be addressed in one of the next patches?
davidb1234
Expert
Posts: 162
Liked: 15 times
Joined: Nov 15, 2011 8:47 pm
Full Name: David Borden
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by davidb1234 »

foggy wrote: This is the reason. There is a known issue that VM disks are not processed in parallel during replication via hotadd.

Will this be fixed in an upcoming patch?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by foggy »

Not in the nearest patch, maybe in the next minor version update, but not sure.
alex.conduit
Novice
Posts: 7
Liked: never
Joined: Aug 21, 2013 8:16 pm
Full Name: Alex Conduit
Location: Auckland, New Zealand
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by alex.conduit »

Thanks for the update Foggy :)
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by Gostev »

davidb1234 wrote:Will this be fixed in an upcoming patch?
No, as this is hot add limitation.
Bunce
Veteran
Posts: 259
Liked: 8 times
Joined: Sep 18, 2009 9:56 am
Full Name: Andrew
Location: Adelaide, Australia
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by Bunce »

Imposed by.. VMware?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by Gostev »

Yes. We had to specifically disable parallel processing at target for hot add proxies based on our v7 testing.
davidb1234
Expert
Posts: 162
Liked: 15 times
Joined: Nov 15, 2011 8:47 pm
Full Name: David Borden
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by davidb1234 »

will we eventually be able to use direct SAN access for the DR proxy so that we can use parallel processing with replicas? I must have missed this and I am sad to hear this feature won't work with replicas.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by Vitaliy S. »

Hi David, If you're referring to target proxy processing mode, then it can only be either network or hotadd. If you want to leverage parallel processing, then you should be using network mode for the DR proxy. Thanks!
alex.conduit
Novice
Posts: 7
Liked: never
Joined: Aug 21, 2013 8:16 pm
Full Name: Alex Conduit
Location: Auckland, New Zealand
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by alex.conduit »

From my testing: direct SAN access at source, network at target works well.
davidb1234
Expert
Posts: 162
Liked: 15 times
Joined: Nov 15, 2011 8:47 pm
Full Name: David Borden
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by davidb1234 »

I'll do some testing. We always assumed network mode was slower than hotadd and not worth pursuing unless you had no choice.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by Gostev »

Correct for 1Gb Ethernet, not so for 10 Gb Ethernet.
davidb1234
Expert
Posts: 162
Liked: 15 times
Joined: Nov 15, 2011 8:47 pm
Full Name: David Borden
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by davidb1234 »

Gostev wrote:Correct for 1Gb Ethernet, not so for 10 Gb Ethernet.
So network mode using a proxy that is a VM with VMXNET3 driver with 10GB connection speed is what you mean? Or are you talking 10GB iscsi to a proxy?
I have a physical server with 8 cores that can be a proxy but it only has 2 1GB nics in a team. My other choice is using a VM as a proxy with VMXNET3 nic driver.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by Gostev »

10Gb Ethernet from backup proxy to ESXi management interface.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by tsightler »

Even if you don't happen to have a 10GbE connection on the target ESXi management interfaces you can be creative by deploying a proxy on every target hosts and having each one have a vNIC on the same virtual network with the ESXi management interface (you can add a second management network and is IP/DNS for force the proxy to use that interface if required). Then you create replication jobs with different hosts for each target, and manually select the proxy that's on that host. This way the "Network mode" traffic on the target actually never leaves the virtual network on the host, which is effectively "unlimited". If you use a secondary management interface you don't even have to connect it to a physical network. Sure, it's a little more trouble to setup, but the replication performance is great.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by Gostev »

+ force proxy to use network processing mode
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 211 guests