Comprehensive data protection for all workloads
Post Reply
alex.conduit
Novice
Posts: 7
Liked: never
Joined: Aug 21, 2013 8:16 pm
Full Name: Alex Conduit
Location: Auckland, New Zealand
Contact:

v7 Parallel Virtual Disk Processing

Post by alex.conduit » Aug 21, 2013 8:21 pm

I have v7 installed at our production & DR sites, both sites have 'Enable Parallel VM & Virtual Disk Processing' enabled.

In Production I run backups & can see multiple disks of the same VM being processed in parallel.
In DR I run replication from Production to DR - I see VM's processed in parallel, but only ever one virtual disk at a time.

Is this by design, or indicative of some problem that needs further investigation?

Thanks!

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by Vitaliy S. » Aug 21, 2013 9:44 pm

Hello Alex,

Can you please confirm that you have available slots on your proxy servers? Do you share proxy servers between your two backup servers?

Thank you!

alex.conduit
Novice
Posts: 7
Liked: never
Joined: Aug 21, 2013 8:16 pm
Full Name: Alex Conduit
Location: Auckland, New Zealand
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by alex.conduit » Aug 21, 2013 9:48 pm

Hi Vitaliy,

Yes, I share the source proxy servers between both backup servers, but no backups are running when replication jobs are active.
Yes, there are available slots on the proxy servers.

foggy
Veeam Software
Posts: 18278
Liked: 1564 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by foggy » Aug 22, 2013 6:47 am

Probably logs investigation is required to verify the proxy selection logic in your particular case, have you opened a case for that?

alex.conduit
Novice
Posts: 7
Liked: never
Joined: Aug 21, 2013 8:16 pm
Full Name: Alex Conduit
Location: Auckland, New Zealand
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by alex.conduit » Aug 22, 2013 6:59 am

I haven't yet - wanted to check if this was expected behaviour or not. If not, I'll open a support case.

veremin
Product Manager
Posts: 16901
Liked: 1439 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by veremin » Aug 22, 2013 10:00 am

Hi, Alex, assuming that you have at least one proxy CPU core available per data processing task (where the task is equal to VM disk) and still experience the said behavior, kindly, open a ticket with our support team and let them investigate it directly. Thanks.

ashemenev
Veeam Software
Posts: 9
Liked: 6 times
Joined: Nov 30, 2011 7:22 am
Full Name: AlexanderSh
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by ashemenev » Aug 22, 2013 6:55 pm

Hi, Alex!
In which transport mode works target proxy in case of replication?
Is it hotadd? You can check this in the session log available in the backup console. Search for something like this - 'Using target proxy VMware Backup Proxy for disk Hard disk 1 [mode]'

Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by Fiskepudding » Aug 22, 2013 7:12 pm

you have checked proxy task number, but have you checked repository task number, that can also be a limiting factor.

alex.conduit
Novice
Posts: 7
Liked: never
Joined: Aug 21, 2013 8:16 pm
Full Name: Alex Conduit
Location: Auckland, New Zealand
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by alex.conduit » Aug 27, 2013 9:19 pm

Support Call has been logged: Case # 00435450

Replication Infrastructure as follows:

Parallel Processing = Enabled
Proxy - Source1 - Virtual Appliance, 8 Tasks
Proxy - Source2 - Direct SAN, 4 Tasks
Proxy - Target1 - Virtual Appliance, 8 Tasks
Proxy - Target2 - Virtual Appliance, 8 Tasks
Proxy - Target3 - Direct SAN, 8 Tasks

Repository - Source - Windows Server, no task limit, no data ingestion limit
Repository - Target - Windows Server, no task limit, no data ingestion limit

Replication Job

1 Virtual Machine - 7 Disks, across 6 Datastores
Source Proxy - Source1, Source2
Target Proxy - Target1, Target2, Target3
Repository - Source
Compression - Optimal
Storage - WAN
CBT Enabled
Seeding - Mapped to Target Replica
Application Aware Processing = True

Replication job run with no other Veeam tasks running either at source or target - disks were processed sequentially, not in parallel

alex.conduit
Novice
Posts: 7
Liked: never
Joined: Aug 21, 2013 8:16 pm
Full Name: Alex Conduit
Location: Auckland, New Zealand
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by alex.conduit » Aug 28, 2013 1:47 am

Further testing carried out:

Job changed so that only HotAdd proxies were used (Source1, Target1) - disks were processed sequentially, not in parallel
Proxies altered to use only Network Transport mode (job proxy configuration restored to original settings) - disks were processed in parallel.

foggy
Veeam Software
Posts: 18278
Liked: 1564 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by foggy » Aug 28, 2013 10:38 am

alex.conduit wrote:Job changed so that only HotAdd proxies were used (Source1, Target1) - disks were processed sequentially, not in parallel
This is the reason. There is a known issue that VM disks are not processed in parallel during replication via hotadd.

Cokovic
Expert
Posts: 295
Liked: 59 times
Joined: Sep 06, 2011 8:45 am
Full Name: Haris Cokovic
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by Cokovic » Aug 28, 2013 12:26 pm

I can see this in my Testlab also on a normal backup job that disks are not processed in parallel when hotadd is used. So this is going to be addressed in one of the next patches?

davidb1234
Expert
Posts: 151
Liked: 15 times
Joined: Nov 15, 2011 8:47 pm
Full Name: David Borden
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by davidb1234 » Aug 28, 2013 3:10 pm

foggy wrote: This is the reason. There is a known issue that VM disks are not processed in parallel during replication via hotadd.

Will this be fixed in an upcoming patch?

foggy
Veeam Software
Posts: 18278
Liked: 1564 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by foggy » Aug 28, 2013 3:14 pm

Not in the nearest patch, maybe in the next minor version update, but not sure.

alex.conduit
Novice
Posts: 7
Liked: never
Joined: Aug 21, 2013 8:16 pm
Full Name: Alex Conduit
Location: Auckland, New Zealand
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by alex.conduit » Aug 28, 2013 8:06 pm

Thanks for the update Foggy :)

Gostev
SVP, Product Management
Posts: 24810
Liked: 3567 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by Gostev » Aug 28, 2013 10:28 pm

davidb1234 wrote:Will this be fixed in an upcoming patch?
No, as this is hot add limitation.

Bunce
Expert
Posts: 259
Liked: 8 times
Joined: Sep 18, 2009 9:56 am
Full Name: Andrew
Location: Adelaide, Australia
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by Bunce » Aug 29, 2013 2:59 am

Imposed by.. VMware?

Gostev
SVP, Product Management
Posts: 24810
Liked: 3567 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by Gostev » Aug 29, 2013 5:39 pm

Yes. We had to specifically disable parallel processing at target for hot add proxies based on our v7 testing.

davidb1234
Expert
Posts: 151
Liked: 15 times
Joined: Nov 15, 2011 8:47 pm
Full Name: David Borden
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by davidb1234 » Aug 29, 2013 6:21 pm

will we eventually be able to use direct SAN access for the DR proxy so that we can use parallel processing with replicas? I must have missed this and I am sad to hear this feature won't work with replicas.

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by Vitaliy S. » Aug 29, 2013 7:01 pm

Hi David, If you're referring to target proxy processing mode, then it can only be either network or hotadd. If you want to leverage parallel processing, then you should be using network mode for the DR proxy. Thanks!

alex.conduit
Novice
Posts: 7
Liked: never
Joined: Aug 21, 2013 8:16 pm
Full Name: Alex Conduit
Location: Auckland, New Zealand
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by alex.conduit » Aug 29, 2013 7:04 pm

From my testing: direct SAN access at source, network at target works well.

davidb1234
Expert
Posts: 151
Liked: 15 times
Joined: Nov 15, 2011 8:47 pm
Full Name: David Borden
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by davidb1234 » Aug 29, 2013 7:14 pm

I'll do some testing. We always assumed network mode was slower than hotadd and not worth pursuing unless you had no choice.

Gostev
SVP, Product Management
Posts: 24810
Liked: 3567 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by Gostev » Aug 29, 2013 8:15 pm

Correct for 1Gb Ethernet, not so for 10 Gb Ethernet.

davidb1234
Expert
Posts: 151
Liked: 15 times
Joined: Nov 15, 2011 8:47 pm
Full Name: David Borden
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by davidb1234 » Aug 29, 2013 8:20 pm

Gostev wrote:Correct for 1Gb Ethernet, not so for 10 Gb Ethernet.
So network mode using a proxy that is a VM with VMXNET3 driver with 10GB connection speed is what you mean? Or are you talking 10GB iscsi to a proxy?
I have a physical server with 8 cores that can be a proxy but it only has 2 1GB nics in a team. My other choice is using a VM as a proxy with VMXNET3 nic driver.

Gostev
SVP, Product Management
Posts: 24810
Liked: 3567 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by Gostev » Aug 29, 2013 8:34 pm

10Gb Ethernet from backup proxy to ESXi management interface.

tsightler
VP, Product Management
Posts: 5421
Liked: 2243 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by tsightler » Aug 29, 2013 9:34 pm

Even if you don't happen to have a 10GbE connection on the target ESXi management interfaces you can be creative by deploying a proxy on every target hosts and having each one have a vNIC on the same virtual network with the ESXi management interface (you can add a second management network and is IP/DNS for force the proxy to use that interface if required). Then you create replication jobs with different hosts for each target, and manually select the proxy that's on that host. This way the "Network mode" traffic on the target actually never leaves the virtual network on the host, which is effectively "unlimited". If you use a secondary management interface you don't even have to connect it to a physical network. Sure, it's a little more trouble to setup, but the replication performance is great.

Gostev
SVP, Product Management
Posts: 24810
Liked: 3567 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v7 Parallel Virtual Disk Processing

Post by Gostev » Aug 29, 2013 9:39 pm

+ force proxy to use network processing mode

Post Reply

Who is online

Users browsing this forum: ChrisTong, Google [Bot], jamcool and 19 guests