Comprehensive data protection for all workloads
marius roma
Veteran
Posts: 459
Liked: 5 times
Joined: Feb 01, 2012 12:04 pm
Full Name: Mario
Contact:

Parallel processing

Post by marius roma »

After upgrading to Veeam Backup & Replication 9.5 Update 4 I don't see anymore the option to enable/disable parallel processing.
How can I set it?
Regards
marius
DGrinev
Veteran
Posts: 1943
Liked: 247 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: Parallel processing

Post by DGrinev »

Hi Mario,

The parallel data processing is always enabled in Update 4 without the ability to disable it. Thanks!
marius roma
Veteran
Posts: 459
Liked: 5 times
Joined: Feb 01, 2012 12:04 pm
Full Name: Mario
Contact:

Re: Parallel processing

Post by marius roma »

In my imfrastructure this is causing some problems, due to concurrent access to repositories with limited bandwidth.
Is there any workaround we can implement?
Regards
marius
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Parallel processing

Post by foggy » 2 people like this post

You can limit max number of concurrent tasks and ingestion rate for these repositories.
ortoscale
Service Provider
Posts: 246
Liked: 20 times
Joined: Aug 02, 2011 9:30 pm
Full Name: Matjaž Antloga
Location: Celje, Slovenia
Contact:

Re: Parallel processing

Post by ortoscale »

What about replications jobs? They are all failing at one of our setups?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Parallel processing

Post by foggy »

Hi Matjaž, could you please be more specific about the problem?
ortoscale
Service Provider
Posts: 246
Liked: 20 times
Joined: Aug 02, 2011 9:30 pm
Full Name: Matjaž Antloga
Location: Celje, Slovenia
Contact:

Re: Parallel processing

Post by ortoscale »

After upgrade to u4 we're getting Unable to perform parallel VM disk processing with replication jobs with one customer.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Parallel processing

Post by foggy »

Have you already contacted support with this?
ortoscale
Service Provider
Posts: 246
Liked: 20 times
Joined: Aug 02, 2011 9:30 pm
Full Name: Matjaž Antloga
Location: Celje, Slovenia
Contact:

Re: Parallel processing

Post by ortoscale » 1 person likes this post

not yet, because the specific customer is not listed in our system when choosing opening support case for them so we are solving that bureaucracy problem.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Parallel processing

Post by foggy »

That would be the most effective way to address the issue, since logs review is required for find out the reasons of this behavior.
ortoscale
Service Provider
Posts: 246
Liked: 20 times
Joined: Aug 02, 2011 9:30 pm
Full Name: Matjaž Antloga
Location: Celje, Slovenia
Contact:

Re: Parallel processing

Post by ortoscale »

machines that have more that 1 disk are failing, parallel processing isn't working, how can we disable that?
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Parallel processing

Post by Gostev »

You cannot disable that - you have to work with support to find and fix the reason for failures. This is not currently a known issue with the product, so it must be something specific to the customer's environment. Thanks!
ortoscale
Service Provider
Posts: 246
Liked: 20 times
Joined: Aug 02, 2011 9:30 pm
Full Name: Matjaž Antloga
Location: Celje, Slovenia
Contact:

Re: Parallel processing

Post by ortoscale »

i will deal with support as soon as we fix what i've mention above, but in the meantime backups are not working ... so I'm digging.
Just found out in logs: <65> Error in c++: Unsupported VDDK version: 6.7
will update vcenter with fresh updates now.
ortoscale
Service Provider
Posts: 246
Liked: 20 times
Joined: Aug 02, 2011 9:30 pm
Full Name: Matjaž Antloga
Location: Celje, Slovenia
Contact:

Re: Parallel processing

Post by ortoscale » 1 person likes this post

case 03403100
teddiejl
Lurker
Posts: 1
Liked: never
Joined: Apr 04, 2019 1:32 pm
Full Name: Teddie Lawler
Contact:

Re: Parallel processing

Post by teddiejl »

Has any progress been made on this issue?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Parallel processing

Post by veremin »

It seems that there were problematic managed and proxy servers reported in the case - they were running obsolete backup components. As soon as they were updated, problem went away.

However, you might want to reach our support team to get direct assistance.

Thanks!
isolated_1
Enthusiast
Posts: 44
Liked: 5 times
Joined: Apr 09, 2015 8:33 pm
Full Name: Simon Chan
Contact:

[MERGED] Disable Parallel Processing Option Removed

Post by isolated_1 » 1 person likes this post

Hello Forums,

I just updated Veeam to 9.5 U4a. Imagine my surprise when I noticed that the option to disable parallel processing has now been removed! We have many jobs that run and we do not want jobs with a lot of VMs in it to hog up all the tasks of the repository leaving the other jobs to wait for resources to be available. Why was this option removed?!

I have submitted a support ticket: 03584215

I love Veeam with a passion. Please do not start removing features without giving users some way to re-enable them if they so choose to unless it severely impacts Veeam development going forward. I now have to spend Saturday night monitoring our backup jobs to see how they will perform because of this enforced change to parallel processing.

EDIT:
I am monitoring the backup jobs and it is exactly as I feared it would be. I have a job that has 20+ or so VMs. My backup repository is set to run 12 concurrent tasks. My other jobs that run after are all in pending state because that job is using up all the tasks. Even for jobs that finished processing VMs, they cannot perform a merge because it is still waiting for the backup resources to be available! The tasks are not getting spread evenly between the jobs that are running. As I am looking at it now, Veeam is dedicating all resources to that backup job with 20+ VMs and as soon as a VM finishes, it allocates the next available task/slot to another VM in that same job. This leaves all my other backup jobs stalling!

This is a nightmare. We have clients that want their VMs to get backed up at a very specific time. Even if parallel processing was disabled and it had to back up each VM within the job one after another, they were okay with it. Now with NO WAY to turn off parallel processing, the timing is all screwed up!
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Parallel processing

Post by Gostev »

isolated_1 wrote: May 26, 2019 5:27 amWe have clients that want their VMs to get backed up at a very specific time. Even if parallel processing was disabled and it had to back up each VM within the job one after another, they were okay with it.
This is a good and valid use case, however sequential disk processing is really a wrong solution for this. Not even a solution, but rather a workaround that allows you to achieve certain behavior in a creative way, using functionality that was intended for something completely different. Let me think how can we address that in a smarter way - I immediately got some ideas here.
isolated_1
Enthusiast
Posts: 44
Liked: 5 times
Joined: Apr 09, 2015 8:33 pm
Full Name: Simon Chan
Contact:

Re: Parallel processing

Post by isolated_1 »

I can understand why you are calling what I did a "workaround". But have you considered exactly why I actually have to do that? There is no job scheduling priority with Veeam so when I have so many jobs that need to run at night, how can such an advanced product not give me an option to do just that? You're looking at it from a technical perspective alone while I have to do that in addition to fulfilling client needs who want their VMs backed up at a certain time at night. I have used Veeam since version 6 and have had to do this workaround ever since to accomplish this.

Commvault, which I hate to mention due to its complexities, gave us the ability to set a priority on backup and restore jobs and honestly that is about one of the only things that I actually loved about it over Veeam. Parallel processing completely disrupted our entire backup setup. It may not have been recommended or the right way to have done things but I ask again, what other options is there besides creating a dedicated VM proxy for each backup job and setting it's concurrent task to 1? I have been scrambling to do just that ever since for the jobs that have larger amount of VMs than others because those jobs when run, hog up all the repository slots so that other VMs are stuck in limbo.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Parallel processing

Post by Gostev »

Yes, of course. And I guess the presence of this workaround is exactly what prevented the request for introducing job priorities from bubbling up to the top requests in the past years. Which in turn is the reason why we don't have this functionality - with hundreds of feature requests pending, we work through them based on the relative customer demand.
rogerdu
Expert
Posts: 148
Liked: 11 times
Joined: Aug 20, 2013 1:16 pm
Full Name: Roger Dufour
Contact:

Re: Parallel processing

Post by rogerdu »

I am in a position now where: I have a replication job that can ONLY process one VM at a time. (support case 03619317). Environment details:
Veeam B&R 9.5 U4a
Both Veeam Server and Repository servers are Hyper-V VMs
Repository Server has a ton of memory and 20 cores.

Client systems are on ESXi 6.7 U2 (latest patch as of a month ago). Where possible, VMtools and virtual hardware are at current levels.
All clients are running various Windows OSes (XP, 2003 all the way up to 2016) and most have multiple defined Hard Drives.

The replication job has 9 VMs listed and unless there is a single drive in the VM, I can ONLY run one VM at a time. This is the initial Sync, and I have a Seed that is less than a week old on the target site for each VM. Application Awareness is ON for 7 of the 9 VMs, but it doesn't make a difference... I can still only run 1 multi disk system at a time. Total disk footprint is 2.34 TB. The jobs that fail do so when calculating the digest.

This is unworkable as the RPO for these systems is 90- minutes... I doubt I'll be able to do that if it has to retry 8 times to get all the systems to run!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Parallel processing

Post by foggy »

Hi Roger, I'm not sure this correlates with the issue described above. What are the max number of concurrent tasks settings for the involved proxies/repository servers?
rogerdu
Expert
Posts: 148
Liked: 11 times
Joined: Aug 20, 2013 1:16 pm
Full Name: Roger Dufour
Contact:

Re: Parallel processing

Post by rogerdu »

I had them at 20 then reduced them to 10 each... No difference
rogerdu
Expert
Posts: 148
Liked: 11 times
Joined: Aug 20, 2013 1:16 pm
Full Name: Roger Dufour
Contact:

Re: Parallel processing

Post by rogerdu » 1 person likes this post

Opened a case with Support (03619317) and it turns out that ESXi doesn't like asynchronous processing of NFC jobs. By default, Veeam uses an asynchronous version of NFC which generally requires more memory for that buffer on the VMware host. To disable this I had to create an entry in the Registry.

Also, once I got all the initial digests completed (only had 25 systems to do), the problem went away... The support one gets from Veeam is outstanding BTW...
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Parallel processing

Post by Gostev »

I am curios if you potentially have any other VMware or 3rd party software potentially doing something that leverages the same buffer? Because we're fully aware of the issue with its limited size, and we did make some specific changes to make sure our activities don't require more memory than the buffer size. After this change, we have never seen this issue in our performance and scalability testing labs - which makes me suspect something else may potentially be using one in your case?
rogerdu
Expert
Posts: 148
Liked: 11 times
Joined: Aug 20, 2013 1:16 pm
Full Name: Roger Dufour
Contact:

Re: Parallel processing

Post by rogerdu »

Gostev, the VMs were anything that had multiple drives defined. Singe drive systems were fine. These included SQL, File servers, Domino and Web (IIS) servers. Also, it only occurred during the initial ingest... when it was updating/calculating the digests.
awsbackups01
Novice
Posts: 5
Liked: never
Joined: Oct 05, 2019 4:02 pm
Full Name: jesse ramos
Contact:

Re: Parallel processing

Post by awsbackups01 »

i am using Veeam B&R 9.5 U4a and trying to use the "Restore to AWS" option and i am getting the parallel processing error. i have one vm with 8 volumes. one volume is 156gb and they rest are 100gb or less. the restore only fails when i try to include the 156gb volume. is there a limit on how big the volume could be for restore?
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Parallel processing

Post by Gostev »

The only idea I have atm is you're selecting EC2 instance type that does not support disks larger than 128GB?
awsbackups01
Novice
Posts: 5
Liked: never
Joined: Oct 05, 2019 4:02 pm
Full Name: jesse ramos
Contact:

Re: Parallel processing

Post by awsbackups01 »

gostev, what ec2 instance types would those be? i have tried m5.2xl and 4xl and both fail.

btw,i have other m5.4xl ec2 instances in AWS that have 1TB drives. the only difference being i created those from AWS rather than importing them from veeam to AWS.
thomasto
Lurker
Posts: 1
Liked: never
Joined: Jan 24, 2020 2:38 am
Full Name: Thomas To
Contact:

Re: Parallel processing

Post by thomasto »

rogerdu wrote: Jun 20, 2019 7:56 pm Opened a case with Support (03619317) and it turns out that ESXi doesn't like asynchronous processing of NFC jobs. By default, Veeam uses an asynchronous version of NFC which generally requires more memory for that buffer on the VMware host. To disable this I had to create an entry in the Registry.

Also, once I got all the initial digests completed (only had 25 systems to do), the problem went away... The support one gets from Veeam is outstanding BTW...
Would you share the registry setting to resume the job?
We are also facing the same issue at this moment :oops:
Post Reply

Who is online

Users browsing this forum: Achraf, Bing [Bot], dbeerts and 160 guests