Comprehensive data protection for all workloads
Post Reply
Oletho
Enthusiast
Posts: 67
Liked: 2 times
Joined: Sep 17, 2010 4:37 am
Full Name: Ole Thomsen
Contact:

Concurrency in a job?

Post by Oletho »

Is there a way to have several VMs backed up at the same time without creating multiple jobs?

If dedup is not global but only covering a backup file I would like to have as many servers in a job as possible. But it seems that a job is strictly sequential?

Sorry if this is a dumb question, but I am just starting to test the product and would like to optimize both concurrency and dedup.

Ole Thomsen
Alexey D.

Re: Concurrency in a job?

Post by Alexey D. »

Hello Ole,

For VMs belonging to same job concurrent processing is not possible. And deduplication works only for VMs in bounds of one job.
But if you want parallel processing there might be a balance point though. For example, create one job and place your Windows VMs inside,
and another job, let's say, for Ubuntu Linux VMs, etc. Dedupe is only effective between similar VMs anyway.

It's up to you which scenario to implement, but the general idea is described above. Hope this helps!
Oletho
Enthusiast
Posts: 67
Liked: 2 times
Joined: Sep 17, 2010 4:37 am
Full Name: Ole Thomsen
Contact:

Re: Concurrency in a job?

Post by Oletho »

Thanks Alexey.

Yes, this helps me to confirm that my understanding of the limitations is correct. That's a good start :-)

But the fact that dedup is most efficient across similar servers does not always cope with the real world needs. Example: we have a system that involves Windows servers (2000, 2003 and 2008), Redhat Linux, SQL and Oracle databases. The goal is to have as small a backup window as possible, and with other products like VDR and esXpress i can run all simultaneously in one job to the same dedup store.

Well, guess I just have to think different.

Thanks again,
Ole Thomsen
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Concurrency in a job?

Post by Gostev »

Hi Ole, I wanted to add that we have a good whitepaper called "Deduplication and VMware Backup Sprawl" going into details of our dedupe approach (per job), and comparing it to a single dedupe store. I recommend that you check it out, it is more technical than marketing (written by our system engineers).

You can download it from here:
http://www.veeam.com/whitepapers.html

By the way, VDR is almost no different from us in terms of end results: while it provides ability to write to a single dedupe store, the dedupe store itself is limited to 2TB in size - so you are going to have to use multiple ones.

Thanks!
Oletho
Enthusiast
Posts: 67
Liked: 2 times
Joined: Sep 17, 2010 4:37 am
Full Name: Ole Thomsen
Contact:

Re: Concurrency in a job?

Post by Oletho »

I'll take a look at the whitepaper, Gostev.

I tested the latest VDR for a couple of months, and while the store limit is 2TB the supported size is only 1TB (500GB for CIFS). Even then it gets unstable the closer one gets to fill the store, and the missing features and weaknesses makes it unusable for all but the smallest environments in my opinion.

Still I wish Veeam had concurrency in each job, this is one thing I really like about esXpress. Is it a design decision or because of the single NTFS file that holds the job data?

Ole Thomsen
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Concurrency in a job?

Post by Gostev »

Hi Ole, it is more of historical design/feature decision.

More detailed answer to the same question:
Why does Veeam do serial VM backups instead of parallel?
Bunce
Veteran
Posts: 259
Liked: 8 times
Joined: Sep 18, 2009 9:56 am
Full Name: Andrew
Location: Adelaide, Australia
Contact:

Re: Concurrency in a job?

Post by Bunce »

Ditto here. We have no issue 'flooding' any pipe when running multiple jobs.

As mentioned frequently, having to run multiple jobs removes the dedupe advantage, and increases management and monitoring requirements.

Not being able to order the backups within a job then adds further complication when trying to backup and replicate the same VM in different jobs (to prevent being backup up at the same time) - while still trying to maintain a small backup window.

If its a technical limitation of the backup engine, perhaps Veem could consider a custom backup 'appliance' target that multiple jobs could write to concurrently? (also of the ilk of esxpress). This may also provide the possibility to provide useful reporting statistics from the target appliance (dedup, compression, VM stats, indexing and searching, accurate transfer rates etc - all the good stuff that we can provide in managerial reports)

Just a thought.

Cheers,
A
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Concurrency in a job?

Post by Gostev »

No, this is not a technical limitation of the backup engine. This is specifically explained in the post referenced in my previous reply.
Oletho
Enthusiast
Posts: 67
Liked: 2 times
Joined: Sep 17, 2010 4:37 am
Full Name: Ole Thomsen
Contact:

Re: Concurrency in a job?

Post by Oletho »

Gostev wrote:Hi Ole, it is more of historical design/feature decision.

More detailed answer to the same question:
Why does Veeam do serial VM backups instead of parallel?
I understand the decision based on those findings, but that is not my experience so far.

We use 10Gbit NFS as primary VM storage (FC for backup store) and I chose to run a 4 vCPU virtual appliance with 4GB RAM on Windows Server 2008 R2.

Each HOTADD backup job uses ~25% CPU (equals 1 vCPU) and gives a processing rate of ~30 MB/s for full backup, higher for CBT.

So I see 2 full jobs using 50% CPU and total processing rate ~60MB/s, 4 full jobs doubles both numbers. I did not add additional vCPUs to see when that flattens :-)

Ole Thomsen
tsightler
VP, Product Management
Posts: 6011
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Concurrency in a job?

Post by tsightler »

Oletho wrote: I understand the decision based on those findings, but that is not my experience so far.
It's not my findings either. I've always seen better performance from multiple jobs than from a single job. Even if a single job saturates the SAN or the CPU that doesn't account for all the "down time" in the job between VM's while snapshots are taken and removed. This can be significant on busy systems (our Exchange server regularly takes 45-60 minutes to remove the snapshot, all the while that Veeam job is basically doing nothing). We see significantly better performance from multiple jobs, even on our relatively old servers.

It also feels like the idea of a single job saturating the SAN or CPU is based on running fulls, but we rarely run a full backup. For CBT based incrementals, a single job rarely uses more than 25% of CPU in our environment, and the SAN in nowhere near saturated. In that scenario the overhead of creation and removal of snapshots is almost half of the entire backup time. Having multiple jobs can cut the overall job time nearly in half for these circumstances. We've found that 4 jobs is generally the best setting for performance without completely saturating the SAN.
MartinSvec
Lurker
Posts: 2
Liked: 1 time
Joined: Apr 17, 2012 3:10 pm
Contact:

Re: Concurrency in a job?

Post by MartinSvec »

Hello, I'd like to reopen this question.

I'm evaluating if Veeam v6 is a suitable backup solution for our vSphere 5 cloud environment and the sequential job processing seems to me like a potential source of troubles. And if I'm right, there is still no option to enable concurrent intra-job processing in v6? Having high number of small VMs with minimal daily CBT changes, I see a lot of time wasted by e.g. waiting for snapshot creations, snapshot removals etc. In this time, storage, CPU and network resources are clearly underutilized and the overall backup window unreasonably increases. Maintaining multiple identical jobs and manually distributing VMs between them just to workaround this behavior doesn't look like an user-friendly idea.

I'm also asking myself if all the great scalability features like backup proxies, storage repositories, automatic load balancing etc. are so great when the level of backup paralellism is primarily limited by the number of (manually created) jobs. With modern 10GE SANs and multi-core backup servers with plenty of RAM and HDDs, I think this deficiency will become more and more obvious.

So, is it true that the job processing is still purely sequential or is there any hidden option that I overlooked? And if it's still not possible, do you have any plans to implement it?

Thank you.

Martin Svec
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Concurrency in a job?

Post by Vitaliy S. »

Hi Martin, there is more recent topic discussing all the pros and cons of this approach, please look it through and continue posting there: Concurrent Tasks
c_manboy
Novice
Posts: 4
Liked: never
Joined: Jul 16, 2012 3:36 am
Contact:

replication job vms not concurrent

Post by c_manboy »

[merged]

I created a replication job with 7 vms. The source proxy and target proxy are different (yet both are local and the destination is local) and configured with 3 "Max concurrent tasks". So my question is, should the vms be processing concurrently within the same job?
dellock6
VeeaMVP
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Concurrency in a job?

Post by dellock6 »

no, all VMs in a single job are processed sequentially. Cuncurrency takes place when you have more than 1 job running at the same time.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Post Reply

Who is online

Users browsing this forum: flakpyro, Google [Bot] and 146 guests