Concurrency in a job?

[Oletho]

Is there a way to have several VMs backed up at the same time without creating multiple jobs?

If dedup is not global but only covering a backup file I would like to have as many servers in a job as possible. But it seems that a job is strictly sequential?

Sorry if this is a dumb question, but I am just starting to test the product and would like to optimize both concurrency and dedup.

Ole Thomsen

[Alexey D.]

Hello Ole,

For VMs belonging to same job concurrent processing is not possible. And deduplication works only for VMs in bounds of one job.
But if you want parallel processing there might be a balance point though. For example, create one job and place your Windows VMs inside,
and another job, let's say, for Ubuntu Linux VMs, etc. Dedupe is only effective between similar VMs anyway.

It's up to you which scenario to implement, but the general idea is described above. Hope this helps!

[Oletho]

Thanks Alexey.

Yes, this helps me to confirm that my understanding of the limitations is correct. That's a good start

But the fact that dedup is most efficient across similar servers does not always cope with the real world needs. Example: we have a system that involves Windows servers (2000, 2003 and 2008), Redhat Linux, SQL and Oracle databases. The goal is to have as small a backup window as possible, and with other products like VDR and esXpress i can run all simultaneously in one job to the same dedup store.

Well, guess I just have to think different.

Thanks again,
Ole Thomsen

[Gostev]

Hi Ole, I wanted to add that we have a good whitepaper called "Deduplication and VMware Backup Sprawl" going into details of our dedupe approach (per job), and comparing it to a single dedupe store. I recommend that you check it out, it is more technical than marketing (written by our system engineers).

You can download it from here:
http://www.veeam.com/whitepapers.html

By the way, VDR is almost no different from us in terms of end results: while it provides ability to write to a single dedupe store, the dedupe store itself is limited to 2TB in size - so you are going to have to use multiple ones.

Thanks!

[Oletho]

I'll take a look at the whitepaper, Gostev.

I tested the latest VDR for a couple of months, and while the store limit is 2TB the supported size is only 1TB (500GB for CIFS). Even then it gets unstable the closer one gets to fill the store, and the missing features and weaknesses makes it unusable for all but the smallest environments in my opinion.

Still I wish Veeam had concurrency in each job, this is one thing I really like about esXpress. Is it a design decision or because of the single NTFS file that holds the job data?

Ole Thomsen

[Gostev]

Hi Ole, it is more of historical design/feature decision.

More detailed answer to the same question:
Why does Veeam do serial VM backups instead of parallel?

[Bunce]

Ditto here. We have no issue 'flooding' any pipe when running multiple jobs.

As mentioned frequently, having to run multiple jobs removes the dedupe advantage, and increases management and monitoring requirements.

Not being able to order the backups within a job then adds further complication when trying to backup and replicate the same VM in different jobs (to prevent being backup up at the same time) - while still trying to maintain a small backup window.

If its a technical limitation of the backup engine, perhaps Veem could consider a custom backup 'appliance' target that multiple jobs could write to concurrently? (also of the ilk of esxpress). This may also provide the possibility to provide useful reporting statistics from the target appliance (dedup, compression, VM stats, indexing and searching, accurate transfer rates etc - all the good stuff that we can provide in managerial reports)

Just a thought.

Cheers,
A

[Gostev]

No, this is not a technical limitation of the backup engine. This is specifically explained in the post referenced in my previous reply.

[Oletho]

Hi Ole, it is more of historical design/feature decision.

More detailed answer to the same question:
Why does Veeam do serial VM backups instead of parallel?

I understand the decision based on those findings, but that is not my experience so far.

We use 10Gbit NFS as primary VM storage (FC for backup store) and I chose to run a 4 vCPU virtual appliance with 4GB RAM on Windows Server 2008 R2.

Each HOTADD backup job uses ~25% CPU (equals 1 vCPU) and gives a processing rate of ~30 MB/s for full backup, higher for CBT.

So I see 2 full jobs using 50% CPU and total processing rate ~60MB/s, 4 full jobs doubles both numbers. I did not add additional vCPUs to see when that flattens

Ole Thomsen

[tsightler]

I understand the decision based on those findings, but that is not my experience so far.

It's not my findings either. I've always seen better performance from multiple jobs than from a single job. Even if a single job saturates the SAN or the CPU that doesn't account for all the "down time" in the job between VM's while snapshots are taken and removed. This can be significant on busy systems (our Exchange server regularly takes 45-60 minutes to remove the snapshot, all the while that Veeam job is basically doing nothing). We see significantly better performance from multiple jobs, even on our relatively old servers.

It also feels like the idea of a single job saturating the SAN or CPU is based on running fulls, but we rarely run a full backup. For CBT based incrementals, a single job rarely uses more than 25% of CPU in our environment, and the SAN in nowhere near saturated. In that scenario the overhead of creation and removal of snapshots is almost half of the entire backup time. Having multiple jobs can cut the overall job time nearly in half for these circumstances. We've found that 4 jobs is generally the best setting for performance without completely saturating the SAN.

[MartinSvec]

Hello, I'd like to reopen this question.

I'm evaluating if Veeam v6 is a suitable backup solution for our vSphere 5 cloud environment and the sequential job processing seems to me like a potential source of troubles. And if I'm right, there is still no option to enable concurrent intra-job processing in v6? Having high number of small VMs with minimal daily CBT changes, I see a lot of time wasted by e.g. waiting for snapshot creations, snapshot removals etc. In this time, storage, CPU and network resources are clearly underutilized and the overall backup window unreasonably increases. Maintaining multiple identical jobs and manually distributing VMs between them just to workaround this behavior doesn't look like an user-friendly idea.

I'm also asking myself if all the great scalability features like backup proxies, storage repositories, automatic load balancing etc. are so great when the level of backup paralellism is primarily limited by the number of (manually created) jobs. With modern 10GE SANs and multi-core backup servers with plenty of RAM and HDDs, I think this deficiency will become more and more obvious.

So, is it true that the job processing is still purely sequential or is there any hidden option that I overlooked? And if it's still not possible, do you have any plans to implement it?

Thank you.

Martin Svec

[Vitaliy S.]

Hi Martin, there is more recent topic discussing all the pros and cons of this approach, please look it through and continue posting there: Concurrent Tasks

[c_manboy]

[merged]

I created a replication job with 7 vms. The source proxy and target proxy are different (yet both are local and the destination is local) and configured with 3 "Max concurrent tasks". So my question is, should the vms be processing concurrently within the same job?

[dellock6]

no, all VMs in a single job are processed sequentially. Cuncurrency takes place when you have more than 1 job running at the same time.

Luca.