Concurrent Tasks

[joost1981]

Hi there,

I have a question about concurrent tasks. We have one backup job which holds about 75 VMs. In version 5 it starts processing them one by one.
I was hoping that the new feature "concurrent tasks" would start processing multiple VMs at a time (from within one backup job). Is this not how "concurrent tasks" are desgined or did I just misunderstood this new feature? Or do we have a problem with our current setup?

Thank you in advance.

Cheers,
Joost Poulissen

[Gostev]

Hi Joost, within the job, each VM is processed sequentially according to the defined processing order. Each proxy can run multiple jobs at once, this is what this setting affects. Thanks!

[joost1981]

Hi Gostev,

Thank you for your answer. The reason why we have only one job, is that we backup the complete VMware cluster. When a new VM is created in VMware it will automatically backup in Veeam.

Is this a feature worth implementing in a new version?

Thank you again!

Cheers,
Joost Poulissen

[Gostev]

Hi Joost, sure - everything is possible depending on demand. I can certainly see how this feature can be useful for monster jobs, however typically customers prefer to create smaller jobs (to keep the backup sizes manageable), and run them in parallel - this is the use case v6 is tuned for.

I am not sure why the fact that you need to backup vSphere cluster forces you to put all VMs in the same job. Most customer seem to prefer to organize the job based on VM folders instead of infrastructure objects for added flexibility. Have you considered using VM folders for backup jobs?

Also, unlike previous version, v6 should be able to fully load your backup infrastructure with consequent processing of VM in the job, so you should be seeing massive improvements from v5 anyway.

Thanks!

[joost1981]

Hi Gostev,

Thank you for explaining this. Of course using VM Folders to backup is an option for us, we can redesign jobs.
However in the future our environment is expected to grow exponentially (about 6/8 backup proxies and 400/500 VMs) and then running mutiple VMs backups simultaneous from one job instead of creating an huge number of jobs would be handy. However of course it is not a requirement!

Thank you again for your help!

Joost.

[mgambetti]

so let me recap if i have understand correctly...if i need to make more parallel backups of VMs inside the SAME job, the OLNY solution is to set more concurrent tasks?
or the other solution is to split VMs on more jobs, and use more proxies, am i right?
thanks.

[foggy]

Please note, that task=job here. So the only way to get more balanced backups is to split jobs into smaller ones, as within the job VMs are processed sequentially.

[averylarry]

Is there a reason for the sequential processing? Even if it only helps a little bit, I have to think it would be more efficient to process multiple VM's simultaneously. There's no way a sequential method will keep all phases (source, destination, target, CPU, ram, network, WAN) saturated.

On the other hand, if it messes with deduplication or compression or something, then I would understand. But I would argue that some people (myself included) have created smaller jobs EXCLUSIVELY for parallel processing.

[andrewilmann]

+1

We are about to split up our Backup job just because of this to!

[jamessa]

+1 here too. This problem for us causes backup jobs to run for a long time when it hits a new 300GB VM. I know the answer could be separate jobs by folders or resource pools, but with multiple people working on our team VMs could and do get moved around between these quite often, which causes them to get duplicated in backups. This is a major problem for us that keeps us from doing multiple jobs per cluster.

[Gostev]

There's no way a sequential method will keep all phases (source, destination, target, CPU, ram, network, WAN) saturated.

Thing is, you do not need to have ALL phases saturated - as soon as the "weakest" phase is saturated, it will become the bottleneck for the job - and as soon as this happens, it no longer matters that other phases are not saturated. Because there is a bottleneck for the job already, it is pointless to start processing of another VM in parallel within the same job.

The ONLY benefit of parallel processing within the job is that while one VM is only preparing for processing, another one can be processed. We are talking about 20-30 seconds of time that is otherwise simply "lost" with sequential method. However, the price you would have to pay for these 20-30 sec of time is twice longer time that each VM lives on snapshot (because with 2 VMs being processed in parallel, processing time for each one doubles), with in turn directly affects snapshot size, and so snapshot commit process that impacts actual VMs. This is exactly why finishing processing of every VM as fast as possible is so important.

The devil is always in details - just because something sounds nice, it does not mean it will work nice in all aspects.

Please note however, that everything I said above applies to v6 mostly, as the previous versions were not nearly as good with saturating backup infrastructure with a single sequential job.

[averylarry]

I just spent 20 minutes writing a response that was disjointed.

Let me just say this -- I'm a small business. I can't afford fancy fast storage. More than anything else -- I think my biggest point is simply -- if I have multiple proxies, why can't I use more than 1 at a time? I use "poor man's aggregation", where I have separate LUNS on my SAN that don't share iSCSI network ports. I have iSCSI ports and LUNS that aren't even getting used because it won't process multiple VMs at the same time.

More jobs? Doesn't that hinder deduplication? Not to mention increasing the administration and complexity for a small business.


Just for the record -- I still love Veeam, and in particular v6. Sometimes I think small businesses (and small budgets) are somewhat forgotten.

[Gostev]

Point taken.

[jamessa]

Very interesting topic and it looks like there are pros and cons for both. I think in future versions there should be a way to allow it, maybe not a default option but one that is an advanced setting. We are also concerned about things like snapshots taking too long, but if we throw enough horsepower at the Veeam server and our backup target is fast enough it seems like a good option to let the more technically inclined people be able to tweak backups to run more than one at a time if they so wish.

JM2C

And btw Gostev, thanks for your participation in these forums!

[vluzhkov]

There's no way a sequential method will keep all phases (source, destination, target, CPU, ram, network, WAN) saturated.

Thing is, you do not need to have ALL phases saturated - as soon as the "weakest" phase is saturated, it will become the bottleneck for the job - and as soon as this happens, it no longer matters that other phases are not saturated. Because there is a bottleneck for the job already, it is pointless to start processing of another VM in parallel within the same job.

You are making main assumption, that backup of one machine should saturate at least one phase in any case. That's not right at all. As it was said in low-budget installations you may have the example of different storage targets. In high-budget installations you have storage and network multipathing, many independent datastores (even storage-drs managed in vsphere 5), high performance network and storage which with CBT can even make data transmission time shorter than preparing to backup VM.
When you are trying to prevent saturation for specific phases you use phase-specific concurrency limits - for proxy servers, for repository servers. These methods are already implemented. Really, i was very surprised after reading pre-v6 infos, that backup jobs still runs sequentially. And really disappointed. Even free vmware Data Recovery has concurrency as one of base features.
Nowdays despite of all new features of v6 we still can't optimize performance of backups without splitting jobs - the concurrence features are almost useless.
It's no way to present jobs as method of managing backup queues - any new instance of any object should be created only when it's properties are different from existing ones. For jobs these are storage policies (expiration, incremental/reserve incremental etc), application processing settings, schedule, proxy/destination, all settings you specify when creating the job. There is no reasons to create different jobs with same settings - expect of software limitations, which force you to manually micromanage things that should not require management at all and add a huge overhead when you should continually inspect job logs to distribute VM's.
Traditional backup software sometimes has sequential approach to processing data because of being tape-orientated - and that is not veeam case again.
I usually don't write any suggestions in support forums, as it has a little to no effect, but in case if veeam, which, as i think, is not a very big company, i hope these opinions can somehow reach the decision makers in veeam...