Concurrent Tasks

[tgiphil]

Another +1.

[sipo75]

+1

Saturation? After five days of reading through the manual and the dozens of related posts I am still stuck with 3MB/s with v6 both with "Virtual Appliance with hot-add" and "Network" for initial backups.

If I could "thread" a job I could work around this issue.

[mysidia]

There's no way a sequential method will keep all phases (source, destination, target, CPU, ram, network, WAN) saturated.

Thing is, you do not need to have ALL phases saturated - as soon as the "weakest" phase is saturated, it will become the bottleneck for the job - and as soon as this happens, it no longer matters that other phases are not saturated. Because there is a bottleneck for the job already, it is pointless to start processing of another VM in parallel within the same job.

I would like to see more concurrency in the Veeam backup processing. In my case, I am backing up large VMs with multiple separate VMDKs. On these VMs, each of their VMDK's is on an entirely different storage array, for the purpose of balancing the load.

What happens is, the backup server's load utilization is very low, and the source is the bottleneck, as in throughput through each storage array, each array tops out at around 30 MB/s with the load that they have; I have multiple source proxies, but nonetheless Veeam backup processes the job VMDK by VMDK.

This serial processing is highly inefficient. If the backup software efficiently processed several of the Virtual machine's virtual disks in parallel, instead of one by one, the VM would finish being backed up much sooner.

[tsightler]

+1
Saturation? After five days of reading through the manual and the dozens of related posts I am still stuck with 3MB/s with v6 both with "Virtual Appliance with hot-add" and "Network" for initial backups.

While I agree that having multiple streams (one per VMDK) might be nice, in your case, 3MB/s is unbelievably slow. It would be best to attempt to figure out why this is the case. My laptop running VMs inside of nested ESX with a regular laptop drive provides, on average 10-20MB/s, and that's backing up to itself, so both reads and writes are on the same, relatively slow laptop hard disk (not an SSD). Backing up to a remote repository over my home wireless is around 15MB/s. There has to be something very, very wrong with your setup.

[MartinSvec]

Hello,

another +1 for me (Vitaliy, thanks for redirecting me to this thread). I evaluate Veeam v6 four our new vSphere 5 cloud environment with dozens (will be hundreds to thousands) of VMs that are relatively small and have minimal CBT changes. Having redundant 10GE iSCSI SAN, enterprise storage arrays managed by Storage DRS and a decent backup server, it's impossible for me to efficiently utilize the backup process with the number of jobs that corresponds to logical groups of VMs. About 10-30% of VM backup time is spent by waiting for quiescing, snapshots and appliance reconfigurations, and during the actual backup there are still plenty of CPU/bandwidth/storage resources available. I was really surprised that despite all the v6 scalability features, the level of concurrency is primarily limited by the number of jobs.

Although we clearly cannot backup hundreds of VMs by only one job and so there will be always some concurrency, I still see this limitation important when considering if Veeam is suitable for our environment. It must be taken into account when planning the structure of vSphere folders, when planning the capacity and partitioning of backup servers, it's necessary to ensure uniform distribution of VMs between multiple jobs to maintain the backup window length, etc.

So, if it's not an architectural issue of the backup engine, I vote for an option to enable and tune concurrency inside jobs. Let users decide which jobs - serial or parallel - are better in their environment

Thank you

Martin Svec

[acasanovanex]

Another +1

[TimJWatts]

[merged]

Hi folks,

Sorry - probably a newbie question...

My early tests with Veeam Backup suggest that if you set up a single backup job with a lot of VMs in, then the system backs up each VM sequentially.

Is that correct?

Assuming so, presumably to get some parallellisation, you need sereral backup jobs with subsets of VMs in running concurrently?

A single backup job with Direct SAN enabled is dumping at a claimed 50MB/s - that's about half of the theoretical max (gig link to very (as in several miles away) remote linux backup store). I suspect a couple of jobs in parallel might get closer to full link utilisation, especially if using 2 proxies running as VMs on separate ESX hosts.

Cheers!

Tim

[TimJWatts]

Thank you Mod - I did not look back this far in the search listing - but just the ticket

[masonit]

Hi!

Is this something you plan to implement? For us this is a big downside that all jobs run serial. It would be alot easier to manage backup window with parallel processing.

\Masonit

[Guido]

I have 4 proxies (hotadd) and configured one backup job for 20 VM's. The VM's are processed one by one. Is that normal? Is that becourse I selected "Automatic Selection" in stead of "use the backup proxy servers specified below"?

How do I know how many proxy servers are used at a given time?

[Vitaliy S.]

How do I know how many proxy servers are used at a given time?

You can either review job session or use Veeam ONE to monitor proxy server load.

[dellock6]

Well, if you have only 1 job, it will only involve one proxy, regardless how many of them you have in place.

Luca.

[ptmartin]

Hello All -
I have what seems to me to a basic question so forgive me if the answer is obvious. ..

When a backup job is being processed that contains multiple VMs, why is it that only one VM is processed at a time? It seems to me that if more than a single VM is processed then the backup time would be reduced .. potentially substantially if several machines can be processed at the same time. If it is because of storage (that is the only reason I can think of) perhaps at least being able to process VMs that are on different LUNs at least would be a good option to have.

Just curious ... I cannot see anywhere in the interface for a regular backup job where I can select how many VMs are processed.
I have that option in a SureBackup job, why don't I have it in a standard backup job?

Thanks,
Paul

[tscott]

I have one VM Proxy with 2 vCPUs x 2 vCores so I have the recommended amount to run 2 concurrent tasks..

I have one backup job that is backing up my datastores with multiple VMs.. When I run the job it only backs up 1 VM at a time.. I'm assuming this is normal?

If so, should I have multiple backup jobs? One for each datastore? And then would I get concurrent backups?

Thanks

[rbrambley]

Yes that is normal

task = job in Veeam speak. Running 2 jobs at the same time would be 2 concurrent tasks.

Veeam processes 1 VM at a time in a job.

If you only have 2 datastores then yes, 1 job per datastore would be a good way to go.