parallel processing

[Tobias_Elfstrom]

I might as well ask this here too
Are there any options related to the new parallel processing or is that just default on if a new job is created in v7 ?

BR Tobias

[veremin]

By default (after the upgrade), the parallel processing is turned off. So, in order to enable it go to the menu -> options -> advanced -> enable parallel VM and virtual disk processing.

Thanks.

[Tobias_Elfstrom]

So there is no per job options for this but rather for one whole instance of Veeam Backup & Replication server then?

[foggy]

Yes, this setting will take effect for all backup and replication jobs on the Veeam B&R server.

[Gostev]

I could not see any value of making it disableable on a job level.

[simwood]

I'm surprised there's no options with this too so can only assume there's loads of intelligence built in. Is there a deep dive explanation anywhere?

Parallel disk backup sounds like a big win and will help with backup related IO issues on heavy VMs. However, backing up multiple heavy VMs in parallel will have the opposite affect. How is B&R managing this decision making?

If there was an option to enable parallel disk processing but disable parallel VM processing I'd be ticking it but there may be way more going on than I know.

[Gostev]

I/O issues can be controlled by the number of concurrent tasks on backup repositories, this capability has been in the product since version 6.0. The whole issue is not particularly new, as you could also run into these issues with concurrent jobs processing multiple heavy VMs at the same time.

Parallel VM processing is only leveraged when you have enough task slots to process all virtual disks of multiple VMs in parallel, so you can easily control the load using standard means. For example, if you set the max task slots on your backup repository to 2, and most of your VMs have two virtual disks, this will effectively disable parallel VM processing completely (except if the job gets hits two single-disk VMs), but 2 virtual disks of the same VM will still be processed in parallel.

[lohelle]

I would like to suggest the possibility to limit max parallel tasks per job. 8 parallel tasks would not be a problem on a SSD-SAN, but it could really hurt VM performance on a small SATA-HDD SAN/array. Remember that the same proxy could be used by many jobs, so then it could be necessary to create a separate proxy for the lowend storage and limit tasks there.

A very nice feature would be to be able to limit parallel tasks on a per datastore-cluster/folder basis in the proxy-configuration.

[lohelle]

Some feedback from a customer I upgraded to v7.
Optimal compression.

Backup:
Job HP - 36 VMs - 3 TB data
v6.5 - 3 hours
v7 - 45 minutes

Job Exchange - 20 VMs - 1,5 TB data
v6.5 - 100 min
v7 - 25 minutes

Replication:
Job Replica HP - 36 VMs - 3 TB data
v6.5 - 4 hours
v7 - 70 minutes

Splitting into more jobs on v6.5 would help, but it would also mean more work/monitoring.
The backup-times would most likely improve even more if data was backed up to a local repository. The backups are sent off-site and the replicas are sent to a local spare SAN.

[Gostev]

I would like to suggest the possibility to limit max parallel tasks per job.

The max parallel tasks per job is implicitly controlled by max parallel tasks setting on the backup repository used by the given job (and real backup repository throughput).

I don't expect your backup repository to be more powerful than your production storage, so if backup storage can keep up with set amount of concurrent tasks, the production storage certainly will too and if backup repository cannot keep up, the engine will be just waiting for it most of the time to write the data, not reading anything from the production storage.

I do have some ideas around controlling load on production and backup storage more intelligently for future versions though.

[lohelle]

The backup repository might usually be slower, but live VMs usually do not run from this storage (even if YOU made it possible!).
Maybe a "number of parallel tasks" selection in the job setup where you select source proxy would be possible in a coming release..?

VERY VERY nice job on v7 btw. Some more vendors supported with SAN integration would be nice. We have mostly "small" customers, so we could like Starwind integration.

[Andreas Neufert]

The backup repository might usually be slower, but live VMs usually do not run from this storage (even if YOU made it possible!).
Maybe a "number of parallel tasks" selection in the job setup where you select source proxy would be possible in a coming release..?

You can do this today by using only proxies with the needed proxy-slot count for the jobs with this heavy load.

[eortega]

Per JOB would be nice. Currently I had a situation where it processed both of my Exchange Mailbox Servers at the same time causing them to kind of freak out on who should Mount the databases.

What would be the best way to stop this from happening besides creating 2 different jobs.

[Gostev]

I would just reorder them within the job to spread out. This is possible even when you configure job with containers.

Say, you currently have your job configured by adding Container 1 (containing both Exchange servers).

In that case, just add both Exchange VMs explicitly as well, but order them like this in the list:

Exchange1
Container1
Exchange2

This should work.

[Andreas Neufert]

Per JOB would be nice. Currently I had a situation where it processed both of my Exchange Mailbox Servers at the same time causing them to kind of freak out on who should Mount the databases.

What would be the best way to stop this from happening besides creating 2 different jobs.

Scary parallel processing. 2 DAG member because of high Dedup (and both because of Surebackup) in same job.

[veremin]

Per JOB would be nice.

Also, if you want to process VMs within certain job sequentially, it might be worth creating a dedicated repository (it might even a new folder on already existing repository) for this job and set 1 as the number of backup tasks this repository can handle. This way, any job that uses this repository will have its concurrency limited.

Thanks.

[Rumple]

I just hit this same thing without realizing it...both CAS Servers and a mailbox server in a job....
they really need a way to disable this per job...so I don't have to create separate jobs just to ensure both sides of an exchange node don't try to backup at the same time.

[veremin]

As mentioned above, for now the best option will be to limit the number of concurrent tasks the job proxy/repository can handle. It should guarantee that VMs will be processed sequentially.

Anyway, thanks for the feedback; much appreciated.

[m.novelli]

Hi, I've been using for some time the new parallel processing option for different customers, and my feeling is that with modern hardware on the backup server the backup process can slow down too much Virtual Machines / datastores / Hosts

I'm asking for a new option that we can call "intelligent parallel processing" with a behavior similar to this:

- check how many task can run at the same time on the proxy server. For instance: 2
- start processing the first VM. If the VM has a single disk, start processing the next VM with a disk NOT located on the same datastore as the first VM. If the backup mode is NBD start processing the next VM with a disk NOT located on the same datastore as the first VM AND not located on the same host as the first VM
- if the first VM has two disk, start processing only the first disk AND start processing another VM with previous logic.

The main logic behind that is that Veeam should backup only one disk per datastore and only one VM per host if the backup mode is NBD
This logic will process the VM slightly out of order but seems to me a better tradeoff than actual parallel processing

Comments are welcome

Marco

[Gostev]

Hi Marco, the main reason why customers asked for parallel processing is that backing up a single disk per datastore almost never saturates typical storage. It was reported that for most arrays require at least 3-4 threads to do this. So, your proposed logic will harm the performance.

I also disagree with processing only one disk at a time for the VM with two disks is a good idea - in fact, the most positive v7 feedback comes from parallel processing of multiple disks of the same VM in parallel (which dramatically reduces time VM runs off the snapshot).

The only times I see backup process slowing down the infrastructure too much is when customer oversubscribe their infrastructure by creating too many powerful backup proxies, which enables tons of VMs to be processed in parallel. In most cases, after upgrading to v7 customers can easily remove half of their proxy server slots, and still get the same or better performance than with previous versions.

And there is a really easy fix. Do not oversubscribe

Now, I am not saying here that the current version is already super smart. There is always room for improvement, and we will certainly have further improvement in the "intelligent" part of our automated load balancing, but they will not be inline with what you are proposing here.

Thanks!

[m.novelli]

mmm so probably we should differentiate "parallel processing for Enterprise" from "parallel processing for SMB"

Unfortunately I work mostly with singles VMware servers or little infrastructures with 2 - 3 hosts and a shared iSCSI SAN Dell PowerVault MD3xxx series and modern Veeam Backup server with local proxy can really slow down quite a bit those SMB infrastructures

Marco

[Gostev]

Limit Veeam proxy to two concurrent tasks, and put ingest rate throttling on the backup repository if backup is still too fast (this is actually a good problem to have!), and the backup will not slow the infrastructure down. While those few dozens of VMs from 2-3 hosts with a still backup in no time.