Feature Request: Multithreaded backup copy jobs

[nezzer]

Hey!


Just figured out with Veeam support that Backup copy does not support mulithreaded read on the backup file.
This is a big deal when dealing with lots of VM's. 200+ and lots of data.
I would like to place a feature request for multithreaded support.


Backstory for anyone interested:
Due to this fact we are looking a 1/8 performance of what our enviroment can achive.

There are 2 possible "dirty" workarounds that has not yet been tested by me.

1. Optimize block size.
2. Split Backup Copy Jobs ( Each job is one process and one thread)

Kindest Regards
Nezzer

[foggy]

Hi Andreas, another workaround is enabling per-VM backup chains on the source repository - in this case each VM will be read by a separate data mover agent in parallel, even though all VMs are added to a single backup copy job.

[mkretzer]

The same problem exists with tape backup - and per-VM does not help and there is no workaround!!

[veremin]

Kindly, try to stick to the original topic where the said issue has been discussed. This helps us to stay focused and not overfill existing thread with irrelevant questions. Thanks.

[danswartz]

Hi Andreas, another workaround is enabling per-VM backup chains on the source repository - in this case each VM will be read by a separate data mover agent in parallel, even though all VMs are added to a single backup copy job.

Sadly, the above requires Enterprise license

[foggy]

Correct, this is one of the advantages you get for additional price.

[nezzer]

Hi Andreas, another workaround is enabling per-VM backup chains on the source repository - in this case each VM will be read by a separate data mover agent in parallel, even though all VMs are added to a single backup copy job.

Heey Foggy!

Could you please explain a bit more about the per VM backup files. I guess the disadvantage of this would be no DDUP at all?

Using mulitple backup copy jobs would atleast give DDUP splitted into two seperate jobs.

I'm using a perperual hosting license Enterprise level.

[mcz]

just wannted to add that per-vm-files could reduce the compression/deduplication ratio (like in our case). Finally you would have to copy more data than you had before (single-threaded)...

[foggy]

Per-VM backup chains is a repository setting. Right, there will be no deduplication between different VMs in this case. What kind of target storage do you have?

[mcz]

What kind of target storage do you have?

Was this question addressed to me?

[foggy]

No, Michael, sorry, it was a question to Andreas.

[jaseruk]

200 VMs in a single chain, that would make me nervous....

I'd definitely be looking at either using per VM backup files (so only one VM at a time is knackered if the chain goes).

Or using smaller jobs - probably this to maintain some level of compression.

I think what I would prefer is multit-hreading of at least disks within a VM on a copy job. Currently if you've got multiple disks then it reads them one after another. Got a couple of customers where this causes issues with huge VMs. The main backup is fine because each disk is read at the same time, but then we hit this bottleneck on the copies.

[nezzer]

Per-VM backup chains is a repository setting. Right, there will be no deduplication between different VMs in this case. What kind of target storage do you have?

Hey Foggy!

I'm using 2 physical front nodes acting Proxy and Repo.
Storage is Local ReFS 64K

1 Physical backend for Copy jobs same config.

[foggy]

Then you could at least benefit from FastClone.

[nezzer]

Yeah FastClone works Great.

It's the health check that works slow.

[foggy]

After switching to per-VM backup chains it will run in parallel as well, so should be faster.

[nezzer]

Hey Foggy!

What is the impact of switching to per-VM backup files?

Will the whole repo run a full backup or will it build a new backup chain resulting in a period of massive backus untill retention is done?

Could you also give me pros and cons with per-VM backup files?


PS this is what concerns me:

Limitations for Per-VM Backup Files

The Use per-VM backup files option cannot be enabled for backup repositories with rotated drives.
If you enable the Use per-VM backup files option, data deduplication between VMs will not work. For more information, see Data Compression and Deduplication.
The Per-VM backup chains functionality is available in Veeam Backup & Replication Enterprise Edition and higher. If you configure backup repositories to produce per-VM backup chains and then install a license that does not support this functionality, you must manually disable the Use per-VM backup files option for backup repositories. Otherwise backup jobs targeted at these backup repositories will be failing.

[foggy]

Will the whole repo run a full backup or will it build a new backup chain resulting in a period of massive backus untill retention is done?

You will need to run a full backup for the per-VM chains setting to take effect.

Could you also give me pros and cons with per-VM backup files?

In your case, switching to per-VM chains will allow for parallel processing of VMs in the job, so you will be trading off parallel processing for Veeam B&R inline deduplication savings.

[nezzer]

Hey, just wanted to update you guys on this.

Switching to Per-VM backup files made all the difference. EXCEPT for a massive Hosted Exchange job which is single threaded as it's a single machine /agent. I will look into this later.
Case closed
Thanks!