Backup Copy Jobs - Parallel Processing?

[mmonroe]

We noticed a huge performance increase with normal jobs when parallel processing was added in Veeam. Is there any way that parallel processing can be added to the Backup Copy Jobs?

We see a good bit of "wasted system time" while Veeam starts/stops each VM. The server never comes close to fully using the network interface, the CPU's or memory on a backup copy job. Server wise, it can do many VM's at the same time as it does with normal backups.

It looks like we could see some massive performance gains in backup copy jobs if we could process several VM's at the same time.

[foggy]

You can always run several backup copy jobs in parallel. What kind of speed do you observe for the backup copy job (and over what kind of link) and what are the bottleneck stats it reports?

[mmonroe]

I watch the machine closely - the CPU usage, the memory, the network interface. I watch the time it takes to finish one VM and then begin the next one. Veeam has "waits" and pauses and does stuff in the background. There are times when the server is actually doing very little "work". This same behavior is what we saw with regular backups prior to the parallel processing being added. Once you get a handful of VM all processing at the same time, things smooth out and you can keep the server under a fairly steady load.

With an 8-core server (with hyperthreading) it is rare for the total CPU usage to go over 30-40% on a Backup Copy Job. The network usage will spike up a good bit but there are long periods where the usage is "zero" or well under 50%. The server needs to be able to process more than one VM at a time during the backup copy jobs. I bet that the total time for our backup copy jobs would drop by half if it would do 8 in parallel versus 1.

[veremin]

What component is identified as a major bottleneck? Is it target by any chance? Thanks.

[mmonroe]

I will also add this - the speed of our exact same backup copy job to the exact same repository on the same servers is much slower with V8 versus V7. It could easily be taking twice as long - or longer - on average. I hope that the upcoming patch will help this. /mm

[namiko78]

Is it possible to enable parallel processing for backup copy jobs, in the future? There is quite a bit of time that the storage at source and destination is idle, while it's gathering info about 1 vm at a time, moving through the various steps. Mine seems to sit at each hard disk for up to 2 minutes before i can see any activity on my storage, not sure what it's doing during this time, but it results in 120 vm's in my job taking a long time to process.

Thank you

[cby]

Yikes, 120 VMs in one job?!

Stating the bleedin' obvious, why not break this into fewer VMs in multiple jobs that run concurrently? Or have I missed something?

[namiko78]

I considered that, but i'm trying to balance manageability with performance, the more VM's in one job, the better dedup I get. The more jobs, the more space it would consume and it would be more complex to manage. The backup jobs themselves work fine, because they process multiple vm's in parallel, it's just the copy jobs that are sluggish.

What is best practice around amount of vm's per job?

[cby]

Agreed, the whole business is a balancing act -- backup storage (shared multi-paths), backup/copy times, scheduling, best dedup model, integration with tape backups, etc. For example, we found that the LUNS (for backups and where the VMs reside) can make a difference in backup times, e.g. where the concurrent backup jobs for several VMs are targetting the same LUN/datastore it can have a negative impact on backup times. We do the obvious by staying with identical operating systems VMs in the same job.

However, we take the view that the most important aspect of any backup process is the restore, i.e. how fast can we restore VM/vdisk/directory/file and is it sound? It does require testing different backup regimes to see which fits best for your infrastructure.

Presumably you run test restores so you could alter the jobs and monitor backup/copy times.

I don't know how typical this is but I have come across Veeam jobs that were set up once and never reviewed or altered to suit the environment, especially a changing environment.

[foggy]

Steven, what are the bottleneck stats for this job?

[namiko78]

I'm looking at the jobs and trying to improve backup times, often Agreed the restore is the most important part, I want to implement sure backup on a schedule when i get the new backup server running. We just switched from daily backups to tape with symantec to daily backups offsite with veeam. anyway, off topic.

Foggy, the job has been running for 6 hrs now, it's 40% complete and the stats are source 26%, proxy 0%, network 8%, target 28%
I can understand if you're going to say my storage is slow and that's the bottleneck, but i'm seeing so much idle time where it's just working on various steps, it seems inefficient. Is it just me that has this problem?

[mmonroe]

For better or worse, I am glad that I am not the only one having backup copy job performance issues. Backup copy jobs are slow to the point - for us - of almost being unusable, however, the exact same VM's with the exact gigs of data to the exact same repository with a normal forever forward incremental backup is fine.

Fine meaning that the forever forward normal backup takes 1 hour and the backup copy job takes 5-7 hours on a daily incremental of 50-70gig.

On a "full" (1.7tb backup) the normal took 8 hours and the backup copy job ran 16 hours and failed since it hit the new copy interval.

[namiko78]

I've brought this up before about why we can't have parallel processing for these jobs, and was told there would be no advantage to doing so. When i look at my source and destination and see next to no activity, that tells me i'm wasting time.

Maybe we should just have two main backup jobs, one that writes to storage at headoffice, another that writes offsite. I don't really want to hit my prod storage twice, and would that have any impact on CBT?

[namiko78]

mmonroe, do you by chance have your SQL database on a remote server? I'm trying to figure out if that could be causing some performance lag.

[mmonroe]

SQL is installed on the local Veeam Server. We just installed it as part of the Veeam load/install. That being said, I built this Veeam box fresh from scratch with 2008R2 Standard early last year and then loaded V7 fresh/clean at that time. I recreated all setup and jobs from scratch and let all of the new VBKs build from scratch/fresh. I then upgraded the installed copy to V8 in November when it came out.

[namiko78]

Ok. in my case, i've got the veeam server on server 2012 R1, built from scratch about a year ago, remote sql DB, remote proxy and repo. Thanks, hopefully they'll come back with some suggestions as i'm in the same boat as you, the copy jobs are too slow to be usable. a regular job would be better as we don't need the GFS stuff.

[foggy]

I've brought this up before about why we can't have parallel processing for these jobs, and was told there would be no advantage to doing so. When i look at my source and destination and see next to no activity, that tells me i'm wasting time.

It is true that generally processing multiple VMs in parallel will not have any impact on backup copy, since it simply copies data blocks from one storage to another. I recommend asking support for assistance in determining what the job actually does during those "no activity" periods.

Maybe we should just have two main backup jobs, one that writes to storage at headoffice, another that writes offsite. I don't really want to hit my prod storage twice, and would that have any impact on CBT?

No impact, CBT is designed to handle such cases.

[namiko78]

Thanks. I'll open a support case. If i set the 2 main backup jobs to be at the same time, i assume veeam is smart enough not to try and back up the same vm at the same time?

[foggy]

Sure, the second job will just wait for the VM to become available for backup.

[namiko78]

Case # 00728394
So far, support has just suggested seeding or breaking the job into smaller pieces, but at the rate that it's running, even with seeding, I don't think it will process the incremental fast enough. I've asked it be escalated.

One thing i noticed, my repo server is a VM, with iSCSI connections into my storage array. Often, i'll see the veeam agent process using 50% of the CPU, and storage doing nothing. I have 2 cores assigned to this VM, but it seems to be only using one of them, could this be where some of these mysterious hangs are coming from? Another case for parallel processing?

Thanks

[namiko78]

Also, since my initial seed has never has a chance to complete from over a week ago, i now have an 800 gig VBK, and a 1TB vib and several multi-hundred gig vib's. Could this also be contributing to the slowness?

[cgoesch]

It is true that generally processing multiple VMs in parallel will not have any impact on backup copy, since it simply copies data blocks from one storage to another. I recommend asking support for assistance in determining what the job actually does during those "no activity" periods.

Perhaps I am missing something but if the argument is that processing multiple vms in parallel will not have any impact on backup copy speeds, then why would breaking into multiple jobs speed things up?


My single daily backup job takes roughly 7 hours to back up up 50TB/417 vms (which is simply amazing). We purchased data domain planning to utilize backup copy feature only to find out it will only process one VM at a time. After 16-17 hours my backup copy is only at 40% when the next daily diff kicks off - so backup copies never get a chance to finish. I'm not sure if I am going to be able to make this work as it is. Hate to say it but we may have to look at Avamar after all...

[namiko78]

Update to this: support suggested i lower my compression on the source job, and i've done that, so my backup copy job now finishes during the day, but my incrementals are 3x larger. Too large for my storage. Because of the lack of parallel processing, my best option is to have two main backup jobs, this should allow me to use the higher compression and have the job complete within the same day.

Backup copy jobs just can't handle too many VM's at once. and still complete within our window.

Still not clear on why we can't have parallel processing on backup copy jobs. They simply don't' perform as fast with sequential processing.

[mmonroe]

We too are having significant performance issues with backup copy jobs. We have a ticket escalated with support but so far we do not have a resolution. Performance with the main backups is fine even to the same repository on the same server with the same VMs. We are on the edge of not having enough time during the day for them to complete as well.

The exact same backup copy jobs took about 1/3 the time on V7 versus V8 and I have a complete backup string with the VBK and 58 VIBs to prove it (two months worth of the backup copy job on V7). V7 was still not as fast as a primary backup but it was still 1/3 the time of the same job done on V8.

[foggy]

Perhaps I am missing something but if the argument is that processing multiple vms in parallel will not have any impact on backup copy speeds, then why would breaking into multiple jobs speed things up?

I said generally, meaning that in most cases it does not have effect (not much where you can parallel something). It should have effect in case of different sources (making network or target to be a bottleneck) or if you need multiple streams to saturate the link (but we already have multi-streaming in backup copy jobs).

My single daily backup job takes roughly 7 hours to back up up 50TB/417 vms (which is simply amazing). We purchased data domain planning to utilize backup copy feature only to find out it will only process one VM at a time. After 16-17 hours my backup copy is only at 40% when the next daily diff kicks off - so backup copies never get a chance to finish. I'm not sure if I am going to be able to make this work as it is. Hate to say it but we may have to look at Avamar after all...

Is it the initial backup copy job run? Then you can seed it to avoid initial transfer of entire data, subsequent increments will take less time.

[cgoesch]

Well I finally split this into 4 backup jobs & 4 backup copy jobs. I'm getting in excess of 2Gbps throughput now for my backup copy to DD. No doubt running 4 concurrent jobs is speeding things up tremendously.

[namiko78]

So, veeam employees, is 130 vm's too much for one job to handle? have i created something against best practice?

[Gostev]

No, 130 VMs is nothing... see feedback from the other user just a few posts above:

My single daily backup job takes roughly 7 hours to back up up 50TB/417 vms (which is simply amazing).

[mmonroe]

Interesting on this thread where it was claimed that there would be no performance improvements with "parallel vm processing" on backup copy jobs yet now its a big feature and improves performance in v9. I guess we told you so, but what do silly customers/users know.

I have to admit that I find it a bit humorous how our Veeam hardware hasn't changed in 2-3 years yet invariably when we have performance issues after loading new Veeam versions, the whole support discussion starts with a hardware review even though none of it has changed - just the Veeam software has changed. Performance of v8 and v9 over versions v6 and v7 has significantly improved but has all been in the Veeam software, fixes, updates and enhancements.

[foggy]

I believe the statement that "processing multiple VMs in parallel will not have any impact on backup copy" made previously in this thread mostly meant the single backup file per job implementation, while what really allows for significant performance improvements with v9 are per-VM backup chains.