Backup Copy and v9

[ChrisGundry]

Hi All,

I have a fairly large backup copy job that takes our backups off to the 2nd/DR site, this job contains most of our VM's. This job runs daily at the moment.
The local daily backup jobs that backup each of the servers to local storage are set to run between 8pm and 8am. The problems I have are:
1. The backup copy job does not seem to start processing SERVER1 because SERVER1 is part of JOB1 and other servers in JOB1 are still being backed up so JOB1 has not finished. So it seems to wait until JOB1 is complete before it will process any of the VM's in JOB1. I can understand this because I guess the copy job cannot take any data from the JOB1 backup file until it is finished. But it means that although a lot of servers are finished backup up between 8pm and 9pm they are not processed until say 6-7am when JOB1 finishes.
2. So the backup copy job starts processing the backed up VM's from JOB1/2/3/4 when the relevant jobs finish at 6-7am and transfers the data to the remote site. This takes a few hours and finishes about 10-11am normally. Then it takes from 10am through until late afternoon to 'merge' the backup files. This is due to poor storage performance at the destination which I am working on.

My questions are:
1. Are either of these issues changed by the new 'per VM' backup chains on Veeam v9
2. If not then I guess the best thing to do is split my large daily local jobs into smaller jobs so that the backup copy job can process them in smaller chunks and possibly split up the backup copy job as well for the same reason, ie ie can do the 'merge' in smaller chunks.
3. The only other thing I can think would be causing my issues is that in the backup copy job I have selected the VMs to backup from type 'Backup job'. If I selected 'from infrastructure' would it change either of the problems above while NOT using 'per VM' backup chains.
4. Would the answer to question 3 change if we used 'per VM' backup chains because the backup file would be available to the backup copy job, or would it still wait for the whole job to finish?

Many thanks

Chris

[Shestakov]

Hi Chris,
Your understanding is correct, while backup job is running, backup copy will not be able to copy any restore points of VM which are still in progress.
Neither "per-VM chain" nor "From infrastructure" options can help here since the files will still be locked while the job is running.
The way to go here is to split the backup copy job and distribute them accordingly over a day.
Thanks!

[ChrisGundry]

Hi Shestakov,

So the issue is that JOB1 has 5 VM's in it and creates a single large file which backup copy cannot access because it is in use.

Why would per-vm chain not help at all? If JOB1 has 5 VM's, then per-vm would create 5 backup files yes? The backup job will finish processing VM1, then go onto VM2 then VM3 etc. I thought that as it goes through the JOB1 it will create a file for VM1 and lock it then when it is finished with VM1 it will move onto VM2 and lock it, releasing VM1 backup file? Or are you saying that VM1 backup file is locked the whole time JOB1 is running, even if it has processed VM1 and is now processing VM3?
In the same way I was hoping that once the backup file for VM1 is created the copy job can process it, before the other 4 VM's in the job can finish. But your saying this wont work?

Thanks

[Shestakov]

It will not help because all files are locked until the job is completely finished.
Thanks!

[ChrisGundry]

Damn, that sucks. What is the reason for them being locked until the job is finished? If the files were only locked as they were used it would mean they could be processed separately and would be a lot more efficient for the working of backup copy jobs.

As it stands I am going to have to separate a daily backup job with 50 VM's into several chunks and then separate a single backup copy job with 50 VM's into several chunks.

Unfortunately individual backups chains wont really benefit us at all from what I am reading which is a real shame as I was hoping it was going to solve these two problems.

[skrause]

The backup copy job probably won't need to be separated out to get you around the file lock issue, just the source backup job(s).

[Shestakov]

Steven is right. That`s another option.
Chris, how long a backup job run takes in average?
Thanks!

[ChrisGundry]

Sorry guys I didn't get a notification that you had posted a reply!

I have separated out the two BCJ's and this has helped already because they can now run side by side onto two different arrays and the write speed and transform speed is doubled compared to how it was previously. They are also easier to manage because they are named and scoped around the type of machines in the job, ie one is SQL servers and associated with the source SQL job and the other is other member servers and associated with the two source jobs that are for the other member servers.

I am not convinced that the BCJ was starting until both the source jobs were complete, unlike you say I think it was waiting for both to finish before copying. Previously the BCJ was finishing at 5-7pm the next day, they are now finishing on average about 1200 noon which is much better.

Since I posted this I have built a new Veeam server on our DR site and new proxies on our source site with more resources. Now I have separated these BCJ's the bottleneck on them is now the throttling we have set on the WAN traffic. And the source job bottleneck is the target performance on the storage box (which we are reviewing with our storage vendor to see what we can do about that). So this is better than it was.

I am just disappointed that with per-vm backup chains you are still saying that the backup files for all chains would be locked until the whole job is finished. That seems like a big oversight because otherwise the BCJ could start as soon as the first restore point is created.

[foggy]

I am just disappointed that with per-vm backup chains you are still saying that the backup files for all chains would be locked until the whole job is finished. That seems like a big oversight because otherwise the BCJ could start as soon as the first restore point is created.

Let's take it as a room for improvement.

[Shestakov]

Foggy is right, it`s a brand new feature. Now we are convinced it works stable and there is a good requests for the improvement.
The logic of scheduling backup and dependent backup copy is pretty straightforward. If you need to keep the backup+backup copy jobs within the day, but it takes too long for now, decrease number of VMs in each job.
Thanks!

[Gostev]

That seems like a big oversight because otherwise the BCJ could start as soon as the first restore point is created.

It's not an oversight so to speak, just a legacy from the original job-based architecture that was hard to change quickly. It certainly was on our radar well before v9 was released, and we're definitely planning to address that down the road.

[ChrisGundry]

Ok thanks guys. Hopefully in a future release this can be changed to not lock the per-vm chains unless that VM is still being processed? Then we wouldn't have to create lots of small BCJ's which get messy to manage.
Thanks

[foggy]

As Anton said, we're looking for improvements in this area, however currently cannot make any estimations on the exact release they will be implemented in.