Backup Copy Job Performance

[mmonroe]

I finally decided to post something to the forum and get some feedback on things..

Backup Copy Job - I have about 30vm's in the job. I have it set to save for 60 days. The first job is the full backup which is about 1.5tb. Each subsequent incremental is about 70-90gig for each day.

The repository for the backup copy job is a network drive over a 1gig network link.

The "first" incremental that is made after the full backup takes about 60min give or take and is about 70-90gig in size. Each subsequent incremental takes longer and longer with each one taking a few minutes longer. At the end of 60 days give or take, each incremental is taking about 4 hours. When I manually copy one of the 70-90gig incrementals to the exact same storage location it only takes about 10-15mins.

Remember, the full backup is not touched until the total number of retained days value is reached. Until then you are only adding a new incremental - basically just copying the incremental data to the new repository.

Why is it taking hours to get a 70-90gig incremental copied over when manually copying the same file only takes 10-15min? Why does each new incremental get slower and slower when the full is not being touched?

I am hoping that Version 8 will resolve some of these issues..

MarkM

[Shestakov]

Hello Mark,

Could you give some clarification, please:
Did you mean that each incremental backup takes more time or each run of backup copy job?
What about processing rate of those incremental jobs, does it slow down for each subsequent run? By the way, are we talking about forward incremental backups?

Note that backup copy job has 4 phases, including transform operations. The transform operations can take additional time doing a backup chain transformation and/or removing deleted VMs from restore points and/or compacting a full backup file, according to it`s own retention policy, which can differ from the source Backup job one.
Thank you.

[mmonroe]

Some clarification..

1) On day 1 that the backup copy job is run a 1.5tb full file is created.

2) On day 2, the backup copy job creates an incremental of about 70-90gig. A transform is not done because the retained date of 60 has not been reached.

3) On day 3, the backup copy job creates a new incremental of about 70-90gig. A transform is not done because the retained date of 60 has not been reached.

4) Etc, Etc - Each nightly incremental is 70-90gig - However, on number #2 above the job took about 60mins to complete. Each subsequent run of the backup copy job takes a few minutes longer - and remember - a transform is not done until the 60 day retained date is reach. After 50 days or so the exact same job copying the same 70-90gig will take 3 hours or longer (versus the 60min on day 2).

5) When I simply copy a 70-90gig file from the Veeam server to the exact same repository it takes 10-15min so I know the network performance is good and not a bottleneck.

[chrisdearden]

What does the job show as the bottleneck? remember that backup copy job is not a straightforward copy of the backups so we are not just robocopying the incremental over the link. I'm guessing you are not using wan accelerators?

you say the target is a network drive - do you have any veeam components on the other side of the WAN? if you dont and there is just //somerservername/somesharename over the wan , then we will use a data mover that isn't local , which is not going to work out well.

[awex]

We are also having some issues with the Backup Copy Job speed. We have 25 VM's, full backup is about 1 TB and a daily reverse incremental is about 80-90 GB set to save for 7 days.

We have been using robocopy to get the backup onto USB 3.0 drives that we rotate and send offsite. It takes about 3-4 hours for the reverse incremental to run and about 5 hours for the robocopy to get the backup onto the USB drive.

With the upgrade to Veeam version 8 and the easier support for rotated drives I thought I would give Backup Copy another try, earlier attempts proved unsuccessful. The new features of version 8 make it easier however we find that it takes too long for the back copy job to get the backup onto the USB drive. What takes robocopy about 5 hours takes the backup copy job about 19-20 hours. I know that it is not doing a simple copy like robocopy but thought it wouldn't be that big of a difference. One test that I did do was take out the largest VM (270 GB) and did a backup copy by itself, I found that the time to copy the VM in the backup copy job went from 3:13 in the copy job with the rest of the VM's to 1:28 if the copy job only contained that VM. Is this expected behaviour? Will separating all of the VMs out improve the time to get all of them onto the USB drive (would prefer not to do this)?

Looking at the throughput graph of the job the speed of the job is quite high for the first 3-4 hours but gradually slows down and towards the end of the job is extremely slow. Is this also expected? The job lists the load as follows:

Load: Source 48% > Proxy 22% > Network 57% > Target 73%

Any help on speeding up the copy job would be appreciated, we would prefer to use the backup copy job if we could.

Nigel.