Backup Copy job performance

[zak2011]

Between both sites , we have a 1Gbps fiber link. When i do a normal file copy between the backupserver and the Data domain, i get a speed of 90MB/s which is good.
However once a backup copy job starts, the processing rate is 15 to 20 MB/s even if only one backup copy job is running.
I have disabled inline deduplication for the backup copy job and used Extreme compression and get a throughput of 30-MB/s.
Is there anything esle that can be done to make the processing rate better? I suspected network issues in the beginning, but doesnt seem to be the case now.

Thanks

[veremin]

Is it direct or WAN accelerated backup copy job? And what are the bottleneck statistics for this job? (target/source WAN Accelerator, I assume). Thanks.

[zak2011]

It is a direct backup copy job. Bottleneck statistics sometimes shows target and sometimes shows source.

Thanks

[foggy]

Please note that backup copy not just copies files but rather synthetically creates restore points in remote location from the changed blocks extracted from the source storage. Also add compression overhead here, so it is not valid to compare its performance with the simple file copy performance and also this is why storage (either source or target) is a typical bottleneck for this kind of job.

[zak2011]

So you mean to say nothing can be done to improve the performance of the backup copy?

[foggy]

Backup copy job performance should not be that critical since its activity is performed outside the backup window. Anyway, if you still want to research whether there is a place for improvement (aside from storage upgrade), you could open a case and review logs with our technical support, probably they will be able to verify what is the most time consuming operation in the entire process and give some recommendations based on that.

[chrisdearden]

I dont think thats what foggy ment , just that there is more to a backup job than simply moving a vbk file.

Think of it as a new backup job that uses the original backup as a source rather than your source VM's. If your primary target storage isn't very quick , then you will extract data slower than you would if you were backing up directly from your source VM's. Can you run a backup job directly to your DD and compare performance?

[zak2011]

@foggy- totally understand your point. Have opened a case for this.
But the thing is I have been struggling for almost more than a week to get the backup copy jobs to the DR site. So what I have been doing is disabling all the backup copy jobs and running a single backup copy job only to find it is able to reach 5-5-15MB/s on a 1Gbps fiber line with inline deduplication and No compression in the backup copy job. We have done almost everything to isolate any possibilities of a network bottleneck by connecting the primary backup storage directly to core swtiches and also teaming the network interface of the Data Domain to provide full throughput. Plus we have mounted the Data Domain as a Linux datastore. After disabling inline deduplication and selection Extreme compression , i get between 25- 30 MB/s running only a single backup copy job.

@ chris- Thanks for the clarification. Tried running the backup job ( Exchange vms) directly to the DD and performance was very good. It touched 156MB/s over the same link.

Thanks!

[chrisdearden]

ok so we know that the link is good and the DD can ingest data directly. try setting the backup copy job to use Dedupe friendly compression

What is your primary backup storage like ?

[zak2011]

The primary backup storage is an HP Proliant DL180 G6 with an Intel Xeon 2.46Ghz ,8GB RAM , 16TB hard disk running 64 bit windows server 2008 R2 with two 1Gb nics.
It is also configured as a backup proxy with Direct SAN access.

[zak2011]

Is there some setting that I am missing for the backup copy job or perhaps something I may need to fine tune to get a better performance?

Thanks

[chrisdearden]

When the copy job is running , is it the only job?

[zak2011]

Well, not necessarily. Sometimes a backup job might also be running.
However most of the tests were done only when a single backup copy job was running.

Thanks!

[zak2011]

After applying the patch1 for Veeam7 there seems to be quite a performance difference. Prior to applying the patch when a single backup copy job was running the processing rate showed only 15-20MB/s
Now when I ran a single backup copy job the processing rate shows as 85-93 MB/s.
Is this because of the changes in patch1.

Thanks!

[Gostev]

No, Patch #1 cannot take credit for that speed increase... something else has probably changed along.

[zak2011]

Surprising. I had addressed this performance problem about backup copy jobs with support and I was asked to install the patch1 as it implements massive improvements in backup copy statistics and reporting possibly as my previous results were not accurate enough.
Also in backup copy jobs, now bottleneck for the jobs shows up as proxy. Does any proxy come into action for a backup copy job?

[Gostev]

True, statistics and reporting were corrected in Patch #1, but no actual speed increase or job duration changes should be observed. I thought you were talking about the latter, but I see now you are only talking about actual counters, and not necessarily faster job completion.

[zak2011]

In my case, a single backup copy job showed processing rate of 15 to 20 MB/s over a 1 Gbps fiber link before the patch was installed. I addressed this problem to support and they advised me to install patch1 if it would make any difference.
After applying patch1 , a single backup copy job shows processing rate of 85 to 90MB/s and job seems to be a bit faster. Our network engineer says also nothing has been done from his side..so dont really know why this happened.
So does that mean the processing rate I am seeing is not accurate?

[veremin]

I'm wondering whether there is significant difference between time backup copy job takes nowadays and the time it used to take or both are approximately equal/differ slightly. The certain discrepancy might show due to the difference between amount of changes that backup copy transfers nowadays and amount of changes it has transferred previously, etc.

Thanks.

[zak2011]

I think I need to observe it more closely again. Let me check it again and confirm to you.

Thanks

[veremin]

As mentioned above, the reported speed might have increased since we’ve changed the way certain counters are calculated. Though, this has nothing to with actual speed improvement. So, what you’re seeing is most likely caused by some side infrastructure changes.

Thanks.

[zak2011]

Okay. So I shouldnt be having any high hopes on performance improvements in backup copy job , just that the statistics or the processing rate may show different numbers now.

[foggy]

Could you please tell whether the time required to process data has changed? You can look up for the corresponding record in the job statistics window. Having both amount of data read and the time required to process it, you can compare the job performance prior and after installing patch and see which numbers are more accurate.

Btw, is this backup copy job WAN accelerated?

[veremin]

Anyway, the point is that backup copy mechanism hasn't undergone any changes in Patch#1 that might result in decreased backup time. Providing that there is a noticeable difference, it must be related to some environmental changes.

Thanks.

[foggy]

I've also split the topic since we're already deep into different issues than discussed in the original thread.

[zak2011]

Yes, Alex. The time required to read the disk has changed. Previously in the job statistics when i used to see the hard disk read at 15 or 20 MB/s..now it shows 90MB/s.
The backup copy job is not WAN accelerated.

[zak2011]

That could be the reason,Vladimir and Alex. It could be some environmental changes then. I wish i knew what it was though.
Apart from this..if backup copy job does not use proxy, why does it say proxy as a bottleneck. Shouldnt it be either source or target?

[Gostev]

Network = Whatever connection there is between source and target data movers (running on source and target backup repositories).

[zak2011]

Did you mean network is the bottleneck? Or did I understand incorrectly.

Thanks

[foggy]

Apart from this..if backup copy job does not use proxy, why does it say proxy as a bottleneck. Shouldnt it be either source or target?

"Proxy" in case of direct backup copy job is Veeam agent installed on the source backup repository (or the backup proxy in the source site in case of a CIFS repo) and performing data compression prior to transfer.