Comprehensive data protection for all workloads
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Backup Copy job performance

Post by zak2011 »

Between both sites , we have a 1Gbps fiber link. When i do a normal file copy between the backupserver and the Data domain, i get a speed of 90MB/s which is good.
However once a backup copy job starts, the processing rate is 15 to 20 MB/s even if only one backup copy job is running.
I have disabled inline deduplication for the backup copy job and used Extreme compression and get a throughput of 30-MB/s.
Is there anything esle that can be done to make the processing rate better? I suspected network issues in the beginning, but doesnt seem to be the case now.

Thanks
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by veremin »

Is it direct or WAN accelerated backup copy job? And what are the bottleneck statistics for this job? (target/source WAN Accelerator, I assume). Thanks.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by zak2011 »

It is a direct backup copy job. Bottleneck statistics sometimes shows target and sometimes shows source.

Thanks
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by foggy »

Please note that backup copy not just copies files but rather synthetically creates restore points in remote location from the changed blocks extracted from the source storage. Also add compression overhead here, so it is not valid to compare its performance with the simple file copy performance and also this is why storage (either source or target) is a typical bottleneck for this kind of job.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by zak2011 »

So you mean to say nothing can be done to improve the performance of the backup copy?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by foggy »

Backup copy job performance should not be that critical since its activity is performed outside the backup window. Anyway, if you still want to research whether there is a place for improvement (aside from storage upgrade), you could open a case and review logs with our technical support, probably they will be able to verify what is the most time consuming operation in the entire process and give some recommendations based on that.
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by chrisdearden »

I dont think thats what foggy ment , just that there is more to a backup job than simply moving a vbk file.

Think of it as a new backup job that uses the original backup as a source rather than your source VM's. If your primary target storage isn't very quick , then you will extract data slower than you would if you were backing up directly from your source VM's. Can you run a backup job directly to your DD and compare performance?
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by zak2011 »

@foggy- totally understand your point. Have opened a case for this.
But the thing is I have been struggling for almost more than a week to get the backup copy jobs to the DR site. So what I have been doing is disabling all the backup copy jobs and running a single backup copy job only to find it is able to reach 5-5-15MB/s on a 1Gbps fiber line with inline deduplication and No compression in the backup copy job. We have done almost everything to isolate any possibilities of a network bottleneck by connecting the primary backup storage directly to core swtiches and also teaming the network interface of the Data Domain to provide full throughput. Plus we have mounted the Data Domain as a Linux datastore. After disabling inline deduplication and selection Extreme compression , i get between 25- 30 MB/s running only a single backup copy job.

@ chris- Thanks for the clarification. Tried running the backup job ( Exchange vms) directly to the DD and performance was very good. It touched 156MB/s over the same link.

Thanks!
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by chrisdearden »

ok so we know that the link is good and the DD can ingest data directly. :) try setting the backup copy job to use Dedupe friendly compression

What is your primary backup storage like ?
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by zak2011 »

The primary backup storage is an HP Proliant DL180 G6 with an Intel Xeon 2.46Ghz ,8GB RAM , 16TB hard disk running 64 bit windows server 2008 R2 with two 1Gb nics.
It is also configured as a backup proxy with Direct SAN access.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by zak2011 »

Is there some setting that I am missing for the backup copy job or perhaps something I may need to fine tune to get a better performance?

Thanks
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by chrisdearden »

When the copy job is running , is it the only job?
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by zak2011 »

Well, not necessarily. Sometimes a backup job might also be running.
However most of the tests were done only when a single backup copy job was running.

Thanks!
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by zak2011 »

After applying the patch1 for Veeam7 there seems to be quite a performance difference. Prior to applying the patch when a single backup copy job was running the processing rate showed only 15-20MB/s
Now when I ran a single backup copy job the processing rate shows as 85-93 MB/s.
Is this because of the changes in patch1.

Thanks!
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by Gostev »

No, Patch #1 cannot take credit for that speed increase... something else has probably changed along.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by zak2011 »

Surprising. I had addressed this performance problem about backup copy jobs with support and I was asked to install the patch1 as it implements massive improvements in backup copy statistics and reporting possibly as my previous results were not accurate enough.
Also in backup copy jobs, now bottleneck for the jobs shows up as proxy. Does any proxy come into action for a backup copy job?
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by Gostev »

True, statistics and reporting were corrected in Patch #1, but no actual speed increase or job duration changes should be observed. I thought you were talking about the latter, but I see now you are only talking about actual counters, and not necessarily faster job completion.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by zak2011 »

In my case, a single backup copy job showed processing rate of 15 to 20 MB/s over a 1 Gbps fiber link before the patch was installed. I addressed this problem to support and they advised me to install patch1 if it would make any difference.
After applying patch1 , a single backup copy job shows processing rate of 85 to 90MB/s and job seems to be a bit faster. Our network engineer says also nothing has been done from his side..so dont really know why this happened.
So does that mean the processing rate I am seeing is not accurate? :?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by veremin »

I'm wondering whether there is significant difference between time backup copy job takes nowadays and the time it used to take or both are approximately equal/differ slightly. The certain discrepancy might show due to the difference between amount of changes that backup copy transfers nowadays and amount of changes it has transferred previously, etc.

Thanks.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by zak2011 »

I think I need to observe it more closely again. Let me check it again and confirm to you.

Thanks
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by veremin »

As mentioned above, the reported speed might have increased since we’ve changed the way certain counters are calculated. Though, this has nothing to with actual speed improvement. So, what you’re seeing is most likely caused by some side infrastructure changes.

Thanks.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by zak2011 »

Okay. So I shouldnt be having any high hopes on performance improvements in backup copy job , just that the statistics or the processing rate may show different numbers now.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by foggy »

Could you please tell whether the time required to process data has changed? You can look up for the corresponding record in the job statistics window. Having both amount of data read and the time required to process it, you can compare the job performance prior and after installing patch and see which numbers are more accurate.

Btw, is this backup copy job WAN accelerated?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by veremin »

Anyway, the point is that backup copy mechanism hasn't undergone any changes in Patch#1 that might result in decreased backup time. Providing that there is a noticeable difference, it must be related to some environmental changes.

Thanks.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Copy job performance

Post by foggy »

I've also split the topic since we're already deep into different issues than discussed in the original thread.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Veeam, DataDomain and Linux NFS share

Post by zak2011 »

Yes, Alex. The time required to read the disk has changed. Previously in the job statistics when i used to see the hard disk read at 15 or 20 MB/s..now it shows 90MB/s.
The backup copy job is not WAN accelerated.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Backup Copy job performance

Post by zak2011 »

That could be the reason,Vladimir and Alex. It could be some environmental changes then. I wish i knew what it was though.
Apart from this..if backup copy job does not use proxy, why does it say proxy as a bottleneck. Shouldnt it be either source or target?
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup Copy job performance

Post by Gostev »

Network = Whatever connection there is between source and target data movers (running on source and target backup repositories).
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Backup Copy job performance

Post by zak2011 »

Did you mean network is the bottleneck? Or did I understand incorrectly.

Thanks
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Copy job performance

Post by foggy »

zak2011 wrote:Apart from this..if backup copy job does not use proxy, why does it say proxy as a bottleneck. Shouldnt it be either source or target?
"Proxy" in case of direct backup copy job is Veeam agent installed on the source backup repository (or the backup proxy in the source site in case of a CIFS repo) and performing data compression prior to transfer.
Post Reply

Who is online

Users browsing this forum: No registered users and 161 guests