-
- Enthusiast
- Posts: 27
- Liked: never
- Joined: Sep 04, 2012 9:26 am
- Full Name: Matt
Trying to increase perfomance
Hi All,
I have spent a lot of time reading different forums/posts and i've tried all different settings to get better performance in my backups.
I'm a little confused with CBT now though, I have enabled virtual appliance as the transport mode so should I be disabling CBT in the backup job?
I've run a job with the virtual appliance as the transport mode and in the backup job status it says
02/01/2014 16:50:41 :: Using backup proxy 10.10.12.247 for disk Hard disk 1 [hotadd]
02/01/2014 16:50:43 :: Hard disk 1 (120.0 GB) read at 21 MB/s [CBT]
Does this mean it isn't using hotadd? Is this why the performance isn't great?
The read speed of the VM seems to go anywhere between 8MB/s to 100MB/s - should it not be a little more steady?
Cheers
I have spent a lot of time reading different forums/posts and i've tried all different settings to get better performance in my backups.
I'm a little confused with CBT now though, I have enabled virtual appliance as the transport mode so should I be disabling CBT in the backup job?
I've run a job with the virtual appliance as the transport mode and in the backup job status it says
02/01/2014 16:50:41 :: Using backup proxy 10.10.12.247 for disk Hard disk 1 [hotadd]
02/01/2014 16:50:43 :: Hard disk 1 (120.0 GB) read at 21 MB/s [CBT]
Does this mean it isn't using hotadd? Is this why the performance isn't great?
The read speed of the VM seems to go anywhere between 8MB/s to 100MB/s - should it not be a little more steady?
Cheers
-
- Chief Product Officer
- Posts: 31800
- Liked: 7298 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Trying to increase perfomance
Hi Matt,
I don't follow why would you want to disable CBT when using hot add (or any other processing mode for that matter).
According to the log you posted, the job is using hot add to backup this particular hard disk.
The read speed will depend on change blocks placement: the more random they are, the slower the changed blocks will be read from source storage. Once the job will hit long sequential segment, the read speed will go up significantly for the duration of that segment. This is basically how hard drive based storage works, it does not "like" random I/O. For example, you can easily reach 100+ MB/s of sequential read speed even on the slowest notebook hard drive available these days, but random I/O would totally kill the performance, and you will be lucky to get a few MB/s.
Thanks.
I don't follow why would you want to disable CBT when using hot add (or any other processing mode for that matter).
According to the log you posted, the job is using hot add to backup this particular hard disk.
The read speed will depend on change blocks placement: the more random they are, the slower the changed blocks will be read from source storage. Once the job will hit long sequential segment, the read speed will go up significantly for the duration of that segment. This is basically how hard drive based storage works, it does not "like" random I/O. For example, you can easily reach 100+ MB/s of sequential read speed even on the slowest notebook hard drive available these days, but random I/O would totally kill the performance, and you will be lucky to get a few MB/s.
Thanks.
-
- Enthusiast
- Posts: 27
- Liked: never
- Joined: Sep 04, 2012 9:26 am
- Full Name: Matt
Re: Trying to increase perfomance
That makes sense, thank you
I read somewhere on a different site that its best to disable CBT when using virtual appliance but I guess they were wrong on that one
I read somewhere on a different site that its best to disable CBT when using virtual appliance but I guess they were wrong on that one
-
- Enthusiast
- Posts: 27
- Liked: never
- Joined: Sep 04, 2012 9:26 am
- Full Name: Matt
Re: Trying to increase perfomance
Hi,
I'm still having a performance issue with our backup copy job though. I currently have it set up as follows:
Site 1 has the veeam server
site 2 has the proxy
A backup copy job is copying the backups from site 1 to site 2 using the site 2 proxy set to virtual appliance transport mode
The read speeds I'm getting are around 3MB/s - and it has taken 14hrs to get to 6%
The bottleneck shows Network: 99%
Site 1 has a dedicated 100mb fiber and site 2 has a dedicated 100mb fiber.
There are no restrictions between the two sites
Is this expected performance?
Cheers
I'm still having a performance issue with our backup copy job though. I currently have it set up as follows:
Site 1 has the veeam server
site 2 has the proxy
A backup copy job is copying the backups from site 1 to site 2 using the site 2 proxy set to virtual appliance transport mode
The read speeds I'm getting are around 3MB/s - and it has taken 14hrs to get to 6%
The bottleneck shows Network: 99%
Site 1 has a dedicated 100mb fiber and site 2 has a dedicated 100mb fiber.
There are no restrictions between the two sites
Is this expected performance?
Cheers
-
- Veeam Software
- Posts: 21138
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Trying to increase perfomance
Yes, completely. Unless they were talking about backing up proxy server itself, where CBT is disabled for this VM automatically.matt19849 wrote:I read somewhere on a different site that its best to disable CBT when using virtual appliance but I guess they were wrong on that one
Actually, proxy backup mode does not affect backup copy job in any way, since data is transferred directly between agents installed on the source and target backup repositories (in direct backup copy job mode) and between WAN accelerators (in WAN accelerated job mode).matt19849 wrote:A backup copy job is copying the backups from site 1 to site 2 using the site 2 proxy set to virtual appliance transport mode
What kind of repositories do you have?
-
- Enthusiast
- Posts: 27
- Liked: never
- Joined: Sep 04, 2012 9:26 am
- Full Name: Matt
Re: Trying to increase perfomance
The repositories are all network shares (Qnap)
i.e.
\\qnap_ip\Backup
i.e.
\\qnap_ip\Backup
-
- Veeam Software
- Posts: 21138
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Trying to increase perfomance
Are you using direct backup copy job mode? Have you specified your site 2 proxy server as a proxying server in repository wizard? Veeam agent cannot be deployed directly on repositories of this type.
-
- Enthusiast
- Posts: 27
- Liked: never
- Joined: Sep 04, 2012 9:26 am
- Full Name: Matt
Re: Trying to increase perfomance
Hi,
It is using direct backup copy mode, it is using the site 2 proxy which is configured on that backup repository
It is using direct backup copy mode, it is using the site 2 proxy which is configured on that backup repository
-
- Veeam Software
- Posts: 21138
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Trying to increase perfomance
What about fiber usage? Is it fully saturated during transfer?
Probably it is worth trying WAN accelerated mode, which was specifically designed for limited bandwidth (100 Mbit or less).
Also, have you considered seeding you backup copy job to avoid initial copy of the full backup files?
Probably it is worth trying WAN accelerated mode, which was specifically designed for limited bandwidth (100 Mbit or less).
Also, have you considered seeding you backup copy job to avoid initial copy of the full backup files?
-
- Enthusiast
- Posts: 27
- Liked: never
- Joined: Sep 04, 2012 9:26 am
- Full Name: Matt
Re: Trying to increase perfomance
HI,
The fiber usage is next to none, it isn't very heavily used at all.
I only have the standard version of Veeam so not sure I can use the WAN accelerator
I haven't tried seeding the backup, could you point me to more information on this?
The fiber usage is next to none, it isn't very heavily used at all.
I only have the standard version of Veeam so not sure I can use the WAN accelerator
I haven't tried seeding the backup, could you point me to more information on this?
-
- Veeam Software
- Posts: 21138
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Trying to increase perfomance
Are you positive there're no throttling rules configured for the pair of source and target agents?matt19849 wrote:The fiber usage is next to none, it isn't very heavily used at all.
Sure, here are a couple of links:matt19849 wrote:I haven't tried seeding the backup, could you point me to more information on this?
Seeding v7 backup copy
Seeding offsite backup files for a Backup Copy Job
-
- Enthusiast
- Posts: 27
- Liked: never
- Joined: Sep 04, 2012 9:26 am
- Full Name: Matt
Re: Trying to increase perfomance
There isn't any throttling on our firewalls, I can check with the ISP still
I'll take a ready of the links - thanks
I'll take a ready of the links - thanks
-
- Chief Product Officer
- Posts: 31800
- Liked: 7298 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Trying to increase perfomance
VM data during Backup Copy in direct mode goes direct from source to target repository, with no backup proxies involved. The transport mode is irrelevant, because VM data is obtained from the source backup repository, not from virtual infrastructure.matt19849 wrote:Site 1 has the veeam server
site 2 has the proxy
A backup copy job is copying the backups from site 1 to site 2 using the site 2 proxy set to virtual appliance transport mode
The read speeds I'm getting are around 3MB/s - and it has taken 14hrs to get to 6%
The bottleneck shows Network: 99%
The Source bottleneck means that the issue is the speed of obtaining data from the source backup repository, this is the issue that needs to be investigated. Firewall settings, WAN acceleration, or even a bigger WAN pipe will not help here.
-
- Enthusiast
- Posts: 27
- Liked: never
- Joined: Sep 04, 2012 9:26 am
- Full Name: Matt
Re: Trying to increase perfomance
I will take a look at the source, the bottle neck in Veeam says network 99%
Could that be the network connection to the source?
Could that be the network connection to the source?
-
- Enthusiast
- Posts: 27
- Liked: never
- Joined: Sep 04, 2012 9:26 am
- Full Name: Matt
Re: Trying to increase perfomance
I've checked the source and it seems a little odd. When the backup job is preparing to read the data i'm getting around 50MB/s on the nic card at the source. Once Veeam starts to read the drive that drops down to 3MB/s on the nic card at the source...
-
- Chief Product Officer
- Posts: 31800
- Liked: 7298 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Trying to increase perfomance
Hi Matt, I am very sorry. Somehow I thought the bottleneck is Source 99%, even though I quoted it. My only excuse is that I am on holiday right now please disregard my comment above, and sorry for sending you in the wrong direction.
Network 99% obviously means that the problem is the speed of connection between source and target data movers.
I suggest simple test of copying files from source to target proxying server, and if it shows much better throughput than the job, then it would be best to open a support case.
Thanks!
Network 99% obviously means that the problem is the speed of connection between source and target data movers.
I suggest simple test of copying files from source to target proxying server, and if it shows much better throughput than the job, then it would be best to open a support case.
This only proves that there are no issues with the speed of obtaining data from the source backup repository. The speed drop you are observing happens because another component of the data processing conveyor becomes a bottleneck, specifically your network.matt19849 wrote:I've checked the source and it seems a little odd. When the backup job is preparing to read the data i'm getting around 50MB/s on the nic card at the source. Once Veeam starts to read the drive that drops down to 3MB/s on the nic card at the source...
Thanks!
-
- Veeam ProPartner
- Posts: 252
- Liked: 26 times
- Joined: Apr 05, 2011 11:44 pm
- Contact:
Re: Trying to increase perfomance
We use QNAPs quite a bit. Sequential writes/reads to and from QNAPs are usually fine. Although we did notice that LAG configured on QNAPs sometimes results in lower overall transfer rates vs single 1Gb link.
The second issue we've seen with this combo is that the reverse incremental speeds are terrible with QNAP. Make sure you are not writing any backups to the QNAP when backup copy job is running. Although Veeam may show network as bottleneck, we've seen and posted here about the mechanism being mistaken about bottlenecks.
The second issue we've seen with this combo is that the reverse incremental speeds are terrible with QNAP. Make sure you are not writing any backups to the QNAP when backup copy job is running. Although Veeam may show network as bottleneck, we've seen and posted here about the mechanism being mistaken about bottlenecks.
-
- Enthusiast
- Posts: 27
- Liked: never
- Joined: Sep 04, 2012 9:26 am
- Full Name: Matt
Re: Trying to increase perfomance
Hi Yuki
Do you use a single network port on your QNAPs or do you use port trunking?
Nothing else is being written to the QNAPs while the backups are running. We have one QNAP for our backup and another QNAP for the backup copy. We are using reverse incremental on the QNAPS too
Do you use a single network port on your QNAPs or do you use port trunking?
Nothing else is being written to the QNAPs while the backups are running. We have one QNAP for our backup and another QNAP for the backup copy. We are using reverse incremental on the QNAPS too
-
- Novice
- Posts: 3
- Liked: never
- Joined: Feb 04, 2010 11:51 pm
- Contact:
Re: Trying to increase perfomance
I believe Yuki said they were using LAG, so multiple connections. Port trunking usually refers to multiple vlans on a single logical connection. So one could use LAG, multiple connections, and set it as a trunk so it could have multiple vlans on that single logical connection. Whether the qnap would be able utilize that type of connection or if it would be of use would be another matter. Seems there are very few LAG type implementations that actually increase the thru put, how ever they have some use as redundant connections, assuming spanning tree is setup correctly. I think one needs to really understand what type of traffic would be utilizing the LAG connection to correctly set it up. For instance, if it is from multiple devices with different mac addresses or maybe different ip addresses or even is initiated on a different physical port, etc.
-
- Enthusiast
- Posts: 27
- Liked: never
- Joined: Sep 04, 2012 9:26 am
- Full Name: Matt
Re: Trying to increase perfomance
Hi,
I have set our QNAPs to use balance-alb which according to QNAP:
Balance-alb is similar to balance-tlb but also attempts to redistribute incoming (receive load balancing) for IPV4 traffic. The receive load balancing is achieved by ARP negotiation sent by the local system on their way out and overwrites the source hardware address with the unique hardware address of one of the Ethernet interfaces in the bond such that different peers use different hardware address for the server. This mode provides load balancing and fault tolerance.
I haven't noticed any performance increase just peace of mind I have a bit of failover
I have set our QNAPs to use balance-alb which according to QNAP:
Balance-alb is similar to balance-tlb but also attempts to redistribute incoming (receive load balancing) for IPV4 traffic. The receive load balancing is achieved by ARP negotiation sent by the local system on their way out and overwrites the source hardware address with the unique hardware address of one of the Ethernet interfaces in the bond such that different peers use different hardware address for the server. This mode provides load balancing and fault tolerance.
I haven't noticed any performance increase just peace of mind I have a bit of failover
-
- Product Manager
- Posts: 20397
- Liked: 2298 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Trying to increase perfomance
Hi, Matt, have you already tried to conduct a series of tests in order to see what performance your network connection is capable of? This should shed a light on existing problem and give some hints about further investigation.
Thanks.
Thanks.
-
- Influencer
- Posts: 11
- Liked: never
- Joined: Apr 19, 2011 10:05 am
- Full Name: Tor c
- Contact:
Re: Trying to increase perfomance
As yuki mentioned, some of the QNAP's have terrible performances especially when using with deduplication and/or transforming. I tried to tune a QNAP and Veeam to perform properly a year ago believeing it was a network bottleneck, but with no success. Iops is the clue when it comes to dedup and transforming.
http://www.virtualtothecore.com/en/veea ... torage-io/
This article probably makes things clearer. We reverted to use a equallogic storage solution.
Regards
Snakes
http://www.virtualtothecore.com/en/veea ... torage-io/
This article probably makes things clearer. We reverted to use a equallogic storage solution.
Regards
Snakes
-
- Product Manager
- Posts: 20397
- Liked: 2298 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Trying to increase perfomance
Even though the statistics suggest that the problem lies in network itself, I'm wondering whether, is there any other device, apart from QNAP, in target location that you can choose as a target repository. This should give us clues whom to blame in this situation - network or target device.
Thanks.
Thanks.
-
- Enthusiast
- Posts: 27
- Liked: never
- Joined: Sep 04, 2012 9:26 am
- Full Name: Matt
Re: Trying to increase perfomance
I can certainly try and back it up else where to see how that performs. I'll give that a go today
-
- Product Manager
- Posts: 20397
- Liked: 2298 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Trying to increase perfomance
And don't forget to test the connectivity itself by copying files from source to target proxying servers. Looking forward to hearing your results. Thanks.
-
- VeeaMVP
- Posts: 6165
- Liked: 1971 times
- Joined: Jul 26, 2009 3:39 pm
- Full Name: Luca Dell'Oca
- Location: Varese, Italy
- Contact:
Re: Trying to increase perfomance
Just a quick Thanks to Snakes for quoting my blog post, I'm glad is still useful after more than a year I wrote it.
Indeed QNAP can be very hard to use on some backup methods; all its functions, even raid, are managed by the central CPU, since it does not have dedicated raid controllers. Basically it's a linux server that uses LVM and dmraid to aggregate the disks, I've seen it live in a large model with VGA output, it can be clearly seen during boot time
Luca.
Indeed QNAP can be very hard to use on some backup methods; all its functions, even raid, are managed by the central CPU, since it does not have dedicated raid controllers. Basically it's a linux server that uses LVM and dmraid to aggregate the disks, I've seen it live in a large model with VGA output, it can be clearly seen during boot time
Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
-
- Enthusiast
- Posts: 27
- Liked: never
- Joined: Sep 04, 2012 9:26 am
- Full Name: Matt
Re: Trying to increase perfomance
I have just tried backup to up a new repository that is an NFS share from a server, this share is on a P2000. I'm getting the same 3-5MB/s speeds again.
I'll start to look at the network now
I'll start to look at the network now
Who is online
Users browsing this forum: Baidu [Spider], Google [Bot], lando_uk and 169 guests