Comprehensive data protection for all workloads
Post Reply
matt19849
Enthusiast
Posts: 27
Liked: never
Joined: Sep 04, 2012 9:26 am
Full Name: Matt

Trying to increase perfomance

Post by matt19849 »

Hi All,

I have spent a lot of time reading different forums/posts and i've tried all different settings to get better performance in my backups.
I'm a little confused with CBT now though, I have enabled virtual appliance as the transport mode so should I be disabling CBT in the backup job?

I've run a job with the virtual appliance as the transport mode and in the backup job status it says

02/01/2014 16:50:41 :: Using backup proxy 10.10.12.247 for disk Hard disk 1 [hotadd]
02/01/2014 16:50:43 :: Hard disk 1 (120.0 GB) read at 21 MB/s [CBT]

Does this mean it isn't using hotadd? Is this why the performance isn't great?
The read speed of the VM seems to go anywhere between 8MB/s to 100MB/s - should it not be a little more steady?

Cheers

Gostev
SVP, Product Management
Posts: 27413
Liked: 4544 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Trying to increase perfomance

Post by Gostev »

Hi Matt,

I don't follow why would you want to disable CBT when using hot add (or any other processing mode for that matter).

According to the log you posted, the job is using hot add to backup this particular hard disk.

The read speed will depend on change blocks placement: the more random they are, the slower the changed blocks will be read from source storage. Once the job will hit long sequential segment, the read speed will go up significantly for the duration of that segment. This is basically how hard drive based storage works, it does not "like" random I/O. For example, you can easily reach 100+ MB/s of sequential read speed even on the slowest notebook hard drive available these days, but random I/O would totally kill the performance, and you will be lucky to get a few MB/s.

Thanks.

matt19849
Enthusiast
Posts: 27
Liked: never
Joined: Sep 04, 2012 9:26 am
Full Name: Matt

Re: Trying to increase perfomance

Post by matt19849 »

That makes sense, thank you

I read somewhere on a different site that its best to disable CBT when using virtual appliance but I guess they were wrong on that one :)

matt19849
Enthusiast
Posts: 27
Liked: never
Joined: Sep 04, 2012 9:26 am
Full Name: Matt

Re: Trying to increase perfomance

Post by matt19849 »

Hi,

I'm still having a performance issue with our backup copy job though. I currently have it set up as follows:

Site 1 has the veeam server
site 2 has the proxy
A backup copy job is copying the backups from site 1 to site 2 using the site 2 proxy set to virtual appliance transport mode

The read speeds I'm getting are around 3MB/s - and it has taken 14hrs to get to 6%
The bottleneck shows Network: 99%

Site 1 has a dedicated 100mb fiber and site 2 has a dedicated 100mb fiber.
There are no restrictions between the two sites

Is this expected performance?

Cheers

foggy
Veeam Software
Posts: 19761
Liked: 1823 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Trying to increase perfomance

Post by foggy »

matt19849 wrote:I read somewhere on a different site that its best to disable CBT when using virtual appliance but I guess they were wrong on that one :)
Yes, completely. Unless they were talking about backing up proxy server itself, where CBT is disabled for this VM automatically.
matt19849 wrote:A backup copy job is copying the backups from site 1 to site 2 using the site 2 proxy set to virtual appliance transport mode
Actually, proxy backup mode does not affect backup copy job in any way, since data is transferred directly between agents installed on the source and target backup repositories (in direct backup copy job mode) and between WAN accelerators (in WAN accelerated job mode).

What kind of repositories do you have?

matt19849
Enthusiast
Posts: 27
Liked: never
Joined: Sep 04, 2012 9:26 am
Full Name: Matt

Re: Trying to increase perfomance

Post by matt19849 »

The repositories are all network shares (Qnap)

i.e.
\\qnap_ip\Backup

foggy
Veeam Software
Posts: 19761
Liked: 1823 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Trying to increase perfomance

Post by foggy »

Are you using direct backup copy job mode? Have you specified your site 2 proxy server as a proxying server in repository wizard? Veeam agent cannot be deployed directly on repositories of this type.

matt19849
Enthusiast
Posts: 27
Liked: never
Joined: Sep 04, 2012 9:26 am
Full Name: Matt

Re: Trying to increase perfomance

Post by matt19849 »

Hi,

It is using direct backup copy mode, it is using the site 2 proxy which is configured on that backup repository

foggy
Veeam Software
Posts: 19761
Liked: 1823 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Trying to increase perfomance

Post by foggy »

What about fiber usage? Is it fully saturated during transfer?

Probably it is worth trying WAN accelerated mode, which was specifically designed for limited bandwidth (100 Mbit or less).

Also, have you considered seeding you backup copy job to avoid initial copy of the full backup files?

matt19849
Enthusiast
Posts: 27
Liked: never
Joined: Sep 04, 2012 9:26 am
Full Name: Matt

Re: Trying to increase perfomance

Post by matt19849 »

HI,

The fiber usage is next to none, it isn't very heavily used at all.
I only have the standard version of Veeam so not sure I can use the WAN accelerator

I haven't tried seeding the backup, could you point me to more information on this?

foggy
Veeam Software
Posts: 19761
Liked: 1823 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Trying to increase perfomance

Post by foggy »

matt19849 wrote:The fiber usage is next to none, it isn't very heavily used at all.
Are you positive there're no throttling rules configured for the pair of source and target agents?
matt19849 wrote:I haven't tried seeding the backup, could you point me to more information on this?
Sure, here are a couple of links:
Seeding v7 backup copy
Seeding offsite backup files for a Backup Copy Job

matt19849
Enthusiast
Posts: 27
Liked: never
Joined: Sep 04, 2012 9:26 am
Full Name: Matt

Re: Trying to increase perfomance

Post by matt19849 »

There isn't any throttling on our firewalls, I can check with the ISP still

I'll take a ready of the links - thanks

Gostev
SVP, Product Management
Posts: 27413
Liked: 4544 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Trying to increase perfomance

Post by Gostev »

matt19849 wrote:Site 1 has the veeam server
site 2 has the proxy
A backup copy job is copying the backups from site 1 to site 2 using the site 2 proxy set to virtual appliance transport mode

The read speeds I'm getting are around 3MB/s - and it has taken 14hrs to get to 6%
The bottleneck shows Network: 99%
VM data during Backup Copy in direct mode goes direct from source to target repository, with no backup proxies involved. The transport mode is irrelevant, because VM data is obtained from the source backup repository, not from virtual infrastructure.

The Source bottleneck means that the issue is the speed of obtaining data from the source backup repository, this is the issue that needs to be investigated. Firewall settings, WAN acceleration, or even a bigger WAN pipe will not help here.

matt19849
Enthusiast
Posts: 27
Liked: never
Joined: Sep 04, 2012 9:26 am
Full Name: Matt

Re: Trying to increase perfomance

Post by matt19849 »

I will take a look at the source, the bottle neck in Veeam says network 99%
Could that be the network connection to the source?

matt19849
Enthusiast
Posts: 27
Liked: never
Joined: Sep 04, 2012 9:26 am
Full Name: Matt

Re: Trying to increase perfomance

Post by matt19849 »

I've checked the source and it seems a little odd. When the backup job is preparing to read the data i'm getting around 50MB/s on the nic card at the source. Once Veeam starts to read the drive that drops down to 3MB/s on the nic card at the source...

Gostev
SVP, Product Management
Posts: 27413
Liked: 4544 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Trying to increase perfomance

Post by Gostev »

Hi Matt, I am very sorry. Somehow I thought the bottleneck is Source 99%, even though I quoted it. My only excuse is that I am on holiday right now :roll: please disregard my comment above, and sorry for sending you in the wrong direction.

Network 99% obviously means that the problem is the speed of connection between source and target data movers.

I suggest simple test of copying files from source to target proxying server, and if it shows much better throughput than the job, then it would be best to open a support case.
matt19849 wrote:I've checked the source and it seems a little odd. When the backup job is preparing to read the data i'm getting around 50MB/s on the nic card at the source. Once Veeam starts to read the drive that drops down to 3MB/s on the nic card at the source...
This only proves that there are no issues with the speed of obtaining data from the source backup repository. The speed drop you are observing happens because another component of the data processing conveyor becomes a bottleneck, specifically your network.

Thanks!

Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Trying to increase perfomance

Post by Yuki »

We use QNAPs quite a bit. Sequential writes/reads to and from QNAPs are usually fine. Although we did notice that LAG configured on QNAPs sometimes results in lower overall transfer rates vs single 1Gb link.

The second issue we've seen with this combo is that the reverse incremental speeds are terrible with QNAP. Make sure you are not writing any backups to the QNAP when backup copy job is running. Although Veeam may show network as bottleneck, we've seen and posted here about the mechanism being mistaken about bottlenecks.

matt19849
Enthusiast
Posts: 27
Liked: never
Joined: Sep 04, 2012 9:26 am
Full Name: Matt

Re: Trying to increase perfomance

Post by matt19849 »

Hi Yuki
Do you use a single network port on your QNAPs or do you use port trunking?

Nothing else is being written to the QNAPs while the backups are running. We have one QNAP for our backup and another QNAP for the backup copy. We are using reverse incremental on the QNAPS too

thornec
Novice
Posts: 3
Liked: never
Joined: Feb 04, 2010 11:51 pm
Contact:

Re: Trying to increase perfomance

Post by thornec »

I believe Yuki said they were using LAG, so multiple connections. Port trunking usually refers to multiple vlans on a single logical connection. So one could use LAG, multiple connections, and set it as a trunk so it could have multiple vlans on that single logical connection. Whether the qnap would be able utilize that type of connection or if it would be of use would be another matter. Seems there are very few LAG type implementations that actually increase the thru put, how ever they have some use as redundant connections, assuming spanning tree is setup correctly. I think one needs to really understand what type of traffic would be utilizing the LAG connection to correctly set it up. For instance, if it is from multiple devices with different mac addresses or maybe different ip addresses or even is initiated on a different physical port, etc.

matt19849
Enthusiast
Posts: 27
Liked: never
Joined: Sep 04, 2012 9:26 am
Full Name: Matt

Re: Trying to increase perfomance

Post by matt19849 »

Hi,

I have set our QNAPs to use balance-alb which according to QNAP:
Balance-alb is similar to balance-tlb but also attempts to redistribute incoming (receive load balancing) for IPV4 traffic. The receive load balancing is achieved by ARP negotiation sent by the local system on their way out and overwrites the source hardware address with the unique hardware address of one of the Ethernet interfaces in the bond such that different peers use different hardware address for the server. This mode provides load balancing and fault tolerance.

I haven't noticed any performance increase just peace of mind I have a bit of failover

veremin
Product Manager
Posts: 18009
Liked: 1717 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Trying to increase perfomance

Post by veremin »

Hi, Matt, have you already tried to conduct a series of tests in order to see what performance your network connection is capable of? This should shed a light on existing problem and give some hints about further investigation.

Thanks.

snakes
Influencer
Posts: 11
Liked: never
Joined: Apr 19, 2011 10:05 am
Full Name: Tor c
Contact:

Re: Trying to increase perfomance

Post by snakes »

As yuki mentioned, some of the QNAP's have terrible performances especially when using with deduplication and/or transforming. I tried to tune a QNAP and Veeam to perform properly a year ago believeing it was a network bottleneck, but with no success. Iops is the clue when it comes to dedup and transforming.

http://www.virtualtothecore.com/en/veea ... torage-io/

This article probably makes things clearer. We reverted to use a equallogic storage solution.

Regards

Snakes

veremin
Product Manager
Posts: 18009
Liked: 1717 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Trying to increase perfomance

Post by veremin »

Even though the statistics suggest that the problem lies in network itself, I'm wondering whether, is there any other device, apart from QNAP, in target location that you can choose as a target repository. This should give us clues whom to blame in this situation - network or target device.

Thanks.

matt19849
Enthusiast
Posts: 27
Liked: never
Joined: Sep 04, 2012 9:26 am
Full Name: Matt

Re: Trying to increase perfomance

Post by matt19849 »

I can certainly try and back it up else where to see how that performs. I'll give that a go today

veremin
Product Manager
Posts: 18009
Liked: 1717 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Trying to increase perfomance

Post by veremin »

And don't forget to test the connectivity itself by copying files from source to target proxying servers. Looking forward to hearing your results. Thanks.

dellock6
Veeam Software
Posts: 5958
Liked: 1780 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Trying to increase perfomance

Post by dellock6 »

Just a quick Thanks to Snakes for quoting my blog post, I'm glad is still useful after more than a year I wrote it.

Indeed QNAP can be very hard to use on some backup methods; all its functions, even raid, are managed by the central CPU, since it does not have dedicated raid controllers. Basically it's a linux server that uses LVM and dmraid to aggregate the disks, I've seen it live in a large model with VGA output, it can be clearly seen during boot time :)

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2020
Veeam VMCE #1

matt19849
Enthusiast
Posts: 27
Liked: never
Joined: Sep 04, 2012 9:26 am
Full Name: Matt

Re: Trying to increase perfomance

Post by matt19849 »

I have just tried backup to up a new repository that is an NFS share from a server, this share is on a P2000. I'm getting the same 3-5MB/s speeds again.

I'll start to look at the network now

Post Reply

Who is online

Users browsing this forum: BartP, Bing [Bot], Google [Bot], joakimp, pirx and 67 guests