Trying to increase perfomance

[matt19849]

Hi All,

I have spent a lot of time reading different forums/posts and i've tried all different settings to get better performance in my backups.
I'm a little confused with CBT now though, I have enabled virtual appliance as the transport mode so should I be disabling CBT in the backup job?

I've run a job with the virtual appliance as the transport mode and in the backup job status it says

02/01/2014 16:50:41 :: Using backup proxy 10.10.12.247 for disk Hard disk 1 [hotadd]
02/01/2014 16:50:43 :: Hard disk 1 (120.0 GB) read at 21 MB/s [CBT]

Does this mean it isn't using hotadd? Is this why the performance isn't great?
The read speed of the VM seems to go anywhere between 8MB/s to 100MB/s - should it not be a little more steady?

Cheers

[Gostev]

Hi Matt,

I don't follow why would you want to disable CBT when using hot add (or any other processing mode for that matter).

According to the log you posted, the job is using hot add to backup this particular hard disk.

The read speed will depend on change blocks placement: the more random they are, the slower the changed blocks will be read from source storage. Once the job will hit long sequential segment, the read speed will go up significantly for the duration of that segment. This is basically how hard drive based storage works, it does not "like" random I/O. For example, you can easily reach 100+ MB/s of sequential read speed even on the slowest notebook hard drive available these days, but random I/O would totally kill the performance, and you will be lucky to get a few MB/s.

Thanks.

[matt19849]

That makes sense, thank you

I read somewhere on a different site that its best to disable CBT when using virtual appliance but I guess they were wrong on that one

[matt19849]

Hi,

I'm still having a performance issue with our backup copy job though. I currently have it set up as follows:

Site 1 has the veeam server
site 2 has the proxy
A backup copy job is copying the backups from site 1 to site 2 using the site 2 proxy set to virtual appliance transport mode

The read speeds I'm getting are around 3MB/s - and it has taken 14hrs to get to 6%
The bottleneck shows Network: 99%

Site 1 has a dedicated 100mb fiber and site 2 has a dedicated 100mb fiber.
There are no restrictions between the two sites

Is this expected performance?

Cheers

[foggy]

I read somewhere on a different site that its best to disable CBT when using virtual appliance but I guess they were wrong on that one

Yes, completely. Unless they were talking about backing up proxy server itself, where CBT is disabled for this VM automatically.

A backup copy job is copying the backups from site 1 to site 2 using the site 2 proxy set to virtual appliance transport mode

Actually, proxy backup mode does not affect backup copy job in any way, since data is transferred directly between agents installed on the source and target backup repositories (in direct backup copy job mode) and between WAN accelerators (in WAN accelerated job mode).

What kind of repositories do you have?

[matt19849]

The repositories are all network shares (Qnap)

i.e.
\\qnap_ip\Backup

[foggy]

Are you using direct backup copy job mode? Have you specified your site 2 proxy server as a proxying server in repository wizard? Veeam agent cannot be deployed directly on repositories of this type.

[matt19849]

Hi,

It is using direct backup copy mode, it is using the site 2 proxy which is configured on that backup repository

[foggy]

What about fiber usage? Is it fully saturated during transfer?

Probably it is worth trying WAN accelerated mode, which was specifically designed for limited bandwidth (100 Mbit or less).

Also, have you considered seeding you backup copy job to avoid initial copy of the full backup files?

[matt19849]

HI,

The fiber usage is next to none, it isn't very heavily used at all.
I only have the standard version of Veeam so not sure I can use the WAN accelerator

I haven't tried seeding the backup, could you point me to more information on this?

[foggy]

The fiber usage is next to none, it isn't very heavily used at all.

Are you positive there're no throttling rules configured for the pair of source and target agents?

I haven't tried seeding the backup, could you point me to more information on this?

Sure, here are a couple of links:
Seeding v7 backup copy
Seeding offsite backup files for a Backup Copy Job

[matt19849]

There isn't any throttling on our firewalls, I can check with the ISP still

I'll take a ready of the links - thanks

[Gostev]

Site 1 has the veeam server
site 2 has the proxy
A backup copy job is copying the backups from site 1 to site 2 using the site 2 proxy set to virtual appliance transport mode

The read speeds I'm getting are around 3MB/s - and it has taken 14hrs to get to 6%
The bottleneck shows Network: 99%

VM data during Backup Copy in direct mode goes direct from source to target repository, with no backup proxies involved. The transport mode is irrelevant, because VM data is obtained from the source backup repository, not from virtual infrastructure.

The Source bottleneck means that the issue is the speed of obtaining data from the source backup repository, this is the issue that needs to be investigated. Firewall settings, WAN acceleration, or even a bigger WAN pipe will not help here.

[matt19849]

I will take a look at the source, the bottle neck in Veeam says network 99%
Could that be the network connection to the source?

[matt19849]

I've checked the source and it seems a little odd. When the backup job is preparing to read the data i'm getting around 50MB/s on the nic card at the source. Once Veeam starts to read the drive that drops down to 3MB/s on the nic card at the source...

[Gostev]

Hi Matt, I am very sorry. Somehow I thought the bottleneck is Source 99%, even though I quoted it. My only excuse is that I am on holiday right now please disregard my comment above, and sorry for sending you in the wrong direction.

Network 99% obviously means that the problem is the speed of connection between source and target data movers.

I suggest simple test of copying files from source to target proxying server, and if it shows much better throughput than the job, then it would be best to open a support case.

I've checked the source and it seems a little odd. When the backup job is preparing to read the data i'm getting around 50MB/s on the nic card at the source. Once Veeam starts to read the drive that drops down to 3MB/s on the nic card at the source...

This only proves that there are no issues with the speed of obtaining data from the source backup repository. The speed drop you are observing happens because another component of the data processing conveyor becomes a bottleneck, specifically your network.

Thanks!

[Yuki]

We use QNAPs quite a bit. Sequential writes/reads to and from QNAPs are usually fine. Although we did notice that LAG configured on QNAPs sometimes results in lower overall transfer rates vs single 1Gb link.

The second issue we've seen with this combo is that the reverse incremental speeds are terrible with QNAP. Make sure you are not writing any backups to the QNAP when backup copy job is running. Although Veeam may show network as bottleneck, we've seen and posted here about the mechanism being mistaken about bottlenecks.

[matt19849]

Hi Yuki
Do you use a single network port on your QNAPs or do you use port trunking?

Nothing else is being written to the QNAPs while the backups are running. We have one QNAP for our backup and another QNAP for the backup copy. We are using reverse incremental on the QNAPS too

[thornec]

I believe Yuki said they were using LAG, so multiple connections. Port trunking usually refers to multiple vlans on a single logical connection. So one could use LAG, multiple connections, and set it as a trunk so it could have multiple vlans on that single logical connection. Whether the qnap would be able utilize that type of connection or if it would be of use would be another matter. Seems there are very few LAG type implementations that actually increase the thru put, how ever they have some use as redundant connections, assuming spanning tree is setup correctly. I think one needs to really understand what type of traffic would be utilizing the LAG connection to correctly set it up. For instance, if it is from multiple devices with different mac addresses or maybe different ip addresses or even is initiated on a different physical port, etc.

[matt19849]

Hi,

I have set our QNAPs to use balance-alb which according to QNAP:
Balance-alb is similar to balance-tlb but also attempts to redistribute incoming (receive load balancing) for IPV4 traffic. The receive load balancing is achieved by ARP negotiation sent by the local system on their way out and overwrites the source hardware address with the unique hardware address of one of the Ethernet interfaces in the bond such that different peers use different hardware address for the server. This mode provides load balancing and fault tolerance.

I haven't noticed any performance increase just peace of mind I have a bit of failover

[veremin]

Hi, Matt, have you already tried to conduct a series of tests in order to see what performance your network connection is capable of? This should shed a light on existing problem and give some hints about further investigation.

Thanks.

[snakes]

As yuki mentioned, some of the QNAP's have terrible performances especially when using with deduplication and/or transforming. I tried to tune a QNAP and Veeam to perform properly a year ago believeing it was a network bottleneck, but with no success. Iops is the clue when it comes to dedup and transforming.

http://www.virtualtothecore.com/en/veea ... torage-io/

This article probably makes things clearer. We reverted to use a equallogic storage solution.

Regards

Snakes

[veremin]

Even though the statistics suggest that the problem lies in network itself, I'm wondering whether, is there any other device, apart from QNAP, in target location that you can choose as a target repository. This should give us clues whom to blame in this situation - network or target device.

Thanks.

[matt19849]

I can certainly try and back it up else where to see how that performs. I'll give that a go today

[veremin]

And don't forget to test the connectivity itself by copying files from source to target proxying servers. Looking forward to hearing your results. Thanks.

[dellock6]

Just a quick Thanks to Snakes for quoting my blog post, I'm glad is still useful after more than a year I wrote it.

Indeed QNAP can be very hard to use on some backup methods; all its functions, even raid, are managed by the central CPU, since it does not have dedicated raid controllers. Basically it's a linux server that uses LVM and dmraid to aggregate the disks, I've seen it live in a large model with VGA output, it can be clearly seen during boot time

Luca.

[matt19849]

I have just tried backup to up a new repository that is an NFS share from a server, this share is on a P2000. I'm getting the same 3-5MB/s speeds again.

I'll start to look at the network now