Comprehensive data protection for all workloads
dellock6
Veeam Software
Posts: 5940
Liked: 1760 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: bottleneck source and proxy

Post by dellock6 »

That's correct, I will pick the second test as a benchmark, the storage can do 300 MB/s.
Since the ESXi servers are connected via FC, this is not only internal speed but also the speed you can pull from the storage via FC. In SAN mode Veeam can potentially reach this speed, why it's not reaching 300 MB/s is the key of the problem.
Reading again the thread, Veeam server is physical with a 4 Gb FC, so the connection is not the bottleneck.
Can you print a recap of a Veeam job, selecting the details of a single VM, so I can see speed and backup mode?
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2020
Veeam VMCE #1

tfloor
Expert
Posts: 270
Liked: 14 times
Joined: Jan 03, 2012 2:02 pm
Full Name: Tristan Floor
Contact:

Re: bottleneck source and proxy

Post by tfloor »

Hi Luca,

Correct, I see no problem on the cables or throughput of the hardware. So there is a problem somewhere.
I will post here the screenshots of an overall backup and with a single one (the one i test a lot).

Don't look at the average speed of the first one (job is waiting for resources for a long time, since i can't run mutiple jobs, because that will divide the speed.

Overall job:
Image

Single Job (my test job on the same lun from all the tests).
Image

It's using SAN (after Patch 2) and can't see any other troubles on this pictures.

Regards,

Tristan.

tfloor
Expert
Posts: 270
Liked: 14 times
Joined: Jan 03, 2012 2:02 pm
Full Name: Tristan Floor
Contact:

Re: bottleneck source and proxy

Post by tfloor »

BTW , is this something where the official veeam support can help ?

tsightler
VP, Product Management
Posts: 5687
Liked: 2508 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: bottleneck source and proxy

Post by tsightler »

I think it's questionable that support could help because I'm not sure you have a problem. You "test" of the storage only proved what the maximum performance of your storage would be running a read-only benchmark. It was not even run on the same hardware that you are running your backups on and most "benchmarks" application also do "forever" read-ahead, basically completely filling the requests queue to eliminate request latency. This is a good theoretical benchmark to prove maximum throughput, but in know way reflect the performance you should expect to receive when running a backup, especially since it is completely different hardware.

A backup must not only read the data, but calculate and perform hash lookups for dedupe, perform compression, and copy that data to another storage device. This places much higher demands on CPU and memory bandwidth than a simple benchmark. I honestly believe that you are very close to the top performance for the hardware. I'm not say that perhaps you couldn't sneak a little more performance by tweaking here and there, but I would estimate that the absolute maximum performance of this hardware would be around 150-180MB/sec assuming typical backup data mix. That would require running multiple jobs. The fact that you state that running multiple jobs cuts your time in half implies that you are pretty much hitting the wall as far as the proxy hardware is concerned.

dellock6
Veeam Software
Posts: 5940
Liked: 1760 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: bottleneck source and proxy

Post by dellock6 »

Right, in fact I suggested 300 MB/s is the theoretical max speed of the production storage.
I was looking at the screenshots, I saw 149 MB/s which is higher than previous numbers, and I also see you were running backups at mid-day. Consider the storage is also running production VMs, and both usage of the storage is completely random.
I agree 130-150 mb/s is a fair result.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2020
Veeam VMCE #1

tfloor
Expert
Posts: 270
Liked: 14 times
Joined: Jan 03, 2012 2:02 pm
Full Name: Tristan Floor
Contact:

Re: bottleneck source and proxy

Post by tfloor »

Luca,

So nothing todo with it.. That's most of the time my experience with troubleshouting performance problems. No usefull conclusion :)
What if i ask "What do i need to change to get a higher performance in veeam"?
:) try it on another way. I'm trying to know what to buy new, to get the performance higher ... If i know that .. i also know the bottleneck and maybe it can answer my question :)

dellock6
Veeam Software
Posts: 5940
Liked: 1760 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: bottleneck source and proxy

Post by dellock6 »

No, wait, the conclusions are there. Remember your tests:

NONE: = 122MB/s Load: Source 63% > Proxy 39% > Network 60% > Target 67%
LOW: = 124 MB/s Load: Source 78% > Proxy 50% > Network 45% > Target 38%
OPTIMAL: = 123 MB/s Load: Source 91% > Proxy 77% > Network 18% > Target 8%
BEST: = 94MB/s Load: Source 54% > Proxy 88% > Network 9% > Target 5%

using optimal compression, the source is the bottleneck, but also proxy. If you want a gain on performances these are the two elements to work on.
Tuning backups is a fine art, as you can see in these results, touching a value removes a bottleneck from one point of the chain, just to raise another bottleneck somewhere else.
You have basically two roads:
- work on source/proxy, myabe having more CPU power on the Veeam Backup server
- leave source untouched, choose a lower compression, and work on target maybe with some dedup appliance. I know you already have DD in place, but this is not the only solution around. Maybe this time you can try to have a "try and buy" before using other money ;)

Probably the first and cheaper way is upgrading CPUs on the Veeam Server. Sadly, and I know this, all these changes require further expenses to be implemented.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2020
Veeam VMCE #1

tfloor
Expert
Posts: 270
Liked: 14 times
Joined: Jan 03, 2012 2:02 pm
Full Name: Tristan Floor
Contact:

Re: bottleneck source and proxy

Post by tfloor »

Luca,

You are right.
I will thank you for your help on this problem.
The backup server will be replaced next year.
The DD610 is a very low end system. The QNAP is fine.
Maybe we take another high end DD the next time ;).

Again,

Thanks.

dellock6
Veeam Software
Posts: 5940
Liked: 1760 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: bottleneck source and proxy

Post by dellock6 »

Don't want to act as a Sales Rep (brrr...), but take also a look at ExaGrid. They do post-processing dedup instead of inline, so ingestion speed is often higher than other appliances, and they also have some cool technologies.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2020
Veeam VMCE #1

Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: bottleneck source and proxy

Post by Fiskepudding »

I have just skimmed your post, but it seems like your speed decreses after how long the job has run.
I se the image after 5 min is 124 MS/s and after 11 hours its 11MB/s..

My problem was the targes cache setting. just a tip: Is my backup speed as expected?

tfloor
Expert
Posts: 270
Liked: 14 times
Joined: Jan 03, 2012 2:02 pm
Full Name: Tristan Floor
Contact:

Re: bottleneck source and proxy

Post by tfloor »

dellock6 wrote:Don't want to act as a Sales Rep (brrr...), but take also a look at ExaGrid. They do post-processing dedup instead of inline, so ingestion speed is often higher than other appliances, and they also have some cool technologies.
I see a lot of "exagrid" indead on twitter...
I will look at it, and i'm wondering if those appliances are also so expensive like DD.
A qnap-look-a-like with dedup is enough ;)

tfloor
Expert
Posts: 270
Liked: 14 times
Joined: Jan 03, 2012 2:02 pm
Full Name: Tristan Floor
Contact:

Re: bottleneck source and proxy

Post by tfloor »

Fiskepudding wrote:I have just skimmed your post, but it seems like your speed decreses after how long the job has run.
I se the image after 5 min is 124 MS/s and after 11 hours its 11MB/s..

My problem was the targes cache setting. just a tip: Is my backup speed as expected?
Hi Fiskepudding,

Which Target Cache setting , do you mean.
My target is a QNAP 1279.
I don't remember that's a bottleneck.
Can you explain where i have to look for?

Thanks

Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: bottleneck source and proxy

Post by Fiskepudding »

I saw that your non compressed backup showed an even bottlenechk on source,proxy,network and target.

This happened to me to, regrdless of compression settigs tho.
After 5-6 minutes my backup speed started to drop, and after 1 TB transfer data, the sped was 22MB/S and it started out with 100 + MB/s

So i would check you cashe settings on your raid controller on TARGET. Make sure it is set to Write Back or whatever the controller terms are.And you would need a battery if you are going to run with that setting. But for testing purpose you can do without.

Worth a shot?

tfloor
Expert
Posts: 270
Liked: 14 times
Joined: Jan 03, 2012 2:02 pm
Full Name: Tristan Floor
Contact:

Re: bottleneck source and proxy

Post by tfloor »

We have the highest qnap model in range. Ec 1279
It has the basic qnap firmware version 3.6 and the only thing I can set from what I see. Is enable or disable write cache.
I have tried to play with it but for backup it doesn't matter.
So I'm wondering if there are more settings. I will ask qnap for that

Post Reply

Who is online

Users browsing this forum: restore-helper and 51 guests