bottleneck source and proxy

[dellock6]

That's correct, I will pick the second test as a benchmark, the storage can do 300 MB/s.
Since the ESXi servers are connected via FC, this is not only internal speed but also the speed you can pull from the storage via FC. In SAN mode Veeam can potentially reach this speed, why it's not reaching 300 MB/s is the key of the problem.
Reading again the thread, Veeam server is physical with a 4 Gb FC, so the connection is not the bottleneck.
Can you print a recap of a Veeam job, selecting the details of a single VM, so I can see speed and backup mode?

[tfloor]

Hi Luca,

Correct, I see no problem on the cables or throughput of the hardware. So there is a problem somewhere.
I will post here the screenshots of an overall backup and with a single one (the one i test a lot).

Don't look at the average speed of the first one (job is waiting for resources for a long time, since i can't run mutiple jobs, because that will divide the speed.

Overall job:


Single Job (my test job on the same lun from all the tests).


It's using SAN (after Patch 2) and can't see any other troubles on this pictures.

Regards,

Tristan.

[tfloor]

BTW , is this something where the official veeam support can help ?

[tsightler]

I think it's questionable that support could help because I'm not sure you have a problem. You "test" of the storage only proved what the maximum performance of your storage would be running a read-only benchmark. It was not even run on the same hardware that you are running your backups on and most "benchmarks" application also do "forever" read-ahead, basically completely filling the requests queue to eliminate request latency. This is a good theoretical benchmark to prove maximum throughput, but in know way reflect the performance you should expect to receive when running a backup, especially since it is completely different hardware.

A backup must not only read the data, but calculate and perform hash lookups for dedupe, perform compression, and copy that data to another storage device. This places much higher demands on CPU and memory bandwidth than a simple benchmark. I honestly believe that you are very close to the top performance for the hardware. I'm not say that perhaps you couldn't sneak a little more performance by tweaking here and there, but I would estimate that the absolute maximum performance of this hardware would be around 150-180MB/sec assuming typical backup data mix. That would require running multiple jobs. The fact that you state that running multiple jobs cuts your time in half implies that you are pretty much hitting the wall as far as the proxy hardware is concerned.

[dellock6]

Right, in fact I suggested 300 MB/s is the theoretical max speed of the production storage.
I was looking at the screenshots, I saw 149 MB/s which is higher than previous numbers, and I also see you were running backups at mid-day. Consider the storage is also running production VMs, and both usage of the storage is completely random.
I agree 130-150 mb/s is a fair result.

[tfloor]

Luca,

So nothing todo with it.. That's most of the time my experience with troubleshouting performance problems. No usefull conclusion
What if i ask "What do i need to change to get a higher performance in veeam"?
try it on another way. I'm trying to know what to buy new, to get the performance higher ... If i know that .. i also know the bottleneck and maybe it can answer my question

[dellock6]

No, wait, the conclusions are there. Remember your tests:

NONE: = 122MB/s Load: Source 63% > Proxy 39% > Network 60% > Target 67%
LOW: = 124 MB/s Load: Source 78% > Proxy 50% > Network 45% > Target 38%
OPTIMAL: = 123 MB/s Load: Source 91% > Proxy 77% > Network 18% > Target 8%
BEST: = 94MB/s Load: Source 54% > Proxy 88% > Network 9% > Target 5%

using optimal compression, the source is the bottleneck, but also proxy. If you want a gain on performances these are the two elements to work on.
Tuning backups is a fine art, as you can see in these results, touching a value removes a bottleneck from one point of the chain, just to raise another bottleneck somewhere else.
You have basically two roads:
- work on source/proxy, myabe having more CPU power on the Veeam Backup server
- leave source untouched, choose a lower compression, and work on target maybe with some dedup appliance. I know you already have DD in place, but this is not the only solution around. Maybe this time you can try to have a "try and buy" before using other money

Probably the first and cheaper way is upgrading CPUs on the Veeam Server. Sadly, and I know this, all these changes require further expenses to be implemented.

[tfloor]

Luca,

You are right.
I will thank you for your help on this problem.
The backup server will be replaced next year.
The DD610 is a very low end system. The QNAP is fine.
Maybe we take another high end DD the next time .

Again,

Thanks.

[dellock6]

Don't want to act as a Sales Rep (brrr...), but take also a look at ExaGrid. They do post-processing dedup instead of inline, so ingestion speed is often higher than other appliances, and they also have some cool technologies.

[Fiskepudding]

I have just skimmed your post, but it seems like your speed decreses after how long the job has run.
I se the image after 5 min is 124 MS/s and after 11 hours its 11MB/s..

My problem was the targes cache setting. just a tip: Is my backup speed as expected?

[tfloor]

Don't want to act as a Sales Rep (brrr...), but take also a look at ExaGrid. They do post-processing dedup instead of inline, so ingestion speed is often higher than other appliances, and they also have some cool technologies.

I see a lot of "exagrid" indead on twitter...
I will look at it, and i'm wondering if those appliances are also so expensive like DD.
A qnap-look-a-like with dedup is enough

[tfloor]

I have just skimmed your post, but it seems like your speed decreses after how long the job has run.
I se the image after 5 min is 124 MS/s and after 11 hours its 11MB/s..

My problem was the targes cache setting. just a tip: Is my backup speed as expected?

Hi Fiskepudding,

Which Target Cache setting , do you mean.
My target is a QNAP 1279.
I don't remember that's a bottleneck.
Can you explain where i have to look for?

Thanks

[Fiskepudding]

I saw that your non compressed backup showed an even bottlenechk on source,proxy,network and target.

This happened to me to, regrdless of compression settigs tho.
After 5-6 minutes my backup speed started to drop, and after 1 TB transfer data, the sped was 22MB/S and it started out with 100 + MB/s

So i would check you cashe settings on your raid controller on TARGET. Make sure it is set to Write Back or whatever the controller terms are.And you would need a battery if you are going to run with that setting. But for testing purpose you can do without.

Worth a shot?

[tfloor]

We have the highest qnap model in range. Ec 1279
It has the basic qnap firmware version 3.6 and the only thing I can set from what I see. Is enable or disable write cache.
I have tried to play with it but for backup it doesn't matter.
So I'm wondering if there are more settings. I will ask qnap for that