Is my backup speed as expected?

[Fiskepudding]

Hi

I was wondering if my system performs ok. I think is a bit slow considdering the hardware....

Source:
VM with 32 GB ram and 4 vCPU's
Intel(R) Xeon(R) CPU E5645 @ 2.40GHz
Datastore is local 12x512GB Crucial M4 SSD's in RAID6 on upgraded HP 410i controller.
(Just used acronis to clone a disk internatly on this array, 2 TB took about 1 hour so the arry performs ok)

Proxy and target:(same VM).
It runs on win 2008 R2 Enterprise, with 8 vCPU and 32 GB RAM
Intel(R) Xeon(R) CPU E5645 @ 2.40GHz
Datastore soncists of 11x3TB Constellation disks in RAID 6 on a LSI controller.

Network:
Its all wired up with 10GbE.

Im donig my first reversed incremental.

It starts at aobut 125MB/s and drops as time goes.
At the start the proxy uses all CPU on all the 8vCPU's.
After 30 minutes and 16% done, the speed is down to 65 MB/s and still dropping.
Tehe spu usage on the proxy jumps uup and down, but i guess averages about 15%
CPU on proxy:

Network on proxy:


The bottelneck tooltip shos up like this:

Source 78%
Proxy 76%
Network 38%
Target 30%

The numbers are quite stable.

Thanx for anny tips.

[Fiskepudding]

Can also add that the "Processed/read/Transfered" seems to freeze, no progress for a long time. Just to ad now, the speed is now at 40 MB/S

[Fiskepudding]

Now one of the VM is finished. The job has moved on to the next VM. (witch is on the same host and datastore)
The speed is now down to 22MB/S for the entire job.
seems like there are no real bottlenecks that really point out:
Source 56%
Proxy 55%
Network 51%
Target 51%



Current cpu load of the proxy (and target)


Any tips on how to make this go faster would be much apriciated!

[Fiskepudding]

I did not post any pictures of the start of the job so here it goes:
After the job has run for about 5 min, the proxy handeles a lot of work, cpu wise.


When i observere after 5 min the speed is about 150 MB/s. and the CPU on proxy is 100%.
This seems Resonable. But during the job the peed gradualy slows down after 5 min, and the cpu usage on proxy dorps (pictures above) to about 15%'ish.

[emachabert]

How fast is the target storage ? Is there write cache on the LSI controler ?

[Gostev]

When you job gets really slow, can you query the current real-time bottlenecks values by hovering over the overall bottleneck value in the top of real-time statistics window? You should get a tool tip pop up.

[Gostev]

When i observe after 5 min the speed is about 150 MB/s. and the CPU on proxy is 100%.
This seems Resonable. But during the job the peed gradualy slows down after 5 min, and the cpu usage on proxy dorps (pictures above) to about 15%'ish.

When you job gets really slow, can you please query the current real-time bottlenecks values by hovering over the overall bottleneck value in the top left of the real-time statistics window? You should get a tool tip pop up. Please post the data you get here. Thanks!

[tsightler]

Sounds like the same issue being discussed here.

[Fiskepudding]

Thanx for the reply. I have created a new job, that backs up about 190 GB.(40 GB VD + 150GB VD)
The results trends are the same, it slows down ( I tried this time without indexing the guest files):

The 40 GB VD ended at 228 MB/S (witch is nice), the 150 GB VD startet out about 150 and ended at 40 MB/S

If we look at the performance of the 40GB drive, i think the target can swallow what it gets.

THE LSI controller has Default write set to "Write Through" I wil test out write back, dont have battery on the controller yet so i need to get that if this proves faster.
Any other tips on the controller setings?

[Fiskepudding]

Thanx for all the replies

When you job gets really slow, can you query the current real-time bottlenecks values by hovering over the overall bottleneck value in the top of real-time statistics window? You should get a tool tip pop up.

There are no real bottlenects, its hard to say. It might lean a few % points towards source, but not much.
Right now the job has actullay speeded up some. But the distribution between source,proxy,network and target styas about the same druing the entire job, UNLESS in the start when it is very fast:
During the job its like this:



After the first 6 minutes its like this..

[emelloul]

Write-through is going to kill that controller, especially considering the type of controller that you're using. Get a battery in there, enable the write cache and that thing should scream.

[Gostev]

Both of the latest screenshots above show very decent performance, considering you are using network processing mode (the data is retrieved through ESXi management interface). I do not see any problems with the numbers above. Source 99% means that source storage data retrieval speed is a bottleneck here.

[Fiskepudding]

Really appriciate the input

Write cache dide help some. So i guess i have to get a battery or condensator.

Gostev: Yea, speed above 100 MB/s is great. But its what happens later on. after 5-6 minuts the speeds gradually drops, and it have an average of 20 MB/s on a 1 TB VM.... So the real speed at the end has to much lower then that. And as you can se from the pictures no bottleneck stands out. (After a while, I am not taking about 5-6 min, thats clearly the target not keeping up, but lets say 1 hour into the backup job).

I CAN live with that, but not if I dont NEED to

The incrimental on top of this are blazing fast!.. I seen numbers in the 2-300MB/s

Is a decrease in speed over time, on large VMs normal?

Thanx

[Gostev]

No, it is not normal. This is why I wanted to see the real-time bottleneck stats numbers for when you have this slowdown, to see where does the issue sit. Thanks!

[Fiskepudding]

(After a while, I am not taking about 5-6 min, thats clearly the target not keeping up, but lets say 1 hour into the backup job).

Should be:
(After a while, I am not taking about 5-6 min, thats clearly the SOURCE not keeping up, but lets say 1 hour into the backup job).

But i guess you figured that out

[emachabert]

By the way, even if source is the bottleneck here, you should definitively have battery backed-up write-cache on the target if you want to have the best throughput on it.

[Fiskepudding]

Hello

I previusly stated that changing write cache made it somewhat better.
I must have been testing somthing wrong. Becase my latest test show that both emachabert and emelloul were SPOT ON!
This has everyting to do with the drop in speed. Cudos to you.

Im currently running a full backup, and after 45 minutes its still going strong with 124MB/S. with the aray set to WRITE BACK.
Identifying thse bottlenecks

Source 99%
Proxy 96%
Network 37%
Target 7%

Just awsome guys!

I delibritly chose to have no battery on these controller since they were soppest do run a SINGLE backup job at at time. Hence WRITE-through.
Because I thougt there were very litle use for that with 1 steady datastream.

I was proven VERY wrong. So now i just have have the option to go enterprise on ESX datasenter on Windows to beefe up the proxyes
Bus as for now I am perfecty happy with 124MB/s.
So i basicly went from 20 MB/s (posibly slower if the job would have lasted longer) to a stable 124MB/S!!! (only at the cost of getting me som batteries for the LSI)

Dont know why Veem did not detect the target as the bottlenech before, but that's another discussion that I wont dive into

[Gostev]

Dont know why Veem did not detect the target as the bottlenech before, but that's another discussion that I wont dive into

You could probably see that in the real-time bottleneck stats, which is why I asked for one... overall bottleneck stats are averaged out across the entire VM processing time, and since your bottleneck was changing as the job was running (shifting from source to target as the job is running), the overall statistics did not really provide us with any good idea on where the problem sits.

[Fiskepudding]

I know you asked for real time stats, and i answered, actually before your post.
Its about 50% on all bottlenect, from the point and forward it slows down and the persentages stabelises at 50% on all, "forever".

I could have made that clearer, sorry.

Hence thats why I wondered why Veeam did not identify Target....

[emachabert]