Comprehensive data protection for all workloads
Post Reply
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Is my backup speed as expected?

Post by Fiskepudding »

Hi

I was wondering if my system performs ok. I think is a bit slow considdering the hardware....

Source:
VM with 32 GB ram and 4 vCPU's
Intel(R) Xeon(R) CPU E5645 @ 2.40GHz
Datastore is local 12x512GB Crucial M4 SSD's in RAID6 on upgraded HP 410i controller.
(Just used acronis to clone a disk internatly on this array, 2 TB took about 1 hour so the arry performs ok)

Proxy and target:(same VM).
It runs on win 2008 R2 Enterprise, with 8 vCPU and 32 GB RAM
Intel(R) Xeon(R) CPU E5645 @ 2.40GHz
Datastore soncists of 11x3TB Constellation disks in RAID 6 on a LSI controller.

Network:
Its all wired up with 10GbE.

Im donig my first reversed incremental.

It starts at aobut 125MB/s and drops as time goes.
At the start the proxy uses all CPU on all the 8vCPU's.
After 30 minutes and 16% done, the speed is down to 65 MB/s and still dropping.
Tehe spu usage on the proxy jumps uup and down, but i guess averages about 15%
CPU on proxy:
Image
Network on proxy:
Image

The bottelneck tooltip shos up like this:

Source 78%
Proxy 76%
Network 38%
Target 30%

The numbers are quite stable.

Thanx for anny tips.
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Is my backup speed as expected?

Post by Fiskepudding »

Can also add that the "Processed/read/Transfered" seems to freeze, no progress for a long time. Just to ad now, the speed is now at 40 MB/S
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Is my backup speed as expected?

Post by Fiskepudding »

Now one of the VM is finished. The job has moved on to the next VM. (witch is on the same host and datastore)
The speed is now down to 22MB/S for the entire job.
seems like there are no real bottlenecks that really point out:
Source 56%
Proxy 55%
Network 51%
Target 51%

Image

Current cpu load of the proxy (and target)
Image

Any tips on how to make this go faster would be much apriciated!
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Is my backup speed as expected?

Post by Fiskepudding »

I did not post any pictures of the start of the job so here it goes:
After the job has run for about 5 min, the proxy handeles a lot of work, cpu wise.
Image

When i observere after 5 min the speed is about 150 MB/s. and the CPU on proxy is 100%.
This seems Resonable. But during the job the peed gradualy slows down after 5 min, and the cpu usage on proxy dorps (pictures above) to about 15%'ish.
emachabert
Veeam Vanguard
Posts: 388
Liked: 168 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: Is my backup speed as expected?

Post by emachabert »

How fast is the target storage ? Is there write cache on the LSI controler ?
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Is my backup speed as expected?

Post by Gostev »

When you job gets really slow, can you query the current real-time bottlenecks values by hovering over the overall bottleneck value in the top of real-time statistics window? You should get a tool tip pop up.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Is my backup speed as expected?

Post by Gostev »

Fiskepudding wrote:When i observe after 5 min the speed is about 150 MB/s. and the CPU on proxy is 100%.
This seems Resonable. But during the job the peed gradualy slows down after 5 min, and the cpu usage on proxy dorps (pictures above) to about 15%'ish.
When you job gets really slow, can you please query the current real-time bottlenecks values by hovering over the overall bottleneck value in the top left of the real-time statistics window? You should get a tool tip pop up. Please post the data you get here. Thanks!
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Is my backup speed as expected?

Post by tsightler »

Sounds like the same issue being discussed here.
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Is my backup speed as expected?

Post by Fiskepudding »

Thanx for the reply. I have created a new job, that backs up about 190 GB.(40 GB VD + 150GB VD)
The results trends are the same, it slows down ( I tried this time without indexing the guest files):

The 40 GB VD ended at 228 MB/S (witch is nice), the 150 GB VD startet out about 150 and ended at 40 MB/S
Image
If we look at the performance of the 40GB drive, i think the target can swallow what it gets.

THE LSI controller has Default write set to "Write Through" I wil test out write back, dont have battery on the controller yet so i need to get that if this proves faster.
Any other tips on the controller setings?
Image
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Is my backup speed as expected?

Post by Fiskepudding »

Thanx for all the replies :)
Gostev wrote:When you job gets really slow, can you query the current real-time bottlenecks values by hovering over the overall bottleneck value in the top of real-time statistics window? You should get a tool tip pop up.
There are no real bottlenects, its hard to say. It might lean a few % points towards source, but not much.
Right now the job has actullay speeded up some. But the distribution between source,proxy,network and target styas about the same druing the entire job, UNLESS in the start when it is very fast:
During the job its like this:
Image


After the first 6 minutes its like this..
Image
emelloul
Novice
Posts: 4
Liked: never
Joined: Sep 21, 2010 6:43 am
Full Name: Ethan Melloul
Location: Houston, TX
Contact:

Re: Is my backup speed as expected?

Post by emelloul »

Write-through is going to kill that controller, especially considering the type of controller that you're using. Get a battery in there, enable the write cache and that thing should scream.
Ethan M. - Engineer III - CCIE #44000 (R/S) - VCP5
Tradentrix - Technology/Cloud Services
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Is my backup speed as expected?

Post by Gostev »

Both of the latest screenshots above show very decent performance, considering you are using network processing mode (the data is retrieved through ESXi management interface). I do not see any problems with the numbers above. Source 99% means that source storage data retrieval speed is a bottleneck here.
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Is my backup speed as expected?

Post by Fiskepudding »

Really appriciate the input :!:

Write cache dide help some. So i guess i have to get a battery or condensator.

Gostev: Yea, speed above 100 MB/s is great. But its what happens later on. after 5-6 minuts the speeds gradually drops, and it have an average of 20 MB/s on a 1 TB VM.... So the real speed at the end has to much lower then that. And as you can se from the pictures no bottleneck stands out. (After a while, I am not taking about 5-6 min, thats clearly the target not keeping up, but lets say 1 hour into the backup job).

I CAN live with that, but not if I dont NEED to :)

The incrimental on top of this are blazing fast!.. I seen numbers in the 2-300MB/s

Is a decrease in speed over time, on large VMs normal?

Thanx
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Is my backup speed as expected?

Post by Gostev »

No, it is not normal. This is why I wanted to see the real-time bottleneck stats numbers for when you have this slowdown, to see where does the issue sit. Thanks!
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Is my backup speed as expected?

Post by Fiskepudding »

Fiskepudding wrote: (After a while, I am not taking about 5-6 min, thats clearly the target not keeping up, but lets say 1 hour into the backup job).
Should be:
(After a while, I am not taking about 5-6 min, thats clearly the SOURCE not keeping up, but lets say 1 hour into the backup job).

But i guess you figured that out :)
emachabert
Veeam Vanguard
Posts: 388
Liked: 168 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: Is my backup speed as expected?

Post by emachabert »

By the way, even if source is the bottleneck here, you should definitively have battery backed-up write-cache on the target if you want to have the best throughput on it.
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Is my backup speed as expected?

Post by Fiskepudding »

Hello :D

I previusly stated that changing write cache made it somewhat better.
I must have been testing somthing wrong. Becase my latest test show that both emachabert and emelloul were SPOT ON!
This has everyting to do with the drop in speed. Cudos to you.

Im currently running a full backup, and after 45 minutes its still going strong with 124MB/S. with the aray set to WRITE BACK.
Identifying thse bottlenecks

Source 99%
Proxy 96%
Network 37%
Target 7%

Just awsome guys!

I delibritly chose to have no battery on these controller since they were soppest do run a SINGLE backup job at at time. Hence WRITE-through.
Because I thougt there were very litle use for that with 1 steady datastream.

I was proven VERY wrong. So now i just have have the option to go enterprise on ESX datasenter on Windows to beefe up the proxyes :)
Bus as for now I am perfecty happy with 124MB/s.
So i basicly went from 20 MB/s (posibly slower if the job would have lasted longer) to a stable 124MB/S!!! (only at the cost of getting me som batteries for the LSI)

Dont know why Veem did not detect the target as the bottlenech before, but that's another discussion that I wont dive into :)
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Is my backup speed as expected?

Post by Gostev »

Fiskepudding wrote:Dont know why Veem did not detect the target as the bottlenech before, but that's another discussion that I wont dive into :)
You could probably see that in the real-time bottleneck stats, which is why I asked for one... overall bottleneck stats are averaged out across the entire VM processing time, and since your bottleneck was changing as the job was running (shifting from source to target as the job is running), the overall statistics did not really provide us with any good idea on where the problem sits.
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Is my backup speed as expected?

Post by Fiskepudding »

I know you asked for real time stats, and i answered, actually before your post.
Its about 50% on all bottlenect, from the point and forward it slows down and the persentages stabelises at 50% on all, "forever".

I could have made that clearer, sorry. :oops:

Hence thats why I wondered why Veeam did not identify Target....
emachabert
Veeam Vanguard
Posts: 388
Liked: 168 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: Is my backup speed as expected?

Post by emachabert »

:D :D
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 174 guests