Slow Restore Speed - 27MB/s - Tips/Ideas?

VMware specific discussions

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby yizhar » Mon Oct 07, 2013 9:50 pm

More info:

I also have a 3par (but smaller machine - 3par 7200 with 12 sata disks in raid1 used for the test),
and I did try to restore a small test VM with about 50gb disk (most part of it empty as this is a test machine).

I got about 70 MB/s during restore, and about 8 minutes to restore 50gb vmdk.

I did try with and without block zero and got same speed, however I still thing you should test it.

This makes me think that one of the differences between my small test and your production VM, is the size of data - VBK size on repository and/or VMDK size on datastore.
Have you checked memory and cpu resources on the backup proxy during restore?

Can you try restoring a smaller VM for the test and compare results?

Yizhar
yizhar
Expert
 
Posts: 179
Liked: 48 times
Joined: Mon Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby tsightler » Mon Oct 07, 2013 11:43 pm

I do agree that this would be interesting to test.
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby kte » Fri Oct 11, 2013 10:19 pm

but veeam calculates his average on the 50 gb not on the 20 gb he restored realy from the 50 gb
So in my case veeam restore 2 tb in 2 minutes but the disk was empty, so no real way to mesure it.
so you also have 60 GB an hour, if you hade a bigger vm that was fulled it ould go slow also.
8 cpu and 32 gb of ram, so no issue there
kte
Expert
 
Posts: 172
Liked: 7 times
Joined: Tue Jul 02, 2013 7:48 pm
Full Name: Koen Teugels

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby tsightler » Sat Oct 12, 2013 2:27 am

The idea would be to test something smaller to see if it's something to do with the size of the VM being restored. For example, I can restore a 100GB VM in my home lab at about 60MB/s. This is 100GB of actual data, not a "fake" speed from restoring blank space. I have recently tested restores on some Netapp hardware in a vendor lab environment and performance was around 110MB/s for restores (real transfer speed). The idea would be to see if there was something impacting restores of very large VMs (perhaps an agent memory issue or something). You can easily measure the restore performance of the actual data by simply watching the bandwidth utilization, or better yet the datastore utilization in vCenter.
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby pgitdept » Fri Feb 14, 2014 11:37 am 6 people like this post

I just thought I'd post some of our findings regarding our slow restores:

We're running ESX 5.5 and Veeam 7R2. Our storage platforms are DELL EQL (4 arrays: 15k RAID10, 10k RAID10 and 2x7.2k RAID6) and a Nimble CS460G. Now we were testing restores from tape on a fairly smallish server here (1.3TB used of 4TB). We were restoring to a new datastore on the EQL SATA storage pool made up of two RAID6 arrays. We were getting about 15MB/s, which is pretty horrendous and it took 29hours to restore. Now that pool is fairly heavily used, but should be giving better performance than that. So we built a smaller VM of around 150GB and tested that - the results were pretty much the same.

We figured it might the storage pool and so we moved it to our 15k RAID10 storage pool. That got to the dizzy heights of 42MB/s. So, we're starting to panic now as it would take us over a month to restore from a full failure - not good. We decided to restore to the Nimble and got the dizzy heights of 80MB/s, not brilliant, but more acceptable. So, we started doing some deeper tests and noticed that our write rates to some VM's were also fairly poor, although it was hard to try and pin it down.

Anyway, I found this thread and started working through it - Veeam restores were consistently poor so we could use that as a measuring stick. We tried HotAdd and other mechanisms and found it to make no difference and were fairly convinced it was more global an issue than Veeam.

So, we got to the post by Yizhar about disabling VAAI functionality on the host. We run vSphere 5.5 Standard edition, so we don't have VAAI support even though our Nimble and EQL support it to differing degrees.

BOOM - our restore to the SATA pool went up to 90MB/s+, the 15K pool around 120MB/s and the Nimble clocked a restore of nearly 200MB/s which is amazing considering it's a large Sequential Write - the workload the Nimble hates chewing on the most.

So, we're still testing this on a spare host, but it appears that disabling VAAI has made a considerable difference to all our systems including restoration operations from Veeam. It looks like ESX has VAAI enabled by default, although I'm not sure what affect it has if you're not licensed to use it. Surely not licensed means disabled?

Anyway, it's made a big difference to us and we'll continue testing. I just wanted to say thanks for the post Yizhar!
pgitdept
Influencer
 
Posts: 17
Liked: 14 times
Joined: Thu Feb 03, 2011 10:29 am
Full Name: PGITDept

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby Frosty » Sat Feb 15, 2014 12:56 am

Really interesting discussion guys!

Can someone perhaps elaborate on what other operations (i.e. other than Veeam Restores) are likely to benefit from this change? Any possible net negatives I should consider?
Frosty
Expert
 
Posts: 138
Liked: 24 times
Joined: Tue Dec 22, 2009 9:00 pm
Full Name: Stephen Frost

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby Frosty » Sat Feb 15, 2014 2:15 am

Digging further, I found this article:
http://kb.vmware.com/selfservice/micros ... Id=1021976

Ran this command: esxcli storage core device vaai status get

Returns me a display like this for each LUN on my Dell MD3200:

naa.6842b2b0004665ba000003964d1101b2
VAAI Plugin Name:
ATS Status: unsupported
Clone Status: unsupported
Zero Status: supported
Delete Status: unsupported
Frosty
Expert
 
Posts: 138
Liked: 24 times
Joined: Tue Dec 22, 2009 9:00 pm
Full Name: Stephen Frost

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby pgitdept » Sat Feb 15, 2014 7:48 pm

So we're still doing testing but we have found that it's definitely the Block Zero setting (DataMover.HardwareAcceleratedInit) that is affecting the restore speed. On the 15K R10 shelf yesterday we started a restore with it turned on and got 45MB/s, during the restore we switched it off and it moved to about 150MB/s before turning it back on again and watching it drop back to ~42MB/s.

We're still doing tests and will be doing more on Monday. Anything of interest, I'll post.
pgitdept
Influencer
 
Posts: 17
Liked: 14 times
Joined: Thu Feb 03, 2011 10:29 am
Full Name: PGITDept

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby ryan1212 » Mon Feb 17, 2014 7:13 pm 2 people like this post

I can confirm that at least parts of the VAAI instruction set are used regardless of the ESXi license level. (I had a recent support case with VMware that involved disabling as a troubleshooting step, even though we have standard)
ryan1212
Influencer
 
Posts: 16
Liked: 5 times
Joined: Tue Feb 26, 2013 3:48 pm
Full Name: Ryan

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby dellock6 » Mon Feb 17, 2014 10:49 pm

Guys, these are really interesting findings from you, thanks for sharing them.
This definitely needs further investigations and testings to address the situation, and confirm it's a constant behaviour. Please keep updating this thread as you go on in your tests. Thanks!

Luca.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5047
Liked: 1330 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby pgitdept » Wed Feb 19, 2014 2:39 pm

Okay, so we built a new 5.5 host and tested it on a eval license to give us Enterprise functionality with full VAAI support. It performed exactly the same as our testing with Standard: Poor restore speeds until we turn off the the Block Zero setting (DataMover.HardwareAcceleratedInit) and then it's between 3-4 times faster. I'm still unable to ascertain if I'm getting a performance hit elsewhere for disabling this.
pgitdept
Influencer
 
Posts: 17
Liked: 14 times
Joined: Thu Feb 03, 2011 10:29 am
Full Name: PGITDept

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby dellock6 » Thu Feb 20, 2014 4:01 pm 2 people like this post

Hi,
as the name implies, the "block zero" primitive is involved when vSphere needs to write zeroes in the storage. It's also called write same. Possible scenarios where this library is called are the creation of thick disks. And if you think, sounds correct, a restore of a VMDK implies zeroing out the disk regions where the VMDK itself is going to be restored.
This is for sure something I'd like to test in my lab once I finished to upgrade to 5.5

PS: block zero is NOT involved in cloning, the library in this case is HardwareAcceleratedMove, and the VAAI type is "Clone Blocks/Full Copy/XCOPY"

Luca.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5047
Liked: 1330 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby tsightler » Thu Feb 20, 2014 7:19 pm

Block zeroing/WRITE_SAME is used with thin disk as new segments are allocated, which of course will happen a lot during a full VM restore of the thin provisioned disk. If the storage supports the call, in theory this should be faster, certainly in cases that are bandwidth constrained as otherwise the ESXi host actually writes zeros to the newly allocated segment prior to writing the actual data (you can see this extra traffic by watching the iSCSI network during a restore, it will generally be 2x the performance of the restore). Perhaps in cases where the bandwidth is not the bottleneck the added latency of waiting for the call to be returned actually makes the process slower as I believe it is a synchronous call that much be acknowledge before sending the actual data. It would be interesting to see the esxtop output of the ZERO parameters from a box that supports zeroing with this feature turned on and off.
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby yizhar » Sun Feb 23, 2014 4:44 pm

Hi.

I would like to add the following:

* I have opened a case with Dell Equallogic about the block zero performance problem with VMware.
Here is the case number:
SR# 877759965
It escalated to Level 2 support, but strangely they were unable to reproduce problem at their labs.
So if you can open another case and refer to mine, it might help them (eqaullogic support) better understand the scale of the problem.

* To whoever experience similar problems, please open a VMware case in addition to storage vendor.

* The problems are not directly related to Veeam restore, but to any operation that needs to write data to not yet allocated VMFS disk space. This triggers the block zero VAAI primitive, but in small increments rather then bulk, and this is probably the cause for performance problem.
For example:
If I create a THICK EAGER ZERO VMDK - no problem it will block zero in seconds.
However if I create a THICK LAZY ZERO, or THIN VMDK, and write data to it (example = full format from within guest OS, or simply copy large file into it), this will cause zero operations in small increments and a noticeable performance hit.

I have noticed this with Dell Equallogic array, but it might also affect other vendors and models.

Yizhar
yizhar
Expert
 
Posts: 179
Liked: 48 times
Joined: Mon Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby dellock6 » Sun Feb 23, 2014 10:31 pm

Guys, just to try and isolate the problems, let me recap:
PGITDept: Equallogic (problems) + Nimble
Yizhar: Equallogic (problems) + 3Par, (no difference)

@PGITDept, I don't understand in your posts if you also tried your tests against the Nimble storage, and if the vaai on/off affects your restores. I'm going to try this week the same tests against ly HP StoreVirtual VSA cluster, I'll let you know the results. *If* numbers will stay the same regardless of VAAI status, and if the Nimble storage is going to perform the same, it would maybe mean it's an EQL problem more than VAAI libraries.

You both have the latest firmware revision on EQL?

Luca.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5047
Liked: 1330 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 16 guests