Host-based backup of VMware vSphere VMs.
moey
Novice
Posts: 4
Liked: never
Joined: Apr 28, 2012 1:18 am
Full Name: Moey
Contact:

Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by moey »

What is the average restore speed that everyone is noticing?

We recently setup a few servers out at our Colo, and was doing some testing recovering VMs from our production site. The backup files are located on a 8 disk array connected to the Veeam VM (Win 7 x64 via MS iSCSI initatior, NTFS, 10GbE). We are restoring to a 16 disk array (iSCSI connection to vSphere, VMFS, 10GbE). Our recovery times seem to get stuck at 27MB/s.


Currently the host that the Veeam VM is running on is not on 10GbE, but I figured we could atleast saturate a 1gb link with a restore. Performance testing (iometer) has been ran on both storage devices and can easily saturate a 1gb pipe.

Specs for the Veeam VM are 4 vCPU and 8gb memory. No cpu/memory contention on host that is running the Veeam VM. vCPU and memory usage are around 50% during a restore.

Anyone have any tips/ideas for why I can't get these restores moving faster? Recovering larger VMs (1tb+) is becoming a pain at this rate.
moey
Novice
Posts: 4
Liked: never
Joined: Apr 28, 2012 1:18 am
Full Name: Moey
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by moey »

Adding some more details:


My Setup:
- Veeam Proxy on a VM - 4 vCPU, 8gb mem, Veeam 6.1
- Veeam Backup Repository - 8 disk raid 5, 10GbE, NTFS, connected to Veeam Proxy via MS iSCSI initiator
- ESXi 4.1 U2
- vSphere datastore - 16 disk raid 10, 10GbE, VMFS, VMware iSCSI initiator
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by foggy »

For fast full VM restores, we recommend using hotadd restores via virtual backup proxy. Make sure that proxy capable of hotadd is picked up during restore (information available in the restore session log).
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by Vitaliy S. »

... or use Instant VM Recovery and Migration jobs to move VM data back to production storage.
moey
Novice
Posts: 4
Liked: never
Joined: Apr 28, 2012 1:18 am
Full Name: Moey
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by moey »

foggy wrote:For fast full VM restores, we recommend using hotadd restores via virtual backup proxy. Make sure that proxy capable of hotadd is picked up during restore (information available in the restore session log).
Can you expand a little more on this? The virtual backup proxy does have access to all the datastores that the VMs are being restored to.
Vitaliy S. wrote:... or use Instant VM Recovery and Migration jobs to move VM data back to production storage.
After doing some digging yesterday, I came across a post mentioning this method. We are not licensed for SvMotion for our DR site, so it would have to be an offline migration.

Any reason why doing an instant restore then storage migration would be quicker than a full Veeam restore?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by Vitaliy S. »

1. If the backup proxy can access datastores you're restoring VMs to, it will automatically attempt to restore selected VMs via ESXi I/O stack directly (Hot Add mode). This approach is a way faster method compared to VM restores that are performed over the network stack.

2. Yes, but using Instant VM Recovery with a Quick Migration functinality (even without Storage vMotion licensed) will allow you to minimize the "non-working" hours for this VM. Please take a look at our User Guide (page 54) for additional details of this process.

Hope this helps!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by foggy »

Vitaliy S. wrote:1. If the backup proxy can access datastores you're restoring VMs to, it will automatically attempt to restore selected VMs via ESXi I/O stack directly (Hot Add mode). This approach is a way faster method compared to VM restores that are performed over the network stack.
Open the session log (right-click the restore session and select the Log tab) and find the "Using target proxy..." record. Check whether it has the [hotadd] label after the name of the proxy. That means that the hotadd transport mode is effectively used to populate VM data on the target host.
moey
Novice
Posts: 4
Liked: never
Joined: Apr 28, 2012 1:18 am
Full Name: Moey
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by moey »

I just checked our restore jobs, and it indeed is using Hot Add Mode.

I will be doing some testing of Instant Recovery with Quick Migration.

Thanks for all the responses. I'll post some updates after I have tested.
trentl
Novice
Posts: 8
Liked: never
Joined: Jul 03, 2012 11:56 pm
Full Name: Trent Lane
Contact:

Restoring. Faster alternative to NBDSSL?

Post by trentl »

[merged]

Hi,
We're restoring a couple of servers from backup.
The proxy is restoring using NBD SSL.
Is there any method faster for restoring VM's?

It backs up using SAN. So I was just curious.

Trent
trentl
Novice
Posts: 8
Liked: never
Joined: Jul 03, 2012 11:56 pm
Full Name: Trent Lane
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by trentl »

Sorry,
I'm not hijacking your thread. My post was merged with this.
I just wanted to clarify my configuration.
We have 4 blades
3 are ESXi hosts
The 4th runs Server 2k8r2. It runs Veeam as a stand alone server (it's the backup proxy)
The 2k8 blade has access to the LUN's used by the 3 hosts for VMs so is able to access backup across via SAN.

Does my backup proxy need to be a VM for Hot Add to work?
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by Gostev » 1 person likes this post

No, you can keep your proxy, but also create additional virtual proxy just for the purposes of hot add restores. Kindly please review the sticky FAQ topic for all information regarding hot add mode and its requirements. Thanks!
trentl
Novice
Posts: 8
Liked: never
Joined: Jul 03, 2012 11:56 pm
Full Name: Trent Lane
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by trentl »

Thanks.
I did read the FAQ just needed that extra bit of info.
Got it going, Proxy is hotadd
Thanks again.
kte
Expert
Posts: 179
Liked: 8 times
Joined: Jul 02, 2013 7:48 pm
Full Name: Koen Teugels
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by kte »

Still slow here after adding a virtual proxy
yizhar
Service Provider
Posts: 181
Liked: 48 times
Joined: Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by yizhar »

Hi.

I suggest that you also try disabling VMware vasa block zero function on the esxi host,
then test if it makes any difference:

VMware KB Disabling the VAAI functionality in ESXi-ESX
http://kb.vmware.com/selfservice/micros ... Id=1033665

See step 7 below:
7.Change the DataMover.HardwareAcceleratedInit setting to 0.

To disable VAAI using the vSphere Client:
1.Open the VMware vSphere Client.
2.In the Inventory pane, select the ESXi/ESX host.
3.Click the Configuration tab.
4.Under Software, click Advanced Settings.
5.Click DataMover.
6.Change the DataMover.HardwareAcceleratedMove setting to 0.
7.Change the DataMover.HardwareAcceleratedInit setting to 0.
8.Click VMFS3.
9.Change the VMFS3.HardwareAcceleratedLocking setting to 0.
10.Click OK to save your changes.
11.Repeat this process for the all ESXi/ESX hosts connected to the storage.

Please tell us if it changes anything for good or bad.

Yizhar
yizhar
Service Provider
Posts: 181
Liked: 48 times
Joined: Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by yizhar »

BTW,

When changing the DataMover.HardwareAcceleratedInit setting,
no restart or further action is needed - so you can easily and quickly experience results.

Yizhar
yizhar
Service Provider
Posts: 181
Liked: 48 times
Joined: Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by yizhar »

More info:

I also have a 3par (but smaller machine - 3par 7200 with 12 sata disks in raid1 used for the test),
and I did try to restore a small test VM with about 50gb disk (most part of it empty as this is a test machine).

I got about 70 MB/s during restore, and about 8 minutes to restore 50gb vmdk.

I did try with and without block zero and got same speed, however I still thing you should test it.

This makes me think that one of the differences between my small test and your production VM, is the size of data - VBK size on repository and/or VMDK size on datastore.
Have you checked memory and cpu resources on the backup proxy during restore?

Can you try restoring a smaller VM for the test and compare results?

Yizhar
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by tsightler »

I do agree that this would be interesting to test.
kte
Expert
Posts: 179
Liked: 8 times
Joined: Jul 02, 2013 7:48 pm
Full Name: Koen Teugels
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by kte »

but veeam calculates his average on the 50 gb not on the 20 gb he restored realy from the 50 gb
So in my case veeam restore 2 tb in 2 minutes but the disk was empty, so no real way to mesure it.
so you also have 60 GB an hour, if you hade a bigger vm that was fulled it ould go slow also.
8 cpu and 32 gb of ram, so no issue there
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by tsightler »

The idea would be to test something smaller to see if it's something to do with the size of the VM being restored. For example, I can restore a 100GB VM in my home lab at about 60MB/s. This is 100GB of actual data, not a "fake" speed from restoring blank space. I have recently tested restores on some Netapp hardware in a vendor lab environment and performance was around 110MB/s for restores (real transfer speed). The idea would be to see if there was something impacting restores of very large VMs (perhaps an agent memory issue or something). You can easily measure the restore performance of the actual data by simply watching the bandwidth utilization, or better yet the datastore utilization in vCenter.
pgitdept
Influencer
Posts: 17
Liked: 14 times
Joined: Feb 03, 2011 10:29 am
Full Name: PGITDept
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by pgitdept » 6 people like this post

I just thought I'd post some of our findings regarding our slow restores:

We're running ESX 5.5 and Veeam 7R2. Our storage platforms are DELL EQL (4 arrays: 15k RAID10, 10k RAID10 and 2x7.2k RAID6) and a Nimble CS460G. Now we were testing restores from tape on a fairly smallish server here (1.3TB used of 4TB). We were restoring to a new datastore on the EQL SATA storage pool made up of two RAID6 arrays. We were getting about 15MB/s, which is pretty horrendous and it took 29hours to restore. Now that pool is fairly heavily used, but should be giving better performance than that. So we built a smaller VM of around 150GB and tested that - the results were pretty much the same.

We figured it might the storage pool and so we moved it to our 15k RAID10 storage pool. That got to the dizzy heights of 42MB/s. So, we're starting to panic now as it would take us over a month to restore from a full failure - not good. We decided to restore to the Nimble and got the dizzy heights of 80MB/s, not brilliant, but more acceptable. So, we started doing some deeper tests and noticed that our write rates to some VM's were also fairly poor, although it was hard to try and pin it down.

Anyway, I found this thread and started working through it - Veeam restores were consistently poor so we could use that as a measuring stick. We tried HotAdd and other mechanisms and found it to make no difference and were fairly convinced it was more global an issue than Veeam.

So, we got to the post by Yizhar about disabling VAAI functionality on the host. We run vSphere 5.5 Standard edition, so we don't have VAAI support even though our Nimble and EQL support it to differing degrees.

BOOM - our restore to the SATA pool went up to 90MB/s+, the 15K pool around 120MB/s and the Nimble clocked a restore of nearly 200MB/s which is amazing considering it's a large Sequential Write - the workload the Nimble hates chewing on the most.

So, we're still testing this on a spare host, but it appears that disabling VAAI has made a considerable difference to all our systems including restoration operations from Veeam. It looks like ESX has VAAI enabled by default, although I'm not sure what affect it has if you're not licensed to use it. Surely not licensed means disabled?

Anyway, it's made a big difference to us and we'll continue testing. I just wanted to say thanks for the post Yizhar!
Frosty
Expert
Posts: 200
Liked: 43 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by Frosty »

Really interesting discussion guys!

Can someone perhaps elaborate on what other operations (i.e. other than Veeam Restores) are likely to benefit from this change? Any possible net negatives I should consider?
Frosty
Expert
Posts: 200
Liked: 43 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by Frosty »

Digging further, I found this article:
http://kb.vmware.com/selfservice/micros ... Id=1021976

Ran this command: esxcli storage core device vaai status get

Returns me a display like this for each LUN on my Dell MD3200:

naa.6842b2b0004665ba000003964d1101b2
VAAI Plugin Name:
ATS Status: unsupported
Clone Status: unsupported
Zero Status: supported
Delete Status: unsupported
pgitdept
Influencer
Posts: 17
Liked: 14 times
Joined: Feb 03, 2011 10:29 am
Full Name: PGITDept
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by pgitdept »

So we're still doing testing but we have found that it's definitely the Block Zero setting (DataMover.HardwareAcceleratedInit) that is affecting the restore speed. On the 15K R10 shelf yesterday we started a restore with it turned on and got 45MB/s, during the restore we switched it off and it moved to about 150MB/s before turning it back on again and watching it drop back to ~42MB/s.

We're still doing tests and will be doing more on Monday. Anything of interest, I'll post.
ryan1212
Influencer
Posts: 16
Liked: 6 times
Joined: Feb 26, 2013 3:48 pm
Full Name: Ryan
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by ryan1212 » 2 people like this post

I can confirm that at least parts of the VAAI instruction set are used regardless of the ESXi license level. (I had a recent support case with VMware that involved disabling as a troubleshooting step, even though we have standard)
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by dellock6 »

Guys, these are really interesting findings from you, thanks for sharing them.
This definitely needs further investigations and testings to address the situation, and confirm it's a constant behaviour. Please keep updating this thread as you go on in your tests. Thanks!

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
pgitdept
Influencer
Posts: 17
Liked: 14 times
Joined: Feb 03, 2011 10:29 am
Full Name: PGITDept
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by pgitdept »

Okay, so we built a new 5.5 host and tested it on a eval license to give us Enterprise functionality with full VAAI support. It performed exactly the same as our testing with Standard: Poor restore speeds until we turn off the the Block Zero setting (DataMover.HardwareAcceleratedInit) and then it's between 3-4 times faster. I'm still unable to ascertain if I'm getting a performance hit elsewhere for disabling this.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by dellock6 » 2 people like this post

Hi,
as the name implies, the "block zero" primitive is involved when vSphere needs to write zeroes in the storage. It's also called write same. Possible scenarios where this library is called are the creation of thick disks. And if you think, sounds correct, a restore of a VMDK implies zeroing out the disk regions where the VMDK itself is going to be restored.
This is for sure something I'd like to test in my lab once I finished to upgrade to 5.5

PS: block zero is NOT involved in cloning, the library in this case is HardwareAcceleratedMove, and the VAAI type is "Clone Blocks/Full Copy/XCOPY"

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by tsightler »

Block zeroing/WRITE_SAME is used with thin disk as new segments are allocated, which of course will happen a lot during a full VM restore of the thin provisioned disk. If the storage supports the call, in theory this should be faster, certainly in cases that are bandwidth constrained as otherwise the ESXi host actually writes zeros to the newly allocated segment prior to writing the actual data (you can see this extra traffic by watching the iSCSI network during a restore, it will generally be 2x the performance of the restore). Perhaps in cases where the bandwidth is not the bottleneck the added latency of waiting for the call to be returned actually makes the process slower as I believe it is a synchronous call that much be acknowledge before sending the actual data. It would be interesting to see the esxtop output of the ZERO parameters from a box that supports zeroing with this feature turned on and off.
yizhar
Service Provider
Posts: 181
Liked: 48 times
Joined: Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by yizhar »

Hi.

I would like to add the following:

* I have opened a case with Dell Equallogic about the block zero performance problem with VMware.
Here is the case number:
SR# 877759965
It escalated to Level 2 support, but strangely they were unable to reproduce problem at their labs.
So if you can open another case and refer to mine, it might help them (eqaullogic support) better understand the scale of the problem.

* To whoever experience similar problems, please open a VMware case in addition to storage vendor.

* The problems are not directly related to Veeam restore, but to any operation that needs to write data to not yet allocated VMFS disk space. This triggers the block zero VAAI primitive, but in small increments rather then bulk, and this is probably the cause for performance problem.
For example:
If I create a THICK EAGER ZERO VMDK - no problem it will block zero in seconds.
However if I create a THICK LAZY ZERO, or THIN VMDK, and write data to it (example = full format from within guest OS, or simply copy large file into it), this will cause zero operations in small increments and a noticeable performance hit.

I have noticed this with Dell Equallogic array, but it might also affect other vendors and models.

Yizhar
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by dellock6 »

Guys, just to try and isolate the problems, let me recap:
PGITDept: Equallogic (problems) + Nimble
Yizhar: Equallogic (problems) + 3Par, (no difference)

@PGITDept, I don't understand in your posts if you also tried your tests against the Nimble storage, and if the vaai on/off affects your restores. I'm going to try this week the same tests against ly HP StoreVirtual VSA cluster, I'll let you know the results. *If* numbers will stay the same regardless of VAAI status, and if the Nimble storage is going to perform the same, it would maybe mean it's an EQL problem more than VAAI libraries.

You both have the latest firmware revision on EQL?

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Post Reply

Who is online

Users browsing this forum: No registered users and 110 guests