Discussions specific to the VMware vSphere hypervisor
moey
Novice
Posts: 4
Liked: never
Joined: Apr 28, 2012 1:18 am
Full Name: Moey
Contact:

Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by moey » Jul 23, 2012 2:40 pm

What is the average restore speed that everyone is noticing?

We recently setup a few servers out at our Colo, and was doing some testing recovering VMs from our production site. The backup files are located on a 8 disk array connected to the Veeam VM (Win 7 x64 via MS iSCSI initatior, NTFS, 10GbE). We are restoring to a 16 disk array (iSCSI connection to vSphere, VMFS, 10GbE). Our recovery times seem to get stuck at 27MB/s.


Currently the host that the Veeam VM is running on is not on 10GbE, but I figured we could atleast saturate a 1gb link with a restore. Performance testing (iometer) has been ran on both storage devices and can easily saturate a 1gb pipe.

Specs for the Veeam VM are 4 vCPU and 8gb memory. No cpu/memory contention on host that is running the Veeam VM. vCPU and memory usage are around 50% during a restore.

Anyone have any tips/ideas for why I can't get these restores moving faster? Recovering larger VMs (1tb+) is becoming a pain at this rate.

moey
Novice
Posts: 4
Liked: never
Joined: Apr 28, 2012 1:18 am
Full Name: Moey
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by moey » Jul 23, 2012 3:32 pm

Adding some more details:


My Setup:
- Veeam Proxy on a VM - 4 vCPU, 8gb mem, Veeam 6.1
- Veeam Backup Repository - 8 disk raid 5, 10GbE, NTFS, connected to Veeam Proxy via MS iSCSI initiator
- ESXi 4.1 U2
- vSphere datastore - 16 disk raid 10, 10GbE, VMFS, VMware iSCSI initiator

foggy
Veeam Software
Posts: 17712
Liked: 1481 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by foggy » Jul 24, 2012 9:30 am

For fast full VM restores, we recommend using hotadd restores via virtual backup proxy. Make sure that proxy capable of hotadd is picked up during restore (information available in the restore session log).

Vitaliy S.
Product Manager
Posts: 22289
Liked: 1413 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by Vitaliy S. » Jul 24, 2012 10:47 am

... or use Instant VM Recovery and Migration jobs to move VM data back to production storage.

moey
Novice
Posts: 4
Liked: never
Joined: Apr 28, 2012 1:18 am
Full Name: Moey
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by moey » Jul 24, 2012 2:16 pm

foggy wrote:For fast full VM restores, we recommend using hotadd restores via virtual backup proxy. Make sure that proxy capable of hotadd is picked up during restore (information available in the restore session log).
Can you expand a little more on this? The virtual backup proxy does have access to all the datastores that the VMs are being restored to.
Vitaliy S. wrote:... or use Instant VM Recovery and Migration jobs to move VM data back to production storage.
After doing some digging yesterday, I came across a post mentioning this method. We are not licensed for SvMotion for our DR site, so it would have to be an offline migration.

Any reason why doing an instant restore then storage migration would be quicker than a full Veeam restore?

Vitaliy S.
Product Manager
Posts: 22289
Liked: 1413 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by Vitaliy S. » Jul 24, 2012 10:29 pm

1. If the backup proxy can access datastores you're restoring VMs to, it will automatically attempt to restore selected VMs via ESXi I/O stack directly (Hot Add mode). This approach is a way faster method compared to VM restores that are performed over the network stack.

2. Yes, but using Instant VM Recovery with a Quick Migration functinality (even without Storage vMotion licensed) will allow you to minimize the "non-working" hours for this VM. Please take a look at our User Guide (page 54) for additional details of this process.

Hope this helps!

foggy
Veeam Software
Posts: 17712
Liked: 1481 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by foggy » Jul 25, 2012 6:47 am

Vitaliy S. wrote:1. If the backup proxy can access datastores you're restoring VMs to, it will automatically attempt to restore selected VMs via ESXi I/O stack directly (Hot Add mode). This approach is a way faster method compared to VM restores that are performed over the network stack.
Open the session log (right-click the restore session and select the Log tab) and find the "Using target proxy..." record. Check whether it has the [hotadd] label after the name of the proxy. That means that the hotadd transport mode is effectively used to populate VM data on the target host.

moey
Novice
Posts: 4
Liked: never
Joined: Apr 28, 2012 1:18 am
Full Name: Moey
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by moey » Jul 25, 2012 2:47 pm

I just checked our restore jobs, and it indeed is using Hot Add Mode.

I will be doing some testing of Instant Recovery with Quick Migration.

Thanks for all the responses. I'll post some updates after I have tested.

trentl
Novice
Posts: 8
Liked: never
Joined: Jul 03, 2012 11:56 pm
Full Name: Trent Lane
Contact:

Restoring. Faster alternative to NBDSSL?

Post by trentl » Aug 09, 2012 9:38 am

[merged]

Hi,
We're restoring a couple of servers from backup.
The proxy is restoring using NBD SSL.
Is there any method faster for restoring VM's?

It backs up using SAN. So I was just curious.

Trent

trentl
Novice
Posts: 8
Liked: never
Joined: Jul 03, 2012 11:56 pm
Full Name: Trent Lane
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by trentl » Aug 09, 2012 11:19 am

Sorry,
I'm not hijacking your thread. My post was merged with this.
I just wanted to clarify my configuration.
We have 4 blades
3 are ESXi hosts
The 4th runs Server 2k8r2. It runs Veeam as a stand alone server (it's the backup proxy)
The 2k8 blade has access to the LUN's used by the 3 hosts for VMs so is able to access backup across via SAN.

Does my backup proxy need to be a VM for Hot Add to work?

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by Gostev » Aug 09, 2012 11:26 am 1 person likes this post

No, you can keep your proxy, but also create additional virtual proxy just for the purposes of hot add restores. Kindly please review the sticky FAQ topic for all information regarding hot add mode and its requirements. Thanks!

trentl
Novice
Posts: 8
Liked: never
Joined: Jul 03, 2012 11:56 pm
Full Name: Trent Lane
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by trentl » Aug 09, 2012 11:57 am

Thanks.
I did read the FAQ just needed that extra bit of info.
Got it going, Proxy is hotadd
Thanks again.

kte
Expert
Posts: 172
Liked: 7 times
Joined: Jul 02, 2013 7:48 pm
Full Name: Koen Teugels
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by kte » Oct 07, 2013 3:48 pm

Still slow here after adding a virtual proxy

yizhar
Service Provider
Posts: 179
Liked: 48 times
Joined: Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by yizhar » Oct 07, 2013 9:10 pm

Hi.

I suggest that you also try disabling VMware vasa block zero function on the esxi host,
then test if it makes any difference:

VMware KB Disabling the VAAI functionality in ESXi-ESX
http://kb.vmware.com/selfservice/micros ... Id=1033665

See step 7 below:
7.Change the DataMover.HardwareAcceleratedInit setting to 0.

To disable VAAI using the vSphere Client:
1.Open the VMware vSphere Client.
2.In the Inventory pane, select the ESXi/ESX host.
3.Click the Configuration tab.
4.Under Software, click Advanced Settings.
5.Click DataMover.
6.Change the DataMover.HardwareAcceleratedMove setting to 0.
7.Change the DataMover.HardwareAcceleratedInit setting to 0.
8.Click VMFS3.
9.Change the VMFS3.HardwareAcceleratedLocking setting to 0.
10.Click OK to save your changes.
11.Repeat this process for the all ESXi/ESX hosts connected to the storage.

Please tell us if it changes anything for good or bad.

Yizhar

yizhar
Service Provider
Posts: 179
Liked: 48 times
Joined: Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by yizhar » Oct 07, 2013 9:14 pm

BTW,

When changing the DataMover.HardwareAcceleratedInit setting,
no restart or further action is needed - so you can easily and quickly experience results.

Yizhar

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 6 guests