Slow Restore Speed - 27MB/s - Tips/Ideas?

VMware specific discussions

Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby moey » Mon Jul 23, 2012 2:40 pm

What is the average restore speed that everyone is noticing?

We recently setup a few servers out at our Colo, and was doing some testing recovering VMs from our production site. The backup files are located on a 8 disk array connected to the Veeam VM (Win 7 x64 via MS iSCSI initatior, NTFS, 10GbE). We are restoring to a 16 disk array (iSCSI connection to vSphere, VMFS, 10GbE). Our recovery times seem to get stuck at 27MB/s.


Currently the host that the Veeam VM is running on is not on 10GbE, but I figured we could atleast saturate a 1gb link with a restore. Performance testing (iometer) has been ran on both storage devices and can easily saturate a 1gb pipe.

Specs for the Veeam VM are 4 vCPU and 8gb memory. No cpu/memory contention on host that is running the Veeam VM. vCPU and memory usage are around 50% during a restore.

Anyone have any tips/ideas for why I can't get these restores moving faster? Recovering larger VMs (1tb+) is becoming a pain at this rate.
moey
Novice
 
Posts: 4
Liked: never
Joined: Sat Apr 28, 2012 1:18 am
Full Name: Moey

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby moey » Mon Jul 23, 2012 3:32 pm

Adding some more details:


My Setup:
- Veeam Proxy on a VM - 4 vCPU, 8gb mem, Veeam 6.1
- Veeam Backup Repository - 8 disk raid 5, 10GbE, NTFS, connected to Veeam Proxy via MS iSCSI initiator
- ESXi 4.1 U2
- vSphere datastore - 16 disk raid 10, 10GbE, VMFS, VMware iSCSI initiator
moey
Novice
 
Posts: 4
Liked: never
Joined: Sat Apr 28, 2012 1:18 am
Full Name: Moey

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby foggy » Tue Jul 24, 2012 9:30 am

For fast full VM restores, we recommend using hotadd restores via virtual backup proxy. Make sure that proxy capable of hotadd is picked up during restore (information available in the restore session log).
foggy
Veeam Software
 
Posts: 14743
Liked: 1081 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby Vitaliy S. » Tue Jul 24, 2012 10:47 am

... or use Instant VM Recovery and Migration jobs to move VM data back to production storage.
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby moey » Tue Jul 24, 2012 2:16 pm

foggy wrote:For fast full VM restores, we recommend using hotadd restores via virtual backup proxy. Make sure that proxy capable of hotadd is picked up during restore (information available in the restore session log).


Can you expand a little more on this? The virtual backup proxy does have access to all the datastores that the VMs are being restored to.

Vitaliy S. wrote:... or use Instant VM Recovery and Migration jobs to move VM data back to production storage.


After doing some digging yesterday, I came across a post mentioning this method. We are not licensed for SvMotion for our DR site, so it would have to be an offline migration.

Any reason why doing an instant restore then storage migration would be quicker than a full Veeam restore?
moey
Novice
 
Posts: 4
Liked: never
Joined: Sat Apr 28, 2012 1:18 am
Full Name: Moey

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby Vitaliy S. » Tue Jul 24, 2012 10:29 pm

1. If the backup proxy can access datastores you're restoring VMs to, it will automatically attempt to restore selected VMs via ESXi I/O stack directly (Hot Add mode). This approach is a way faster method compared to VM restores that are performed over the network stack.

2. Yes, but using Instant VM Recovery with a Quick Migration functinality (even without Storage vMotion licensed) will allow you to minimize the "non-working" hours for this VM. Please take a look at our User Guide (page 54) for additional details of this process.

Hope this helps!
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby foggy » Wed Jul 25, 2012 6:47 am

Vitaliy S. wrote:1. If the backup proxy can access datastores you're restoring VMs to, it will automatically attempt to restore selected VMs via ESXi I/O stack directly (Hot Add mode). This approach is a way faster method compared to VM restores that are performed over the network stack.

Open the session log (right-click the restore session and select the Log tab) and find the "Using target proxy..." record. Check whether it has the [hotadd] label after the name of the proxy. That means that the hotadd transport mode is effectively used to populate VM data on the target host.
foggy
Veeam Software
 
Posts: 14743
Liked: 1081 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby moey » Wed Jul 25, 2012 2:47 pm

I just checked our restore jobs, and it indeed is using Hot Add Mode.

I will be doing some testing of Instant Recovery with Quick Migration.

Thanks for all the responses. I'll post some updates after I have tested.
moey
Novice
 
Posts: 4
Liked: never
Joined: Sat Apr 28, 2012 1:18 am
Full Name: Moey

Restoring. Faster alternative to NBDSSL?

Veeam Logoby trentl » Thu Aug 09, 2012 9:38 am

[merged]

Hi,
We're restoring a couple of servers from backup.
The proxy is restoring using NBD SSL.
Is there any method faster for restoring VM's?

It backs up using SAN. So I was just curious.

Trent
trentl
Novice
 
Posts: 8
Liked: never
Joined: Tue Jul 03, 2012 11:56 pm
Full Name: Trent Lane

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby trentl » Thu Aug 09, 2012 11:19 am

Sorry,
I'm not hijacking your thread. My post was merged with this.
I just wanted to clarify my configuration.
We have 4 blades
3 are ESXi hosts
The 4th runs Server 2k8r2. It runs Veeam as a stand alone server (it's the backup proxy)
The 2k8 blade has access to the LUN's used by the 3 hosts for VMs so is able to access backup across via SAN.

Does my backup proxy need to be a VM for Hot Add to work?
trentl
Novice
 
Posts: 8
Liked: never
Joined: Tue Jul 03, 2012 11:56 pm
Full Name: Trent Lane

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby Gostev » Thu Aug 09, 2012 11:26 am 1 person likes this post

No, you can keep your proxy, but also create additional virtual proxy just for the purposes of hot add restores. Kindly please review the sticky FAQ topic for all information regarding hot add mode and its requirements. Thanks!
Gostev
Veeam Software
 
Posts: 21396
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby trentl » Thu Aug 09, 2012 11:57 am

Thanks.
I did read the FAQ just needed that extra bit of info.
Got it going, Proxy is hotadd
Thanks again.
trentl
Novice
 
Posts: 8
Liked: never
Joined: Tue Jul 03, 2012 11:56 pm
Full Name: Trent Lane

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby kte » Mon Oct 07, 2013 3:48 pm

Still slow here after adding a virtual proxy
kte
Expert
 
Posts: 172
Liked: 7 times
Joined: Tue Jul 02, 2013 7:48 pm
Full Name: Koen Teugels

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby yizhar » Mon Oct 07, 2013 9:10 pm

Hi.

I suggest that you also try disabling VMware vasa block zero function on the esxi host,
then test if it makes any difference:

VMware KB Disabling the VAAI functionality in ESXi-ESX
http://kb.vmware.com/selfservice/micros ... Id=1033665

See step 7 below:
7.Change the DataMover.HardwareAcceleratedInit setting to 0.

To disable VAAI using the vSphere Client:
1.Open the VMware vSphere Client.
2.In the Inventory pane, select the ESXi/ESX host.
3.Click the Configuration tab.
4.Under Software, click Advanced Settings.
5.Click DataMover.
6.Change the DataMover.HardwareAcceleratedMove setting to 0.
7.Change the DataMover.HardwareAcceleratedInit setting to 0.
8.Click VMFS3.
9.Change the VMFS3.HardwareAcceleratedLocking setting to 0.
10.Click OK to save your changes.
11.Repeat this process for the all ESXi/ESX hosts connected to the storage.

Please tell us if it changes anything for good or bad.

Yizhar
yizhar
Expert
 
Posts: 179
Liked: 48 times
Joined: Mon Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby yizhar » Mon Oct 07, 2013 9:14 pm

BTW,

When changing the DataMover.HardwareAcceleratedInit setting,
no restart or further action is needed - so you can easily and quickly experience results.

Yizhar
yizhar
Expert
 
Posts: 179
Liked: 48 times
Joined: Mon Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz

Next

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 11 guests