Discussions specific to the VMware vSphere hypervisor
chjones
Enthusiast
Posts: 98
Liked: 26 times
Joined: Oct 30, 2012 7:53 pm
Full Name: Chris Jones
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by chjones » May 12, 2015 10:39 pm

This is the exact same results we have seen Daryl. Running an unmap on a 16TB VV with maybe 5-6TB free before doing a restore takes too long.

What we are doing for now as a workaround is to create a new VV big enough for the restore, restore the VM to it via NBD (since SAN restores are a third the speed for us) and then using VMware to Storage vMotion the VM back to the original volume.

I tested a restore of a VM to a it's original 3PAR Thin VV and it ran at 12MB/sec. I then restored the same VM to a new 3PAR VV and it was over 150MB/sec. I then Storage vMotioned the VM back to the same original thin VV and based on the start and end times of the tasks in vCenter I was able to calculate that the 3PAR itself (since the 3PAR is VAAI capable) moved the VM at over 330MB/sec. This was with doing no unmaps.

It's crazy that the 3PAR can move the data that fast but if Veeam is restoring we get 12MB/sec.

isaako
Service Provider
Posts: 25
Liked: never
Joined: Sep 15, 2010 11:31 am
Full Name: Isaac González
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by isaako » Jun 15, 2015 8:27 am

Hi,

My restore is about 12 - 20 MB/sec.
All my Veeam components are connected to a 10Gb network and the backup source is idle.
I've also tested disabing and enabling /DataMover/HardwareAcceleratedInit

Veeam do you have a solution for this issue?

Isaac

foggy
Veeam Software
Posts: 16841
Liked: 1361 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by foggy » Jun 15, 2015 4:20 pm

Isaac, please contact technical support directly. We need to collect more information regarding this performance issue. Thanks!

agrob
Expert
Posts: 140
Liked: 12 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by agrob » Jul 01, 2015 2:18 pm

Are there any generall news on this issue? i'm experience the same problem
Restore to a existing Virtual Volume on a 3Par 7200 with hotadd mode works with about 12MB/s
Restore to a new Virtual Volume on the 3Par works with about 75MB/s. If i repeate restores to this Volume it slows down..
Restore to a Raid 10 Array with 4x 10k SAS Disk local on the ESX Host works with about 85MB/s

Thanks

foggy
Veeam Software
Posts: 16841
Liked: 1361 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by foggy » Jul 01, 2015 2:30 pm

No news as of yet, please contact support for a closer look at your environment.

agrob
Expert
Posts: 140
Liked: 12 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by agrob » Jul 02, 2015 7:20 am

Thanks foggy. I'll do some more tests and then open a case.
btw, when i restore i let disk type "same as source"
If i check disk type after restore i have "Thick Provisioning Lazy Zeroed" but original Server has "Thick Provisioning Eager Zeroed"....

foggy
Veeam Software
Posts: 16841
Liked: 1361 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by foggy » Jul 02, 2015 10:12 am

Yes, currently disks are always restored as Lazy Zeroed.

agrob
Expert
Posts: 140
Liked: 12 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by agrob » Jul 03, 2015 2:29 pm

i did a lot of testing so far and also changed the backup infrastructure a bit to which i think should give the best restore speed for your environment. For testing i used the following setup

Proxy/Repsoitory is the same VM -> hotadd mode
Source: Repository is a vmdk file on a SAN attached EVA 4400 Lun (Diskgroup with 36 x 15k Disk - there is nearly no other load on this dg)
Destination: is a empty datastore on a SAN attached 3par VV (7200 with 56 sas 10k disk)
Testvm to restore is a 2012 r2 vm with a 60GB Disk C and a 10GB Disk E

if i start the restore now, the proxy/repository vm reads the data over fc from EVA System and writes it over FC (vmware io stack) to the datastore on the 3par. Performance is about 60-70MB/s! In my opinion this should be much faster because if i do storage v-motion beteween EVA and 3Par i get about 200-300MB/s or more. Also there is no 1 gbit nic involved which could be a bottleneck (only the veeam br server is on another location which is 1gbit connected but this should not matter because restore traffic is "inside" the prox/repository vm)
btw if i restore the same vm to another datastore where other vms are running, performance is worse (about 10 or 20mb/s)

there are other threads in this forum affecting restore performance problems. Do we have a generall problem here?
thanks

Christian33
Influencer
Posts: 10
Liked: never
Joined: Dec 29, 2014 8:01 am
Contact:

[MERGED] Slow restore rate

Post by Christian33 » Jul 10, 2015 7:09 am

Hello together,

we have a slow restorerate (about 16 MB/s). We use a physical server as backup server. The server has all veeam roles (proxy,..) and the backups are stored at the local harddrives.
This server is with 10GBit/s connected to the VMware ESX 6 hostserver.

Have you an idea what we need to change at the configuration ?

Thanks for your help.

agrob
Expert
Posts: 140
Liked: 12 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: Slow restorerate

Post by agrob » Jul 10, 2015 7:37 am

Hi Christian

There are several Threads here at the moment regarding restore speed. I also have restore spead about 12MB/s if i restore to an existing datastore where other vms are running. I made a lot of tests (Restore NBD, Hotadd, changed Hosts etc). I get better restore rates (about 60-70MB/s - Better but not good) if i restore to a new empty datastore without any other vms running on it. Do you have the oportunity to create a new datastore and do a restore on it?

Regards

Christian33
Influencer
Posts: 10
Liked: never
Joined: Dec 29, 2014 8:01 am
Contact:

Re: Slow restorerate

Post by Christian33 » Jul 10, 2015 8:41 am

Hi agrob,

i have tested the restore to a datastore without running vms. The datastore (local harddrives) is at a esx host server. It is the same result as with the restore to the san.

agrob
Expert
Posts: 140
Liked: 12 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: Slow restorerate

Post by agrob » Jul 10, 2015 10:29 am

do you have the possibility to create a new datastore from san storage and restore to this (formatted with latest vmfs)?

Shestakov
Veeam Software
Posts: 5973
Liked: 521 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by Shestakov » Jul 10, 2015 10:55 am

Hello Christian,
I would try to make a restore in a hot-add mode. You need to grant a proxy role to one of VMs.
Please review the thread, it contains a lot of relevant information. Thanks!

chjones
Enthusiast
Posts: 98
Liked: 26 times
Joined: Oct 30, 2012 7:53 pm
Full Name: Chris Jones
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by chjones » Jul 13, 2015 2:16 am 3 people like this post

My support case I had for this has concluded (Case 00911701). Veeam opened a case with VMware and a Veeam KB has been created as a result: http://www.veeam.com/kb2052.

The KB explains why the SAN restores are so slow. Basically it's because Veeam must constantly check with the ESXi host before every block is written to work out which block to write to on the disk next. This causes the slow down. If the restored VM was THICK EAGER then this wouldn't be an issue. When you use Network Mode for a restore the ESXi host preforms the write to the Datastore and it already has knowledge for the layout of the VMDK so doesn't have to keep checking which block to use next.

For now the best solution is to make sure your datastores are OFFLINE on your backup proxy and are not mounted. This allows a SAN mode backup to work, but a restore will failover to Network Mode. You get a warning during a restore that SAN mode is unavailable, but it then fails over network and the speeds are good.

It would be good if the restore wizard prompted which restore mode to use, but for now this is the best we have. At least Veeam and VMware have confirmed the cause and haven't found something we're all doing wrong.

Christian33
Influencer
Posts: 10
Liked: never
Joined: Dec 29, 2014 8:01 am
Contact:

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Post by Christian33 » Jul 15, 2015 9:45 am

Hi,

we not restore to the san.
We are using a physical backup server with local harddrives. The backup server has all veeam roles (proxy,..).
We restore a backup to a new datastore at local harddrives from a esx host. This esx host and the datastore has no running vms. This server are connected with 10GBit/s.

The restore speed is about 10 MB/s.

Must we create a vm as backup proxy at the esx host ?

Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 26 guests