Slow Restore Speed - 27MB/s - Tips/Ideas?

VMware specific discussions

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby chjones » Tue May 12, 2015 10:39 pm

This is the exact same results we have seen Daryl. Running an unmap on a 16TB VV with maybe 5-6TB free before doing a restore takes too long.

What we are doing for now as a workaround is to create a new VV big enough for the restore, restore the VM to it via NBD (since SAN restores are a third the speed for us) and then using VMware to Storage vMotion the VM back to the original volume.

I tested a restore of a VM to a it's original 3PAR Thin VV and it ran at 12MB/sec. I then restored the same VM to a new 3PAR VV and it was over 150MB/sec. I then Storage vMotioned the VM back to the same original thin VV and based on the start and end times of the tasks in vCenter I was able to calculate that the 3PAR itself (since the 3PAR is VAAI capable) moved the VM at over 330MB/sec. This was with doing no unmaps.

It's crazy that the 3PAR can move the data that fast but if Veeam is restoring we get 12MB/sec.
chjones
Enthusiast
 
Posts: 83
Liked: 25 times
Joined: Tue Oct 30, 2012 7:53 pm
Full Name: Chris Jones

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby isaako » Mon Jun 15, 2015 8:27 am

Hi,

My restore is about 12 - 20 MB/sec.
All my Veeam components are connected to a 10Gb network and the backup source is idle.
I've also tested disabing and enabling /DataMover/HardwareAcceleratedInit

Veeam do you have a solution for this issue?

Isaac
isaako
Influencer
 
Posts: 19
Liked: never
Joined: Wed Sep 15, 2010 11:31 am
Full Name: Isaac González

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby foggy » Mon Jun 15, 2015 4:20 pm

Isaac, please contact technical support directly. We need to collect more information regarding this performance issue. Thanks!
foggy
Veeam Software
 
Posts: 14728
Liked: 1076 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby agrob » Wed Jul 01, 2015 2:18 pm

Are there any generall news on this issue? i'm experience the same problem
Restore to a existing Virtual Volume on a 3Par 7200 with hotadd mode works with about 12MB/s
Restore to a new Virtual Volume on the 3Par works with about 75MB/s. If i repeate restores to this Volume it slows down..
Restore to a Raid 10 Array with 4x 10k SAS Disk local on the ESX Host works with about 85MB/s

Thanks
agrob
Enthusiast
 
Posts: 90
Liked: 8 times
Joined: Mon Sep 05, 2011 1:31 pm
Full Name: Andre

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby foggy » Wed Jul 01, 2015 2:30 pm

No news as of yet, please contact support for a closer look at your environment.
foggy
Veeam Software
 
Posts: 14728
Liked: 1076 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby agrob » Thu Jul 02, 2015 7:20 am

Thanks foggy. I'll do some more tests and then open a case.
btw, when i restore i let disk type "same as source"
If i check disk type after restore i have "Thick Provisioning Lazy Zeroed" but original Server has "Thick Provisioning Eager Zeroed"....
agrob
Enthusiast
 
Posts: 90
Liked: 8 times
Joined: Mon Sep 05, 2011 1:31 pm
Full Name: Andre

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby foggy » Thu Jul 02, 2015 10:12 am

Yes, currently disks are always restored as Lazy Zeroed.
foggy
Veeam Software
 
Posts: 14728
Liked: 1076 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby agrob » Fri Jul 03, 2015 2:29 pm

i did a lot of testing so far and also changed the backup infrastructure a bit to which i think should give the best restore speed for your environment. For testing i used the following setup

Proxy/Repsoitory is the same VM -> hotadd mode
Source: Repository is a vmdk file on a SAN attached EVA 4400 Lun (Diskgroup with 36 x 15k Disk - there is nearly no other load on this dg)
Destination: is a empty datastore on a SAN attached 3par VV (7200 with 56 sas 10k disk)
Testvm to restore is a 2012 r2 vm with a 60GB Disk C and a 10GB Disk E

if i start the restore now, the proxy/repository vm reads the data over fc from EVA System and writes it over FC (vmware io stack) to the datastore on the 3par. Performance is about 60-70MB/s! In my opinion this should be much faster because if i do storage v-motion beteween EVA and 3Par i get about 200-300MB/s or more. Also there is no 1 gbit nic involved which could be a bottleneck (only the veeam br server is on another location which is 1gbit connected but this should not matter because restore traffic is "inside" the prox/repository vm)
btw if i restore the same vm to another datastore where other vms are running, performance is worse (about 10 or 20mb/s)

there are other threads in this forum affecting restore performance problems. Do we have a generall problem here?
thanks
agrob
Enthusiast
 
Posts: 90
Liked: 8 times
Joined: Mon Sep 05, 2011 1:31 pm
Full Name: Andre

[MERGED] Slow restore rate

Veeam Logoby Christian33 » Fri Jul 10, 2015 7:09 am

Hello together,

we have a slow restorerate (about 16 MB/s). We use a physical server as backup server. The server has all veeam roles (proxy,..) and the backups are stored at the local harddrives.
This server is with 10GBit/s connected to the VMware ESX 6 hostserver.

Have you an idea what we need to change at the configuration ?

Thanks for your help.
Christian33
Influencer
 
Posts: 10
Liked: never
Joined: Mon Dec 29, 2014 8:01 am

Re: Slow restorerate

Veeam Logoby agrob » Fri Jul 10, 2015 7:37 am

Hi Christian

There are several Threads here at the moment regarding restore speed. I also have restore spead about 12MB/s if i restore to an existing datastore where other vms are running. I made a lot of tests (Restore NBD, Hotadd, changed Hosts etc). I get better restore rates (about 60-70MB/s - Better but not good) if i restore to a new empty datastore without any other vms running on it. Do you have the oportunity to create a new datastore and do a restore on it?

Regards
agrob
Enthusiast
 
Posts: 90
Liked: 8 times
Joined: Mon Sep 05, 2011 1:31 pm
Full Name: Andre

Re: Slow restorerate

Veeam Logoby Christian33 » Fri Jul 10, 2015 8:41 am

Hi agrob,

i have tested the restore to a datastore without running vms. The datastore (local harddrives) is at a esx host server. It is the same result as with the restore to the san.
Christian33
Influencer
 
Posts: 10
Liked: never
Joined: Mon Dec 29, 2014 8:01 am

Re: Slow restorerate

Veeam Logoby agrob » Fri Jul 10, 2015 10:29 am

do you have the possibility to create a new datastore from san storage and restore to this (formatted with latest vmfs)?
agrob
Enthusiast
 
Posts: 90
Liked: 8 times
Joined: Mon Sep 05, 2011 1:31 pm
Full Name: Andre

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby Shestakov » Fri Jul 10, 2015 10:55 am

Hello Christian,
I would try to make a restore in a hot-add mode. You need to grant a proxy role to one of VMs.
Please review the thread, it contains a lot of relevant information. Thanks!
Shestakov
Veeam Software
 
Posts: 4856
Liked: 394 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby chjones » Mon Jul 13, 2015 2:16 am 3 people like this post

My support case I had for this has concluded (Case 00911701). Veeam opened a case with VMware and a Veeam KB has been created as a result: http://www.veeam.com/kb2052.

The KB explains why the SAN restores are so slow. Basically it's because Veeam must constantly check with the ESXi host before every block is written to work out which block to write to on the disk next. This causes the slow down. If the restored VM was THICK EAGER then this wouldn't be an issue. When you use Network Mode for a restore the ESXi host preforms the write to the Datastore and it already has knowledge for the layout of the VMDK so doesn't have to keep checking which block to use next.

For now the best solution is to make sure your datastores are OFFLINE on your backup proxy and are not mounted. This allows a SAN mode backup to work, but a restore will failover to Network Mode. You get a warning during a restore that SAN mode is unavailable, but it then fails over network and the speeds are good.

It would be good if the restore wizard prompted which restore mode to use, but for now this is the best we have. At least Veeam and VMware have confirmed the cause and haven't found something we're all doing wrong.
chjones
Enthusiast
 
Posts: 83
Liked: 25 times
Joined: Tue Oct 30, 2012 7:53 pm
Full Name: Chris Jones

Re: Slow Restore Speed - 27MB/s - Tips/Ideas?

Veeam Logoby Christian33 » Wed Jul 15, 2015 9:45 am

Hi,

we not restore to the san.
We are using a physical backup server with local harddrives. The backup server has all veeam roles (proxy,..).
We restore a backup to a new datastore at local harddrives from a esx host. This esx host and the datastore has no running vms. This server are connected with 10GBit/s.

The restore speed is about 10 MB/s.

Must we create a vm as backup proxy at the esx host ?
Christian33
Influencer
 
Posts: 10
Liked: never
Joined: Mon Dec 29, 2014 8:01 am

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: Google [Bot] and 13 guests