Direct SAN restore speed

Availability for the Always-On Enterprise

Direct SAN restore speed

Veeam Logoby xy16644 » Thu Dec 22, 2016 9:55 pm

I am using Veeam 9.5 and it is installed on my Windows Server 2016 server. This server is also my SAN running Starwind Virtual SAN and is an iSCSI target for my two ESXi 6 hosts.

I have setup Direct SAN access for my Veeam backups and the speeds are just spectacular! I get 300-400MB/s speeds when the backups happen and I am very happy :D

The problem is when I do an actual restore. The other day I had to restore my entire environment so was restoring all my VMs (one at a time) and got speeds between 25-50MB/s which I thought was really slow? Is this normal? I'm using 1Gb for my management and iSCSI network so I was expecting to get at least 90-110MB/s?

Is there anything i can do to speed up Veeam restores? Is Direct SAN writes possible?

Thank you.
xy16644
Novice
 
Posts: 7
Liked: never
Joined: Fri Dec 04, 2015 10:37 pm
Full Name: SW

Re: Direct SAN restore speed

Veeam Logoby marco.horstmann » Thu Dec 22, 2016 10:16 pm

Hi,
backup is normally faster than restore. We had some weeks ago 2GB/s for backup and 1.2GB/s restore. ;-)

Restore speed can be limited in some factors:
1) thin provisioned VM disks needs more management overhead in Direct SAN Restore. This should be avoided.
One option is here to use a virtual HotAdd Proxy and using it for restoring VMs. Maybe you have thin disks?
2) The storagesystem write performance.

See this help page: https://helpcenter.veeam.com/backup/vsp ... iting.html

Regards
Marco
Marco Horstmann
System Engineer @ Veeam Software

@marcohorstmann
https://horstmann.in
VMware VCP
NetApp NCIE-SAN for 7-Mode and Clustered Ontap
marco.horstmann
Veeam Software
 
Posts: 300
Liked: 44 times
Joined: Wed Dec 31, 2014 3:05 pm
Location: Hannover, Germany
Full Name: Marco Horstmann

Re: Direct SAN restore speed

Veeam Logoby xy16644 » Thu Dec 22, 2016 10:21 pm

Those are some good numbers!

Yes, I am using thin provisioned disks for ALL my VMs. I should mention that I am NOT using Direct SAN for restores. I think in Veeam is said it was using NBD mode when it did the restore unlike when a backup ran it said "SAN mode* (I forget the exact wording now).

I'm using all flash/SSD storage so I know the write performance is good (400+MB/s) so I'm pretty sure it isn't this so are you saying that my poor/slow restore speeds are due to me using thinly provisioned disks?
xy16644
Novice
 
Posts: 7
Liked: never
Joined: Fri Dec 04, 2015 10:37 pm
Full Name: SW

Re: Direct SAN restore speed

Veeam Logoby marco.horstmann » Thu Dec 22, 2016 10:30 pm

you can test it. Install a new Windows VM with access to the datastore you want to restore.
Install proxy role to it and select this proxy while the Entire VM Restore Wizard. What are
now the data rates?

The overhead I've mentioned isn't storage IO overhead. It's management overhead with your vcenter.

P.S: The limitating factor with my 2GB/sec example was the 2x 10GbE. ;-)
Marco Horstmann
System Engineer @ Veeam Software

@marcohorstmann
https://horstmann.in
VMware VCP
NetApp NCIE-SAN for 7-Mode and Clustered Ontap
marco.horstmann
Veeam Software
 
Posts: 300
Liked: 44 times
Joined: Wed Dec 31, 2014 3:05 pm
Location: Hannover, Germany
Full Name: Marco Horstmann

Re: Direct SAN restore speed

Veeam Logoby tsightler » Thu Dec 22, 2016 10:38 pm 1 person likes this post

NBD mode is quite limited in performance, especially for 1Gb. You might want to configure a hotadd proxy that can be used just for restores.

Because of the way Direct SAN restore works, restoring thin provisioned disks requires constant calls to vCenter to lock areas of the disk, which in turn are proxied to the hosts. Regardless of the speed of the storage system, this adds a lot of latency to the Direct SAN restore process when disk are thin provisioned because locking has to occur inline during the restore. This isn't as much of an issue with thick provisioned disk because the entire disk is allocated at once and can be reserved up front.
tsightler
Veeam Software
 
Posts: 4872
Liked: 1819 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Direct SAN restore speed

Veeam Logoby xy16644 » Thu Dec 22, 2016 10:43 pm

marco.horstmann wrote:you can test it. Install a new Windows VM with access to the datastore you want to restore.
Install proxy role to it and select this proxy while the Entire VM Restore Wizard. What are
now the data rates?


So do I configure the access to the iSCSI target like I have done for Direct SAN access with the test Windows VM? Or maybe I have misunderstood what you are suggrsting :o

marco.horstmann wrote:The overhead I've mentioned isn't storage IO overhead. It's management overhead with your vcenter.

P.S: The limitating factor with my 2GB/sec example was the 2x 10GbE. ;-)


I have four SSDs and each one is a datastore (no RAID used). vCenter is running on one of these SSDs alongside a couple other VMs. Basically my VMs are spread across all 4 SSDs (including vCenter). It didn't matter which VM I was restoring when I was doing a full restore but my speeds were in the 25-50Mb/s range.

Is it a bad idea (or not possible) to do a Direct SAN restore with thin disks? I'm sure I read somewhere that it wasn't supported. Correct me if I am wrong :D
xy16644
Novice
 
Posts: 7
Liked: never
Joined: Fri Dec 04, 2015 10:37 pm
Full Name: SW

Re: Direct SAN restore speed

Veeam Logoby marco.horstmann » Thu Dec 22, 2016 11:09 pm

xy16644 wrote:So do I configure the access to the iSCSI target like I have done for Direct SAN access with the test Windows VM? Or maybe I have misunderstood what you are suggrsting :o

No. The additional proxy VM must be on a esx server which has access to the datastore, where the restore should run. Otherwise we need to restore e.g. via NBD, which is in the most cases not the best performing mode.

xy16644 wrote:Is it a bad idea (or not possible) to do a Direct SAN restore with thin disks? I'm sure I read somewhere that it wasn't supported. Correct me if I am wrong :D

I would say it's not Best Practise. Best Practise is to use a virtual proxy to restore thin provisioned disks.

Most people today uses thick disk format in VMware and thin provisioning is done inside of the storage array.
Marco Horstmann
System Engineer @ Veeam Software

@marcohorstmann
https://horstmann.in
VMware VCP
NetApp NCIE-SAN for 7-Mode and Clustered Ontap
marco.horstmann
Veeam Software
 
Posts: 300
Liked: 44 times
Joined: Wed Dec 31, 2014 3:05 pm
Location: Hannover, Germany
Full Name: Marco Horstmann

Re: Direct SAN restore speed

Veeam Logoby Net Runner » Fri Dec 23, 2016 9:37 am

I second marco.horstmann for sure.

Try to compare the storage performance between thin provisioned virtual machine and thick provisioned eager zero one. You will be surprised :-)
Net Runner
Novice
 
Posts: 5
Liked: 1 time
Joined: Tue Apr 19, 2016 5:07 am

Re: Direct SAN restore speed

Veeam Logoby xy16644 » Fri Dec 23, 2016 4:22 pm

Thank you all. I will be trying the HotAdd Proxy option and be looking into comverting my thin disks to thick to see how this performs!

Can you have thick disks set in your VM in ESXi and then on the SAN side (Starwind Virtual SAN in this case) have thin disks enabled? ie: best of both worlds
xy16644
Novice
 
Posts: 7
Liked: never
Joined: Fri Dec 04, 2015 10:37 pm
Full Name: SW

Re: Direct SAN restore speed

Veeam Logoby Gostev » Mon Dec 26, 2016 1:44 am

Absolutely, and moreover this is the recommended approach.
Gostev
Veeam Software
 
Posts: 21607
Liked: 2405 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Direct SAN restore speed

Veeam Logoby xy16644 » Fri Dec 30, 2016 4:18 pm

So I converted all my VMs to "thick eager zero" disks in vSphere 6. I have just restored two VMs as a test and here are the results:

1) VM1 ( single 40GB in size) - restored at an everage of 143MB/s using [san] mode and took 5min to complete the restore

2) VM2 (has 4 disks, see below) - restored as follows and took 18min to restore 110GB of disks):

Using proxy VMware Backup Proxy for restoring disk Hard disk 1
Restoring Hard disk 1 (50.0 GB) : 20.3 GB restored at 114 MB/s [san]
Using proxy VMware Backup Proxy for restoring disk Hard disk 2
Restoring Hard disk 2 (30.0 GB) : 17.6 GB restored at 91 MB/s [san]
Using proxy VMware Backup Proxy for restoring disk Hard disk 3
Restoring Hard disk 3 (20.0 GB) : 8.3 GB restored at 84 MB/s [san]
Using proxy VMware Backup Proxy for restoring disk Hard disk 4
Restoring Hard disk 4 (10.0 GB) : 8.5 GB restored at 92 MB/s [san]

These results are much better than the 25-40MB/s I was getting when using thin disks and nbd mode when doing a restore. Are the speeds I am getting acceptable? I thought with Direct SAN restores I would get better speeds as I am restoring from and to SSD storage.

I haven't changed anything else and haven't installed the "virtual HotAdd Proxy" anywhere. Is this still necessary? Will this give even better speeds?

Thanks for all the help with this!
xy16644
Novice
 
Posts: 7
Liked: never
Joined: Fri Dec 04, 2015 10:37 pm
Full Name: SW

Re: Direct SAN restore speed

Veeam Logoby foggy » Tue Jan 03, 2017 10:17 am

Speeds look more expected, however, you can try with virtual proxies as well, to see if hotadd provides better restore performance.
foggy
Veeam Software
 
Posts: 15275
Liked: 1131 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: anthonyspiteri79, Bing [Bot], DGrinev, Google [Bot], gpeachey, JuA, mats.jansson@atea.se, n.vanuffelen, vitaliy.shende and 1 guest