Comprehensive data protection for all workloads
Post Reply
xy16644
Novice
Posts: 7
Liked: never
Joined: Dec 04, 2015 10:37 pm
Full Name: SW

Direct SAN restore speed

Post by xy16644 »

I am using Veeam 9.5 and it is installed on my Windows Server 2016 server. This server is also my SAN running Starwind Virtual SAN and is an iSCSI target for my two ESXi 6 hosts.

I have setup Direct SAN access for my Veeam backups and the speeds are just spectacular! I get 300-400MB/s speeds when the backups happen and I am very happy :D

The problem is when I do an actual restore. The other day I had to restore my entire environment so was restoring all my VMs (one at a time) and got speeds between 25-50MB/s which I thought was really slow? Is this normal? I'm using 1Gb for my management and iSCSI network so I was expecting to get at least 90-110MB/s?

Is there anything i can do to speed up Veeam restores? Is Direct SAN writes possible?

Thank you.
marco.horstmann
Veeam Software
Posts: 594
Liked: 105 times
Joined: Dec 31, 2014 3:05 pm
Full Name: Marco Horstmann
Location: Hannover, Germany
Contact:

Re: Direct SAN restore speed

Post by marco.horstmann »

Hi,
backup is normally faster than restore. We had some weeks ago 2GB/s for backup and 1.2GB/s restore. ;-)

Restore speed can be limited in some factors:
1) thin provisioned VM disks needs more management overhead in Direct SAN Restore. This should be avoided.
One option is here to use a virtual HotAdd Proxy and using it for restoring VMs. Maybe you have thin disks?
2) The storagesystem write performance.

See this help page: https://helpcenter.veeam.com/backup/vsp ... iting.html

Regards
Marco
Marco Horstmann
Senior System Engineer @ Veeam Software

@marcohorstmann
https://horstmann.in
VMware VCP
NetApp NCIE-SAN for 7-Mode and Clustered Ontap
xy16644
Novice
Posts: 7
Liked: never
Joined: Dec 04, 2015 10:37 pm
Full Name: SW

Re: Direct SAN restore speed

Post by xy16644 »

Those are some good numbers!

Yes, I am using thin provisioned disks for ALL my VMs. I should mention that I am NOT using Direct SAN for restores. I think in Veeam is said it was using NBD mode when it did the restore unlike when a backup ran it said "SAN mode* (I forget the exact wording now).

I'm using all flash/SSD storage so I know the write performance is good (400+MB/s) so I'm pretty sure it isn't this so are you saying that my poor/slow restore speeds are due to me using thinly provisioned disks?
marco.horstmann
Veeam Software
Posts: 594
Liked: 105 times
Joined: Dec 31, 2014 3:05 pm
Full Name: Marco Horstmann
Location: Hannover, Germany
Contact:

Re: Direct SAN restore speed

Post by marco.horstmann »

you can test it. Install a new Windows VM with access to the datastore you want to restore.
Install proxy role to it and select this proxy while the Entire VM Restore Wizard. What are
now the data rates?

The overhead I've mentioned isn't storage IO overhead. It's management overhead with your vcenter.

P.S: The limitating factor with my 2GB/sec example was the 2x 10GbE. ;-)
Marco Horstmann
Senior System Engineer @ Veeam Software

@marcohorstmann
https://horstmann.in
VMware VCP
NetApp NCIE-SAN for 7-Mode and Clustered Ontap
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Direct SAN restore speed

Post by tsightler » 1 person likes this post

NBD mode is quite limited in performance, especially for 1Gb. You might want to configure a hotadd proxy that can be used just for restores.

Because of the way Direct SAN restore works, restoring thin provisioned disks requires constant calls to vCenter to lock areas of the disk, which in turn are proxied to the hosts. Regardless of the speed of the storage system, this adds a lot of latency to the Direct SAN restore process when disk are thin provisioned because locking has to occur inline during the restore. This isn't as much of an issue with thick provisioned disk because the entire disk is allocated at once and can be reserved up front.
xy16644
Novice
Posts: 7
Liked: never
Joined: Dec 04, 2015 10:37 pm
Full Name: SW

Re: Direct SAN restore speed

Post by xy16644 »

marco.horstmann wrote:you can test it. Install a new Windows VM with access to the datastore you want to restore.
Install proxy role to it and select this proxy while the Entire VM Restore Wizard. What are
now the data rates?
So do I configure the access to the iSCSI target like I have done for Direct SAN access with the test Windows VM? Or maybe I have misunderstood what you are suggrsting :o
marco.horstmann wrote: The overhead I've mentioned isn't storage IO overhead. It's management overhead with your vcenter.

P.S: The limitating factor with my 2GB/sec example was the 2x 10GbE. ;-)
I have four SSDs and each one is a datastore (no RAID used). vCenter is running on one of these SSDs alongside a couple other VMs. Basically my VMs are spread across all 4 SSDs (including vCenter). It didn't matter which VM I was restoring when I was doing a full restore but my speeds were in the 25-50Mb/s range.

Is it a bad idea (or not possible) to do a Direct SAN restore with thin disks? I'm sure I read somewhere that it wasn't supported. Correct me if I am wrong :D
marco.horstmann
Veeam Software
Posts: 594
Liked: 105 times
Joined: Dec 31, 2014 3:05 pm
Full Name: Marco Horstmann
Location: Hannover, Germany
Contact:

Re: Direct SAN restore speed

Post by marco.horstmann »

xy16644 wrote: So do I configure the access to the iSCSI target like I have done for Direct SAN access with the test Windows VM? Or maybe I have misunderstood what you are suggrsting :o
No. The additional proxy VM must be on a esx server which has access to the datastore, where the restore should run. Otherwise we need to restore e.g. via NBD, which is in the most cases not the best performing mode.
xy16644 wrote: Is it a bad idea (or not possible) to do a Direct SAN restore with thin disks? I'm sure I read somewhere that it wasn't supported. Correct me if I am wrong :D
I would say it's not Best Practise. Best Practise is to use a virtual proxy to restore thin provisioned disks.

Most people today uses thick disk format in VMware and thin provisioning is done inside of the storage array.
Marco Horstmann
Senior System Engineer @ Veeam Software

@marcohorstmann
https://horstmann.in
VMware VCP
NetApp NCIE-SAN for 7-Mode and Clustered Ontap
Net Runner
Influencer
Posts: 20
Liked: 2 times
Joined: Apr 19, 2016 5:07 am
Contact:

Re: Direct SAN restore speed

Post by Net Runner »

I second marco.horstmann for sure.

Try to compare the storage performance between thin provisioned virtual machine and thick provisioned eager zero one. You will be surprised :-)
xy16644
Novice
Posts: 7
Liked: never
Joined: Dec 04, 2015 10:37 pm
Full Name: SW

Re: Direct SAN restore speed

Post by xy16644 »

Thank you all. I will be trying the HotAdd Proxy option and be looking into comverting my thin disks to thick to see how this performs!

Can you have thick disks set in your VM in ESXi and then on the SAN side (Starwind Virtual SAN in this case) have thin disks enabled? ie: best of both worlds
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Direct SAN restore speed

Post by Gostev »

Absolutely, and moreover this is the recommended approach.
xy16644
Novice
Posts: 7
Liked: never
Joined: Dec 04, 2015 10:37 pm
Full Name: SW

Re: Direct SAN restore speed

Post by xy16644 »

So I converted all my VMs to "thick eager zero" disks in vSphere 6. I have just restored two VMs as a test and here are the results:

1) VM1 ( single 40GB in size) - restored at an everage of 143MB/s using [san] mode and took 5min to complete the restore

2) VM2 (has 4 disks, see below) - restored as follows and took 18min to restore 110GB of disks):

Using proxy VMware Backup Proxy for restoring disk Hard disk 1
Restoring Hard disk 1 (50.0 GB) : 20.3 GB restored at 114 MB/s [san]
Using proxy VMware Backup Proxy for restoring disk Hard disk 2
Restoring Hard disk 2 (30.0 GB) : 17.6 GB restored at 91 MB/s [san]
Using proxy VMware Backup Proxy for restoring disk Hard disk 3
Restoring Hard disk 3 (20.0 GB) : 8.3 GB restored at 84 MB/s [san]
Using proxy VMware Backup Proxy for restoring disk Hard disk 4
Restoring Hard disk 4 (10.0 GB) : 8.5 GB restored at 92 MB/s [san]

These results are much better than the 25-40MB/s I was getting when using thin disks and nbd mode when doing a restore. Are the speeds I am getting acceptable? I thought with Direct SAN restores I would get better speeds as I am restoring from and to SSD storage.

I haven't changed anything else and haven't installed the "virtual HotAdd Proxy" anywhere. Is this still necessary? Will this give even better speeds?

Thanks for all the help with this!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Direct SAN restore speed

Post by foggy »

Speeds look more expected, however, you can try with virtual proxies as well, to see if hotadd provides better restore performance.
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Brian.Knoblauch and 183 guests