I am using Veeam 9.5 and it is installed on my Windows Server 2016 server. This server is also my SAN running Starwind Virtual SAN and is an iSCSI target for my two ESXi 6 hosts.
I have setup Direct SAN access for my Veeam backups and the speeds are just spectacular! I get 300-400MB/s speeds when the backups happen and I am very happy
The problem is when I do an actual restore. The other day I had to restore my entire environment so was restoring all my VMs (one at a time) and got speeds between 25-50MB/s which I thought was really slow? Is this normal? I'm using 1Gb for my management and iSCSI network so I was expecting to get at least 90-110MB/s?
Is there anything i can do to speed up Veeam restores? Is Direct SAN writes possible?
Hi,
backup is normally faster than restore. We had some weeks ago 2GB/s for backup and 1.2GB/s restore.
Restore speed can be limited in some factors:
1) thin provisioned VM disks needs more management overhead in Direct SAN Restore. This should be avoided.
One option is here to use a virtual HotAdd Proxy and using it for restoring VMs. Maybe you have thin disks?
2) The storagesystem write performance.
Yes, I am using thin provisioned disks for ALL my VMs. I should mention that I am NOT using Direct SAN for restores. I think in Veeam is said it was using NBD mode when it did the restore unlike when a backup ran it said "SAN mode* (I forget the exact wording now).
I'm using all flash/SSD storage so I know the write performance is good (400+MB/s) so I'm pretty sure it isn't this so are you saying that my poor/slow restore speeds are due to me using thinly provisioned disks?
you can test it. Install a new Windows VM with access to the datastore you want to restore.
Install proxy role to it and select this proxy while the Entire VM Restore Wizard. What are
now the data rates?
The overhead I've mentioned isn't storage IO overhead. It's management overhead with your vcenter.
P.S: The limitating factor with my 2GB/sec example was the 2x 10GbE.
Marco Horstmann Senior System Engineer @ Veeam Software
@marcohorstmann
https://horstmann.in
VMware VCP
NetApp NCIE-SAN for 7-Mode and Clustered Ontap
NBD mode is quite limited in performance, especially for 1Gb. You might want to configure a hotadd proxy that can be used just for restores.
Because of the way Direct SAN restore works, restoring thin provisioned disks requires constant calls to vCenter to lock areas of the disk, which in turn are proxied to the hosts. Regardless of the speed of the storage system, this adds a lot of latency to the Direct SAN restore process when disk are thin provisioned because locking has to occur inline during the restore. This isn't as much of an issue with thick provisioned disk because the entire disk is allocated at once and can be reserved up front.
marco.horstmann wrote:you can test it. Install a new Windows VM with access to the datastore you want to restore.
Install proxy role to it and select this proxy while the Entire VM Restore Wizard. What are
now the data rates?
So do I configure the access to the iSCSI target like I have done for Direct SAN access with the test Windows VM? Or maybe I have misunderstood what you are suggrsting
marco.horstmann wrote:
The overhead I've mentioned isn't storage IO overhead. It's management overhead with your vcenter.
P.S: The limitating factor with my 2GB/sec example was the 2x 10GbE.
I have four SSDs and each one is a datastore (no RAID used). vCenter is running on one of these SSDs alongside a couple other VMs. Basically my VMs are spread across all 4 SSDs (including vCenter). It didn't matter which VM I was restoring when I was doing a full restore but my speeds were in the 25-50Mb/s range.
Is it a bad idea (or not possible) to do a Direct SAN restore with thin disks? I'm sure I read somewhere that it wasn't supported. Correct me if I am wrong
xy16644 wrote:
So do I configure the access to the iSCSI target like I have done for Direct SAN access with the test Windows VM? Or maybe I have misunderstood what you are suggrsting
No. The additional proxy VM must be on a esx server which has access to the datastore, where the restore should run. Otherwise we need to restore e.g. via NBD, which is in the most cases not the best performing mode.
xy16644 wrote:
Is it a bad idea (or not possible) to do a Direct SAN restore with thin disks? I'm sure I read somewhere that it wasn't supported. Correct me if I am wrong
I would say it's not Best Practise. Best Practise is to use a virtual proxy to restore thin provisioned disks.
Most people today uses thick disk format in VMware and thin provisioning is done inside of the storage array.
Marco Horstmann Senior System Engineer @ Veeam Software
@marcohorstmann
https://horstmann.in
VMware VCP
NetApp NCIE-SAN for 7-Mode and Clustered Ontap
Thank you all. I will be trying the HotAdd Proxy option and be looking into comverting my thin disks to thick to see how this performs!
Can you have thick disks set in your VM in ESXi and then on the SAN side (Starwind Virtual SAN in this case) have thin disks enabled? ie: best of both worlds
So I converted all my VMs to "thick eager zero" disks in vSphere 6. I have just restored two VMs as a test and here are the results:
1) VM1 ( single 40GB in size) - restored at an everage of 143MB/s using [san] mode and took 5min to complete the restore
2) VM2 (has 4 disks, see below) - restored as follows and took 18min to restore 110GB of disks):
Using proxy VMware Backup Proxy for restoring disk Hard disk 1
Restoring Hard disk 1 (50.0 GB) : 20.3 GB restored at 114 MB/s [san]
Using proxy VMware Backup Proxy for restoring disk Hard disk 2
Restoring Hard disk 2 (30.0 GB) : 17.6 GB restored at 91 MB/s [san]
Using proxy VMware Backup Proxy for restoring disk Hard disk 3
Restoring Hard disk 3 (20.0 GB) : 8.3 GB restored at 84 MB/s [san]
Using proxy VMware Backup Proxy for restoring disk Hard disk 4
Restoring Hard disk 4 (10.0 GB) : 8.5 GB restored at 92 MB/s [san]
These results are much better than the 25-40MB/s I was getting when using thin disks and nbd mode when doing a restore. Are the speeds I am getting acceptable? I thought with Direct SAN restores I would get better speeds as I am restoring from and to SSD storage.
I haven't changed anything else and haven't installed the "virtual HotAdd Proxy" anywhere. Is this still necessary? Will this give even better speeds?
