Slow VM restore speeds from QNAP

[jonc]

We currently have 4 ESX hosts that back files to a QNAP via Virtual Appliance. Backup speeds range between 50MB/s to 150MB/s and that's fine for us. When it comes to restoring whole VMs back to ESX, we can only average between 10MB/s to 20MB/s. The QNAP is connected via a Gigabit link and copying files directly from "host to QNAP" and "Backup VirtualAppliance to QNAP" and vice versa reaches Gigabit transfer speeds, so the QNAP isn't the bottleneck.

Also, even more strange, the restores are being done to an SSD RAID datastore so speed shouldn't be an issue at all. This happens both with small and large VM restores.

Any help on what to check would be greatly appreciated!

[ashman70]

What type of drives are in the QNAP and what kind of RAID are they configured for?

[jonc]

This particular Volume has 3x 2TB Seagate Barracuda 7200rpm in RAID5.

[dellock6]

Restore operations are made with random IO, since Veeam has to read deduplicated blocks that are distrubuted all over the backup file. A random IO cannot really be compared at all with a sequential IO like all the tests you did, for example a file copy that streams data sequentially. Even a single HDD can reach 100 MBs on sequential. Have you tested some random IO activity against the QNAP to see the performances? Sorry but my guess is that the storage is indeed the bottleneck, simply the sequential activities are not good ways to test it.

[alanbolte]

For such a small number of disks in the array, that sounds about right. If you can add more disks to the volume that might be the simplest fix. Switching to 10K RPM drives wouldn't be too bad either, but you'd be sacrificing capacity or need more disks anyway.

Last I worked with a QNAP its user interface didn't offer an option of setting stripe size, I believe they default to 64KB. It's basically a Linux server though so I believe you could create arrays with other settings from CLI (e.g. the --chunk parameter in mdadm) but I've never done it myself. I mention this because Veeam uses fairly large I/O sizes, so on an array that small you'd want a large stripe size to get decent random I/O.

Have you been using forward incrementals with active fulls? I'd think on an array like that you'd see long durations for for forever-incremental merge or synthetic fulls.

[jonc]

I have tried various setups to test out the speeds. Basically:

Incremental (As it was) - 15MB/s Average
Reverse Incremental - 20MB/s Average
Reverse Incremental (No DeDupe / No Compression) - 15MB/s at best

Wouldn't disabling dedupe avoid Random reads? These tests were done on a new backup job with restores directly off a full single backup. What gives?

[dellock6]

Reversed Incremental is always random regardless deduplication because of the way it works, by injecting new blocks in different places of the VBK file.
In oder to reduce the backup time, on those low-end device, you should switch from reversed to one of the forward modes.

[jonc]

Reversed Incremental is always random regardless deduplication because of the way it works, by injecting new blocks in different places of the VBK file.
In oder to reduce the backup time, on those low-end device, you should switch from reversed to one of the forward modes.

I was currently using Normal Incremental backups and was having these slow restore speeds. Is there a backup method that will provide me with a sequential backup file I can test with to restore so I'll know if it's a random read issue?

[dellock6]

Restores of entire VMs as you wrote in the opening post are always going to be random reads, as the blocks are spread in different positions of the backup files, both because of deduplication and the incremental chain.
You can test random reads IO using tools like IOmeter or FIO, we published a paper on this topic:
http://www.veeam.com/wp-veeam-backup-re ... mance.html
You can find information about IO pattern for the different backups and the block size used in different scenarios.

[jonc]

Sorry for re-posting in my old dormant thread, but I have done some more tests and would like to check a few things with you guys. We have now an enterprise grade QNAP TS-879U-RP over 1G link. As before, sequential read and writes max out the link. I have tried backing up 1 VM in one File (Reverse Incremental) and maxed write speed at around 50MB/s (The backup had dedupe off, no compression, no encryption, and anything that might have tried to make it more efficient turned off in hopes to eliminate the randomness of it).

Restore of this VM backup to an ESX with an all SSD volume runs at around 10MB/s-20MB/s. Is there anyway this can be improved? Recently we had a problem and we had to restore a machine that failed and this took hours. We'd like a way to improve speeds or at least make it more efficient.

[NightBird]

ESX with SSD volume, and a crapy QNAP NAS for backup ? It sounds crazy for me.

Buy a physical server for the Veeam repo with 12 (or 24)x 1TB spindles with a good raid controller (2 ou 4GB FBWC), you will see the improvement.

[jonc]

I guess you're right about the QNAP for backup. Is there a way for Veeam to directly back up data to a local datastore so all the transfers stay within the hosts?

[foggy]

Backups cannot be stored on host's local storage, unless you put them inside some VM, making it a backup repository (which, however is not considered as best practice).

[agrob]

jonc, did you restore to an existing datastore? if yes please try to create a new empty datastore and restore the vm to this. What is your performance then? (i ask because i have the issue if i restore to a existing datastore with other vms on it i can restore with only about 12MB/s. if i create a new empty datastore i restore with about 70MB/s)

[dellock6]

This can happen because of metadata updates happening in VMFS filesystem. One quick trick is to use an NFS datastore for restore operations, and then storage vmotion the VM to the VMFS if needed. NFS does not have metadata updates that slow down the upload. This one is two years old but still totally valid: http://cormachogan.com/2013/07/18/why-i ... s-so-slow/

Luca

[jonc]

I went for a local drive in Windows pointing to iSCSI storage within the Backup VM. Backups speeds have improved a lot, reaching some 160MB/s. A full 60GB VM restore from this backup to an SSD Volume still goes at around 10-20MB/s and I can't wrap my head around this. Logs show that it's using hotadd for both backup and restore...

[agrob]

Interessting Post @dellock6. i added an nfs datastore and tried a restore to this store... but because we have 1gbit nics and as far as i have seen data transfer from esx host to nfs runs over vmotion/management nics which are limited, i cant reach more speed i think... so if i have a fc storage which performs (with vmotion testet) several 100MB/s it can slow down restore process to 70MB/s because of the metadata "overload"? Is there no other way to speed things up with restoring to vmfs filesystem stores?

[dellock6]

We usually suggest to have at least one virtual proxy in order to leverage hotadd mode exactly for these reasons, the ESXi host is faster than VDDK when writing to VMFSand hotadd uses the ESXi stack to do the writes. Obviously if the VM network is also at 1GB, the final speed will never exceed 125 MBs, but it's already an improvement compared to network mode or VMFS direct writes.

[agrob]

Thanks Luca. But i never get over 60-70MB/s even with proxy in hottadd mode. it also slows down to about 12MB/s (proxy with hotadd) if i restore to a vmfs datastore (san fc storage) which has odther vm's on it...