Discussions specific to the VMware vSphere hypervisor
Post Reply
jonc
Novice
Posts: 9
Liked: never
Joined: May 19, 2015 1:47 pm
Contact:

Slow VM restore speeds from QNAP

Post by jonc » May 19, 2015 1:54 pm

We currently have 4 ESX hosts that back files to a QNAP via Virtual Appliance. Backup speeds range between 50MB/s to 150MB/s and that's fine for us. When it comes to restoring whole VMs back to ESX, we can only average between 10MB/s to 20MB/s. The QNAP is connected via a Gigabit link and copying files directly from "host to QNAP" and "Backup VirtualAppliance to QNAP" and vice versa reaches Gigabit transfer speeds, so the QNAP isn't the bottleneck.

Also, even more strange, the restores are being done to an SSD RAID datastore so speed shouldn't be an issue at all. This happens both with small and large VM restores.

Any help on what to check would be greatly appreciated!

ashman70
Expert
Posts: 203
Liked: 12 times
Joined: Dec 04, 2012 2:18 pm
Full Name: Both
Contact:

Re: Slow VM restore speeds from QNAP

Post by ashman70 » May 19, 2015 2:25 pm

What type of drives are in the QNAP and what kind of RAID are they configured for?

jonc
Novice
Posts: 9
Liked: never
Joined: May 19, 2015 1:47 pm
Contact:

Re: Slow VM restore speeds from QNAP

Post by jonc » May 19, 2015 2:29 pm

This particular Volume has 3x 2TB Seagate Barracuda 7200rpm in RAID5.

dellock6
Veeam Software
Posts: 5734
Liked: 1626 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Slow VM restore speeds from QNAP

Post by dellock6 » May 19, 2015 9:20 pm

Restore operations are made with random IO, since Veeam has to read deduplicated blocks that are distrubuted all over the backup file. A random IO cannot really be compared at all with a sequential IO like all the tests you did, for example a file copy that streams data sequentially. Even a single HDD can reach 100 MBs on sequential. Have you tested some random IO activity against the QNAP to see the performances? Sorry but my guess is that the storage is indeed the bottleneck, simply the sequential activities are not good ways to test it.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

alanbolte
Expert
Posts: 635
Liked: 172 times
Joined: Jun 18, 2012 8:58 pm
Full Name: Alan Bolte
Contact:

Re: Slow VM restore speeds from QNAP

Post by alanbolte » May 20, 2015 5:34 am

For such a small number of disks in the array, that sounds about right. If you can add more disks to the volume that might be the simplest fix. Switching to 10K RPM drives wouldn't be too bad either, but you'd be sacrificing capacity or need more disks anyway.

Last I worked with a QNAP its user interface didn't offer an option of setting stripe size, I believe they default to 64KB. It's basically a Linux server though so I believe you could create arrays with other settings from CLI (e.g. the --chunk parameter in mdadm) but I've never done it myself. I mention this because Veeam uses fairly large I/O sizes, so on an array that small you'd want a large stripe size to get decent random I/O.

Have you been using forward incrementals with active fulls? I'd think on an array like that you'd see long durations for for forever-incremental merge or synthetic fulls.

jonc
Novice
Posts: 9
Liked: never
Joined: May 19, 2015 1:47 pm
Contact:

Re: Slow VM restore speeds from QNAP

Post by jonc » May 20, 2015 1:39 pm

I have tried various setups to test out the speeds. Basically:

Incremental (As it was) - 15MB/s Average
Reverse Incremental - 20MB/s Average
Reverse Incremental (No DeDupe / No Compression) - 15MB/s at best

Wouldn't disabling dedupe avoid Random reads? These tests were done on a new backup job with restores directly off a full single backup. What gives?

dellock6
Veeam Software
Posts: 5734
Liked: 1626 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Slow VM restore speeds from QNAP

Post by dellock6 » May 20, 2015 2:48 pm

Reversed Incremental is always random regardless deduplication because of the way it works, by injecting new blocks in different places of the VBK file.
In oder to reduce the backup time, on those low-end device, you should switch from reversed to one of the forward modes.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

jonc
Novice
Posts: 9
Liked: never
Joined: May 19, 2015 1:47 pm
Contact:

Re: Slow VM restore speeds from QNAP

Post by jonc » May 20, 2015 3:06 pm

dellock6 wrote:Reversed Incremental is always random regardless deduplication because of the way it works, by injecting new blocks in different places of the VBK file.
In oder to reduce the backup time, on those low-end device, you should switch from reversed to one of the forward modes.
I was currently using Normal Incremental backups and was having these slow restore speeds. Is there a backup method that will provide me with a sequential backup file I can test with to restore so I'll know if it's a random read issue?

dellock6
Veeam Software
Posts: 5734
Liked: 1626 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Slow VM restore speeds from QNAP

Post by dellock6 » May 20, 2015 8:21 pm

Restores of entire VMs as you wrote in the opening post are always going to be random reads, as the blocks are spread in different positions of the backup files, both because of deduplication and the incremental chain.
You can test random reads IO using tools like IOmeter or FIO, we published a paper on this topic:
http://www.veeam.com/wp-veeam-backup-re ... mance.html
You can find information about IO pattern for the different backups and the block size used in different scenarios.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

jonc
Novice
Posts: 9
Liked: never
Joined: May 19, 2015 1:47 pm
Contact:

Re: Slow VM restore speeds from QNAP

Post by jonc » Jul 07, 2015 7:05 am

Sorry for re-posting in my old dormant thread, but I have done some more tests and would like to check a few things with you guys. We have now an enterprise grade QNAP TS-879U-RP over 1G link. As before, sequential read and writes max out the link. I have tried backing up 1 VM in one File (Reverse Incremental) and maxed write speed at around 50MB/s (The backup had dedupe off, no compression, no encryption, and anything that might have tried to make it more efficient turned off in hopes to eliminate the randomness of it).

Restore of this VM backup to an ESX with an all SSD volume runs at around 10MB/s-20MB/s. Is there anyway this can be improved? Recently we had a problem and we had to restore a machine that failed and this took hours. We'd like a way to improve speeds or at least make it more efficient.

NightBird
Service Provider
Posts: 178
Liked: 32 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: Slow VM restore speeds from QNAP

Post by NightBird » Jul 07, 2015 7:40 am

ESX with SSD volume, and a crapy QNAP NAS for backup ? It sounds crazy for me.

Buy a physical server for the Veeam repo with 12 (or 24)x 1TB spindles with a good raid controller (2 ou 4GB FBWC), you will see the improvement.

jonc
Novice
Posts: 9
Liked: never
Joined: May 19, 2015 1:47 pm
Contact:

Re: Slow VM restore speeds from QNAP

Post by jonc » Jul 07, 2015 11:36 am

I guess you're right about the QNAP for backup. Is there a way for Veeam to directly back up data to a local datastore so all the transfers stay within the hosts?

foggy
Veeam Software
Posts: 18287
Liked: 1568 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow VM restore speeds from QNAP

Post by foggy » Jul 07, 2015 1:47 pm

Backups cannot be stored on host's local storage, unless you put them inside some VM, making it a backup repository (which, however is not considered as best practice).

agrob
Expert
Posts: 189
Liked: 20 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: Slow VM restore speeds from QNAP

Post by agrob » Jul 08, 2015 6:58 am

jonc, did you restore to an existing datastore? if yes please try to create a new empty datastore and restore the vm to this. What is your performance then? (i ask because i have the issue if i restore to a existing datastore with other vms on it i can restore with only about 12MB/s. if i create a new empty datastore i restore with about 70MB/s)

dellock6
Veeam Software
Posts: 5734
Liked: 1626 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Slow VM restore speeds from QNAP

Post by dellock6 » Jul 08, 2015 2:38 pm

This can happen because of metadata updates happening in VMFS filesystem. One quick trick is to use an NFS datastore for restore operations, and then storage vmotion the VM to the VMFS if needed. NFS does not have metadata updates that slow down the upload. This one is two years old but still totally valid: http://cormachogan.com/2013/07/18/why-i ... s-so-slow/

Luca
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

jonc
Novice
Posts: 9
Liked: never
Joined: May 19, 2015 1:47 pm
Contact:

Re: Slow VM restore speeds from QNAP

Post by jonc » Jul 09, 2015 8:15 am

I went for a local drive in Windows pointing to iSCSI storage within the Backup VM. Backups speeds have improved a lot, reaching some 160MB/s. A full 60GB VM restore from this backup to an SSD Volume still goes at around 10-20MB/s and I can't wrap my head around this. Logs show that it's using hotadd for both backup and restore...

agrob
Expert
Posts: 189
Liked: 20 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: Slow VM restore speeds from QNAP

Post by agrob » Jul 09, 2015 8:54 am

Interessting Post @dellock6. i added an nfs datastore and tried a restore to this store... but because we have 1gbit nics and as far as i have seen data transfer from esx host to nfs runs over vmotion/management nics which are limited, i cant reach more speed i think... so if i have a fc storage which performs (with vmotion testet) several 100MB/s it can slow down restore process to 70MB/s because of the metadata "overload"? Is there no other way to speed things up with restoring to vmfs filesystem stores?

dellock6
Veeam Software
Posts: 5734
Liked: 1626 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Slow VM restore speeds from QNAP

Post by dellock6 » Jul 12, 2015 9:28 pm

We usually suggest to have at least one virtual proxy in order to leverage hotadd mode exactly for these reasons, the ESXi host is faster than VDDK when writing to VMFSand hotadd uses the ESXi stack to do the writes. Obviously if the VM network is also at 1GB, the final speed will never exceed 125 MBs, but it's already an improvement compared to network mode or VMFS direct writes.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

agrob
Expert
Posts: 189
Liked: 20 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: Slow VM restore speeds from QNAP

Post by agrob » Jul 13, 2015 5:59 am

Thanks Luca. But i never get over 60-70MB/s even with proxy in hottadd mode. it also slows down to about 12MB/s (proxy with hotadd) if i restore to a vmfs datastore (san fc storage) which has odther vm's on it...

Post Reply

Who is online

Users browsing this forum: Andreas Neufert and 23 guests