Slow VM restore speeds from QNAP

VMware specific discussions

Slow VM restore speeds from QNAP

Veeam Logoby jonc » Tue May 19, 2015 1:54 pm

We currently have 4 ESX hosts that back files to a QNAP via Virtual Appliance. Backup speeds range between 50MB/s to 150MB/s and that's fine for us. When it comes to restoring whole VMs back to ESX, we can only average between 10MB/s to 20MB/s. The QNAP is connected via a Gigabit link and copying files directly from "host to QNAP" and "Backup VirtualAppliance to QNAP" and vice versa reaches Gigabit transfer speeds, so the QNAP isn't the bottleneck.

Also, even more strange, the restores are being done to an SSD RAID datastore so speed shouldn't be an issue at all. This happens both with small and large VM restores.

Any help on what to check would be greatly appreciated!
jonc
Novice
 
Posts: 9
Liked: never
Joined: Tue May 19, 2015 1:47 pm

Re: Slow VM restore speeds from QNAP

Veeam Logoby ashman70 » Tue May 19, 2015 2:25 pm

What type of drives are in the QNAP and what kind of RAID are they configured for?
ashman70
Expert
 
Posts: 195
Liked: 12 times
Joined: Tue Dec 04, 2012 2:18 pm
Full Name: Both

Re: Slow VM restore speeds from QNAP

Veeam Logoby jonc » Tue May 19, 2015 2:29 pm

This particular Volume has 3x 2TB Seagate Barracuda 7200rpm in RAID5.
jonc
Novice
 
Posts: 9
Liked: never
Joined: Tue May 19, 2015 1:47 pm

Re: Slow VM restore speeds from QNAP

Veeam Logoby dellock6 » Tue May 19, 2015 9:20 pm

Restore operations are made with random IO, since Veeam has to read deduplicated blocks that are distrubuted all over the backup file. A random IO cannot really be compared at all with a sequential IO like all the tests you did, for example a file copy that streams data sequentially. Even a single HDD can reach 100 MBs on sequential. Have you tested some random IO activity against the QNAP to see the performances? Sorry but my guess is that the storage is indeed the bottleneck, simply the sequential activities are not good ways to test it.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5055
Liked: 1336 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: Slow VM restore speeds from QNAP

Veeam Logoby alanbolte » Wed May 20, 2015 5:34 am

For such a small number of disks in the array, that sounds about right. If you can add more disks to the volume that might be the simplest fix. Switching to 10K RPM drives wouldn't be too bad either, but you'd be sacrificing capacity or need more disks anyway.

Last I worked with a QNAP its user interface didn't offer an option of setting stripe size, I believe they default to 64KB. It's basically a Linux server though so I believe you could create arrays with other settings from CLI (e.g. the --chunk parameter in mdadm) but I've never done it myself. I mention this because Veeam uses fairly large I/O sizes, so on an array that small you'd want a large stripe size to get decent random I/O.

Have you been using forward incrementals with active fulls? I'd think on an array like that you'd see long durations for for forever-incremental merge or synthetic fulls.
alanbolte
Expert
 
Posts: 635
Liked: 170 times
Joined: Mon Jun 18, 2012 8:58 pm
Full Name: Alan Bolte

Re: Slow VM restore speeds from QNAP

Veeam Logoby jonc » Wed May 20, 2015 1:39 pm

I have tried various setups to test out the speeds. Basically:

Incremental (As it was) - 15MB/s Average
Reverse Incremental - 20MB/s Average
Reverse Incremental (No DeDupe / No Compression) - 15MB/s at best

Wouldn't disabling dedupe avoid Random reads? These tests were done on a new backup job with restores directly off a full single backup. What gives?
jonc
Novice
 
Posts: 9
Liked: never
Joined: Tue May 19, 2015 1:47 pm

Re: Slow VM restore speeds from QNAP

Veeam Logoby dellock6 » Wed May 20, 2015 2:48 pm

Reversed Incremental is always random regardless deduplication because of the way it works, by injecting new blocks in different places of the VBK file.
In oder to reduce the backup time, on those low-end device, you should switch from reversed to one of the forward modes.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5055
Liked: 1336 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: Slow VM restore speeds from QNAP

Veeam Logoby jonc » Wed May 20, 2015 3:06 pm

dellock6 wrote:Reversed Incremental is always random regardless deduplication because of the way it works, by injecting new blocks in different places of the VBK file.
In oder to reduce the backup time, on those low-end device, you should switch from reversed to one of the forward modes.


I was currently using Normal Incremental backups and was having these slow restore speeds. Is there a backup method that will provide me with a sequential backup file I can test with to restore so I'll know if it's a random read issue?
jonc
Novice
 
Posts: 9
Liked: never
Joined: Tue May 19, 2015 1:47 pm

Re: Slow VM restore speeds from QNAP

Veeam Logoby dellock6 » Wed May 20, 2015 8:21 pm

Restores of entire VMs as you wrote in the opening post are always going to be random reads, as the blocks are spread in different positions of the backup files, both because of deduplication and the incremental chain.
You can test random reads IO using tools like IOmeter or FIO, we published a paper on this topic:
http://www.veeam.com/wp-veeam-backup-re ... mance.html
You can find information about IO pattern for the different backups and the block size used in different scenarios.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5055
Liked: 1336 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: Slow VM restore speeds from QNAP

Veeam Logoby jonc » Tue Jul 07, 2015 7:05 am

Sorry for re-posting in my old dormant thread, but I have done some more tests and would like to check a few things with you guys. We have now an enterprise grade QNAP TS-879U-RP over 1G link. As before, sequential read and writes max out the link. I have tried backing up 1 VM in one File (Reverse Incremental) and maxed write speed at around 50MB/s (The backup had dedupe off, no compression, no encryption, and anything that might have tried to make it more efficient turned off in hopes to eliminate the randomness of it).

Restore of this VM backup to an ESX with an all SSD volume runs at around 10MB/s-20MB/s. Is there anyway this can be improved? Recently we had a problem and we had to restore a machine that failed and this took hours. We'd like a way to improve speeds or at least make it more efficient.
jonc
Novice
 
Posts: 9
Liked: never
Joined: Tue May 19, 2015 1:47 pm

Re: Slow VM restore speeds from QNAP

Veeam Logoby NightBird » Tue Jul 07, 2015 7:40 am

ESX with SSD volume, and a crapy QNAP NAS for backup ? It sounds crazy for me.

Buy a physical server for the Veeam repo with 12 (or 24)x 1TB spindles with a good raid controller (2 ou 4GB FBWC), you will see the improvement.
NightBird
Service Provider
 
Posts: 148
Liked: 20 times
Joined: Tue Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE

Re: Slow VM restore speeds from QNAP

Veeam Logoby jonc » Tue Jul 07, 2015 11:36 am

I guess you're right about the QNAP for backup. Is there a way for Veeam to directly back up data to a local datastore so all the transfers stay within the hosts?
jonc
Novice
 
Posts: 9
Liked: never
Joined: Tue May 19, 2015 1:47 pm

Re: Slow VM restore speeds from QNAP

Veeam Logoby foggy » Tue Jul 07, 2015 1:47 pm

Backups cannot be stored on host's local storage, unless you put them inside some VM, making it a backup repository (which, however is not considered as best practice).
foggy
Veeam Software
 
Posts: 14754
Liked: 1083 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Slow VM restore speeds from QNAP

Veeam Logoby agrob » Wed Jul 08, 2015 6:58 am

jonc, did you restore to an existing datastore? if yes please try to create a new empty datastore and restore the vm to this. What is your performance then? (i ask because i have the issue if i restore to a existing datastore with other vms on it i can restore with only about 12MB/s. if i create a new empty datastore i restore with about 70MB/s)
agrob
Enthusiast
 
Posts: 91
Liked: 8 times
Joined: Mon Sep 05, 2011 1:31 pm
Full Name: Andre

Re: Slow VM restore speeds from QNAP

Veeam Logoby dellock6 » Wed Jul 08, 2015 2:38 pm

This can happen because of metadata updates happening in VMFS filesystem. One quick trick is to use an NFS datastore for restore operations, and then storage vmotion the VM to the VMFS if needed. NFS does not have metadata updates that slow down the upload. This one is two years old but still totally valid: http://cormachogan.com/2013/07/18/why-i ... s-so-slow/

Luca
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5055
Liked: 1336 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Next

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 33 guests