Comprehensive data protection for all workloads
briggs
Enthusiast
Posts: 76
Liked: 3 times
Joined: Mar 22, 2013 11:01 am
Full Name: Alan Thompson
Contact:

Re: Speed of backup/restore jobs

Post by briggs »

Alex, your backup stats, was that a FULL backup, with "optimal compression"? Did you set "optimize for LAN target" on the job? I just wanted to know, so I compare with my stats.
I really wish there was a way of me trialling this before I implemented it. Virtualising is great, It was the right thing to do, Veeam is also a fantastic product, but for my DR purposes, the performance of the restores (backup speed less important - Alex, how are you restores? Do you have a standalone host to restore to at all?) will cripple me should I ever need to do a restore urgently, and in all honesty, my LTO4 restores were quicker. I do need to get to the bottom of this, though the more help I get from Veeam (I logged a call on this), the more it looks like the performance of the LAN, NAS, or the datastore could be more to blame. I really am no closer to solving this.

What are people's restore speeds in general, in MB/s. I mean the true restore speed of actual used data, ie how many MB/s for the data in use, as Veeam I think gives you the stats for the whole disk, not just the data (used blocks)?
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: Speed of backup/restore jobs

Post by chrisdearden »

If Restore RTO is critical - why not consider using replication ?
briggs
Enthusiast
Posts: 76
Liked: 3 times
Joined: Mar 22, 2013 11:01 am
Full Name: Alan Thompson
Contact:

Re: Speed of backup/restore jobs

Post by briggs »

I may well look into replication, my problem though is lack of disk space, and I still need backups. I guess I need a seperate 'window' to do replication otherwise I impact the backups?
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: Speed of backup/restore jobs

Post by chrisdearden » 1 person likes this post

local Disk space on a standalone host may provide your with the DR compute and storage - dont forget you can put your replicas to work - use them for file and application level restores.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Speed of backup/restore jobs

Post by tsightler » 1 person likes this post

As I mentioned earlier in this thread, VMFS write semantics are by far the biggest factor when performing restores. There are two primary reasons:

1. When performing a restore, the disk is either thin provisioned, or thick lazy zeroed, based on the source, both of those options require twice the writes to disk, i.e. the block must be written with zeros prior to writing any data. You can view this by watching the write rate on the datastore from the host level. If you're seeing 35MB/s restore, you'll see 70MB/s writes to the datastore

2. I'm not 100% sure why (perhaps due to the zeroing or to the high frequency of metadata updates during the restore), but every write to VMFS during a restore operation appears to be flagged as synchronous. This means that the write must be flushed to disk by the storage before it is acknowledged which causes write performance of most storage systems to suffer.

For SMB class storage arrays I've generally seen restores to VMFS from Veeam top out around 35-50MB/s (100MB/s synchronous writes on the backend). For higher-end storage systems, I've seen faster. One thing that's very interesting, restores to NFS datastores are generally MUCH faster, because they are not limited by these VMFS write performance issues mentioned above. If you have the ability mount an NFS datastore from your storage, this might be an option to allow for faster restores.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Speed of backup/restore jobs

Post by dellock6 » 2 people like this post

Tom, for your second point, Cormac Hogan from VMware just wrote a blog article explaining exactly the reason for slow VMFS upload slowness:
http://cormachogan.com/2013/07/18/why-i ... s-so-slow/
I jumped into the article with some comments, and he confirmed the same behaviour is expected even for backup restores done with network mode.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
briggs
Enthusiast
Posts: 76
Liked: 3 times
Joined: Mar 22, 2013 11:01 am
Full Name: Alan Thompson
Contact:

Re: Speed of backup/restore jobs

Post by briggs »

Hi Tom, I've just added an NFS datastore, which is the most basic of NAS boxes (2 bay, mirrored disks, home user spec) and restored to that instead of the local datastore. 30MB/s before, 69MB/s now. This is still too slow for us to be back up and running in a reasonable time, but as it's over twice as fast, with a cheap low performance NAS, it proves your points perfectly. Thanks for mentioning this, it may well be our only option for restoring in a hurry.
I don't suppose you can have local disk datastores as anything other than VMFS can you?
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Speed of backup/restore jobs

Post by dellock6 »

Hi Alan, unfortunately for your need, no, local datastore in an ESXi server can only be VMFS.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: Speed of backup/restore jobs

Post by chrisdearden »

dellock6 wrote:Hi Alan, unfortunately for your need, no, local datastore in an ESXi server can only be VMFS.

Luca.
Run a VSA on the local datastore ? :)
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Speed of backup/restore jobs

Post by dellock6 »

Could be a possible solution indeed, nice idea. Maybe too much abstraction layers, but worthy if the need for fast restores are paramount. Then by using svmotion or quick migration you can always move it to the final datastore with ease.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Speed of backup/restore jobs

Post by tsightler » 1 person likes this post

Believe it or not, this actually works pretty well as that's exactly the setup I have in my lab, a Linux systems sharing out an NFS datastore to the ESXi host. Restores to this datastore are roughly 2x faster that restores to the underlying VMFS datastore, even with hotadd (hotadd restores to VMFS are still fairly slow, I think because they create a snapshot, which increases locking overhead).

But with this setup there's actually an even faster restore option. If you add the Linux host to Veeam, you can perform a "Restore VM Files" directly to the Linux host. With this method we deploy a VeeamAgent to the Linux system, and thus we can restore directly to the underlying Linux filesystem instead of restoring via NFS, then one the restore of the VM files is completed, you use the datastore browser, select the VMX file, and add it to the vCenter Inventory.

If your backup respository itself happens to be a Linux server, and you perform a "Restore VM Files" to a different set of disks on the same server, you effectively get disk-to-disk restores that can be exceptionally fast. You could probably also do this with Windows using the vPower NFS servers, but I haven't tested it. The idea would be to configure the vPower NFS "cache" area to a location backed by fairly fast disks, perform a "Restore VM Files" directly to this cache area, and then manually register the VMs.

Of course, I'm not really sure what speeds the OP is expecting. I believe he mentioned he has 1Gb network so the theoretical max is ~125MB/s, but in reality you'd have to pay very close attention to the data flow to actually get this.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Speed of backup/restore jobs

Post by dellock6 »

nice setup indeed!
In a previous reply Alan said he reached 69 MBs by switching to NFS, other than that there are also the usual considerations about the underlying physical storage to be made, being a "cheap NAS" we are probably reaching its limits.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
briggs
Enthusiast
Posts: 76
Liked: 3 times
Joined: Mar 22, 2013 11:01 am
Full Name: Alan Thompson
Contact:

Re: Speed of backup/restore jobs

Post by briggs »

Hi Tom, I'm afraid you lost me a little bit... Is the Linux server a VM? Any particular distribution of Linux? (preferably free!) I need to utilise the internal RAID5 as my datastore, so do I build the Linux server as a VM, on the local VMFS datastore, then share out the same storage as another NFS datastore? That's kind of bizarre, so I guess I got that wrong?
Thanks, Alan
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Speed of backup/restore jobs

Post by dellock6 »

No Alan, you got it right! :)
Among the free linux distro, CentOS or Ubuntu are among the most common ones, depending on your preferences.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Speed of backup/restore jobs

Post by tsightler »

I'm not necessarily suggesting you should do this, unless you really understand the what and why you'd be doing it. I was just pointing out some cool things you can do with the right setup. It really sounds like you have very tight RTO's so you'd might be better off considering replication instead. I mean, how much data do you have and what do you consider an acceptable restore time.

If you're just playing around to understand the concept, and see what performance is like, I'll tell you what I currently have setup. I'm a Redhat/CentOS guy at heart, but I happen to be using Ubuntu Server 12.04 in my lab right now, just to stay up with my Linux skills. Pretty much any Linux distro should work fine to create an NFS share though, especially since it's just a temporary staging area. Of course, you'd have to have enough disk space to basically restore the server, and then eventually move it to the "real" location, so that's important to understand.
gwhite
Novice
Posts: 8
Liked: never
Joined: May 11, 2011 9:28 pm
Full Name: Greg White
Contact:

Re: Speed of backup/restore jobs

Post by gwhite »

tsightler wrote:If you add the Linux host to Veeam, you can perform a "Restore VM Files" directly to the Linux host. With this method we deploy a VeeamAgent to the Linux system, and thus we can restore directly to the underlying Linux filesystem instead of restoring via NFS, then one the restore of the VM files is completed, you use the datastore browser, select the VMX file, and add it to the vCenter Inventory.

If your backup respository itself happens to be a Linux server, and you perform a "Restore VM Files" to a different set of disks on the same server, you effectively get disk-to-disk restores that can be exceptionally fast. You could probably also do this with Windows using the vPower NFS servers, but I haven't tested it. The idea would be to configure the vPower NFS "cache" area to a location backed by fairly fast disks, perform a "Restore VM Files" directly to this cache area, and then manually register the VMs.
Any docs or instructions on how to do these setups? I'd really love to try them out!

Thanks,

G
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Speed of backup/restore jobs

Post by veremin »

Which particular ones you're interested in? Restoring VM Files directly to the Linux host or configuring the vPower NFS "cache" area? Thanks.
gwhite
Novice
Posts: 8
Liked: never
Joined: May 11, 2011 9:28 pm
Full Name: Greg White
Contact:

Re: Speed of backup/restore jobs

Post by gwhite »

v.Eremin wrote:Which particular ones you're interested in? Restoring VM Files directly to the Linux host or configuring the vPower NFS "cache" area? Thanks.
Both...
Post Reply

Who is online

Users browsing this forum: Google [Bot], rjv1971 and 213 guests