Comprehensive data protection for all workloads
stephaneb
Influencer
Posts: 21
Liked: never
Joined: Feb 04, 2010 9:19 am
Full Name: Stephane Bourdeaud
Contact:

Re: Veeam Restore Speed

Post by stephaneb » Apr 02, 2010 3:32 pm

I forgot to add that in all my scenarios, the network send rate matches perfectly the backup storage disk read rate.
I tested that disk read rate with IO Meter and I know I can get a lot more than the average 55 MB read rate I get when running a restore job from Veeam B&R.

The question is: why is Veeam B&R solliciting more the disk?

tsightler
VP, Product Management
Posts: 5298
Liked: 2153 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam Restore Speed

Post by tsightler » Apr 02, 2010 4:42 pm

Restore to Linux Physical Host (ext4 filesystem, NFS datastore): 213MB/sec -- 1min 36sec

Restore to Linux VM (ext3 filesystem on VMFS datastore): 155MB/sec -- 2min 12sec

Restore to ESX Console (VMFS filesystem/datastore): 41MB/sec -- 8min 19sec

Restore to ESX Console (NFS mounted datastore): 96MB/sec -- 3min 35sec

So this shows that, as Anton theorized, the ESX console is not a significant bottleneck. When restoring to an NFS mounted datastore the restore was more than twice as fast, and I believe this was limited by the network connectivity. Currently my console OS is sharing the same physical NIC with the NFS mount since we're only doing this for testing. That means that Veeam was sending the restore data, and the ESX server was writing the restored files over the same 1Gb NIC, not exactly optimal. I suspect that if I changed my configuration to make sure the NFS datastore traffic used a separate NIC from the COS the restore would have been closer to the 155MB/sec number. As it was we were completely saturating a 1Gb link. Still, the results show that the COS is capable of good performance during restores, just not when writing to a VMFS volumes.

I'm certainly not a VMFS expert, it's possible it's not SCSI reservations, but rather locking behavior. I believe that VMFS supports file level locking per host at the filesystem level. When a VM is powered on it locks it's VMDK file blocks by "region" and thus can write to it's blocks without acquiring a filesystem wide lock. On the other hand, an console OS using traditional unix "open" and "write" syscalls requires a lock for each write. You can see the locking behavior of a VMFS by attempting to delete a flat VMDK file for a running host from another host. It will hang for a few seconds and then eventually give a "device or resource busy" error. You don't get the same behavior if you try to delete a file while Veeam is restoring it. In that scenario you'll get a persistent hang until the restore is complete and, if you don't cancel the delete, it will be removed the instant Veeam completes the restore. There's obviously a locking difference between the two.

stephaneb
Influencer
Posts: 21
Liked: never
Joined: Feb 04, 2010 9:19 am
Full Name: Stephane Bourdeaud
Contact:

Re: Veeam Restore Speed

Post by stephaneb » Apr 06, 2010 10:07 am

I also wanted to see if there was any benefit to use the Veeam agent in the COS.

When using data that is neither compressed nor deduped, there is about 10% overhead when using the Veeam agent (based on 5 tests using the same 215 GB restore data, restoring a single VM to a single host, it restores on average in 83 minutes with the Veeam agent, and on average in 75 minutes without the Veeam agent).

When the data is heavily compressed and slightly deduped (60% compression and 5% deduped), there is about 25% overhead when using the Veeam agent (based on 5 tests using the same 215 GB restore data, restoring a single VM to a single host, it restores on average in 72 minutes with the Veeam agent, and on average in 55 minutes without the Veeam agent).

In conclusion: there seems to be a performance hit when using the Veeam agent during restore operations (probably because by Veeam server is decompressind and a lot faster with its 4 Nehalem cores than my COS with its single 7440 virtual processor). Probably using the Veeam agent is beneficial when restoring multiple virtual machines to multiple hosts because then you definitely want to save the network bandwidth and decompress and dedupe at the ESX host level.

vbussiro
Enthusiast
Posts: 64
Liked: never
Joined: Feb 18, 2009 10:05 pm
Contact:

Re: Veeam Restore Speed

Post by vbussiro » Apr 07, 2010 6:44 am

Very interesting post. So in case of hurry, it should be better to restore on nfs datastore then svmotion it to final datastore ?

tsightler
VP, Product Management
Posts: 5298
Liked: 2153 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam Restore Speed

Post by tsightler » Apr 07, 2010 1:21 pm

That is certainly the case in our environment. Restoring to an NFS datastore is 2-3x faster than restoring to a VMFS datastore and our environment is not optimized for restoring to NFS. I'm going to try to reconfigure one of our systems to mount the NFS datastore on a different NIC and see if performance gets even better.

So far our best performance is achieved by restoring to a Linux system that acts as an NFS server. This is the perfect solution for us because we already use Linux servers as our backup targets. We've created a "Veeam Recovery Area" on our Linux backup targets and shared them out via NFS to the ESX servers and now, when we do a restore, we simply restore to this "Veeam Recovery Area" and SVmotion from there. Our restores are now 4-5x our previous performance levels, hitting 150-200MB/sec even for very large, compressed VM's. We restored a 750GB VM in just under an hour, where that took almost 8 hours restoring to VMFS. Even better, we can easily run multiple restores to the Linux NFS server because it has so much more native horsepower than the ESX console meaning that we can get aggregate restore performance that completely saturates our bandwidth. Sweet!

jzemaitis
Influencer
Posts: 17
Liked: never
Joined: Apr 27, 2009 11:08 pm
Full Name: Joe Zemaitis
Contact:

Re: Veeam Restore Speed

Post by jzemaitis » May 14, 2010 2:50 pm

I'm running into the same problem. The backups are very fast but restores are causing a big problem. 2.5 hours to restore 60GB when it's backup takes less then 45 minutes. All my hardware is good... the backup server isn't breaking a sweat. Neither is the ESX host. It's always been this way... whats going on?

Maybe you can help me out. I posted something here:
http://www.veeam.com/forums/viewtopic.php?f=2&t=3733

clocatel
Novice
Posts: 3
Liked: never
Joined: Feb 18, 2010 2:59 pm
Full Name: Cedric Locatelli
Contact:

Re: Veeam Restore Speed

Post by clocatel » May 18, 2010 2:20 pm

Is there any project to restore with SAN Mode. Other backup products use vStorage API with SAN Mode to restore. With VEEAM Backup, restore is about 30 MB/s. With other product, it's about 70 MB/s and i can launch several restore jobs (240 MB/s with 4 restore jobs). One or 8 VEEAM restore jobs, the total speed is always 30 MB/s.
My client uses backups for his Disaster Recovery Plan. So he has about 1,5 TB to restore. 30 MB/s versus 240 MB/s is a significant difference with this amount of data.

Gostev
SVP, Product Management
Posts: 24022
Liked: 3257 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Restore Speed

Post by Gostev » May 18, 2010 2:32 pm

Yes, we are planning to implement direct restores to SAN.

sunshineb
Novice
Posts: 6
Liked: never
Joined: Mar 16, 2011 7:18 pm
Contact:

Re: Veeam Restore Speed

Post by sunshineb » Jun 07, 2012 6:55 pm

When do they plan on offering direct restore to SAN? I"m testing Symantec Backup Exec's product and you are able to do restores with the HotAdd. This is restoring directly to SAN, instead of traversing the network. The speeds are noticeably faster than Veeam. We are planning on using Symantec Backup Exec visualization product, since they can do physical machines too. The pricing has also dropped to start planning on replacing veeam. Their Application recovery is more reliable than Veeams. Veeams works sometimes and sometimes it doesn't. We can not have a product like that backing up SQL Servers, Exchange Servers, and Domain Controllers.

Thanks,
Sunshine

Gostev
SVP, Product Management
Posts: 24022
Liked: 3257 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Restore Speed

Post by Gostev » Jun 07, 2012 7:30 pm

We have ended up implementing direct SAN restores via ESXi I/O stack (aka hot add restore), because "normal" direct SAN restores do not work well with thin-provisioned disk, which is what majority of our customers are using for most of their VMs. Specifically, it goes extremely slow (slower than restore over the network), and causes a storm of events that may impact vCenter.

As I explained earlier, we had Direct SAN restores implemented over 2 years ago, but decided not to enable it because of performance issues and potential impact on the environment.

For fastest possible restores, we provide Instant VM Recovery, which Symantec does not provide. No restore type can beat that (granted you backup storage provides decent performance). Also, consider this or this most recent feedback from actual BackupExec 2012 users. That said, you should certainly go with the product that works well for your specific environment, storage and addresses your needs best.

By the way, since you mention application item recovery - one thing that is commonly overlooked with Symantec, is that their file level and application item level recoveries require that backups are stored as uncompressed. As a result, depending on your retention policy, the additional storage cost may be a few times larger than the cost of Veeam licenses. Since you mentioned the price of both solutions, I understand that TCO is a significant factor for you?

tsightler
VP, Product Management
Posts: 5298
Liked: 2153 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam Restore Speed

Post by tsightler » Jun 08, 2012 12:34 am

sunshineb wrote:I'm testing Symantec Backup Exec's product and you are able to do restores with the HotAdd.
Veeam also support restores via hotadd.

sunshineb
Novice
Posts: 6
Liked: never
Joined: Mar 16, 2011 7:18 pm
Contact:

Re: Veeam Restore Speed

Post by sunshineb » Jun 11, 2012 8:54 pm

I will continue testing both products. Thanks for the responds.

Post Reply

Who is online

Users browsing this forum: FredrikJ, Karinne, LDuffy, wishr and 48 guests