Comprehensive data protection for all workloads
Post Reply
SBarrett847
Service Provider
Posts: 315
Liked: 41 times
Joined: Feb 02, 2016 5:02 pm
Full Name: Stephen Barrett
Contact:

Feature Request - SureBackup staging area.

Post by SBarrett847 »

I love the Idea of Surebackup, however my main Backup repo is a StoreOnce who's Random I/O performance is abysmal. This prevents the VM being able to boot within any reasonable time frame. However it does have a reasonable restore rate (still not great though).

I would like if Surebackup could fully restore the VM to local storage or primary storage first, then boot and test the VM from there.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Feature Request - SureBackup staging area.

Post by foggy »

Hi Stephen, what SO firmware version does your storage run? StoreOnce restore performance was significantly increased in 3.15.1, specifically to allow SureBackup/Instant Recovery possible. So make sure you're running the recent one.
SBarrett847
Service Provider
Posts: 315
Liked: 41 times
Joined: Feb 02, 2016 5:02 pm
Full Name: Stephen Barrett
Contact:

Re: Feature Request - SureBackup staging area.

Post by SBarrett847 »

Hi Foggy - I'm on 3.16.3-1730.1 - however it is a CIFS share, not catalyst if that makes any difference.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Feature Request - SureBackup staging area.

Post by foggy »

It could. But also the overall setup might not be optimal. Some hints can be found here. Btw, what kind of VM's are you restoring - do they have a single or multiple disks?
SBarrett847
Service Provider
Posts: 315
Liked: 41 times
Joined: Feb 02, 2016 5:02 pm
Full Name: Stephen Barrett
Contact:

Re: Feature Request - SureBackup staging area.

Post by SBarrett847 »

Testing now and I'm Getting about 15-20 MB/s per stream from the my StoreOnce (4500) I've never ever seen anything like 275MB/s restores as in that link. I have been able get up to 250/300 MB/s going into the StoreOnce, but coming out it is very slow.

From the Link it would seem that if you aren't using catalyst, you should probably not be using incrementals, as the performance is better with Full backups. Also it recommends using primary storage snapshots to allow for lightning restores. However as I use Hyper-V, and Veeam can't Leverage native 3Par snapshots for restores on Hyper-V, I'm hamstrung and have to rely on the StoreOnce to perform all backup tasks.

StoreOnce IMO is not really suitable for Primary Backup storage, and in places I have Backups going to an old SAN for a few VM I really need to be able to use Instant recovery.

However - I digress - If I could get Surebackup to restore to a staging area (local HDD or SSD array for example) and then boot and perform tests, I could get the Surebackup jobs to complete.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Feature Request - SureBackup staging area.

Post by foggy »

Well, using any dedupe device as primary backup storage is actually against our reference architecture.
SBarrett847
Service Provider
Posts: 315
Liked: 41 times
Joined: Feb 02, 2016 5:02 pm
Full Name: Stephen Barrett
Contact:

Re: Feature Request - SureBackup staging area.

Post by SBarrett847 » 1 person likes this post

Yeah, I know that now :)

It is my fault for letting HP design the solution originally. Will consider Exagrid for the future I think.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Feature Request - SureBackup staging area.

Post by foggy »

I didn't mean that actually, but rather introducing some raw device in-between, to better comply with the 3-2-1 rule.
FedericoV
Technology Partner
Posts: 35
Liked: 37 times
Joined: Aug 21, 2017 3:27 pm
Full Name: Federico Venier
Contact:

Re: Feature Request - SureBackup staging area.

Post by FedericoV » 1 person likes this post

Hi Stephen, StoreOnce random IO has been optimized for Catalyst only. If you are using CIFS, than IVMR is quite slower than using Catalyst and, sorry to say, it is even not supported.
If you can, I suggest you to test IVMR with a Backup Repository created on a Catalyst Store.
Regarding the Restore Performance, your 15-20MB/s per stream are quite lower then the expected performance for your SO4500. For a single stream restore I would expect about 200MB/s.
If you can, try mounting the CIFS share from a server and make a command line test such as "copy /b MyLargeFile.vbk NUL" ? Write NUL not NULL, it works similarly to the unix /dev/null.
Another point to check is the "per VM file". we know that the restore (and the dedupe) is lower when there is one file per job rather than per VM.
I hope this can help you discover the root cause of your slow full restore performance.
SBarrett847
Service Provider
Posts: 315
Liked: 41 times
Joined: Feb 02, 2016 5:02 pm
Full Name: Stephen Barrett
Contact:

Re: Feature Request - SureBackup staging area.

Post by SBarrett847 »

Much obliged Frederico - I do indeed see ~200MB/s using the NUL copy (and Copying to a local Flash array)- however I get no where near that in Veeam. I wonder what Veeam is doing during a restore that kills performance so much.

I'll Switch to Per VM and see if that helps.
FedericoV
Technology Partner
Posts: 35
Liked: 37 times
Joined: Aug 21, 2017 3:27 pm
Full Name: Federico Venier
Contact:

Re: Feature Request - SureBackup staging area.

Post by FedericoV »

I guess that per-VM will work much better. Without the per-VM there is only one writing stream for all the VM in a job. The result is that the data from a VM is interleaved with the data of all the other VMs in the job. The backup is slower and you also get less dedupe. As you have tested, even the restore is quite slower. StoreOnce is optimized for sequential IO, but when you restore a VM from a multiplexed file, Veeam has to read a chunk of data from StoreOnce, than it will seek forward to avoid reading the other VMs blocks, and so on. This causes a lot of random IO on StoreOnce and makes the internal StoreOnce read-ahead less effective. I suggest disabling Veeam own dedupe and compression as well. Finally, if you can, make a test using Catalyst over LAN, it is even faster and, by the way, IVMR has been certified with it.
Post Reply

Who is online

Users browsing this forum: ante_704, Bing [Bot], JTT, vtsybin and 264 guests