Feature Request - SureBackup staging area.

[SBarrett847]

I love the Idea of Surebackup, however my main Backup repo is a StoreOnce who's Random I/O performance is abysmal. This prevents the VM being able to boot within any reasonable time frame. However it does have a reasonable restore rate (still not great though).

I would like if Surebackup could fully restore the VM to local storage or primary storage first, then boot and test the VM from there.

[foggy]

Hi Stephen, what SO firmware version does your storage run? StoreOnce restore performance was significantly increased in 3.15.1, specifically to allow SureBackup/Instant Recovery possible. So make sure you're running the recent one.

[SBarrett847]

Hi Foggy - I'm on 3.16.3-1730.1 - however it is a CIFS share, not catalyst if that makes any difference.

[foggy]

It could. But also the overall setup might not be optimal. Some hints can be found here. Btw, what kind of VM's are you restoring - do they have a single or multiple disks?

[SBarrett847]

Testing now and I'm Getting about 15-20 MB/s per stream from the my StoreOnce (4500) I've never ever seen anything like 275MB/s restores as in that link. I have been able get up to 250/300 MB/s going into the StoreOnce, but coming out it is very slow.

From the Link it would seem that if you aren't using catalyst, you should probably not be using incrementals, as the performance is better with Full backups. Also it recommends using primary storage snapshots to allow for lightning restores. However as I use Hyper-V, and Veeam can't Leverage native 3Par snapshots for restores on Hyper-V, I'm hamstrung and have to rely on the StoreOnce to perform all backup tasks.

StoreOnce IMO is not really suitable for Primary Backup storage, and in places I have Backups going to an old SAN for a few VM I really need to be able to use Instant recovery.

However - I digress - If I could get Surebackup to restore to a staging area (local HDD or SSD array for example) and then boot and perform tests, I could get the Surebackup jobs to complete.

[foggy]

Well, using any dedupe device as primary backup storage is actually against our reference architecture.

[SBarrett847]

Yeah, I know that now

It is my fault for letting HP design the solution originally. Will consider Exagrid for the future I think.

[foggy]

I didn't mean that actually, but rather introducing some raw device in-between, to better comply with the 3-2-1 rule.

[FedericoV]

Hi Stephen, StoreOnce random IO has been optimized for Catalyst only. If you are using CIFS, than IVMR is quite slower than using Catalyst and, sorry to say, it is even not supported.
If you can, I suggest you to test IVMR with a Backup Repository created on a Catalyst Store.
Regarding the Restore Performance, your 15-20MB/s per stream are quite lower then the expected performance for your SO4500. For a single stream restore I would expect about 200MB/s.
If you can, try mounting the CIFS share from a server and make a command line test such as "copy /b MyLargeFile.vbk NUL" ? Write NUL not NULL, it works similarly to the unix /dev/null.
Another point to check is the "per VM file". we know that the restore (and the dedupe) is lower when there is one file per job rather than per VM.
I hope this can help you discover the root cause of your slow full restore performance.

[SBarrett847]

Much obliged Frederico - I do indeed see ~200MB/s using the NUL copy (and Copying to a local Flash array)- however I get no where near that in Veeam. I wonder what Veeam is doing during a restore that kills performance so much.

I'll Switch to Per VM and see if that helps.

[FedericoV]

I guess that per-VM will work much better. Without the per-VM there is only one writing stream for all the VM in a job. The result is that the data from a VM is interleaved with the data of all the other VMs in the job. The backup is slower and you also get less dedupe. As you have tested, even the restore is quite slower. StoreOnce is optimized for sequential IO, but when you restore a VM from a multiplexed file, Veeam has to read a chunk of data from StoreOnce, than it will seek forward to avoid reading the other VMs blocks, and so on. This causes a lot of random IO on StoreOnce and makes the internal StoreOnce read-ahead less effective. I suggest disabling Veeam own dedupe and compression as well. Finally, if you can, make a test using Catalyst over LAN, it is even faster and, by the way, IVMR has been certified with it.