-
- Service Provider
- Posts: 315
- Liked: 41 times
- Joined: Feb 02, 2016 5:02 pm
- Full Name: Stephen Barrett
- Contact:
Feature Request - SureBackup staging area.
I love the Idea of Surebackup, however my main Backup repo is a StoreOnce who's Random I/O performance is abysmal. This prevents the VM being able to boot within any reasonable time frame. However it does have a reasonable restore rate (still not great though).
I would like if Surebackup could fully restore the VM to local storage or primary storage first, then boot and test the VM from there.
I would like if Surebackup could fully restore the VM to local storage or primary storage first, then boot and test the VM from there.
-
- Veeam Software
- Posts: 21138
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Feature Request - SureBackup staging area.
Hi Stephen, what SO firmware version does your storage run? StoreOnce restore performance was significantly increased in 3.15.1, specifically to allow SureBackup/Instant Recovery possible. So make sure you're running the recent one.
-
- Service Provider
- Posts: 315
- Liked: 41 times
- Joined: Feb 02, 2016 5:02 pm
- Full Name: Stephen Barrett
- Contact:
Re: Feature Request - SureBackup staging area.
Hi Foggy - I'm on 3.16.3-1730.1 - however it is a CIFS share, not catalyst if that makes any difference.
-
- Veeam Software
- Posts: 21138
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Feature Request - SureBackup staging area.
It could. But also the overall setup might not be optimal. Some hints can be found here. Btw, what kind of VM's are you restoring - do they have a single or multiple disks?
-
- Service Provider
- Posts: 315
- Liked: 41 times
- Joined: Feb 02, 2016 5:02 pm
- Full Name: Stephen Barrett
- Contact:
Re: Feature Request - SureBackup staging area.
Testing now and I'm Getting about 15-20 MB/s per stream from the my StoreOnce (4500) I've never ever seen anything like 275MB/s restores as in that link. I have been able get up to 250/300 MB/s going into the StoreOnce, but coming out it is very slow.
From the Link it would seem that if you aren't using catalyst, you should probably not be using incrementals, as the performance is better with Full backups. Also it recommends using primary storage snapshots to allow for lightning restores. However as I use Hyper-V, and Veeam can't Leverage native 3Par snapshots for restores on Hyper-V, I'm hamstrung and have to rely on the StoreOnce to perform all backup tasks.
StoreOnce IMO is not really suitable for Primary Backup storage, and in places I have Backups going to an old SAN for a few VM I really need to be able to use Instant recovery.
However - I digress - If I could get Surebackup to restore to a staging area (local HDD or SSD array for example) and then boot and perform tests, I could get the Surebackup jobs to complete.
From the Link it would seem that if you aren't using catalyst, you should probably not be using incrementals, as the performance is better with Full backups. Also it recommends using primary storage snapshots to allow for lightning restores. However as I use Hyper-V, and Veeam can't Leverage native 3Par snapshots for restores on Hyper-V, I'm hamstrung and have to rely on the StoreOnce to perform all backup tasks.
StoreOnce IMO is not really suitable for Primary Backup storage, and in places I have Backups going to an old SAN for a few VM I really need to be able to use Instant recovery.
However - I digress - If I could get Surebackup to restore to a staging area (local HDD or SSD array for example) and then boot and perform tests, I could get the Surebackup jobs to complete.
-
- Veeam Software
- Posts: 21138
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Feature Request - SureBackup staging area.
Well, using any dedupe device as primary backup storage is actually against our reference architecture.
-
- Service Provider
- Posts: 315
- Liked: 41 times
- Joined: Feb 02, 2016 5:02 pm
- Full Name: Stephen Barrett
- Contact:
Re: Feature Request - SureBackup staging area.
Yeah, I know that now
It is my fault for letting HP design the solution originally. Will consider Exagrid for the future I think.
It is my fault for letting HP design the solution originally. Will consider Exagrid for the future I think.
-
- Veeam Software
- Posts: 21138
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Feature Request - SureBackup staging area.
I didn't mean that actually, but rather introducing some raw device in-between, to better comply with the 3-2-1 rule.
-
- Technology Partner
- Posts: 36
- Liked: 38 times
- Joined: Aug 21, 2017 3:27 pm
- Full Name: Federico Venier
- Contact:
Re: Feature Request - SureBackup staging area.
Hi Stephen, StoreOnce random IO has been optimized for Catalyst only. If you are using CIFS, than IVMR is quite slower than using Catalyst and, sorry to say, it is even not supported.
If you can, I suggest you to test IVMR with a Backup Repository created on a Catalyst Store.
Regarding the Restore Performance, your 15-20MB/s per stream are quite lower then the expected performance for your SO4500. For a single stream restore I would expect about 200MB/s.
If you can, try mounting the CIFS share from a server and make a command line test such as "copy /b MyLargeFile.vbk NUL" ? Write NUL not NULL, it works similarly to the unix /dev/null.
Another point to check is the "per VM file". we know that the restore (and the dedupe) is lower when there is one file per job rather than per VM.
I hope this can help you discover the root cause of your slow full restore performance.
If you can, I suggest you to test IVMR with a Backup Repository created on a Catalyst Store.
Regarding the Restore Performance, your 15-20MB/s per stream are quite lower then the expected performance for your SO4500. For a single stream restore I would expect about 200MB/s.
If you can, try mounting the CIFS share from a server and make a command line test such as "copy /b MyLargeFile.vbk NUL" ? Write NUL not NULL, it works similarly to the unix /dev/null.
Another point to check is the "per VM file". we know that the restore (and the dedupe) is lower when there is one file per job rather than per VM.
I hope this can help you discover the root cause of your slow full restore performance.
-
- Service Provider
- Posts: 315
- Liked: 41 times
- Joined: Feb 02, 2016 5:02 pm
- Full Name: Stephen Barrett
- Contact:
Re: Feature Request - SureBackup staging area.
Much obliged Frederico - I do indeed see ~200MB/s using the NUL copy (and Copying to a local Flash array)- however I get no where near that in Veeam. I wonder what Veeam is doing during a restore that kills performance so much.
I'll Switch to Per VM and see if that helps.
I'll Switch to Per VM and see if that helps.
-
- Technology Partner
- Posts: 36
- Liked: 38 times
- Joined: Aug 21, 2017 3:27 pm
- Full Name: Federico Venier
- Contact:
Re: Feature Request - SureBackup staging area.
I guess that per-VM will work much better. Without the per-VM there is only one writing stream for all the VM in a job. The result is that the data from a VM is interleaved with the data of all the other VMs in the job. The backup is slower and you also get less dedupe. As you have tested, even the restore is quite slower. StoreOnce is optimized for sequential IO, but when you restore a VM from a multiplexed file, Veeam has to read a chunk of data from StoreOnce, than it will seek forward to avoid reading the other VMs blocks, and so on. This causes a lot of random IO on StoreOnce and makes the internal StoreOnce read-ahead less effective. I suggest disabling Veeam own dedupe and compression as well. Finally, if you can, make a test using Catalyst over LAN, it is even faster and, by the way, IVMR has been certified with it.
Who is online
Users browsing this forum: Bing [Bot] and 264 guests