Comprehensive data protection for all workloads
Post Reply
BrandonH
Influencer
Posts: 21
Liked: 2 times
Joined: Jul 10, 2013 4:37 pm
Full Name: Brandon Hogue
Contact:

HP StoreOnce 4500 and Veeam

Post by BrandonH »

I'm wanting to find out peoples impressions of using the StoreOnce devices and Veeam. Currently I'm using three Drobo b1200i's that just aren't cutting it anymore. With 7/14 day retention I can scrape by with around 2TB free space, assuming I run disk cleanup fairly often.

In the end, I want something that lets me combine both my VM and Physical server backups to a single system as well. We also want longer retention times, as 14 days just isn't enough. We're considering two replicating 4500's, one at our main office, one at our backup data center connected via 10g Ethernet and 10g FC (Both DWDM). We currently use two 3PAR 7400's as our SP's, sitting on a Brocade SAN.

So what do you think, is the StoreOnce worth it?
Rick.Vanover
Veeam Software
Posts: 708
Liked: 167 times
Joined: Nov 30, 2010 3:19 pm
Full Name: Rick Vanover
Location: Columbus, Ohio USA
Contact:

Re: HP StoreOnce 4500 and Veeam

Post by Rick.Vanover »

The StoreOnce would have to be better than the Drobos...
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: HP StoreOnce 4500 and Veeam

Post by veremin »

Once you have StoreOnce in place, set it in NFS mode, as the latter gives the better performance in comparison to CIFS. If the usage of CIFS is mandatory, make sure to install the latest firmware, once it's released to general public:
Gostev wrote:Kudos to HP engineers, who were very quick to fix the StoreOnce CIFS performance issue with modern Windows clients that we have ran into earlier this month. We've got a hold of the newer firmware build (version 3.11.1) and have confirmed that the CIFS performance was significantly improved in both of our labs. I don't believe this build is generally available yet, but at least you know now what to look for. Based on our very limited and small scale testing, we suspect that NFS will easily beat CIFS in top throughput even with this firmware, but at least this update brings them fairly on par with less than 10% difference, as opposed to 10x difference previously.
Thanks.
BrandonH
Influencer
Posts: 21
Liked: 2 times
Joined: Jul 10, 2013 4:37 pm
Full Name: Brandon Hogue
Contact:

Re: HP StoreOnce 4500 and Veeam

Post by BrandonH »

It'll be in next year's budget, so by then the firmware should be out. If it's not, I doubt we'd have an issue getting it. We're currently running beta firmware on one of our 7400's for Luns larger than 16TiB.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: HP StoreOnce 4500 and Veeam

Post by veremin »

Then, you shouldn't be worried as the said firmware significantly increases the backup performance. Though, NFS still seems to be slightly better. Thanks.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: HP StoreOnce 4500 and Veeam

Post by Gostev »

Slightly as in 50% in some cases ;) your mileage may vary!
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: HP StoreOnce 4500 and Veeam

Post by tsightler »

The new firmware definitely makes a huge difference in CIFS performance. In both lab and field testing we've seen anywhere from 5-10x faster, however, the top end throughput of NFS is still faster than CIFS (this is generally true on a other dedupe appliances as well). The difference so far has varied from 10% to 50% in various environments and testing scenarios. If you're looking for absolute best performance, NFS is still likely the way to go, if you're looking for simplicity and can deal with not quite getting top speeds from a single I/O stream, CIFS is probably "good enough" with the updated firmware. With old firmware the difference between CIFS and NFS was more like 500% to 1000%, so the gap has narrowed nicely.
emachabert
Veeam Vanguard
Posts: 388
Liked: 168 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: HP StoreOnce 4500 and Veeam

Post by emachabert »

I'm really curious to know what bandwidth you get with the older firmware that suffer the performance issue.

I have many Storeonce (G2 and G3) in the wild working with veeam on firmware 2.X and 3X (3.9 for the most recent one).
I have never seen "real" performance problem with those arrays. Proxies are Win2K8R2. The main problems have been wip files and hidden used space.

With gigabit models, over CIFS, I'm writing at 115MB/s and reading at 70/80 MB/s when offloading to tape. Going further than 115MB/s on one Gigabit would be hard. Perhaps I could boost the read performance ?
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: HP StoreOnce 4500 and Veeam

Post by tsightler » 1 person likes this post

Most of the performance benefits of NFS are during restore, well, technically, any read operation. As always if you are happy with the performance you are seeing it really doesn't matter. Most of the units I'm working with are large units with multiple disk shelves and 10GbE connectivity. With those units I was seeing around 150-170MB/s writes, but with some long pauses in the job between tasks and files. With NFS the performance increased to 220-250MB/s, and is almost 100% consistent during the job. In this case we were running mulitple jobs, and those performance numbers are "per-job". As jobs were ramped up on CIFS, performance per-stream would slow, for example, with two streams each might write at 125-150MB/s, so total throughput of 300MB/s for both jobs. With NFS, performance was 450-500MB/s with two jobs. Of course if you're bandwidth constrained none of that will really matter as your right, 115MB/s over 1GbE is already quite good.

Restores/tape offload were the biggest improvements. In the worst examples we were seeing full VM restore speeds of only 5-20MB/s, even when backup speeds were 100MB/s or more. Tape speeds were usually better but roughly half of what they should be. Using NFS drastically changed that to restore performance being very similar to backup performance.

I really don't remember seeing this issue with 2.X firmware, but that testing was quite some time ago. The results were reproducible across a wide range of solutions, from a StoreOnce VSA running in my home lab, to entry level StoreOnce appliances, to large B6200 units running in enterprise environments.

So in summary, the primary driver was restore performance for full VM restore and tape offload, backup speeds via CIFS were generally acceptable, although the customers that made the switch and had available bandwidth still saw a nice boost there as well.
emachabert
Veeam Vanguard
Posts: 388
Liked: 168 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: HP StoreOnce 4500 and Veeam

Post by emachabert »

For having worked with Storeonce devs at Bristol labs at the time of 2.X for a denial of service problem on the CIFS engine, I can say that the two generations can't be compared since the CIFS engine is not the same. Before 3.X, CIFS engine was an OEM integration built by Likewise, the same as Datadomain by the time. But EMC bought Likewise, so HP decided to build its own CIFS engine, first released with 3.X firmware. That should explain that the problem didn't show up in 2.x
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: HP StoreOnce 4500 and Veeam

Post by Gostev »

I believe Tom actually saw Likewise of some really old and weird version when digging inside of the 3.x firmware...
emachabert
Veeam Vanguard
Posts: 388
Liked: 168 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: HP StoreOnce 4500 and Veeam

Post by emachabert » 1 person likes this post

3.11 has been released and include the fix.
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
dcolpitts
Veeam ProPartner
Posts: 119
Liked: 24 times
Joined: Apr 01, 2011 10:36 am
Full Name: Dean Colpitts
Location: Atlantic coast of Canada
Contact:

Re: HP StoreOnce 4500 and Veeam

Post by dcolpitts » 2 people like this post

Yeah - upgrading our 4700s from 3.9.4 to 3.11.0 made a HUGE difference in performance, and call me crazy, I think replication has sped up in our environment too. I didn't measure it yet though, and we didn't didn't increase bandwidth during the past couple of months, but after a 10 days on 3.11.0, replication seems to be completed before I ever manage to crawl out of bed in the morning and check in. With 3.9.4, replication was typically ending an hour or two after I would first get logged in around 7 am, and now replication is completed when I log in.

These 4700s are my first production experience with the StoreOnce product line - but they certainly won't be my last. I only wish we had started implementing StoreOnce sooner at our client locations. We are also targeting Backup Exec 2012 & 2014 for physical servers to the 4700s. I had to restore a a several hundred gig folder the other day from BE2014 off one of the 4700s - wow is all I can say. It was done before I got back with my coffee 30 seconds after I hit submit. We are connected via bonded 10GbE for all servers (ESXi and 2008 R2) and the 4700s connected via a ProCurve 5406 ZL switch.

The 4700s we have have 48TB raw (38.something TB usable), and have been in production since early March. Our combined BE / Veeam dedup ratio is 6.45:1, and according to the StoreOnce Enterprise Manager, 95% utilization of capacity won't hit for another 8+ months, so we'll have darn close to 1 year of full daily backups of approximately 4TB raw data per day there. Very impressive in my opinion.

dcc
Jack1874
Enthusiast
Posts: 95
Liked: 5 times
Joined: Oct 17, 2015 3:32 pm
Full Name: Stuart Little
Location: Canada
Contact:

Re: HP StoreOnce 4500 and Veeam

Post by Jack1874 »

tsightler wrote:Most of the performance benefits of NFS are during restore, well, technically, any read operation. As always if you are happy with the performance you are seeing it really doesn't matter. Most of the units I'm working with are large units with multiple disk shelves and 10GbE connectivity. With those units I was seeing around 150-170MB/s writes, but with some long pauses in the job between tasks and files. With NFS the performance increased to 220-250MB/s, and is almost 100% consistent during the job. In this case we were running mulitple jobs, and those performance numbers are "per-job". As jobs were ramped up on CIFS, performance per-stream would slow, for example, with two streams each might write at 125-150MB/s, so total throughput of 300MB/s for both jobs. With NFS, performance was 450-500MB/s with two jobs. Of course if you're bandwidth constrained none of that will really matter as your right, 115MB/s over 1GbE is already quite good.

Hi tsightler

"With those units I was seeing around 150-170MB/s writes, but with some long pauses in the job between tasks and files".

Did you ever manage to solve the long pause issue? We have a StoreOnce 4900 with NFS exports\shares, Windows 2012 R2 (physical Proxies) using Storage Snapshots.

I see the long pause and was wondering if there is a root cause for it?

TIA
emachabert
Veeam Vanguard
Posts: 388
Liked: 168 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: HP StoreOnce 4500 and Veeam

Post by emachabert »

Long pause can be caused by ethernet flow control kicking in. If you do a network capture and see pause frames being transmitted then try disabling flowcontrol and let tcp/ip manage the flow speed. I've seen a lot of this kind of long pause when you have a mixed speed network (gig/10gig) in the backup path.
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
sbojic
Lurker
Posts: 1
Liked: never
Joined: Jun 04, 2014 3:21 pm
Full Name: Sergej Bojic
Contact:

Re: HP StoreOnce 4500 and Veeam

Post by sbojic »

Jack1874 wrote:
"With those units I was seeing around 150-170MB/s writes, but with some long pauses in the job between tasks and files".

Did you ever manage to solve the long pause issue? We have a StoreOnce 4900 with NFS exports\shares, Windows 2012 R2 (physical Proxies) using Storage Snapshots.

I see the long pause and was wondering if there is a root cause for it?

TIA
Jack, I had this exact problem and was directly linked to the Repository Type i.e. change it to Local Store 16+TB. Larger blocks will bring this delay to the minimum. Also, I noticed that if I have too many concurrent jobs (like 5-6 per Job) it will also slow down, that is delays will be longer, so I settled to 3 HDs/Job with 2 Jobs running at the same time.

Hope this helps
mmugaiga
Lurker
Posts: 2
Liked: never
Joined: Feb 19, 2020 1:30 pm
Full Name: Michael Mugaiga
Contact:

Re: HP StoreOnce 4500 and Veeam

Post by mmugaiga »

I'm experiencing a challenge with HP Store once 4500 Firmware upgrade from version 3.9.4 to version 3.13.9, it shows update failed due to system error.

Kindly see error below:

Initializing.
Verifying staged package repository integrity.
Package stage area verification successful.
Updating cluster from version '3.9.4-1411.2' to '3.13.9-1843.1'.
Reboot will occur after update completes.
Installing update tool packages. Can take 5 minutes to complete.
Installing: D2D_UpdateTools
Install successful
Update tool package installation successful. 0 minutes elapsed.
Updating from version '3.9.4-1411.2' to '3.13.9-1843.1'.
Reboot will occur after update completes.
ERROR Upgrading from 3.9.4 to 3.13.9 is not supported.
Removing update tools packages.
Update encountered system error(s) before starting the update.

Please if you have encountered this challenge before and found a way to go about it or have an idea on how to go about it, kindly assist.

Thanks and Regards,
Michael
Steve-nIP
Service Provider
Posts: 117
Liked: 49 times
Joined: Feb 06, 2018 10:08 am
Full Name: Steve
Contact:

Re: HP StoreOnce 4500 and Veeam

Post by Steve-nIP »

I'd definitely create a case with HPE as soon as possible.
ianbutton1
Enthusiast
Posts: 58
Liked: 18 times
Joined: Oct 14, 2016 3:54 pm
Full Name: Ian Button
Contact:

Re: HP StoreOnce 4500 and Veeam

Post by ianbutton1 » 1 person likes this post

I'd check HPE documentation about what upgrade paths are supported - probably one or more intermediate upgrade stages are necessary.
Read https://support.hpe.com/hpesc/public/do ... 38994en_us - it says upgrade to 3.11 first.
Good luck!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: HP StoreOnce 4500 and Veeam

Post by foggy »

Hi Michael, this is definitely a question for HPE support, Veeam is out of the picture. Please also keep in mind that to be able to use Veeam B&R with this array, you would need to upgrade at least to 3.15.1. Thanks!
evilaedmin
Expert
Posts: 176
Liked: 30 times
Joined: Jul 26, 2018 8:04 pm
Full Name: Eugene V
Contact:

Re: HP StoreOnce 4500 and Veeam

Post by evilaedmin »

mmugaiga wrote: Feb 20, 2020 5:51 am I'm experiencing a challenge with HP Store once 4500 Firmware upgrade from version 3.9.4 to version 3.13.9, it shows update failed due to system error.
Michael,

If you sign into support center and check release notes for 3.13.9 you will find requirements:
.In order to upgrade to 3.13.9 software you should first be running 3.13.8, 3.13.5, 3.13.4, 3.13.3, 3.13.2, 3.13.1 or 3.13.0 software or one of the two previous software versions: 3.12.x and 3.11.x.
You will need to review release notes of all releases since 3.9.4 to see the order in which you must do upgrade.
mmugaiga
Lurker
Posts: 2
Liked: never
Joined: Feb 19, 2020 1:30 pm
Full Name: Michael Mugaiga
Contact:

Re: HP StoreOnce 4500 and Veeam

Post by mmugaiga »

Thank you all, i appreciate your assistance.
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 159 guests