HP StoreOnce 4500 and Veeam

[BrandonH]

I'm wanting to find out peoples impressions of using the StoreOnce devices and Veeam. Currently I'm using three Drobo b1200i's that just aren't cutting it anymore. With 7/14 day retention I can scrape by with around 2TB free space, assuming I run disk cleanup fairly often.

In the end, I want something that lets me combine both my VM and Physical server backups to a single system as well. We also want longer retention times, as 14 days just isn't enough. We're considering two replicating 4500's, one at our main office, one at our backup data center connected via 10g Ethernet and 10g FC (Both DWDM). We currently use two 3PAR 7400's as our SP's, sitting on a Brocade SAN.

So what do you think, is the StoreOnce worth it?

[Rick.Vanover]

The StoreOnce would have to be better than the Drobos...

[veremin]

Once you have StoreOnce in place, set it in NFS mode, as the latter gives the better performance in comparison to CIFS. If the usage of CIFS is mandatory, make sure to install the latest firmware, once it's released to general public:

Kudos to HP engineers, who were very quick to fix the StoreOnce CIFS performance issue with modern Windows clients that we have ran into earlier this month. We've got a hold of the newer firmware build (version 3.11.1) and have confirmed that the CIFS performance was significantly improved in both of our labs. I don't believe this build is generally available yet, but at least you know now what to look for. Based on our very limited and small scale testing, we suspect that NFS will easily beat CIFS in top throughput even with this firmware, but at least this update brings them fairly on par with less than 10% difference, as opposed to 10x difference previously.

Thanks.

[BrandonH]

It'll be in next year's budget, so by then the firmware should be out. If it's not, I doubt we'd have an issue getting it. We're currently running beta firmware on one of our 7400's for Luns larger than 16TiB.

[veremin]

Then, you shouldn't be worried as the said firmware significantly increases the backup performance. Though, NFS still seems to be slightly better. Thanks.

[Gostev]

Slightly as in 50% in some cases your mileage may vary!

[tsightler]

The new firmware definitely makes a huge difference in CIFS performance. In both lab and field testing we've seen anywhere from 5-10x faster, however, the top end throughput of NFS is still faster than CIFS (this is generally true on a other dedupe appliances as well). The difference so far has varied from 10% to 50% in various environments and testing scenarios. If you're looking for absolute best performance, NFS is still likely the way to go, if you're looking for simplicity and can deal with not quite getting top speeds from a single I/O stream, CIFS is probably "good enough" with the updated firmware. With old firmware the difference between CIFS and NFS was more like 500% to 1000%, so the gap has narrowed nicely.

[emachabert]

I'm really curious to know what bandwidth you get with the older firmware that suffer the performance issue.

I have many Storeonce (G2 and G3) in the wild working with veeam on firmware 2.X and 3X (3.9 for the most recent one).
I have never seen "real" performance problem with those arrays. Proxies are Win2K8R2. The main problems have been wip files and hidden used space.

With gigabit models, over CIFS, I'm writing at 115MB/s and reading at 70/80 MB/s when offloading to tape. Going further than 115MB/s on one Gigabit would be hard. Perhaps I could boost the read performance ?

[tsightler]

Most of the performance benefits of NFS are during restore, well, technically, any read operation. As always if you are happy with the performance you are seeing it really doesn't matter. Most of the units I'm working with are large units with multiple disk shelves and 10GbE connectivity. With those units I was seeing around 150-170MB/s writes, but with some long pauses in the job between tasks and files. With NFS the performance increased to 220-250MB/s, and is almost 100% consistent during the job. In this case we were running mulitple jobs, and those performance numbers are "per-job". As jobs were ramped up on CIFS, performance per-stream would slow, for example, with two streams each might write at 125-150MB/s, so total throughput of 300MB/s for both jobs. With NFS, performance was 450-500MB/s with two jobs. Of course if you're bandwidth constrained none of that will really matter as your right, 115MB/s over 1GbE is already quite good.

Restores/tape offload were the biggest improvements. In the worst examples we were seeing full VM restore speeds of only 5-20MB/s, even when backup speeds were 100MB/s or more. Tape speeds were usually better but roughly half of what they should be. Using NFS drastically changed that to restore performance being very similar to backup performance.

I really don't remember seeing this issue with 2.X firmware, but that testing was quite some time ago. The results were reproducible across a wide range of solutions, from a StoreOnce VSA running in my home lab, to entry level StoreOnce appliances, to large B6200 units running in enterprise environments.

So in summary, the primary driver was restore performance for full VM restore and tape offload, backup speeds via CIFS were generally acceptable, although the customers that made the switch and had available bandwidth still saw a nice boost there as well.

[emachabert]

For having worked with Storeonce devs at Bristol labs at the time of 2.X for a denial of service problem on the CIFS engine, I can say that the two generations can't be compared since the CIFS engine is not the same. Before 3.X, CIFS engine was an OEM integration built by Likewise, the same as Datadomain by the time. But EMC bought Likewise, so HP decided to build its own CIFS engine, first released with 3.X firmware. That should explain that the problem didn't show up in 2.x

[Gostev]

I believe Tom actually saw Likewise of some really old and weird version when digging inside of the 3.x firmware...

[emachabert]

3.11 has been released and include the fix.

[dcolpitts]

Yeah - upgrading our 4700s from 3.9.4 to 3.11.0 made a HUGE difference in performance, and call me crazy, I think replication has sped up in our environment too. I didn't measure it yet though, and we didn't didn't increase bandwidth during the past couple of months, but after a 10 days on 3.11.0, replication seems to be completed before I ever manage to crawl out of bed in the morning and check in. With 3.9.4, replication was typically ending an hour or two after I would first get logged in around 7 am, and now replication is completed when I log in.

These 4700s are my first production experience with the StoreOnce product line - but they certainly won't be my last. I only wish we had started implementing StoreOnce sooner at our client locations. We are also targeting Backup Exec 2012 & 2014 for physical servers to the 4700s. I had to restore a a several hundred gig folder the other day from BE2014 off one of the 4700s - wow is all I can say. It was done before I got back with my coffee 30 seconds after I hit submit. We are connected via bonded 10GbE for all servers (ESXi and 2008 R2) and the 4700s connected via a ProCurve 5406 ZL switch.

The 4700s we have have 48TB raw (38.something TB usable), and have been in production since early March. Our combined BE / Veeam dedup ratio is 6.45:1, and according to the StoreOnce Enterprise Manager, 95% utilization of capacity won't hit for another 8+ months, so we'll have darn close to 1 year of full daily backups of approximately 4TB raw data per day there. Very impressive in my opinion.

dcc

[Jack1874]

Most of the performance benefits of NFS are during restore, well, technically, any read operation. As always if you are happy with the performance you are seeing it really doesn't matter. Most of the units I'm working with are large units with multiple disk shelves and 10GbE connectivity. With those units I was seeing around 150-170MB/s writes, but with some long pauses in the job between tasks and files. With NFS the performance increased to 220-250MB/s, and is almost 100% consistent during the job. In this case we were running mulitple jobs, and those performance numbers are "per-job". As jobs were ramped up on CIFS, performance per-stream would slow, for example, with two streams each might write at 125-150MB/s, so total throughput of 300MB/s for both jobs. With NFS, performance was 450-500MB/s with two jobs. Of course if you're bandwidth constrained none of that will really matter as your right, 115MB/s over 1GbE is already quite good.

Hi tsightler

"With those units I was seeing around 150-170MB/s writes, but with some long pauses in the job between tasks and files".

Did you ever manage to solve the long pause issue? We have a StoreOnce 4900 with NFS exports\shares, Windows 2012 R2 (physical Proxies) using Storage Snapshots.

I see the long pause and was wondering if there is a root cause for it?

TIA

[emachabert]

Long pause can be caused by ethernet flow control kicking in. If you do a network capture and see pause frames being transmitted then try disabling flowcontrol and let tcp/ip manage the flow speed. I've seen a lot of this kind of long pause when you have a mixed speed network (gig/10gig) in the backup path.

[sbojic]

"With those units I was seeing around 150-170MB/s writes, but with some long pauses in the job between tasks and files".

Did you ever manage to solve the long pause issue? We have a StoreOnce 4900 with NFS exports\shares, Windows 2012 R2 (physical Proxies) using Storage Snapshots.

I see the long pause and was wondering if there is a root cause for it?

TIA

Jack, I had this exact problem and was directly linked to the Repository Type i.e. change it to Local Store 16+TB. Larger blocks will bring this delay to the minimum. Also, I noticed that if I have too many concurrent jobs (like 5-6 per Job) it will also slow down, that is delays will be longer, so I settled to 3 HDs/Job with 2 Jobs running at the same time.

Hope this helps

[mmugaiga]

I'm experiencing a challenge with HP Store once 4500 Firmware upgrade from version 3.9.4 to version 3.13.9, it shows update failed due to system error.

Kindly see error below:

Initializing.
Verifying staged package repository integrity.
Package stage area verification successful.
Updating cluster from version '3.9.4-1411.2' to '3.13.9-1843.1'.
Reboot will occur after update completes.
Installing update tool packages. Can take 5 minutes to complete.
Installing: D2D_UpdateTools
Install successful
Update tool package installation successful. 0 minutes elapsed.
Updating from version '3.9.4-1411.2' to '3.13.9-1843.1'.
Reboot will occur after update completes.
ERROR Upgrading from 3.9.4 to 3.13.9 is not supported.
Removing update tools packages.
Update encountered system error(s) before starting the update.

Please if you have encountered this challenge before and found a way to go about it or have an idea on how to go about it, kindly assist.

Thanks and Regards,
Michael

[Steve-nIP]

I'd definitely create a case with HPE as soon as possible.

[ianbutton1]

I'd check HPE documentation about what upgrade paths are supported - probably one or more intermediate upgrade stages are necessary.
Read https://support.hpe.com/hpesc/public/do ... 38994en_us - it says upgrade to 3.11 first.
Good luck!

[foggy]

Hi Michael, this is definitely a question for HPE support, Veeam is out of the picture. Please also keep in mind that to be able to use Veeam B&R with this array, you would need to upgrade at least to 3.15.1. Thanks!

[evilaedmin]

I'm experiencing a challenge with HP Store once 4500 Firmware upgrade from version 3.9.4 to version 3.13.9, it shows update failed due to system error.

Michael,

If you sign into support center and check release notes for 3.13.9 you will find requirements:

.In order to upgrade to 3.13.9 software you should first be running 3.13.8, 3.13.5, 3.13.4, 3.13.3, 3.13.2, 3.13.1 or 3.13.0 software or one of the two previous software versions: 3.12.x and 3.11.x.

You will need to review release notes of all releases since 3.9.4 to see the order in which you must do upgrade.

[mmugaiga]

Thank you all, i appreciate your assistance.