HP Storeonce - Catalyst Shares (CoFC)

[gdavid16]

We currently have a Veeam installation ( 2 physical proxies) with HP Storeonce 6500 over CoFC, 3PAR storage snapshots.
Added on is encryption on the shares. This required a firmware upgrade on the storeonce as it threw CoFC errors when we had it enabled. 3PAR storage snapshots work great. The whole thing pretty much works over fiber channel which is great for us.

We started backing up about 125 VMs. 2 weeks worth of daily, backup copy (local - back to the same storeonce for monthlys/yrly), a remote copy to our secondary datacenter (secondary storeonce). Synthetic fulls on Saturdays.
The storeonce (catalyst) cannot seem to handle the number of data streams & command sessions required to handle this. Veeam also has no mechanism to deal with how many sessions are currently in use because it bursts over the total in a matter of seconds of it checking. this becomes worse since we have 2 proxies and it needs to deal with how both proxies are hitting the storeonce at a single time.

we currently have HP and Veeam developers working on fixes for this.
is anyone else using CoFC? how many proxies? how many VMs in a given job? Do you have encryption enabled?

thanks for any feedback

Gd

[gdavid16]

from another thread i hijacked

Hi George, you want to avoid pointing too many tasks to the same StoreOnce unit, because its limit of concurrently open files is very low comparing to other storage devices. I voiced my concern about this to HP back when we started working on the integration, and was assured this is something they will address in close future. But for now, from reliability perspective it is best not to run too many simultaneous tasks against StoreOnce, or you will keep hitting the limit. Thanks!

[vdellachiesa]

I think we can conclude we are the only suckers at this point.

[Gostev]

Well, our integration with StoreOnce was just released anyway, and many of our larger customers have not even upgraded to v9 yet...

[SyNtAxx]

Sorry, I don't feel that is a legitimate excuse. My company was very disappointed with the level of actual integration of the Catalyst/StoreOnce, so much so we've abandoned using it and went back to SMB shares of the HPE StoreOnce 6500s. We certainly aren't the largest user of veeam, but we are not the smallest either. Hundreds of terabytes of vm data backed up and over 100 of Ent+ licenses.....

[Gostev]

Sorry, I don't feel that is a legitimate excuse.

Perhaps because it was not an excuse to start with?
I was merely explaining to the previous poster why no other users are chiming in.

I know you are upset with the fact that there is no support for Catalyst replication, however to be completely fair we have pre-announced every v9 feature months before the actual release with excessive level of details on the actual functionality we're delivering, so it's not like we over-promised and under-delivered. Anyway, Catalyst replication is an off-topic to this discussion, let's please stay on topic with future posts in this thread.

[vdellachiesa]

Veeam/HP and our Veeam Partner have identified some logic errors and have put forth a lot of work to correct this for us. At the end of the day we are definitely happy about the effort put forth to solve this problem for us. I'm still concerned that the solution won't scale to meet our future needs.

[justin23]

@vdellachiesa Can you elaborate about what errors were found. I've just taken over a Veeam System with Catalyst StoreOnce integration and noting a lot of CoFC errors. I thought it may have been the number of VMs as a single job with a single VM works fine, yet a job with as few as 10-15 VM's still has this error. I've also noted its only at one datacentre mostly where as the other one is running pretty well. I just spent a day reconfiguring the StoreOnce as well.

[vdellachiesa]

Sorry for the late and short reply, getting over a rough sinus infection. I believe we were receiving cofc 1404/03 error messages. Very high level was Veeam was trying to use more threads then the StoreOnce would allow. We received a private hf for v9 as well as redesign suggestion (from Veeam) around our SO catalyst shares and the way we were utilizing backup copy jobs. We have a 2 node SO. Originally we had proxy 1 as the gateway server for all the shares on node 1. When a job ran that could utilize both our proxies we were limiting our connections/througput to node 1 by not letting proxy 2 connect directly to a share on node 1, instead the data flow was proxy 2->proxy 1->SO node 1.

If you are receiving 1404/1403 errors give your Veeam rep a call(feel free to reference mine and gdavids posts) and have them review the logs to determine if you are hitting a thread limit issue. If so, you'll need the hf and maybe a job review to see if you can reduce your thread count.

[vdellachiesa]

Are the units the same in both datacenters? Do you have the same amount of fiber connections to both units and the same amount of devices per login(setting on SO).

[justin23]

The setup is identical at both datacentres, but I'm still seeing errors at both datacentres. Its just one of them, the job basically was all CoFC 1404 error messages, the other one only occasional. I've got around it by adding the same Catalyst store as a non CoFC repository in Veeam. It looks as though the Backup job can handle the change from CoFC to LAN without breaking the chain. Of course performance isn't as good, but i also need all VM's backing up!

I ran a test job with 1 VM and it worked, meaning the infrastructure itself was at least configured to work. So my conclusion is there were perhaps something different between datacentres, but yet to find that, however the more i research the move it looks like both datacentres have the same issue. I've got my local Veeam Reps business cards so I'll contact them today. Did you need any assistance from HP?

[vdellachiesa]

We worked with both HP\Veeam and a 3rd party vendor who was doing the install for us. At the end of the day it was Veeam who created the private HF to "correct" the 1404 errors for us. We were told there is a possibility the logic to address the 1404 error messages in V9.5 (but you still need a registry entry to make it work). The name of the registry key (reg dword) is StoreOnceMaxFcFileSessions, hopefully this information will escalate your case.

[justin23]

So an update regarding my issue. Sounds very similar fix to just done on the Storeonce.
On the Storeonce we set the max initiators per port to 64 on the fibre channel settings (It was 4 which I though was the correct setting). This seems to have resolved my problems, although waiting to see how it performs on the weekend full backups

[foggy]

Right, the recommendation is to set this number to a maximum value, unless there's some specific need to limit the number of target LUNs the client server sees. This should be mentioned in the configuration guide.

[justin23]

Just confirming after a month or so, the CoFC errors have not come back since I set the max initiators per port to 64 on the Storeonce.

[Gostev]

Great, thanks for following up!