HP Storeonce - Catalyst Shares (CoFC)

VMware specific discussions

HP Storeonce - Catalyst Shares (CoFC)

Veeam Logoby gdavid16 » Wed May 04, 2016 12:44 am

We currently have a Veeam installation ( 2 physical proxies) with HP Storeonce 6500 over CoFC, 3PAR storage snapshots.
Added on is encryption on the shares. This required a firmware upgrade on the storeonce as it threw CoFC errors when we had it enabled. 3PAR storage snapshots work great. The whole thing pretty much works over fiber channel which is great for us.

We started backing up about 125 VMs. 2 weeks worth of daily, backup copy (local - back to the same storeonce for monthlys/yrly), a remote copy to our secondary datacenter (secondary storeonce). Synthetic fulls on Saturdays.
The storeonce (catalyst) cannot seem to handle the number of data streams & command sessions required to handle this. Veeam also has no mechanism to deal with how many sessions are currently in use because it bursts over the total in a matter of seconds of it checking. this becomes worse since we have 2 proxies and it needs to deal with how both proxies are hitting the storeonce at a single time.

we currently have HP and Veeam developers working on fixes for this.
is anyone else using CoFC? how many proxies? how many VMs in a given job? Do you have encryption enabled?

thanks for any feedback

Gd
gdavid16
Influencer
 
Posts: 16
Liked: never
Joined: Thu Aug 27, 2015 2:05 pm
Full Name: George D.

Re: HP Storeonce - Catalyst Shares (CoFC)

Veeam Logoby gdavid16 » Wed May 04, 2016 12:46 am

from another thread i hijacked
Gostev wrote:Hi George, you want to avoid pointing too many tasks to the same StoreOnce unit, because its limit of concurrently open files is very low comparing to other storage devices. I voiced my concern about this to HP back when we started working on the integration, and was assured this is something they will address in close future. But for now, from reliability perspective it is best not to run too many simultaneous tasks against StoreOnce, or you will keep hitting the limit. Thanks!
gdavid16
Influencer
 
Posts: 16
Liked: never
Joined: Thu Aug 27, 2015 2:05 pm
Full Name: George D.

Re: HP Storeonce - Catalyst Shares (CoFC)

Veeam Logoby vdellachiesa » Sun May 22, 2016 11:36 pm

I think we can conclude we are the only suckers at this point.
vdellachiesa
Novice
 
Posts: 7
Liked: 2 times
Joined: Wed Jun 11, 2014 11:39 pm

Re: HP Storeonce - Catalyst Shares (CoFC)

Veeam Logoby Gostev » Tue May 24, 2016 1:14 am

Well, our integration with StoreOnce was just released anyway, and many of our larger customers have not even upgraded to v9 yet...
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: HP Storeonce - Catalyst Shares (CoFC)

Veeam Logoby SyNtAxx » Wed May 25, 2016 7:38 pm

Sorry, I don't feel that is a legitimate excuse. My company was very disappointed with the level of actual integration of the Catalyst/StoreOnce, so much so we've abandoned using it and went back to SMB shares of the HPE StoreOnce 6500s. We certainly aren't the largest user of veeam, but we are not the smallest either. Hundreds of terabytes of vm data backed up and over 100 of Ent+ licenses.....
SyNtAxx
Expert
 
Posts: 127
Liked: 14 times
Joined: Fri Jan 02, 2015 7:12 pm

Re: HP Storeonce - Catalyst Shares (CoFC)

Veeam Logoby Gostev » Thu May 26, 2016 12:48 pm 1 person likes this post

SyNtAxx wrote:Sorry, I don't feel that is a legitimate excuse.

Perhaps because it was not an excuse to start with? :D
I was merely explaining to the previous poster why no other users are chiming in.

I know you are upset with the fact that there is no support for Catalyst replication, however to be completely fair we have pre-announced every v9 feature months before the actual release with excessive level of details on the actual functionality we're delivering, so it's not like we over-promised and under-delivered. Anyway, Catalyst replication is an off-topic to this discussion, let's please stay on topic with future posts in this thread.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: HP Storeonce - Catalyst Shares (CoFC)

Veeam Logoby vdellachiesa » Sun May 29, 2016 11:26 pm 1 person likes this post

Veeam/HP and our Veeam Partner have identified some logic errors and have put forth a lot of work to correct this for us. At the end of the day we are definitely happy about the effort put forth to solve this problem for us. I'm still concerned that the solution won't scale to meet our future needs.
vdellachiesa
Novice
 
Posts: 7
Liked: 2 times
Joined: Wed Jun 11, 2014 11:39 pm

Re: HP Storeonce - Catalyst Shares (CoFC)

Veeam Logoby justin23 » Fri Jun 03, 2016 2:16 am

@vdellachiesa Can you elaborate about what errors were found. I've just taken over a Veeam System with Catalyst StoreOnce integration and noting a lot of CoFC errors. I thought it may have been the number of VMs as a single job with a single VM works fine, yet a job with as few as 10-15 VM's still has this error. I've also noted its only at one datacentre mostly where as the other one is running pretty well. I just spent a day reconfiguring the StoreOnce as well.
justin23
Influencer
 
Posts: 22
Liked: 2 times
Joined: Mon Apr 07, 2014 5:49 am
Full Name: Justin Watson

Re: HP Storeonce - Catalyst Shares (CoFC)

Veeam Logoby vdellachiesa » Sun Jun 05, 2016 11:07 pm

Sorry for the late and short reply, getting over a rough sinus infection. I believe we were receiving cofc 1404/03 error messages. Very high level was Veeam was trying to use more threads then the StoreOnce would allow. We received a private hf for v9 as well as redesign suggestion (from Veeam) around our SO catalyst shares and the way we were utilizing backup copy jobs. We have a 2 node SO. Originally we had proxy 1 as the gateway server for all the shares on node 1. When a job ran that could utilize both our proxies we were limiting our connections/througput to node 1 by not letting proxy 2 connect directly to a share on node 1, instead the data flow was proxy 2->proxy 1->SO node 1.

If you are receiving 1404/1403 errors give your Veeam rep a call(feel free to reference mine and gdavids posts) and have them review the logs to determine if you are hitting a thread limit issue. If so, you'll need the hf and maybe a job review to see if you can reduce your thread count.
vdellachiesa
Novice
 
Posts: 7
Liked: 2 times
Joined: Wed Jun 11, 2014 11:39 pm

Re: HP Storeonce - Catalyst Shares (CoFC)

Veeam Logoby vdellachiesa » Sun Jun 05, 2016 11:12 pm

Are the units the same in both datacenters? Do you have the same amount of fiber connections to both units and the same amount of devices per login(setting on SO).
vdellachiesa
Novice
 
Posts: 7
Liked: 2 times
Joined: Wed Jun 11, 2014 11:39 pm

Re: HP Storeonce - Catalyst Shares (CoFC)

Veeam Logoby justin23 » Sun Jun 05, 2016 11:43 pm

The setup is identical at both datacentres, but I'm still seeing errors at both datacentres. Its just one of them, the job basically was all CoFC 1404 error messages, the other one only occasional. I've got around it by adding the same Catalyst store as a non CoFC repository in Veeam. It looks as though the Backup job can handle the change from CoFC to LAN without breaking the chain. Of course performance isn't as good, but i also need all VM's backing up!

I ran a test job with 1 VM and it worked, meaning the infrastructure itself was at least configured to work. So my conclusion is there were perhaps something different between datacentres, but yet to find that, however the more i research the move it looks like both datacentres have the same issue. I've got my local Veeam Reps business cards so I'll contact them today. Did you need any assistance from HP?
justin23
Influencer
 
Posts: 22
Liked: 2 times
Joined: Mon Apr 07, 2014 5:49 am
Full Name: Justin Watson

Re: HP Storeonce - Catalyst Shares (CoFC)

Veeam Logoby vdellachiesa » Wed Jun 08, 2016 7:59 pm

We worked with both HP\Veeam and a 3rd party vendor who was doing the install for us. At the end of the day it was Veeam who created the private HF to "correct" the 1404 errors for us. We were told there is a possibility the logic to address the 1404 error messages in V9.5 (but you still need a registry entry to make it work). The name of the registry key (reg dword) is StoreOnceMaxFcFileSessions, hopefully this information will escalate your case.
vdellachiesa
Novice
 
Posts: 7
Liked: 2 times
Joined: Wed Jun 11, 2014 11:39 pm

Re: HP Storeonce - Catalyst Shares (CoFC)

Veeam Logoby justin23 » Fri Jun 10, 2016 12:16 am 1 person likes this post

So an update regarding my issue. Sounds very similar fix to just done on the Storeonce.
On the Storeonce we set the max initiators per port to 64 on the fibre channel settings (It was 4 which I though was the correct setting). This seems to have resolved my problems, although waiting to see how it performs on the weekend full backups :-)
justin23
Influencer
 
Posts: 22
Liked: 2 times
Joined: Mon Apr 07, 2014 5:49 am
Full Name: Justin Watson

Re: HP Storeonce - Catalyst Shares (CoFC)

Veeam Logoby foggy » Fri Jun 10, 2016 2:51 pm

Right, the recommendation is to set this number to a maximum value, unless there's some specific need to limit the number of target LUNs the client server sees. This should be mentioned in the configuration guide.
foggy
Veeam Software
 
Posts: 14728
Liked: 1078 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: HP Storeonce - Catalyst Shares (CoFC)

Veeam Logoby justin23 » Fri Jul 08, 2016 3:08 am 1 person likes this post

Just confirming after a month or so, the CoFC errors have not come back since I set the max initiators per port to 64 on the Storeonce.
justin23
Influencer
 
Posts: 22
Liked: 2 times
Joined: Mon Apr 07, 2014 5:49 am
Full Name: Justin Watson

Next

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 13 guests