Comprehensive data protection for all workloads
bthorne
Novice
Posts: 5
Liked: never
Joined: Jul 31, 2017 5:27 am
Full Name: Bernard Thorne
Contact:

3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by bthorne »

Hello all,


We've run into an issue with Veeam and 3Par storage integration since upgrading the SANs to 3.3.1 MU3

We can no longer get Veeam to scan the Luns on 3Par SANs that are running 3.3.1 MU3 and that don’t have an RCOPY license
The errors below are from the 3Par console and it's not just the Veeam server. SSMC 3.4 is also generating the same error on the 3Par console
We were able to resolve issue by installing a temp evauluation rcopy licence from HP.

Has anyone else experianced this issue ?


Model : 8200
Case # 03261891

Seeing this error in the 3Par logs

Code: Select all

Time     : 2018-10-29 12:23:33 AEDT
Severity : Informational
Type     : Command executed
Message  : {3paradm super all {{0 8}} -1 127.127.0.1:39790 55674} {setclienthost 10.7.2.61:51471} {}

Time     : 2018-10-29 12:23:33 AEDT
Severity : Minor
Type     : Command error arguments
Message  : {3paradm super all {{0 8}} -1 10.7.2.61:51471 55674} Command: getrcopy  Error: Error: This system is not licensed for Remote Copy..

Time     : 2018-10-29 12:23:33 AEDT
Severity : Informational
Type     : Command executed
Message  : {3paradm super all {{0 8}} -1 127.127.0.1:44812 55679} {setclienthost 10.7.2.61:51474} {}

Time     : 2018-10-29 12:23:33 AEDT
Severity : Minor
Type     : Command error arguments
Message  : {3paradm super all {{0 8}} -1 10.7.2.61:51474 55679} Command: getrcopy  Error: Error: This system is not licensed for Remote Copy..
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by foggy »

Hi Bernard, yes, we are aware of the issues with this update and are currently upgrading our labs to perform closer investigation. Thanks for opening a case, that should give us more information about the problem. We will share the news once we have any definitive information.
ejenner
Veteran
Posts: 636
Liked: 100 times
Joined: Mar 23, 2018 4:43 pm
Full Name: EJ
Location: London
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by ejenner »

Is this just simply HPE removing a feature so you have to pay for it if you want it?

Whilst I'm sure their license agreement allows them to do whatever they want, I do think it's a bit unfair to take away functionality which was in the product when it was released?
KelKnowles
Service Provider
Posts: 30
Liked: 2 times
Joined: Sep 15, 2012 8:01 pm
Full Name: Kelly Michael Knowles
Contact:

[MERGED] 3PAR 3.3.1 MU3 Storage Integration Broken

Post by KelKnowles »

Hi,

I opened ticket 03225593 about a month ago and am not getting traction on it. I am having issues with Veeam 9.5.0.1922 working with HPE 3PAR 3.3.1 MU3 which is . I have an HPE ticket 5333036065 as well. Can anyone please confirm if this is an issue which is being worked on?

Thanks,

Kelly
Kelly Knowles
Principal Systems Architect at PNJ Technology Partners
Veeam Certified Architect and Veeam Certified Engineer - Advanced: Design & Optimization
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by foggy »

Hi Kelly, yes, this is an issue we are currently investigating.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by foggy »

So far we haven't been able to reproduce the issue internally. Please continue investigating with your support engineers, they should suggest you to perform a couple of checks to track the issue down.
mecki
Veeam ProPartner
Posts: 16
Liked: 2 times
Joined: Sep 08, 2015 3:48 am
Full Name: Martin Eckart-W.
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by mecki »

Hi,

my customer got also an issue with a 3PAR8440 3.3.1 mu3. Here is the wsapi service crashing on the 3PAR when veeam is doing the first scan of the 3PAR volumes.
In the veeam logs you can see there a succesful login to the wsapi service. VBR is on 9.5 update 3a. A 2nd 3PAR connected to the same VBR is working perfect but has an older FW.
Case 03272933 opened at Veeam. With "So far we haven't been able to reproduce the issue internally" do you mean, you didn't got a 3PAR with 3.3.1 mu3 on it to test or you can't see the issue on a 3PAR (3.3.1mu3) yet?

Thanking in advance,

Martin
Martin
VMCE, VMCT
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by foggy »

Hi Martin, we've got two of our storages upgraded to MU3 and the issue doesn't show up in our tests with both them. However, it does show up if we run a simple PowerShell connectivity test script for 3PAR that uses the same APIs, so seems that this update introduced some changes around these APIs. As far as I know, our engineers are already in contact with HPE regarding this. Please escalate the case and wait for our engineers response based on their communication with HPE.
KelKnowles
Service Provider
Posts: 30
Liked: 2 times
Joined: Sep 15, 2012 8:01 pm
Full Name: Kelly Michael Knowles
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by KelKnowles »

Alexander,

I had gotten a powershell script a while ago to test against my storage from the Veeam support team. The script had a syntax error in it but I looked again and I think I fixed the syntax error and ran the script.

I get the following results running 3ParConnTest.ps1 Can I please get an update on this issue? I believe this has been broken for about two months now.
Kelly Knowles
Principal Systems Architect at PNJ Technology Partners
Veeam Certified Architect and Veeam Certified Engineer - Advanced: Design & Optimization
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by foggy »

Kelly, please ask your support engineer for an update, since their team should be in contact with HPE support regarding this issue (and I believe the ball is on HPE side now).
mecki
Veeam ProPartner
Posts: 16
Liked: 2 times
Joined: Sep 08, 2015 3:48 am
Full Name: Martin Eckart-W.
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by mecki » 1 person likes this post

Installing a Remote Copy Trial License is now a workaround to get the 3PAR integration with VBR working. According to HPE there will be a hotfix for 3.3.1 mu3 and future versions will be fixed.
Martin
VMCE, VMCT
perseusco
Influencer
Posts: 19
Liked: never
Joined: May 08, 2018 8:17 pm
Full Name: Perseus Coronado
Contact:

[MERGED] 3Par OS upgrade 3.3.1.460 (MU3)

Post by perseusco »

We had our HPE_3PAR 7200 OS upraded to 3.3.1.460 (MU3) just recently, a few days after(that tells you how good i am at paying attention to our backups) i noticed our backups were failing with the following error : "Failed to create snapshot for LUN (Lun name) Error: Internal server error. I went back on the job history and noticed it coincided with the date the Array was upgraded.
After looking around i went to VBR Storage Infrastructure and tried a Rescan of the 3Par Array and i get the following error: "Failed to refresh HPE 3PAR StoreServ storage 3par.domainname.com configuration Error: Internal server error".
"Failed to rescan HPE 3PAR StoreServ storage 3par.domainname.com configuration Error: Internal server error".
I have a Veeam ticket open and i also have an HPE support ticket opened, this is what the backup logs show when trying to connect to the array:
Veeam had me switch our jobs to Failover to standard backup job and that has me going for now.

Code: Select all

[12.12.2018 07:37:52] <01> Info     [Hpe3PAR] Getting information about volumes
[12.12.2018 07:37:52] <01> Info     [WSAPI] Connectiong to HPE 3PAR: 'https://3par.domainname.com:8080'.
[12.12.2018 07:37:52] <01> Info     [WSAPI Connection] Successfully connected to URL 'https://3par.domainname.com:8080'. Connection key prefix: '0'
[12.12.2018 07:37:52] <01> Info     [WSAPI] Getting volumes from 'https://3par.domainname.com:8080'
[12.12.2018 07:37:52] <01> Error    [Hpe3PAR] Generate Hp3PAR exception. ErrorCode: '1', Description: 'internal server error', Reference: '<null>'.
[12.12.2018 07:37:52] <01> Error    Failed to call web service methods. RetryCount: '1'. MaxRetryCount: '3'.
[12.12.2018 07:37:52] <01> Error    Exception has been thrown by the target of an invocation. (System.Reflection.TargetInvocationException)
[12.12.2018 07:37:52] <01> Error       at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
[12.12.2018 07:37:52] <01> Error       at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
[12.12.2018 07:37:52] <01> Error       at System.Delegate.DynamicInvokeImpl(Object[] args)
[12.12.2018 07:37:52] <01> Error       at Veeam.Backup.SanPlugin.Hp3PARLib.CHp3PARReconnectableMethod.DoInvoke(CHp3PARWsApiInvoker apiInvoker, Delegate dlg, Object[] args)
[12.12.2018 07:37:52] <01> Error    internal server error (Veeam.Backup.SanPlugin.Hp3PARLib.CHp3PARException)
Has anyone seen this?

thanks

Perseus
perseusco
Influencer
Posts: 19
Liked: never
Joined: May 08, 2018 8:17 pm
Full Name: Perseus Coronado
Contact:

Re: 3Par OS upgrade 3.3.1.460 (MU3)

Post by perseusco »

sorry forgot to add some additional info about my environment:
VBR 9.5.0.1922
VMWare 6.7 U1
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by foggy »

Hi Perseus, we are aware of the issues with this update, please see above for some additional details.

I'd also mention that 6.7 U1 is not supported by Veeam B&R current version (9.5 U3a).
perseusco
Influencer
Posts: 19
Liked: never
Joined: May 08, 2018 8:17 pm
Full Name: Perseus Coronado
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by perseusco »

Hi,
Yes, that was my mistake, I actually had to do the registry fix in in order for the backups to work under failover.

thanks

Perseus
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by foggy » 1 person likes this post

The discussed behavior was confirmed by HPE as an issue on their side (inability to connect via REST API without Remote Copy license), waiting for the bug ID for reference.
adamdb
Novice
Posts: 4
Liked: never
Joined: Sep 20, 2018 1:39 pm
Full Name: Ads
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by adamdb »

Can you post the bugID and any other information you've had from HPE once received? We are experiencing the same issue and currently have a call open with the 3PAR team.

thank you.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by foggy »

Yes, sure. At the moment you can refer to the HPE case ID #5334105693 though.
adamdb
Novice
Posts: 4
Liked: never
Joined: Sep 20, 2018 1:39 pm
Full Name: Ads
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by adamdb »

thanks. We managed get issued with a remote copy licence in the interim so at least things are working again.
Hopefully they'll issue a fix in Q1.
dcolpitts
Veeam ProPartner
Posts: 119
Liked: 24 times
Joined: Apr 01, 2011 10:36 am
Full Name: Dean Colpitts
Location: Atlantic coast of Canada
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by dcolpitts »

Heads up!!! I just installed 3.3.1 MU3 P50 and it completely broke all SSH and Veeam connectivity to the array. I'm on the phone with HPE Support now. According to the documentation, you can't revert it either.

dcc
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by Andreas Neufert »

Asked our HPE solution architect to comment. Please open a Veeam Support case and upload logs, to give us a chance to analyse what happend.

Can you connect to it outside of Veeam?

Without having analysed the logs... please open the 3Par Veeam wizard again to check if you can then connect.
dcolpitts
Veeam ProPartner
Posts: 119
Liked: 24 times
Joined: Apr 01, 2011 10:36 am
Full Name: Dean Colpitts
Location: Atlantic coast of Canada
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by dcolpitts »

Andreas - I don't think opening a Veeam case is necessary - 3Par 3.3.1 MU3 P50 broke more than just Veeam connectivity...
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by Andreas Neufert »

ok... but if you can connect to management and only veeam connection not possible, please run the 3par wizard again where you configure connection, this should update certificates/authentication if not updated automatically.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by Andreas Neufert »

and the wizard will enable some management stuff (need to confirm the veeam popup again) if it was disabled by the firmware update
dcolpitts
Veeam ProPartner
Posts: 119
Liked: 24 times
Joined: Apr 01, 2011 10:36 am
Full Name: Dean Colpitts
Location: Atlantic coast of Canada
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by dcolpitts »

I'll try that, but if I can't open Putty and connect, then there are other issues. And right now, I can't open Putty from the VBR server and connect. But interestingly enough I can from other 2012R2 and 2016 servers, so I'm beginning to think it is a Windows protocol issue.

dcc
JasonL
Veeam Software
Posts: 114
Liked: 9 times
Joined: Jan 17, 2011 4:04 pm
Full Name: Jason Leiva
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by JasonL »

Agreed that not being able to connect via Putty on this one server sounds like some other issue. As for the specific issue occurring after 3PAR updates with storage snapshot integration, I can confirm that HPE has confirmed there is a latch coming (P58) that should remedy this. They tentative ETA for this is end of March.
dcolpitts
Veeam ProPartner
Posts: 119
Liked: 24 times
Joined: Apr 01, 2011 10:36 am
Full Name: Dean Colpitts
Location: Atlantic coast of Canada
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by dcolpitts »

I reinstalled Windows on my Veeam server this morning. It didn't help any with the issue. I then proceeded to reboot each node, one at a time, starting with Node 1 (Node 0 was the master at the time). Once Node 1 was back online, as soon as I rebooted Node 0, I was able to putty the array from the server. Once Node 0 was online again, I rebooted Node 1 and it took maybe 30 to 45 seconds before I could putty the array again (via Node 0) - this was likely just related to the arp tables on the switch being stale for a few seconds after Node 0 became master again.

So I don't know if there was a service that failed to start on Node 0 after the upgrade or what…

I do know I'm still not very impressed with the QA of 3Par patches lately, even though they keep telling us they have cleaned up their QA process. I shouldn't have had to reinstall Windows on my server to determine a resolution to this issue. 3.3.1 MU3 has been nothing but troubles all the way around on this particular array too.

For anyone else reading this and not on 3.3.1 MU3 yet, despite what HPE Pointnext and their Global Deployment Center tells you, I would recommend you stay as far away from 3.3.1 MU3 as you can at this time.

dcc
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by Andreas Neufert »

Based on the fact that you was able to access SSH from other server but not from the server used with backup, I think something is going on in your network.
Please check with the switch and router admins.

We saw it in the past in different environments, were the ARP caching was so long that it never updated. We had to manually start some network connections on the target side to reach the target again from source.
ejenner
Veteran
Posts: 636
Liked: 100 times
Joined: Mar 23, 2018 4:43 pm
Full Name: EJ
Location: London
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by ejenner »

It's very much the sort of thing I think is open to failure where you have three vendors (Microsoft, HPE & Veeam) coming together to make something work. Microsoft and Veeam I'm not too worried about, but HPE I fear could easily change something which breaks it all. With Microsoft and Veeam updates I can always reboot and remove updates which are problematic but it is very difficult to schedule 3PAR downtime as they're always in use.

The way I've been evaluating it is to see if updates to the 3PAR are security critical. If not patching vulnerabilities then I don't want the update as the system works perfectly without it.
dcolpitts
Veeam ProPartner
Posts: 119
Liked: 24 times
Joined: Apr 01, 2011 10:36 am
Full Name: Dean Colpitts
Location: Atlantic coast of Canada
Contact:

Re: 3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

Post by dcolpitts »

I sit on a couple of panels and programs within HPE and see a fair bit of NDA material, plans and roadmaps and have direct access to several of the execs in the Storage BU. They keep telling and showing us storage partners that QA is priority number one, but stuff like this still gives me concern. Heck - even the initial reason for this thread hasn't been publicly resolved yet and Pointnext wasn't even aware of the issue in mid January - I had to explain it to them - so that is very concerning.

Andreas - I thought it was arp caching too, but clearing the tables on the switches should have resolved it, and it didn't. I also thought it might have been related to NIC teaming on the VBR server, but even after breaking the team it didn't help. It specifically appeared that something wasn't running on Node 0 and it took a node reboot to clear it up. I also moved the 3Par patch cables to a completely different switch (from a FlexFabric 5700 where the VBR server & 8200 were to an Aruba 2920 where all the ILOs and other management devices were located) to test that as well (before rebooting the nodes).

dcc
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 173 guests