3par Storage Snapshots fail after upgrade to 3Par 3.3.1 MU3

[bthorne]

Hello all,


We've run into an issue with Veeam and 3Par storage integration since upgrading the SANs to 3.3.1 MU3

We can no longer get Veeam to scan the Luns on 3Par SANs that are running 3.3.1 MU3 and that don’t have an RCOPY license
The errors below are from the 3Par console and it's not just the Veeam server. SSMC 3.4 is also generating the same error on the 3Par console
We were able to resolve issue by installing a temp evauluation rcopy licence from HP.

Has anyone else experianced this issue ?


Model : 8200
Case # 03261891

Seeing this error in the 3Par logs

Code: Select all

Time     : 2018-10-29 12:23:33 AEDT
Severity : Informational
Type     : Command executed
Message  : {3paradm super all {{0 8}} -1 127.127.0.1:39790 55674} {setclienthost 10.7.2.61:51471} {}

Time     : 2018-10-29 12:23:33 AEDT
Severity : Minor
Type     : Command error arguments
Message  : {3paradm super all {{0 8}} -1 10.7.2.61:51471 55674} Command: getrcopy  Error: Error: This system is not licensed for Remote Copy..

Time     : 2018-10-29 12:23:33 AEDT
Severity : Informational
Type     : Command executed
Message  : {3paradm super all {{0 8}} -1 127.127.0.1:44812 55679} {setclienthost 10.7.2.61:51474} {}

Time     : 2018-10-29 12:23:33 AEDT
Severity : Minor
Type     : Command error arguments
Message  : {3paradm super all {{0 8}} -1 10.7.2.61:51474 55679} Command: getrcopy  Error: Error: This system is not licensed for Remote Copy..

[foggy]

Hi Bernard, yes, we are aware of the issues with this update and are currently upgrading our labs to perform closer investigation. Thanks for opening a case, that should give us more information about the problem. We will share the news once we have any definitive information.

[ejenner]

Is this just simply HPE removing a feature so you have to pay for it if you want it?

Whilst I'm sure their license agreement allows them to do whatever they want, I do think it's a bit unfair to take away functionality which was in the product when it was released?

[KelKnowles]

Hi,

I opened ticket 03225593 about a month ago and am not getting traction on it. I am having issues with Veeam 9.5.0.1922 working with HPE 3PAR 3.3.1 MU3 which is . I have an HPE ticket 5333036065 as well. Can anyone please confirm if this is an issue which is being worked on?

Thanks,

Kelly

[foggy]

Hi Kelly, yes, this is an issue we are currently investigating.

[foggy]

So far we haven't been able to reproduce the issue internally. Please continue investigating with your support engineers, they should suggest you to perform a couple of checks to track the issue down.

[mecki]

Hi,

my customer got also an issue with a 3PAR8440 3.3.1 mu3. Here is the wsapi service crashing on the 3PAR when veeam is doing the first scan of the 3PAR volumes.
In the veeam logs you can see there a succesful login to the wsapi service. VBR is on 9.5 update 3a. A 2nd 3PAR connected to the same VBR is working perfect but has an older FW.
Case 03272933 opened at Veeam. With "So far we haven't been able to reproduce the issue internally" do you mean, you didn't got a 3PAR with 3.3.1 mu3 on it to test or you can't see the issue on a 3PAR (3.3.1mu3) yet?

Thanking in advance,

Martin

[foggy]

Hi Martin, we've got two of our storages upgraded to MU3 and the issue doesn't show up in our tests with both them. However, it does show up if we run a simple PowerShell connectivity test script for 3PAR that uses the same APIs, so seems that this update introduced some changes around these APIs. As far as I know, our engineers are already in contact with HPE regarding this. Please escalate the case and wait for our engineers response based on their communication with HPE.

[KelKnowles]

Alexander,

I had gotten a powershell script a while ago to test against my storage from the Veeam support team. The script had a syntax error in it but I looked again and I think I fixed the syntax error and ran the script.

I get the following results running 3ParConnTest.ps1 Can I please get an update on this issue? I believe this has been broken for about two months now.

[foggy]

Kelly, please ask your support engineer for an update, since their team should be in contact with HPE support regarding this issue (and I believe the ball is on HPE side now).

[mecki]

Installing a Remote Copy Trial License is now a workaround to get the 3PAR integration with VBR working. According to HPE there will be a hotfix for 3.3.1 mu3 and future versions will be fixed.

[perseusco]

We had our HPE_3PAR 7200 OS upraded to 3.3.1.460 (MU3) just recently, a few days after(that tells you how good i am at paying attention to our backups) i noticed our backups were failing with the following error : "Failed to create snapshot for LUN (Lun name) Error: Internal server error. I went back on the job history and noticed it coincided with the date the Array was upgraded.
After looking around i went to VBR Storage Infrastructure and tried a Rescan of the 3Par Array and i get the following error: "Failed to refresh HPE 3PAR StoreServ storage 3par.domainname.com configuration Error: Internal server error".
"Failed to rescan HPE 3PAR StoreServ storage 3par.domainname.com configuration Error: Internal server error".
I have a Veeam ticket open and i also have an HPE support ticket opened, this is what the backup logs show when trying to connect to the array:
Veeam had me switch our jobs to Failover to standard backup job and that has me going for now.

Code: Select all

[12.12.2018 07:37:52] <01> Info     [Hpe3PAR] Getting information about volumes
[12.12.2018 07:37:52] <01> Info     [WSAPI] Connectiong to HPE 3PAR: 'https://3par.domainname.com:8080'.
[12.12.2018 07:37:52] <01> Info     [WSAPI Connection] Successfully connected to URL 'https://3par.domainname.com:8080'. Connection key prefix: '0'
[12.12.2018 07:37:52] <01> Info     [WSAPI] Getting volumes from 'https://3par.domainname.com:8080'
[12.12.2018 07:37:52] <01> Error    [Hpe3PAR] Generate Hp3PAR exception. ErrorCode: '1', Description: 'internal server error', Reference: '<null>'.
[12.12.2018 07:37:52] <01> Error    Failed to call web service methods. RetryCount: '1'. MaxRetryCount: '3'.
[12.12.2018 07:37:52] <01> Error    Exception has been thrown by the target of an invocation. (System.Reflection.TargetInvocationException)
[12.12.2018 07:37:52] <01> Error       at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
[12.12.2018 07:37:52] <01> Error       at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
[12.12.2018 07:37:52] <01> Error       at System.Delegate.DynamicInvokeImpl(Object[] args)
[12.12.2018 07:37:52] <01> Error       at Veeam.Backup.SanPlugin.Hp3PARLib.CHp3PARReconnectableMethod.DoInvoke(CHp3PARWsApiInvoker apiInvoker, Delegate dlg, Object[] args)
[12.12.2018 07:37:52] <01> Error    internal server error (Veeam.Backup.SanPlugin.Hp3PARLib.CHp3PARException)

Has anyone seen this?

thanks

Perseus

[perseusco]

sorry forgot to add some additional info about my environment:
VBR 9.5.0.1922
VMWare 6.7 U1

[foggy]

Hi Perseus, we are aware of the issues with this update, please see above for some additional details.

I'd also mention that 6.7 U1 is not supported by Veeam B&R current version (9.5 U3a).

[perseusco]

Hi,
Yes, that was my mistake, I actually had to do the registry fix in in order for the backups to work under failover.

thanks

Perseus

[foggy]

The discussed behavior was confirmed by HPE as an issue on their side (inability to connect via REST API without Remote Copy license), waiting for the bug ID for reference.

[adamdb]

Can you post the bugID and any other information you've had from HPE once received? We are experiencing the same issue and currently have a call open with the 3PAR team.

thank you.

[foggy]

Yes, sure. At the moment you can refer to the HPE case ID #5334105693 though.

[adamdb]

thanks. We managed get issued with a remote copy licence in the interim so at least things are working again.
Hopefully they'll issue a fix in Q1.

[dcolpitts]

Heads up!!! I just installed 3.3.1 MU3 P50 and it completely broke all SSH and Veeam connectivity to the array. I'm on the phone with HPE Support now. According to the documentation, you can't revert it either.

dcc

[Andreas Neufert]

Asked our HPE solution architect to comment. Please open a Veeam Support case and upload logs, to give us a chance to analyse what happend.

Can you connect to it outside of Veeam?

Without having analysed the logs... please open the 3Par Veeam wizard again to check if you can then connect.

[dcolpitts]

Andreas - I don't think opening a Veeam case is necessary - 3Par 3.3.1 MU3 P50 broke more than just Veeam connectivity...

[Andreas Neufert]

ok... but if you can connect to management and only veeam connection not possible, please run the 3par wizard again where you configure connection, this should update certificates/authentication if not updated automatically.

[Andreas Neufert]

and the wizard will enable some management stuff (need to confirm the veeam popup again) if it was disabled by the firmware update

[dcolpitts]

I'll try that, but if I can't open Putty and connect, then there are other issues. And right now, I can't open Putty from the VBR server and connect. But interestingly enough I can from other 2012R2 and 2016 servers, so I'm beginning to think it is a Windows protocol issue.

dcc

[JasonL]

Agreed that not being able to connect via Putty on this one server sounds like some other issue. As for the specific issue occurring after 3PAR updates with storage snapshot integration, I can confirm that HPE has confirmed there is a latch coming (P58) that should remedy this. They tentative ETA for this is end of March.

[dcolpitts]

I reinstalled Windows on my Veeam server this morning. It didn't help any with the issue. I then proceeded to reboot each node, one at a time, starting with Node 1 (Node 0 was the master at the time). Once Node 1 was back online, as soon as I rebooted Node 0, I was able to putty the array from the server. Once Node 0 was online again, I rebooted Node 1 and it took maybe 30 to 45 seconds before I could putty the array again (via Node 0) - this was likely just related to the arp tables on the switch being stale for a few seconds after Node 0 became master again.

So I don't know if there was a service that failed to start on Node 0 after the upgrade or what…

I do know I'm still not very impressed with the QA of 3Par patches lately, even though they keep telling us they have cleaned up their QA process. I shouldn't have had to reinstall Windows on my server to determine a resolution to this issue. 3.3.1 MU3 has been nothing but troubles all the way around on this particular array too.

For anyone else reading this and not on 3.3.1 MU3 yet, despite what HPE Pointnext and their Global Deployment Center tells you, I would recommend you stay as far away from 3.3.1 MU3 as you can at this time.

dcc

[Andreas Neufert]

Based on the fact that you was able to access SSH from other server but not from the server used with backup, I think something is going on in your network.
Please check with the switch and router admins.

We saw it in the past in different environments, were the ARP caching was so long that it never updated. We had to manually start some network connections on the target side to reach the target again from source.

[ejenner]

It's very much the sort of thing I think is open to failure where you have three vendors (Microsoft, HPE & Veeam) coming together to make something work. Microsoft and Veeam I'm not too worried about, but HPE I fear could easily change something which breaks it all. With Microsoft and Veeam updates I can always reboot and remove updates which are problematic but it is very difficult to schedule 3PAR downtime as they're always in use.

The way I've been evaluating it is to see if updates to the 3PAR are security critical. If not patching vulnerabilities then I don't want the update as the system works perfectly without it.

[dcolpitts]

I sit on a couple of panels and programs within HPE and see a fair bit of NDA material, plans and roadmaps and have direct access to several of the execs in the Storage BU. They keep telling and showing us storage partners that QA is priority number one, but stuff like this still gives me concern. Heck - even the initial reason for this thread hasn't been publicly resolved yet and Pointnext wasn't even aware of the issue in mid January - I had to explain it to them - so that is very concerning.

Andreas - I thought it was arp caching too, but clearing the tables on the switches should have resolved it, and it didn't. I also thought it might have been related to NIC teaming on the VBR server, but even after breaking the team it didn't help. It specifically appeared that something wasn't running on Node 0 and it took a node reboot to clear it up. I also moved the 3Par patch cables to a completely different switch (from a FlexFabric 5700 where the VBR server & 8200 were to an Aruba 2920 where all the ILOs and other management devices were located) to test that as well (before rebooting the nodes).

dcc