3PAR 3.2.2MU3 breaks Veeam B&R integration?

[wally.nl]

This morning HP upgraded 1 of our 3PAR 7400 systems. The upgrade itself went without issues but it seems the upgrade breaks the Veeam B&R integration. When creating snapshots we get a "user authentication failed" error. When we go to the users and issue a check we get a "fingerprint validation failed". On the 3par we see a different error: Description "ssl handshake error sock7: handshake failed: wrong version number" type "SSL error between CLI client and server".

The "old" 3PAR 3.1.2 CLI utility also can't connect after the FW upgrade (to 3.2.2MU3) and shows the same error on the 3PAR but the newer 3.2.2 CLI client connects without issues to this newer version and the "old" 3.2.1MU2 version that is running on our other campus.

We're waiting for completion of the reboot after having tried the "SshFingerprintCheck" reghack, will update this post if this fixes things but the error on the 3PAR makes me think it won't.

Anybody else having (dealt with) this issue?

[gpomanti]

We just experienced the same issue after we upgraded our 3PAR to 3.2.2 MU3 last weekend. I contacted Veeam support and they told me that 3PAR recently changed their authentication method, which makes sense considering the error. Veeam provided me with a hotfix pretty quickly that solved the problem. I suggest contacting support and you should have it resolved quickly.

[foggy]

Correct, you can directly refer the support engineer to Issue ID 84029.

[Khue]

Hey everyone,

I plan on opening a ticket with Veeam on this in the morning during business hours, but I am noticing a potential issue with Veeam and InServ 3.2.2 MU3. I recently just upgraded my 7450s to this release and Veeam has lost the ability to snapshot these devices. My initial thought was that either the WSAPI service was inactive or the SSH fingerprint got changed with the update. After checking on both of those two issues and still seeing Veeam fail hinting at authentication issues, I then went on to test the Storage Infrastructure integration. Upon attempting to rescan I started to receive a fingerprinting issue which I had suspected. I removed the device from Veeam and attempted to readd it accepting the newer fingerprint. After accepting the fingerprint, I still received issues about authentication. I've tried the built in superuser account for the InServ OS and our native AD integrated credentials and neither seem to be working. So again on the checklist:

• Verified that the WSAPI service was up and running. Both HTTP and HTTPS access work
• Verified that credentials work to InServ OS SSH session. Both integrated AD auth and native InServ root access accounts working
• Attempted to remove the storage and re-add it to Veeam. Accepted fingerprint, authentication still failed.

In the meantime, I plan on grabbing some Wireshark dumps. Has anyone else run into this?

[PTide]

Hi,

That is a known issue, please contact our support team and refer to the issue ID 84029 so they can provide you with a fix.

Thanks

[Khue]

I went ahead and ran the fix however, still no luck. I have reported back on my open ticket and I am awaiting a response. Might have to do with a subsequent patch for MU3. Patch version is P41.

[Khue]

Interestingly enough, on my Server 2012 R2 box, I have two different folders where there were files. The folder from the issue ID wasn't accurate. When I searched the Veeam root directory, I found another folder with the same DLLs in it and updated those and that seems to have worked. I can provide details if necessary.

[Mike Resseler]

Hi,

Happy to hear that it worked. And yes, we would love to hear some additional details. I am not sure what you mean by the fact that there are two folders.

Thanks
Mike

[veremin]

I have two different folders where there were files

Do you refer to \Backup and \Console folders where the said DLLs are present? Which one did you patch originally? Thanks.

[Khue]

Veeam instructions from the IssueID (Actually, I was just following directions from an email on my case) references the following folder:

Code: Select all

C:\Program Files\Veeam\Backup and Replication\Backup

That folder exists and the DLLs referenced exist there. We redirect non Microsoft based installs to the D: drive, specifically the "Program Files (x86)" folder set as we have specific permissions on that folder and our D: drive is substantially larger. In this specific case, I therefore modified:

Code: Select all

D:\Program Files (x86)\Veeam\Backup and Replication\Backup\

So after modifying the DLLs in that folder, nothing changed. I was still not able to do hardware based snapshots from Veeam. I dug around a bit and found that the DLLs needing to be replaced, also exist in the following folder:

Code: Select all

D:\Program Files (x86)\Veeam\Backup\

To test, I ran through the process on that folder and then started Veeam again and I was then able to do backups. I am not sure what type of user error I did to generate that problem, but clearly it must be something that I've done along the way. What's odd is that in my development environment on my Windows 2008 R2 environment, the process went as described. In production on my Windows 2012 R2 server, I had to modify the "D:\Program Files (x86)\Veeam\Backup" folder instead.

Like I said before, most likely, this is probably something I screwed up at some point.

[Mike Resseler]

Thanks for coming back to us with the explanation. It is appreciated.

Mike

[Ejdesgaard]

Does this also affect 3.2.2MU2 on a new installation ?

[Gostev]

Yes.

[dweide]

We had the same issue here. Opened a support case with the case Id mentioned above and received the fix within 2 hours.

The fix immediately worked.

Perfect!

[aschepers]

We will be upgrading to 3.2.1 MU5 P39 and P49 this week. Will we also need the hotfix?

[Jack1874]

Case # 01976501

Guys, we updated our 3PAR to 3.2.2 andare having intermittent backup issues. We are getting the SSH timeout issue

We have applied the "3par hotfix" as advised by Veeam Support.. but still we are seeing backups fail.

Support tell us that they only way forward is to move to ver 9.5

Anyone else updated and applied the fix.. but still having issues ??

[emak]

Hi

We use Veeam 9.5 Update 1 and updated last thursday from HPE 3PAR OS 3.2.1 MU1 (patch 4) to 3.2.2 MU3 (patch 52).

Since then we are using more than double the time for backups because - as HPE support says "It does seem that Veeam is unable to detect the snapshot that is created on the 3PAR." In fact the storage snapshots are taken but not exported for approx. 10h and then veeam seems to continue (like waiting for a timeout) and the backup runs fast as it used to be.

I am just wondering that I was in touch with Veeam Support (# 5317390161) for more than a week now and delivered all necessary details but until now they did not seem able to discover that. I just added the details to the support case and am looking forward to get the support hotfix and test - will keep you updated.

[foggy]

Could you please verify the Veeam support case ID, since the one provided doesn't look valid. Thanks.

[emak]

Sorry - right support ID# 02072656

[foggy]

Thanks, I'll pass it to QA guys for a closer look.

[emak]

Hi there

Well I will know soon if our backup runs fine with 3.2.2 MU3 and Veeam 9.5 Update 1 as I am going to run it tonight again. If it does not it is most proably due to erroneous WsAPI on HPE's side (check also https://h20565.www2.hpe.com/hpsc/doc/pu ... -c05225720 and look for WSAPI and MU 3.2.2 MU4).

On the other hand - if you folk encounter a problem with the iSCSI connection discovery (will appear after updating HPE 3PAR) then please read that:
HPE implemented an iSCSI specifiation feature called STGT (Send-Target Group Tag) with 3.2.2 MU3 which basically adds a new tag for connection availabilities. After the update the STGT values are per default set to the TPGT values (e.g. 21, 22, 121, 122) but the kernel on the 3PAR cages are not made aware of that breaking all new iSCSI discoveries (and dynamic discoveries). So you will have to change the STGT values forth (and it is recommended to set it back to default again) in order to make iSCSI discovery work again. Be aware though if you have active Smart SAN target driven zones, to do a removehost first. If not done first, you will not be able to manage the zone on the switch associated with this port. But now, check the actual values of STGT and set them to (e.g. 1 and then back to 21) for port 0:2:1

Code: Select all

cli% showport -iscsi
cli% controliscsiport stgt 1  0:2:1
cli% controliscsiport stgt 21  0:2:1

You are done and iSCSI dynamic discovery will work again

[emak]

Since then we are using more than double the time for backups because - as HPE support says "It does seem that Veeam is unable to detect the snapshot that is created on the 3PAR." In fact the storage snapshots are taken but not exported for approx. 10h and then veeam seems to continue (like waiting for a timeout) and the backup runs fast as it used to be.

I can confirm that the backup works fine and without initial delay after:
- having changed the STGT values on 3PAR forth and back to default
- rediscovered the iSCSI target on 3PAR from the source server
- rescanned the storage in Veeam's storage infrastructure

[oli4]

Hello all

We have a 3PAR Storeserv with Software version 3.3.1. We can successfully integrate this into Veeam 9.5
but not into Veeam 9.0 Update 2.

Is there a solution for that?

Regards
oli4

[foggy]

Hi, please contact support, there should be hotfix available for Veeam B&R v9U2. Thanks.