Comprehensive data protection for all workloads
Post Reply
wally.nl
Lurker
Posts: 1
Liked: never
Joined: Oct 14, 2016 2:59 pm
Contact:

3PAR 3.2.2MU3 breaks Veeam B&R integration?

Post by wally.nl »

This morning HP upgraded 1 of our 3PAR 7400 systems. The upgrade itself went without issues but it seems the upgrade breaks the Veeam B&R integration. When creating snapshots we get a "user authentication failed" error. When we go to the users and issue a check we get a "fingerprint validation failed". On the 3par we see a different error: Description "ssl handshake error sock7: handshake failed: wrong version number" type "SSL error between CLI client and server".

The "old" 3PAR 3.1.2 CLI utility also can't connect after the FW upgrade (to 3.2.2MU3) and shows the same error on the 3PAR but the newer 3.2.2 CLI client connects without issues to this newer version and the "old" 3.2.1MU2 version that is running on our other campus.

We're waiting for completion of the reboot after having tried the "SshFingerprintCheck" reghack, will update this post if this fixes things but the error on the 3PAR makes me think it won't.

Anybody else having (dealt with) this issue?
gpomanti
Enthusiast
Posts: 30
Liked: 1 time
Joined: Mar 03, 2015 4:11 pm
Full Name: Greg Pomanti
Contact:

Re: 3PAR 3.2.2MU3 breaks Veeam B&R integration?

Post by gpomanti »

We just experienced the same issue after we upgraded our 3PAR to 3.2.2 MU3 last weekend. I contacted Veeam support and they told me that 3PAR recently changed their authentication method, which makes sense considering the error. Veeam provided me with a hotfix pretty quickly that solved the problem. I suggest contacting support and you should have it resolved quickly.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: 3PAR 3.2.2MU3 breaks Veeam B&R integration?

Post by foggy »

Correct, you can directly refer the support engineer to Issue ID 84029.
Khue
Enthusiast
Posts: 67
Liked: 3 times
Joined: Sep 26, 2013 6:01 pm
Contact:

[MERGED] HPE 3PAR/StoreServ 3.2.2 MU3 Integration

Post by Khue »

Hey everyone,

I plan on opening a ticket with Veeam on this in the morning during business hours, but I am noticing a potential issue with Veeam and InServ 3.2.2 MU3. I recently just upgraded my 7450s to this release and Veeam has lost the ability to snapshot these devices. My initial thought was that either the WSAPI service was inactive or the SSH fingerprint got changed with the update. After checking on both of those two issues and still seeing Veeam fail hinting at authentication issues, I then went on to test the Storage Infrastructure integration. Upon attempting to rescan I started to receive a fingerprinting issue which I had suspected. I removed the device from Veeam and attempted to readd it accepting the newer fingerprint. After accepting the fingerprint, I still received issues about authentication. I've tried the built in superuser account for the InServ OS and our native AD integrated credentials and neither seem to be working. So again on the checklist:
  • Verified that the WSAPI service was up and running. Both HTTP and HTTPS access work
  • Verified that credentials work to InServ OS SSH session. Both integrated AD auth and native InServ root access accounts working
  • Attempted to remove the storage and re-add it to Veeam. Accepted fingerprint, authentication still failed.
In the meantime, I plan on grabbing some Wireshark dumps. Has anyone else run into this?
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: 3PAR 3.2.2MU3 breaks Veeam B&R integration?

Post by PTide »

Hi,

That is a known issue, please contact our support team and refer to the issue ID 84029 so they can provide you with a fix.

Thanks
Khue
Enthusiast
Posts: 67
Liked: 3 times
Joined: Sep 26, 2013 6:01 pm
Contact:

Re: 3PAR 3.2.2MU3 breaks Veeam B&R integration?

Post by Khue »

I went ahead and ran the fix however, still no luck. I have reported back on my open ticket and I am awaiting a response. Might have to do with a subsequent patch for MU3. Patch version is P41.
Khue
Enthusiast
Posts: 67
Liked: 3 times
Joined: Sep 26, 2013 6:01 pm
Contact:

Re: 3PAR 3.2.2MU3 breaks Veeam B&R integration?

Post by Khue »

Interestingly enough, on my Server 2012 R2 box, I have two different folders where there were files. The folder from the issue ID wasn't accurate. When I searched the Veeam root directory, I found another folder with the same DLLs in it and updated those and that seems to have worked. I can provide details if necessary.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 3PAR 3.2.2MU3 breaks Veeam B&R integration?

Post by Mike Resseler »

Hi,

Happy to hear that it worked. And yes, we would love to hear some additional details. I am not sure what you mean by the fact that there are two folders.

Thanks
Mike
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: 3PAR 3.2.2MU3 breaks Veeam B&R integration?

Post by veremin »

I have two different folders where there were files
Do you refer to \Backup and \Console folders where the said DLLs are present? Which one did you patch originally? Thanks.
Khue
Enthusiast
Posts: 67
Liked: 3 times
Joined: Sep 26, 2013 6:01 pm
Contact:

Re: 3PAR 3.2.2MU3 breaks Veeam B&R integration?

Post by Khue »

Veeam instructions from the IssueID (Actually, I was just following directions from an email on my case) references the following folder:

Code: Select all

C:\Program Files\Veeam\Backup and Replication\Backup
That folder exists and the DLLs referenced exist there. We redirect non Microsoft based installs to the D: drive, specifically the "Program Files (x86)" folder set as we have specific permissions on that folder and our D: drive is substantially larger. In this specific case, I therefore modified:

Code: Select all

D:\Program Files (x86)\Veeam\Backup and Replication\Backup\
So after modifying the DLLs in that folder, nothing changed. I was still not able to do hardware based snapshots from Veeam. I dug around a bit and found that the DLLs needing to be replaced, also exist in the following folder:

Code: Select all

D:\Program Files (x86)\Veeam\Backup\
To test, I ran through the process on that folder and then started Veeam again and I was then able to do backups. I am not sure what type of user error I did to generate that problem, but clearly it must be something that I've done along the way. What's odd is that in my development environment on my Windows 2008 R2 environment, the process went as described. In production on my Windows 2012 R2 server, I had to modify the "D:\Program Files (x86)\Veeam\Backup" folder instead.

Like I said before, most likely, this is probably something I screwed up at some point.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 3PAR 3.2.2MU3 breaks Veeam B&R integration?

Post by Mike Resseler »

Thanks for coming back to us with the explanation. It is appreciated.

Mike
Ejdesgaard
Enthusiast
Posts: 43
Liked: 8 times
Joined: Aug 24, 2012 11:59 am
Contact:

Re: 3PAR 3.2.2MU3 breaks Veeam B&R integration?

Post by Ejdesgaard »

Does this also affect 3.2.2MU2 on a new installation ?
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 3PAR 3.2.2MU3 breaks Veeam B&R integration?

Post by Gostev »

Yes.
dweide
Enthusiast
Posts: 38
Liked: 9 times
Joined: Mar 29, 2012 1:57 pm
Full Name: D. Weide
Contact:

Re: 3PAR 3.2.2MU3 breaks Veeam B&R integration?

Post by dweide » 1 person likes this post

We had the same issue here. Opened a support case with the case Id mentioned above and received the fix within 2 hours.

The fix immediately worked.

Perfect!
aschepers
Enthusiast
Posts: 28
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

Re: 3PAR 3.2.2MU3 breaks Veeam B&R integration?

Post by aschepers »

We will be upgrading to 3.2.1 MU5 P39 and P49 this week. Will we also need the hotfix?
Jack1874
Enthusiast
Posts: 95
Liked: 5 times
Joined: Oct 17, 2015 3:32 pm
Full Name: Stuart Little
Location: Canada
Contact:

Re: 3PAR 3.2.2MU3 breaks Veeam B&R integration?

Post by Jack1874 »

Case # 01976501

Guys, we updated our 3PAR to 3.2.2 andare having intermittent backup issues. We are getting the SSH timeout issue

We have applied the "3par hotfix" as advised by Veeam Support.. but still we are seeing backups fail.

Support tell us that they only way forward is to move to ver 9.5

Anyone else updated and applied the fix.. but still having issues ??
emak
Influencer
Posts: 10
Liked: 2 times
Joined: Sep 20, 2016 6:48 am
Contact:

Re: 3PAR 3.2.2MU3 breaks Veeam B&R integration?

Post by emak »

Hi

We use Veeam 9.5 Update 1 and updated last thursday from HPE 3PAR OS 3.2.1 MU1 (patch 4) to 3.2.2 MU3 (patch 52).

Since then we are using more than double the time for backups because - as HPE support says "It does seem that Veeam is unable to detect the snapshot that is created on the 3PAR." In fact the storage snapshots are taken but not exported for approx. 10h and then veeam seems to continue (like waiting for a timeout) and the backup runs fast as it used to be.

I am just wondering that I was in touch with Veeam Support (# 5317390161) for more than a week now and delivered all necessary details but until now they did not seem able to discover that. I just added the details to the support case and am looking forward to get the support hotfix and test - will keep you updated.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: 3PAR 3.2.2MU3 breaks Veeam B&R integration?

Post by foggy »

Could you please verify the Veeam support case ID, since the one provided doesn't look valid. Thanks.
emak
Influencer
Posts: 10
Liked: 2 times
Joined: Sep 20, 2016 6:48 am
Contact:

Re: 3PAR 3.2.2MU3 breaks Veeam B&R integration?

Post by emak »

Sorry - right support ID# 02072656
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: 3PAR 3.2.2MU3 breaks Veeam B&R integration?

Post by foggy »

Thanks, I'll pass it to QA guys for a closer look.
emak
Influencer
Posts: 10
Liked: 2 times
Joined: Sep 20, 2016 6:48 am
Contact:

Re: 3PAR 3.2.2MU3 breaks Veeam B&R integration?

Post by emak » 1 person likes this post

Hi there

Well I will know soon if our backup runs fine with 3.2.2 MU3 and Veeam 9.5 Update 1 as I am going to run it tonight again. If it does not it is most proably due to erroneous WsAPI on HPE's side (check also https://h20565.www2.hpe.com/hpsc/doc/pu ... -c05225720 and look for WSAPI and MU 3.2.2 MU4).

On the other hand - if you folk encounter a problem with the iSCSI connection discovery (will appear after updating HPE 3PAR) then please read that:
HPE implemented an iSCSI specifiation feature called STGT (Send-Target Group Tag) with 3.2.2 MU3 which basically adds a new tag for connection availabilities. After the update the STGT values are per default set to the TPGT values (e.g. 21, 22, 121, 122) but the kernel on the 3PAR cages are not made aware of that breaking all new iSCSI discoveries (and dynamic discoveries). So you will have to change the STGT values forth (and it is recommended to set it back to default again) in order to make iSCSI discovery work again. Be aware though if you have active Smart SAN target driven zones, to do a removehost first. If not done first, you will not be able to manage the zone on the switch associated with this port. But now, check the actual values of STGT and set them to (e.g. 1 and then back to 21) for port 0:2:1

Code: Select all

cli% showport -iscsi
cli% controliscsiport stgt 1  0:2:1
cli% controliscsiport stgt 21  0:2:1
You are done and iSCSI dynamic discovery will work again :D
emak
Influencer
Posts: 10
Liked: 2 times
Joined: Sep 20, 2016 6:48 am
Contact:

Re: 3PAR 3.2.2MU3 breaks Veeam B&R integration?

Post by emak » 1 person likes this post

emak wrote: Since then we are using more than double the time for backups because - as HPE support says "It does seem that Veeam is unable to detect the snapshot that is created on the 3PAR." In fact the storage snapshots are taken but not exported for approx. 10h and then veeam seems to continue (like waiting for a timeout) and the backup runs fast as it used to be.
I can confirm that the backup works fine and without initial delay after:
- having changed the STGT values on 3PAR forth and back to default
- rediscovered the iSCSI target on 3PAR from the source server
- rescanned the storage in Veeam's storage infrastructure
oli4
Enthusiast
Posts: 63
Liked: 3 times
Joined: Aug 14, 2012 11:34 am
Contact:

[MERGED] Veeam 9,0 Update 2 and 3PAR Storserv 3.3.1

Post by oli4 »

Hello all

We have a 3PAR Storeserv with Software version 3.3.1. We can successfully integrate this into Veeam 9.5
but not into Veeam 9.0 Update 2.

Is there a solution for that?

Regards
oli4
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam 9,0 Update 2 and 3PAR Storserv 3.3.1

Post by foggy »

Hi, please contact support, there should be hotfix available for Veeam B&R v9U2. Thanks.
Post Reply

Who is online

Users browsing this forum: ante_704 and 243 guests