3Par Storage Snapshots left over

Availability for the Always-On Enterprise

3Par Storage Snapshots left over

Veeam Logoby jkoyle » Thu Feb 12, 2015 9:58 pm

If I have 2 different backup jobs both using storage snapshots, and they both kick off at the same time, I get a chain of snaps like this:

3Par_virtual_vol
-> 3par_virtual_vol_snap.
-> 3par_virtual_vol_snap._1

If the backup that created the first snap completes first, it doesn't/can clean up the snapshot from the 2nd running job because it's based off the HW snap from the first. However, when the 2nd job completes, it only cleans up it's snapshot, and I'm left with a lingering HW snap on the array: 3par_virtual_vol_snap. Is this expected, or shall I open a support ticket?
jkoyle
Novice
 
Posts: 7
Liked: 2 times
Joined: Fri May 27, 2011 8:25 pm
Full Name: John Koyle

Re: 3Par Storage Snapshots left over

Veeam Logoby foggy » Fri Feb 13, 2015 10:10 am

John, this does not look expected, so please open a case and provide its ID here for reference. Thanks!
foggy
Veeam Software
 
Posts: 14590
Liked: 1063 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: 3Par Storage Snapshots left over

Veeam Logoby jkoyle » Fri Feb 27, 2015 4:28 pm

I've just opened case 00818913 for this issue. I see the copy space growing and growing because the initial snapshot isn't deleted as expected.
jkoyle
Novice
 
Posts: 7
Liked: 2 times
Joined: Fri May 27, 2011 8:25 pm
Full Name: John Koyle

Re: 3Par Storage Snapshots left over

Veeam Logoby foggy » Fri Feb 27, 2015 4:29 pm

Thanks, please keep us informed on how the investigation goes.
foggy
Veeam Software
 
Posts: 14590
Liked: 1063 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: 3Par Storage Snapshots left over

Veeam Logoby jkoyle » Sat Feb 28, 2015 12:07 am

It looks like Veeam is complaining that the snapshot volume doesn't exist when it attempts to remove the snapshot:
Code: Select all
[27.02.2015 09:05:21] <01> Info     [WSAPI Connection] Disconnecting from HP 3PAR 'https://X.X.X.X:8080'. Connection key prefix: '0'
[27.02.2015 09:05:21] <01> Error    [Hp3PAR] Generate Hp3PAR exception. ErrorCode: '23', Description: 'volume does not exist', Reference: '<null>'.
[27.02.2015 09:05:21] <01> Error    Failed to call web service methods. RetryCount: '1'. MaxRetryCount: '3'.
[27.02.2015 09:05:21] <01> Error    Exception has been thrown by the target of an invocation. (System.Reflection.TargetInvocationException)
[27.02.2015 09:05:21] <01> Error       at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
[27.02.2015 09:05:21] <01> Error       at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
[27.02.2015 09:05:21] <01> Error       at System.Delegate.DynamicInvokeImpl(Object[] args)
[27.02.2015 09:05:21] <01> Error       at Veeam.Backup.SanPlugin.Hp3PARLib.CHp3PARReconnectableMethod.DoInvoke(CHp3PARWsApiInvoker apiInvoker, Delegate dlg, Object[] args)
[27.02.2015 09:05:21] <01> Error    volume does not exist (Veeam.Backup.SanPlugin.Hp3PARLib.CHp3PARException)
[27.02.2015 09:05:21] <01> Error       at Veeam.Backup.SanPlugin.Hp3PARLib.CHp3PARWsApiInvoker.CallAuthorizedWebServiceInternal(String commandUrl, String method)
[27.02.2015 09:05:21] <01> Info     [Hp3PAR] Reconnectable method: Retry is not needed.
[27.02.2015 09:05:21] <01> Info     [Hp3PAR] Snapshot 'Veeam_prd.vmw.r6.datastore3.' is no longer exists.


However, the volume clearly is still present on the Array. Is it possible that there's some limit to the size/name of the volume that's causing B&R to not see the snapshot? In the instance above, the snapshot created is named: Veeam_prd.vmw.r6.datastore3. (yes, with the trailing .). If I leave this snapshot in place, and re-run the backup, Veeam creates a snapshot off this Snap labeled: Veeam_prd.vmw.r6.datastore3._1 - after backing up from this snapshot, the snapshot ending in _1 is removed, but the initial snapshot still remains.

Support is trying to tell me this is an HP issue, but I'm skeptical about this. Thanks!
jkoyle
Novice
 
Posts: 7
Liked: 2 times
Joined: Fri May 27, 2011 8:25 pm
Full Name: John Koyle

Re: 3Par Storage Snapshots left over

Veeam Logoby jkoyle » Mon Mar 02, 2015 11:27 pm

I believe this is a bug in the snapshot functionality. My ticket was escalated to level 2 so I'll chase with them when they reach out to me. As stated previously, my original 3par virtual volume name was: prd.vmw.r6.datastore3.vv, and when snapshots were created, they would be named: Veeam_prd.vmw.r6.datastore3. (with the trailing period).

Today I renamed the datastore to simply be: prd.vmw.r6.ds3.vv and rescanned the 3par, when my test backup ran the snapshot was named Veeam_prd.vmw.r6.ds3.vv (no trailing period). When this backup completed, the snapshot was removed as expected.

I'm unsure what the issue is here. As per my previous post, subsequent backup runs would create/remove snapshots as expected. It was simply the first run snapshot that was never removed.
jkoyle
Novice
 
Posts: 7
Liked: 2 times
Joined: Fri May 27, 2011 8:25 pm
Full Name: John Koyle

Re: 3Par Storage Snapshots left over

Veeam Logoby Gostev » Mon Mar 02, 2015 11:49 pm

This looks like there is a datastore name length limit in API or elsewhere.
Gostev
Veeam Software
 
Posts: 21364
Liked: 2336 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: 3Par Storage Snapshots left over

Veeam Logoby jkoyle » Mon Mar 02, 2015 11:53 pm 1 person likes this post

It feels like some sort of limitation, but is it in the API or elsewhere - I'm not sure. Once B&R took an initial snapshot of the volume it was able to work with the longer snapshot just fine (adding the _1 suffix, see post 1). Definitely an issue somewhere though as I've now run several storage snapshot based backups and all are OK.
jkoyle
Novice
 
Posts: 7
Liked: 2 times
Joined: Fri May 27, 2011 8:25 pm
Full Name: John Koyle

Re: 3Par Storage Snapshots left over

Veeam Logoby jheleniak » Sat Dec 03, 2016 5:28 pm

Jkoyle did shortening your virtual volume less than 16 characters fix your problem? I'm getting similar behavior with 3par 3.2.2 MU3 and Veeam 9.5. Tons of snapshots left behind. Weird is that it's not all just some.
jheleniak
Lurker
 
Posts: 2
Liked: never
Joined: Sat Nov 26, 2016 3:54 pm
Full Name: John Heleniak

Re: 3Par Storage Snapshots left over

Veeam Logoby jkoyle » Mon Dec 05, 2016 3:31 pm 1 person likes this post

Yes, shortening the length took care of the issue.
jkoyle
Novice
 
Posts: 7
Liked: 2 times
Joined: Fri May 27, 2011 8:25 pm
Full Name: John Koyle

Re: 3Par Storage Snapshots left over

Veeam Logoby jheleniak » Wed Dec 07, 2016 1:51 am

for me too. i had my 3par virtual volume named VMWARE_DATASTORE_DEDUPED and changed it to VMWARE_DEDUPED and it properly deletes the snapshots now...i think under 16 characters for the 3par virtual volume name is safe and fixes the issue. must be somewhere either on the api or veeam code. i am letting them know to save someone else hours of banging their head against their box.
jheleniak
Lurker
 
Posts: 2
Liked: never
Joined: Sat Nov 26, 2016 3:54 pm
Full Name: John Heleniak

Re: 3Par Storage Snapshots left over

Veeam Logoby znabela » Tue Jun 06, 2017 8:43 am

We are having the same issue on our 3PAR - all our volumes are named with 16 characters and above.

3PAR: MU 3.2.2 mu4
Veeam: 9.5 u2.

Also please note that the naming convention on 3PAR allows up to 31 characters. ref page 29-30, "HPE 3PAR StoreServe Storage best practices guide", May 2017 rev 13.
znabela
Influencer
 
Posts: 12
Liked: 5 times
Joined: Thu Dec 04, 2014 7:09 am
Full Name: Robert Christiansen

Re: 3Par Storage Snapshots left over

Veeam Logoby foggy » Wed Jun 07, 2017 5:21 pm

Yes, the length limitation on 3PAR is the culprit. We've checked on 3.3.1 though and it doesn't seem to have such issues (snapshot names were cut to comply with the length limit and everything was cleaned up).
foggy
Veeam Software
 
Posts: 14590
Liked: 1063 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: 3Par Storage Snapshots left over

Veeam Logoby jdixon » Fri Jul 21, 2017 2:28 am

I opened a case on this and i'm running 3PAR 3.3.1.215 with Patch 01 and 02. I have an issue with snapshots not being deleted.

The hardware provider is at the latest 2.7.0 also:

[logs removed by moderator]
jdixon
Novice
 
Posts: 4
Liked: never
Joined: Wed Nov 23, 2016 4:30 pm
Full Name: Jacob Dixon

Re: 3Par Storage Snapshots left over

Veeam Logoby foggy » Fri Jul 21, 2017 2:41 pm

Jacob, please provide your support case ID number for reference. Thanks.
foggy
Veeam Software
 
Posts: 14590
Liked: 1063 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Codehouse, foggy, Google [Bot], hasayeretFMG, v.Eremin and 74 guests