3Par Storage Snapshots left over

[jkoyle]

If I have 2 different backup jobs both using storage snapshots, and they both kick off at the same time, I get a chain of snaps like this:

3Par_virtual_vol
-> 3par_virtual_vol_snap.
-> 3par_virtual_vol_snap._1

If the backup that created the first snap completes first, it doesn't/can clean up the snapshot from the 2nd running job because it's based off the HW snap from the first. However, when the 2nd job completes, it only cleans up it's snapshot, and I'm left with a lingering HW snap on the array: 3par_virtual_vol_snap. Is this expected, or shall I open a support ticket?

[foggy]

John, this does not look expected, so please open a case and provide its ID here for reference. Thanks!

[jkoyle]

I've just opened case 00818913 for this issue. I see the copy space growing and growing because the initial snapshot isn't deleted as expected.

[foggy]

Thanks, please keep us informed on how the investigation goes.

[jkoyle]

It looks like Veeam is complaining that the snapshot volume doesn't exist when it attempts to remove the snapshot:

Code: Select all

[27.02.2015 09:05:21] <01> Info     [WSAPI Connection] Disconnecting from HP 3PAR 'https://X.X.X.X:8080'. Connection key prefix: '0'
[27.02.2015 09:05:21] <01> Error    [Hp3PAR] Generate Hp3PAR exception. ErrorCode: '23', Description: 'volume does not exist', Reference: '<null>'.
[27.02.2015 09:05:21] <01> Error    Failed to call web service methods. RetryCount: '1'. MaxRetryCount: '3'.
[27.02.2015 09:05:21] <01> Error    Exception has been thrown by the target of an invocation. (System.Reflection.TargetInvocationException)
[27.02.2015 09:05:21] <01> Error       at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
[27.02.2015 09:05:21] <01> Error       at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
[27.02.2015 09:05:21] <01> Error       at System.Delegate.DynamicInvokeImpl(Object[] args)
[27.02.2015 09:05:21] <01> Error       at Veeam.Backup.SanPlugin.Hp3PARLib.CHp3PARReconnectableMethod.DoInvoke(CHp3PARWsApiInvoker apiInvoker, Delegate dlg, Object[] args)
[27.02.2015 09:05:21] <01> Error    volume does not exist (Veeam.Backup.SanPlugin.Hp3PARLib.CHp3PARException)
[27.02.2015 09:05:21] <01> Error       at Veeam.Backup.SanPlugin.Hp3PARLib.CHp3PARWsApiInvoker.CallAuthorizedWebServiceInternal(String commandUrl, String method)
[27.02.2015 09:05:21] <01> Info     [Hp3PAR] Reconnectable method: Retry is not needed.
[27.02.2015 09:05:21] <01> Info     [Hp3PAR] Snapshot 'Veeam_prd.vmw.r6.datastore3.' is no longer exists.

However, the volume clearly is still present on the Array. Is it possible that there's some limit to the size/name of the volume that's causing B&R to not see the snapshot? In the instance above, the snapshot created is named: Veeam_prd.vmw.r6.datastore3. (yes, with the trailing .). If I leave this snapshot in place, and re-run the backup, Veeam creates a snapshot off this Snap labeled: Veeam_prd.vmw.r6.datastore3._1 - after backing up from this snapshot, the snapshot ending in _1 is removed, but the initial snapshot still remains.

Support is trying to tell me this is an HP issue, but I'm skeptical about this. Thanks!

[jkoyle]

I believe this is a bug in the snapshot functionality. My ticket was escalated to level 2 so I'll chase with them when they reach out to me. As stated previously, my original 3par virtual volume name was: prd.vmw.r6.datastore3.vv, and when snapshots were created, they would be named: Veeam_prd.vmw.r6.datastore3. (with the trailing period).

Today I renamed the datastore to simply be: prd.vmw.r6.ds3.vv and rescanned the 3par, when my test backup ran the snapshot was named Veeam_prd.vmw.r6.ds3.vv (no trailing period). When this backup completed, the snapshot was removed as expected.

I'm unsure what the issue is here. As per my previous post, subsequent backup runs would create/remove snapshots as expected. It was simply the first run snapshot that was never removed.

[Gostev]

This looks like there is a datastore name length limit in API or elsewhere.

[jkoyle]

It feels like some sort of limitation, but is it in the API or elsewhere - I'm not sure. Once B&R took an initial snapshot of the volume it was able to work with the longer snapshot just fine (adding the _1 suffix, see post 1). Definitely an issue somewhere though as I've now run several storage snapshot based backups and all are OK.

[jheleniak]

Jkoyle did shortening your virtual volume less than 16 characters fix your problem? I'm getting similar behavior with 3par 3.2.2 MU3 and Veeam 9.5. Tons of snapshots left behind. Weird is that it's not all just some.

[jkoyle]

Yes, shortening the length took care of the issue.

[jheleniak]

for me too. i had my 3par virtual volume named VMWARE_DATASTORE_DEDUPED and changed it to VMWARE_DEDUPED and it properly deletes the snapshots now...i think under 16 characters for the 3par virtual volume name is safe and fixes the issue. must be somewhere either on the api or veeam code. i am letting them know to save someone else hours of banging their head against their box.

[znabela]

We are having the same issue on our 3PAR - all our volumes are named with 16 characters and above.

3PAR: MU 3.2.2 mu4
Veeam: 9.5 u2.

Also please note that the naming convention on 3PAR allows up to 31 characters. ref page 29-30, "HPE 3PAR StoreServe Storage best practices guide", May 2017 rev 13.

[foggy]

Yes, the length limitation on 3PAR is the culprit. We've checked on 3.3.1 though and it doesn't seem to have such issues (snapshot names were cut to comply with the length limit and everything was cleaned up).

[jdixon]

I opened a case on this and i'm running 3PAR 3.3.1.215 with Patch 01 and 02. I have an issue with snapshots not being deleted.

The hardware provider is at the latest 2.7.0 also:

[logs removed by moderator]

[foggy]

Jacob, please provide your support case ID number for reference. Thanks.

[jdixon]

Jacob, please provide your support case ID number for reference. Thanks.

Veeam CASE: 02248867
HP CASE:5321539117

I made some progress with HP. I'm on version 3.3.1 and the latest available download for the 3PAR CLI is 3.2.2. HOWEVER apparently they do have a 3.3.1 CLI version but it isn't publicly available for some reason. I'm testing the new 3PAR cli version right now (since last night.. too soon for update)