Comprehensive data protection for all workloads
Post Reply
jkoyle
Novice
Posts: 7
Liked: 2 times
Joined: May 27, 2011 8:25 pm
Full Name: John Koyle
Contact:

3Par Storage Snapshots left over

Post by jkoyle »

If I have 2 different backup jobs both using storage snapshots, and they both kick off at the same time, I get a chain of snaps like this:

3Par_virtual_vol
-> 3par_virtual_vol_snap.
-> 3par_virtual_vol_snap._1

If the backup that created the first snap completes first, it doesn't/can clean up the snapshot from the 2nd running job because it's based off the HW snap from the first. However, when the 2nd job completes, it only cleans up it's snapshot, and I'm left with a lingering HW snap on the array: 3par_virtual_vol_snap. Is this expected, or shall I open a support ticket?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: 3Par Storage Snapshots left over

Post by foggy »

John, this does not look expected, so please open a case and provide its ID here for reference. Thanks!
jkoyle
Novice
Posts: 7
Liked: 2 times
Joined: May 27, 2011 8:25 pm
Full Name: John Koyle
Contact:

Re: 3Par Storage Snapshots left over

Post by jkoyle »

I've just opened case 00818913 for this issue. I see the copy space growing and growing because the initial snapshot isn't deleted as expected.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: 3Par Storage Snapshots left over

Post by foggy »

Thanks, please keep us informed on how the investigation goes.
jkoyle
Novice
Posts: 7
Liked: 2 times
Joined: May 27, 2011 8:25 pm
Full Name: John Koyle
Contact:

Re: 3Par Storage Snapshots left over

Post by jkoyle »

It looks like Veeam is complaining that the snapshot volume doesn't exist when it attempts to remove the snapshot:

Code: Select all

[27.02.2015 09:05:21] <01> Info     [WSAPI Connection] Disconnecting from HP 3PAR 'https://X.X.X.X:8080'. Connection key prefix: '0'
[27.02.2015 09:05:21] <01> Error    [Hp3PAR] Generate Hp3PAR exception. ErrorCode: '23', Description: 'volume does not exist', Reference: '<null>'.
[27.02.2015 09:05:21] <01> Error    Failed to call web service methods. RetryCount: '1'. MaxRetryCount: '3'.
[27.02.2015 09:05:21] <01> Error    Exception has been thrown by the target of an invocation. (System.Reflection.TargetInvocationException)
[27.02.2015 09:05:21] <01> Error       at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
[27.02.2015 09:05:21] <01> Error       at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
[27.02.2015 09:05:21] <01> Error       at System.Delegate.DynamicInvokeImpl(Object[] args)
[27.02.2015 09:05:21] <01> Error       at Veeam.Backup.SanPlugin.Hp3PARLib.CHp3PARReconnectableMethod.DoInvoke(CHp3PARWsApiInvoker apiInvoker, Delegate dlg, Object[] args)
[27.02.2015 09:05:21] <01> Error    volume does not exist (Veeam.Backup.SanPlugin.Hp3PARLib.CHp3PARException)
[27.02.2015 09:05:21] <01> Error       at Veeam.Backup.SanPlugin.Hp3PARLib.CHp3PARWsApiInvoker.CallAuthorizedWebServiceInternal(String commandUrl, String method)
[27.02.2015 09:05:21] <01> Info     [Hp3PAR] Reconnectable method: Retry is not needed.
[27.02.2015 09:05:21] <01> Info     [Hp3PAR] Snapshot 'Veeam_prd.vmw.r6.datastore3.' is no longer exists.
However, the volume clearly is still present on the Array. Is it possible that there's some limit to the size/name of the volume that's causing B&R to not see the snapshot? In the instance above, the snapshot created is named: Veeam_prd.vmw.r6.datastore3. (yes, with the trailing .). If I leave this snapshot in place, and re-run the backup, Veeam creates a snapshot off this Snap labeled: Veeam_prd.vmw.r6.datastore3._1 - after backing up from this snapshot, the snapshot ending in _1 is removed, but the initial snapshot still remains.

Support is trying to tell me this is an HP issue, but I'm skeptical about this. Thanks!
jkoyle
Novice
Posts: 7
Liked: 2 times
Joined: May 27, 2011 8:25 pm
Full Name: John Koyle
Contact:

Re: 3Par Storage Snapshots left over

Post by jkoyle »

I believe this is a bug in the snapshot functionality. My ticket was escalated to level 2 so I'll chase with them when they reach out to me. As stated previously, my original 3par virtual volume name was: prd.vmw.r6.datastore3.vv, and when snapshots were created, they would be named: Veeam_prd.vmw.r6.datastore3. (with the trailing period).

Today I renamed the datastore to simply be: prd.vmw.r6.ds3.vv and rescanned the 3par, when my test backup ran the snapshot was named Veeam_prd.vmw.r6.ds3.vv (no trailing period). When this backup completed, the snapshot was removed as expected.

I'm unsure what the issue is here. As per my previous post, subsequent backup runs would create/remove snapshots as expected. It was simply the first run snapshot that was never removed.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 3Par Storage Snapshots left over

Post by Gostev »

This looks like there is a datastore name length limit in API or elsewhere.
jkoyle
Novice
Posts: 7
Liked: 2 times
Joined: May 27, 2011 8:25 pm
Full Name: John Koyle
Contact:

Re: 3Par Storage Snapshots left over

Post by jkoyle » 1 person likes this post

It feels like some sort of limitation, but is it in the API or elsewhere - I'm not sure. Once B&R took an initial snapshot of the volume it was able to work with the longer snapshot just fine (adding the _1 suffix, see post 1). Definitely an issue somewhere though as I've now run several storage snapshot based backups and all are OK.
jheleniak
Lurker
Posts: 2
Liked: never
Joined: Nov 26, 2016 3:54 pm
Full Name: John Heleniak
Contact:

Re: 3Par Storage Snapshots left over

Post by jheleniak »

Jkoyle did shortening your virtual volume less than 16 characters fix your problem? I'm getting similar behavior with 3par 3.2.2 MU3 and Veeam 9.5. Tons of snapshots left behind. Weird is that it's not all just some.
jkoyle
Novice
Posts: 7
Liked: 2 times
Joined: May 27, 2011 8:25 pm
Full Name: John Koyle
Contact:

Re: 3Par Storage Snapshots left over

Post by jkoyle » 1 person likes this post

Yes, shortening the length took care of the issue.
jheleniak
Lurker
Posts: 2
Liked: never
Joined: Nov 26, 2016 3:54 pm
Full Name: John Heleniak
Contact:

Re: 3Par Storage Snapshots left over

Post by jheleniak »

for me too. i had my 3par virtual volume named VMWARE_DATASTORE_DEDUPED and changed it to VMWARE_DEDUPED and it properly deletes the snapshots now...i think under 16 characters for the 3par virtual volume name is safe and fixes the issue. must be somewhere either on the api or veeam code. i am letting them know to save someone else hours of banging their head against their box.
znabela
Service Provider
Posts: 13
Liked: 5 times
Joined: Dec 04, 2014 7:09 am
Full Name: Robert Christiansen
Contact:

Re: 3Par Storage Snapshots left over

Post by znabela »

We are having the same issue on our 3PAR - all our volumes are named with 16 characters and above.

3PAR: MU 3.2.2 mu4
Veeam: 9.5 u2.

Also please note that the naming convention on 3PAR allows up to 31 characters. ref page 29-30, "HPE 3PAR StoreServe Storage best practices guide", May 2017 rev 13.
Robert Christiansen
Infrastructure Specialist @ Danoffice IT
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: 3Par Storage Snapshots left over

Post by foggy »

Yes, the length limitation on 3PAR is the culprit. We've checked on 3.3.1 though and it doesn't seem to have such issues (snapshot names were cut to comply with the length limit and everything was cleaned up).
jdixon
Novice
Posts: 8
Liked: 1 time
Joined: Nov 23, 2016 4:30 pm
Full Name: Jacob Dixon
Contact:

Re: 3Par Storage Snapshots left over

Post by jdixon »

I opened a case on this and i'm running 3PAR 3.3.1.215 with Patch 01 and 02. I have an issue with snapshots not being deleted.

The hardware provider is at the latest 2.7.0 also:

[logs removed by moderator]
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: 3Par Storage Snapshots left over

Post by foggy »

Jacob, please provide your support case ID number for reference. Thanks.
jdixon
Novice
Posts: 8
Liked: 1 time
Joined: Nov 23, 2016 4:30 pm
Full Name: Jacob Dixon
Contact:

Re: 3Par Storage Snapshots left over

Post by jdixon » 1 person likes this post

foggy wrote:Jacob, please provide your support case ID number for reference. Thanks.
Veeam CASE: 02248867
HP CASE:5321539117

I made some progress with HP. I'm on version 3.3.1 and the latest available download for the 3PAR CLI is 3.2.2. HOWEVER apparently they do have a 3.3.1 CLI version but it isn't publicly available for some reason. I'm testing the new 3PAR cli version right now (since last night.. too soon for update)
Post Reply

Who is online

Users browsing this forum: justin.hendren and 165 guests