-
- Veteran
- Posts: 515
- Liked: 100 times
- Joined: Sep 17, 2017 3:20 am
- Full Name: Franc
- Contact:
NetApp tamperproof snapshots cause veeam_aux volume not being deleted
Case #07108016
On our NetApp system running Ontap 9.13.1 we’ve enabled the new tamperproof snapshot copies feature. We’ve set a snapshot schedule to automatically create snapshots every hour. As a test we’ve set the Snaplock retention to 1 day.
For backups using Veeam we use storage integration.
The problem we are facing now is that Veeam sometimes is unable to delete its storage snapshot because it’s locked by the Snaplock retention. This happens when Veeam creates its snapshot around the time the automatic snapshot schedule on the NetApp runs and creates a tamperproof snapshot. By default a volume clone inherits its parents snapshot settings. This results in automatic tamperproof snapshots being taken of the temporary clone also.
In NetApp system manager we now have multiple veeam_aux volumes which can’t be deleted yet since the Snaplock retention has not expired.
The affected Veeam jobs return the following warning:
Failed to delete snapshot VeeamAUX<volme name> of volume VeeamAUX<volume name>. Details: Failed to delete volume. The volume has one or more unexpired locked Snapshot copies.
The solution would be that Veeam removes the snapshot schedules from the cloned volume so that no automatic snapshots will be created and Veeam can delete the volume after the backup completes.
The situation currently is that you end up with many veeam_aux volumes on the storage system which can only be deleted when the retention expires. Depending on the Snaplock configuration this can take weeks or months. Also these volumes need to be manually deleted. Not a pleasant situation to say the least
On our NetApp system running Ontap 9.13.1 we’ve enabled the new tamperproof snapshot copies feature. We’ve set a snapshot schedule to automatically create snapshots every hour. As a test we’ve set the Snaplock retention to 1 day.
For backups using Veeam we use storage integration.
The problem we are facing now is that Veeam sometimes is unable to delete its storage snapshot because it’s locked by the Snaplock retention. This happens when Veeam creates its snapshot around the time the automatic snapshot schedule on the NetApp runs and creates a tamperproof snapshot. By default a volume clone inherits its parents snapshot settings. This results in automatic tamperproof snapshots being taken of the temporary clone also.
In NetApp system manager we now have multiple veeam_aux volumes which can’t be deleted yet since the Snaplock retention has not expired.
The affected Veeam jobs return the following warning:
Failed to delete snapshot VeeamAUX<volme name> of volume VeeamAUX<volume name>. Details: Failed to delete volume. The volume has one or more unexpired locked Snapshot copies.
The solution would be that Veeam removes the snapshot schedules from the cloned volume so that no automatic snapshots will be created and Veeam can delete the volume after the backup completes.
The situation currently is that you end up with many veeam_aux volumes on the storage system which can only be deleted when the retention expires. Depending on the Snaplock configuration this can take weeks or months. Also these volumes need to be manually deleted. Not a pleasant situation to say the least
-
- Product Manager
- Posts: 9385
- Liked: 2500 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: NetApp tamperproof snapshots cause veeam_aux volume not being deleted
Hi Franc
Veeam Backup & Replication is not yet aware of NetApp tamperproof snapshots. Therefore I expect such side affects. We have official support on our roadmap, but no ETA yet.
When we introduce official support, we will make sure that such use cases are also covered.
Best,
Fabian
Veeam Backup & Replication is not yet aware of NetApp tamperproof snapshots. Therefore I expect such side affects. We have official support on our roadmap, but no ETA yet.
When we introduce official support, we will make sure that such use cases are also covered.
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Veteran
- Posts: 515
- Liked: 100 times
- Joined: Sep 17, 2017 3:20 am
- Full Name: Franc
- Contact:
Re: NetApp tamperproof snapshots cause veeam_aux volume not being deleted
ok, thanks. In the meantime I've contacted NetApp support to ask if it's possible to disable the automatic inheritance of snapshot schedules when mounting a snapshot as a temporary volume.
-
- Veteran
- Posts: 515
- Liked: 100 times
- Joined: Sep 17, 2017 3:20 am
- Full Name: Franc
- Contact:
Re: NetApp tamperproof snapshots cause veeam_aux volume not being deleted
Netapp support stated that it’s not possible to disable the automatic inheritance of the tamperproof snapshot settings and schedule. So for now, it’s not possible for us to activate this feature until Veeam is compatible with it. Bummer…
-
- Veeam Software
- Posts: 21111
- Liked: 2132 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: NetApp tamperproof snapshots cause veeam_aux volume not being deleted
Hi Franc, we've researched this a bit and it is possible to disable automatic schedule inheritance right after creating the clone on the Veeam B&R side. We will implement this in one of the upcoming updates as a workaround/fix for the behavior you're seeing. Thanks for the heads up!
-
- Veteran
- Posts: 515
- Liked: 100 times
- Joined: Sep 17, 2017 3:20 am
- Full Name: Franc
- Contact:
Re: NetApp tamperproof snapshots cause veeam_aux volume not being deleted
Ok, great!
I think you may also need to clear the 'enable snapshot locking' setting on the cloned volume or else the volume still can't be deleted even when there are no locked snapshots present.
I think you may also need to clear the 'enable snapshot locking' setting on the cloned volume or else the volume still can't be deleted even when there are no locked snapshots present.
-
- Veteran
- Posts: 515
- Liked: 100 times
- Joined: Sep 17, 2017 3:20 am
- Full Name: Franc
- Contact:
Re: NetApp tamperproof snapshots cause veeam_aux volume not being deleted
thanks, we aren't using snapmirror, but these were exactly the steps I had to take earlier to be able to delete the _aux volumes.
-
- Veeam Software
- Posts: 21111
- Liked: 2132 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: NetApp tamperproof snapshots cause veeam_aux volume not being deleted
This may be a different issue, e.g. this one: https://kb.netapp.com/onprem/ontap/dp/S ... hot_copies
Without a schedule being inherited, the clone volume shouldn't get locked and can be deleted even with the 'Enable snapshot locking' setting enabled - at least this is what we're seeing on 9.14.
Without a schedule being inherited, the clone volume shouldn't get locked and can be deleted even with the 'Enable snapshot locking' setting enabled - at least this is what we're seeing on 9.14.
-
- Veteran
- Posts: 515
- Liked: 100 times
- Joined: Sep 17, 2017 3:20 am
- Full Name: Franc
- Contact:
Re: NetApp tamperproof snapshots cause veeam_aux volume not being deleted
We are on 13.1P6 and are planning to install P7. 9.14.1 is quite new, so we wait a couple of patches before updating to that one.
-
- Veteran
- Posts: 515
- Liked: 100 times
- Joined: Sep 17, 2017 3:20 am
- Full Name: Franc
- Contact:
Re: NetApp tamperproof snapshots cause veeam_aux volume not being deleted
Is this fix/workaround implemented in the latest cumulative update 12.1.2.172?foggy wrote: ↑Feb 12, 2024 2:05 pm Hi Franc, we've researched this a bit and it is possible to disable automatic schedule inheritance right after creating the clone on the Veeam B&R side. We will implement this in one of the upcoming updates as a workaround/fix for the behavior you're seeing. Thanks for the heads up!
-
- Product Manager
- Posts: 9385
- Liked: 2500 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: NetApp tamperproof snapshots cause veeam_aux volume not being deleted
Hi Franc
I checked our internal system and the fix is scheduled for V12.2.
Best,
Fabian
I checked our internal system and the fix is scheduled for V12.2.
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Veteran
- Posts: 515
- Liked: 100 times
- Joined: Sep 17, 2017 3:20 am
- Full Name: Franc
- Contact:
Re: NetApp tamperproof snapshots cause veeam_aux volume not being deleted
Hi Fabian,
ok, thanks. Then we'll stick to non-storage snapshot backups for the time being. Tamperproof snapshots are more important to us.
ok, thanks. Then we'll stick to non-storage snapshot backups for the time being. Tamperproof snapshots are more important to us.
-
- Veteran
- Posts: 515
- Liked: 100 times
- Joined: Sep 17, 2017 3:20 am
- Full Name: Franc
- Contact:
Re: NetApp tamperproof snapshots cause veeam_aux volume not being deleted
We have another issue with this. After we enabled tamperproof snapshots on our NetApp there's an issue now with Storage Rescan. When starting the storage rescan, Veeam creates a clone of the volume it wants to scan for VMs. Creating a clone also inherits the snapshot settings and also the snaplock setting. When the storage rescan creates a volume clone around the same time an automatic snapshot is taken by the NetApp, this results in a locked snapshot and as a result, Veeam cannot delete the volume anymore. I have to manually clean up the volume after the snaplock retention period, which can be days or even weeks if the NetApp also created a weekly snapshot.
-
- Veeam Software
- Posts: 21111
- Liked: 2132 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: NetApp tamperproof snapshots cause veeam_aux volume not being deleted
Hi Franc, we're aware of the issue and it will be addressed in the next update - we're already testing the solution.
-
- Service Provider
- Posts: 4
- Liked: 1 time
- Joined: Nov 02, 2022 9:58 am
- Full Name: Francois DEFROCOURT
- Contact:
Re: NetApp tamperproof snapshots cause veeam_aux volume not being deleted
Hi All,
Have you info about this issue is fixed in 12.2 ?
i have ONTAP 9.14.1 and 12.2 and the issue is still here.
Best regards,
Have you info about this issue is fixed in 12.2 ?
i have ONTAP 9.14.1 and 12.2 and the issue is still here.
Best regards,
-
- Veeam Software
- Posts: 21111
- Liked: 2132 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: NetApp tamperproof snapshots cause veeam_aux volume not being deleted
Hi Francois, this should have been fixed in 12.2. please open a support case for a closer look.
Who is online
Users browsing this forum: Bing [Bot], Gostev, Semrush [Bot] and 240 guests