Pure SafeMode and Veeam HW Snapshot Retention

[Spex]

We use Veeam snapshots supported on pure storage HW for backup jobs. This works great and avoids the need to present all LUNs to our HW proxies. Since 12.3 and the SanNoBFSSIfSnapshotExists option, there are even no more issues with "Failed to collect disk file location data. Timeout exceeded."

But now I want to share an issue we observed in this combination. We have been wondering about the high load on pure storage (CPU and capacity) for some time and had no idea where it was coming from. We also did not use pure safe snapshots (turned off) for our protection groups.

But now we have noticed that with the introduction of a new pure firmware in May 2024, any LUN without an explicit setting for secure snapshots will become a member of a default protection group that stores the snapshots in safe mode for a few days. I knew about this firmware change but thought we were unaffected because all of our LUNs were configured to not use secure snapshots...

But I didn't account for the snapshots created by Veeam and presented to our backup proxies during Veeam's backup. All of them where configured with the default safe snapshot policy from pure and stored for in our case 5 days. Maybe your storage is also doing things you don't know

[Andreas Neufert]

Thanks for sharing Stefan.
This is more of a question for Pure Storage as they are writing the storage integration directly.
I will ask our Pure Alliance team to comment and ask Pure to comment here as well.
Thanks.

[markdpolin]

Hi Stefan,

My contact at Pure Storage is saying this is likely a known issue with auto-on SafeMode that is planned to be resolved in their upcoming V2.1 USAPI plugin. He did recommend opening a support case with Pure just to be sure. I hope this helps.

[Andreas Neufert]

@foggy FYI

[Spex]

Hi Stefan,

My contact at Pure Storage is saying this is likely a known issue with auto-on SafeMode that is planned to be resolved in their upcoming V2.1 USAPI plugin. He did recommend opening a support case with Pure just to be sure. I hope this helps.

Thanks for this info. I'm already in contact with pure support for cleanup (not that quick&easy).

[foggy]

Hi Stefan, you're most likely referring to clones, not snapshots. Indeed, with the current plug-in version, clones become members of a default protection group (pgroup-auto) and safe snapshot policy is applied to them. This is not the case with the plug-in version that is currently going through our QA cycles, so this seems to be fixed there.

[SteveK821]

For Backup from Storage Snapshots, Veeam creates a clone on the FlashArray so it can mount it on the proxy host. You can't just mount a snapshot on another host, you have to create a volume from it first (which is a clone). And that automatically gets added to pgroup-auto.
Veeam also deletes the volume after the backup, but this volume and its snapshots are retained in the Flasharray's "recycle bin" and are not eradicated until they expire. Without Safemode that period is 24 hours (but you can manually eradicate them as well), on a Safemode-enabled volume it's 8 days by default and cannot be manually eradicated. With one of my customers, they had a high change-rate on a couple of their volumes, which significantly increased the used capacity of the array. So we had to get Pure support to disable Array-Wide pgroup-auto, they were already using other Safemode protection groups for their production volumes. This was on a new FlashArray XR4, which come pre-installed with a Purity version with auto-on Safemode activated by default (6.4.10 or higher).

But it seems the new plugin for VBR will alleviate this issue, so that's good to know.

[foggy]

The new plug-in version has just been released: https://www.veeam.com/kb4129