Veeam w/ Pure Storage Snapshot Integration Orphaned Snapshot

[jtupeck]

Morning All -

A couple weeks ago we installed the Pure Storage plugin for snapshot integration into two of our environments. Both are Veeam 9.5 U3 with Purity code at 4.8.11, with physical proxy servers running on a Cisco UCS platform (one UCS 3260 and one is a B200 blade series server). In one environment everything works perfectly. In the other, we are experiencing orphaned snapshots at the array level upon job completion. Somewhere, it appears that the process to delete the snap is timing out. Whether the failure is on the Veeam side and its not communicating with the array, or it's on the Pure side and is taking too long to delete, I am unsure. I have Veeam Support case #03030115 opened, as well as a Pure Storage support case.

From everything I can see, this began when one particular VM had a VMDK disk sizing issue and Veeam reported the "Virtual disk size not a multiple of 1kb" error. This caused the backup of this one VM to fail, and we also saw the array snapshot error at the same time. This was resolved a day later and the subsequent retry worked perfectly. I thought we were in the clear and reported that to support yesterday. Last night, the snapshot delete error came up again on the same job and on another job the snapshot creation event ALSO timed out.

I am posting here for informational purposes as well as to just see if there have been any other issues that anyone has seen with this integration since it was released. Support is looking at it, it has been escalated and new logs from last night's job runs were uploaded moments ago. Hopefully it's something that can be figured out soon!

Thanks for reading.

[Gostev]

I am posting here for informational purposes as well as to just see if there have been any other issues that anyone has seen with this integration since it was released.

Last I checked with our T3 support (at around 1000 Pure plug-in downloads), and there were no known issues around this particular integration... however, this was a few weeks ago already. Thanks for opening the support case!

[jtupeck]

Latest update from Support is to swap the MPIO policy on the physical proxy to 'Fail Over Only'. I am waiting on confirmation that this is what was meant and will make the changes once I know it won't affect anything negatively. (I suspect it will not, but I am paranoid about this sort of stuff, seeing as its directly connected to the production volumes on the array, haha!) This setting is supposedly related to a snapshot CREATION event failure, which i did experience last night, but we are unsure how it will affect the deletion timeout issue. Also, logging level for integrated plugins was turned on as well via the key below and I will report on any additional failures that occur this weekend so we can get the more detailed logs looked at.

Code: Select all

Location: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
DWORD: LogPublicPluginMethods
Value: 1

[foggy]

There was an issue where network problems resulted in similar behavior - snapshots were not deleted. It was addressed in the upcoming Veeam B&R v9.5 U3a, so it worth checking when it is available.