Equallogic On Latest Firmware-Volume Snapshots Not Deleting

[stevedarl1]

We have just upgraded to the latest Dell Equallogic firmware V7.0.1 on our iSCSI SANs and now when a backup runs it snapshots the volumes fine but does not remove them after completion. Anyone else had similar issues after upgrading the firmware? It means multiple snapshots are being created for VMs that are backing up from the same Volume. It is a manual process at the moment to remove the snapshots every day.

Environment:

Hyper-V 2012 and Hyper-V 2012 R2 cluster
Veeam V7 R2
Equallogic iSCSI SANs running V7.0.1

[veremin]

Hi, Steve,

Have you already contacted support team with this issue? They will be able to determine whether this behaviour is caused by the latest firmware and not by other environmental factors/changes that passed unnoticed.

Thanks.

[stevedarl1]

Hi,

Yes I have, the Veeam logs show the commands being sent to create/delete the snapshot. I believe this to be an Equallogic issue, however I was just seeing if anyone else has had similar issues.

[foggy]

I believe no one has reported similar issues here so far, so it is worth contacting Dell support for clarification. Thanks.

[stevedarl1]

Hi Foggy, do you know the command or syntax that is used to take/remove a snapshot? We use on host backups with Dell Auto Snapshot Manager to take Hardware VSS snapshots.

Thanks.

[foggy]

Steve, the snapshot is created with the IVssBackupComponents::DoSnapshotSet method and should be automatically deleted on releasing the IVssBackupComponents object.

[matt306]

I work with Steve and we've had a reply from Equallogic today. It transpires they have missed something in VSS support, hence why the snapshots aren't being deleted. They hinted that we were not the only customers affected and that a patch "may" be available soon, although it wasn't confirmed if this would be an official release or an early deployment/availability patch.

[foggy]

Thanks for these news, Matt! Much appreciated.

[Argos]

I am seeing the same issue with firmware v7.01 on Equallogic. Snapshots not being automatically deleted.

[foggy]

I believe you could also contact Dell support to be notified once the above mentioned fix is available. Thanks.

[stevedarl1]

Not only are the snapshots not deleting automatically still, but we are now seeing snapshots failing to online once created within Equallogic group manager. If anyone sees this issue in Veeam the job will just sit there until you manually online the snapshot within group manager, it then resumes the backup.

Dell are releasing the new V7.0.2 firmware within the next week or 2 and they have confirmed the code has returned. However HIT kit 4.7 will not be released until March sometime and apparently this issue is related to HIT kit 4.6 on the hosts which is where the VSS credentials are stored. The new HIT kit will also be 2012 R2 compatible however we ran V4.6 for 3 months on 2012 R2 hosts using the old Equallogic firmware without issues.

[bcscholl]

Any updates on this?

I've set my Snapshot Reserve down to 10%, turned off borrowing. This seems to work, just deletes the oldest snapshot when it needs more space.

[matt306]

No news yet from Dell, still the same 7.0.2 is due in the next couple of weeks and HIT 4.7 by March.

Originally we did the same with limiting the snapshots, but had a couple of instances of losing iSCSI connections to volumes once the snapshot ran out of space. Since then we've sent a load of logs to Dell who found a controller panic on one of the SAN's so it's possible it was this not the snapshot that caused our issues.

As we can't be sure we're still deleting shortly after they're used, although recently we've got braver and allowed a whole job to complete before removing them.

[bcscholl]

I’ve seen a few MPIO and disk errors on one of my Hyper-V nodes as the snapshot is started. During that time I cannot browse to C:\ClusterVolumes folder. All the VMs stay running and still look to write data to the disk. No Cluster events are recorded, no CSV ownership is changed. My worry is that at some point it will break something. Holding my breath and hoping 4.7 HIT kit will fix the MPIO errors.

[matt306]

7.0.2 is now available, which allegedly adds the VSS commands back in (although nothing is mentioned in the release notes). However it will not fix the deleting snapshots until HIT 4.7 is released.

[estark]

It seems like this is a problem with Dell Equallogic HIT Kit 4.6. After working for a day with Dell, it seems like they have two documented bugs regarding the removal of snapshots with HIT Kit 4.6 which is required for PS Firmware 7.0.1. There is a problem with VEEAM and DPM (and I think others) where the programs are unable to set snapshots offline and delete them. There first documented this at the end of January and are planning to address this in the next HIT Kit scheduled to be released on February 28. Bugzilla has this bug reported as #60369 according to Dell.

[matt306]

Even though Dell are saying it's HIT kit related I think it's only half the issue. When we were running 6.0.4 with HIT 4.6 on a 2012R2 Cluster it didn't have any problems removing the snapshots. The issue only presented itself once 7.0.1 was installed. I think they've added some of the code back in for 7.0.2 but are relying on the HIT to provide the correct syntax to the SAN, which is why we have to wait for 4.7

[ryanpsulion]

So glad we went ahead and updated all of are arrays to 7.0.1 and HIT 4.6 two weekends ago in anticipation for Veeam deployment. To only learn we now have hundreds of online snapshots that are not being trimmed at all. Superb!

[matt306]

Ryan - Other people are leaving their snapshots and configuring the Equallogic to remove the oldest one once the volume runs out of snapshot reserve. We are deleting ours manually as we had a number of CSV pauses/disconnects once the snapshot reserve was consumed. However we were still getting pause/disconnects outside of the backup window, just not as frequently (or cluster breaking). We turned ODX off and since doing so have not had any further CSV problems. I suspect that if I left the snapshots we wouldn't have a problem, but it's not the sort of environment to be testing that hypothesis!

[matt306]

From a Dell Employee:
http://en.community.dell.com/techcenter ... 43118.aspx

4.7 is due out very soon. That should help.

There was a change made to 7.0.x firmware based on customer feedback. If a snapshot is online or online with connections then don't just delete it. There were cases when customers were running from a snapshot and a schedule max keep or to reclaim snapshot reserve would delete that active snapshot. When VSS goes to delete the snapshot, it's not putting it offline first, preventing the snapshot from being deleted.

Looking at the thread it also affects MS DPM users. It probably affects Symantec V-Ray customers too (if you can find any).

[matt306]

HIT kit 4.7 EPA was released on 27/02:

https://eqlsupport.dell.com/support/dow ... px?id=2425

From the release notes:

Miscellaneous

• The EqualLogic VSS Hardware Provider now properly deletes snapshots created by third-party backup applications.

I would say that it's an EPA so probably best not to use in production, but given the recent quality in SAN firmware and HIT kit, it's perhaps no more risky than a GA release.

[matt306]

We've just tested a single VM backup using hit 4.7 EPA with the SAN on 7.0.1 and it's removed the snapshot \o/

[kennocha]

We've just tested a single VM backup using hit 4.7 EPA with the SAN on 7.0.1 and it's removed the snapshot \o/

Dell advised me the other day to roll to 4.7 EPA because of this. It is fully supported by Dell and the guy assured me 70 different ways it would work fine.

We shall see in tonight's job!

[matt306]

We've been running 4.7 EPA for just under a week now on our production environment (as it is fully supported by Dell).

Pleased to say that snapshots are removed and the occasional blip where it wouldn't online a snapshot is also gone.

Result!

[foggy]

Thanks for taking your time to test and update us on the results.

[bullerj]

I know this has already been solved, but wanted to share a solution I use after each job. It helps keep the snapshots cleaned up. I create this PS script and run it after the backup job finishes.(Post job activity)

Code: Select all

powershell.exe -Noninteractive -File "C:\PS Scripts\Delete_Snapshots.ps1"
****Script***
$date = Get-Date -format M.d.yyyy
$file = "c:\PS Scripts\logs\DeleteSnap_$date`.txt"
$GrpAddr = "192.168.1.1"
$VolNames = @("HyperVCluster-Vol1", "HyperVCluster-Vol2", "HyperVCluster-Vol3", "HyperVCluster-Vol4", "HyperVCluster-Vol5", "HyperVCluster-Vol6")

Import-Module "c:\program files\EqualLogic\bin\EqlPSTools.dll"

Connect-EqlGroup -GroupAddress $GrpAddr -UserName "Skippy" -Password "MyPassword" -ignoreSavedCredentials | out-file $file

foreach ($VolName in $VolNames) {
	Get-EqlSnapshot -GroupAddress $GrpAddr -VolumeName $VolName |Sort-Object CreationTimeStamp -descending | Select-Object SnapshotName |Set-EqlSnapshot -VolumeName $VolName -OnlineStatus offline| out-file -append $file
	Remove-EqlSnapshot -VolumeName $VolName | out-file -append $file
	}
****Script***

[houstek]

Correction :

Code: Select all

$date = Get-Date -format M.d.yyyy
$file = "c:\jobs\log\DeleteSnap_$date`.txt"
$GrpAddr = "192.168.1.xxx"
$VolNames = @("Lun-Prod-1", "Lun-Prod-2")

Import-Module "c:\program files\EqualLogic\bin\EqlPSTools.dll"

Connect-EqlGroup -GroupAddress $GrpAddr -UserName "toto" -Password "mypass" -ignoreSavedCredentials | out-file $file
foreach ($VolName in $VolNames) {
	Write-host $VolName
   Get-EqlSnapshot -GroupAddress $GrpAddr -VolumeName $VolName |Sort-Object CreationTimeStamp -descending | Select-Object SnapshotName |Set-EqlSnapshot -GroupAddress $GrpAddr -VolumeName $VolName -OnlineStatus offline| out-file -append $file
   Remove-EqlSnapshot -GroupAddress $GrpAddr -VolumeName $VolName | out-file -append $file
   }