Host-based backup of Microsoft Hyper-V VMs.
Post Reply
stevedarl1
Influencer
Posts: 14
Liked: 1 time
Joined: Apr 19, 2013 1:22 pm
Full Name: Steve Darlington
Contact:

Equallogic On Latest Firmware-Volume Snapshots Not Deleting

Post by stevedarl1 »

We have just upgraded to the latest Dell Equallogic firmware V7.0.1 on our iSCSI SANs and now when a backup runs it snapshots the volumes fine but does not remove them after completion. Anyone else had similar issues after upgrading the firmware? It means multiple snapshots are being created for VMs that are backing up from the same Volume. It is a manual process at the moment to remove the snapshots every day.

Environment:

Hyper-V 2012 and Hyper-V 2012 R2 cluster
Veeam V7 R2
Equallogic iSCSI SANs running V7.0.1
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by veremin »

Hi, Steve,

Have you already contacted support team with this issue? They will be able to determine whether this behaviour is caused by the latest firmware and not by other environmental factors/changes that passed unnoticed.

Thanks.
stevedarl1
Influencer
Posts: 14
Liked: 1 time
Joined: Apr 19, 2013 1:22 pm
Full Name: Steve Darlington
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by stevedarl1 »

Hi,

Yes I have, the Veeam logs show the commands being sent to create/delete the snapshot. I believe this to be an Equallogic issue, however I was just seeing if anyone else has had similar issues.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by foggy »

I believe no one has reported similar issues here so far, so it is worth contacting Dell support for clarification. Thanks.
stevedarl1
Influencer
Posts: 14
Liked: 1 time
Joined: Apr 19, 2013 1:22 pm
Full Name: Steve Darlington
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by stevedarl1 »

Hi Foggy, do you know the command or syntax that is used to take/remove a snapshot? We use on host backups with Dell Auto Snapshot Manager to take Hardware VSS snapshots.

Thanks.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by foggy »

Steve, the snapshot is created with the IVssBackupComponents::DoSnapshotSet method and should be automatically deleted on releasing the IVssBackupComponents object.
matt306
Influencer
Posts: 19
Liked: 11 times
Joined: Jan 06, 2014 2:29 pm
Full Name: Matthew Brown
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by matt306 » 2 people like this post

I work with Steve and we've had a reply from Equallogic today. It transpires they have missed something in VSS support, hence why the snapshots aren't being deleted. They hinted that we were not the only customers affected and that a patch "may" be available soon, although it wasn't confirmed if this would be an official release or an early deployment/availability patch.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by foggy »

Thanks for these news, Matt! Much appreciated.
Argos
Lurker
Posts: 1
Liked: never
Joined: Feb 03, 2014 8:43 pm
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by Argos »

I am seeing the same issue with firmware v7.01 on Equallogic. Snapshots not being automatically deleted.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by foggy »

I believe you could also contact Dell support to be notified once the above mentioned fix is available. Thanks.
stevedarl1
Influencer
Posts: 14
Liked: 1 time
Joined: Apr 19, 2013 1:22 pm
Full Name: Steve Darlington
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by stevedarl1 »

Not only are the snapshots not deleting automatically still, but we are now seeing snapshots failing to online once created within Equallogic group manager. If anyone sees this issue in Veeam the job will just sit there until you manually online the snapshot within group manager, it then resumes the backup.

Dell are releasing the new V7.0.2 firmware within the next week or 2 and they have confirmed the code has returned. However HIT kit 4.7 will not be released until March sometime and apparently this issue is related to HIT kit 4.6 on the hosts which is where the VSS credentials are stored. The new HIT kit will also be 2012 R2 compatible however we ran V4.6 for 3 months on 2012 R2 hosts using the old Equallogic firmware without issues.
bcscholl
Enthusiast
Posts: 31
Liked: 3 times
Joined: Dec 04, 2013 3:56 pm
Full Name: Brian Scholl
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by bcscholl »

Any updates on this?

I've set my Snapshot Reserve down to 10%, turned off borrowing. This seems to work, just deletes the oldest snapshot when it needs more space.
matt306
Influencer
Posts: 19
Liked: 11 times
Joined: Jan 06, 2014 2:29 pm
Full Name: Matthew Brown
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by matt306 »

No news yet from Dell, still the same 7.0.2 is due in the next couple of weeks and HIT 4.7 by March.

Originally we did the same with limiting the snapshots, but had a couple of instances of losing iSCSI connections to volumes once the snapshot ran out of space. Since then we've sent a load of logs to Dell who found a controller panic on one of the SAN's so it's possible it was this not the snapshot that caused our issues.

As we can't be sure we're still deleting shortly after they're used, although recently we've got braver and allowed a whole job to complete before removing them.
bcscholl
Enthusiast
Posts: 31
Liked: 3 times
Joined: Dec 04, 2013 3:56 pm
Full Name: Brian Scholl
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by bcscholl »

I’ve seen a few MPIO and disk errors on one of my Hyper-V nodes as the snapshot is started. During that time I cannot browse to C:\ClusterVolumes folder. All the VMs stay running and still look to write data to the disk. No Cluster events are recorded, no CSV ownership is changed. My worry is that at some point it will break something. Holding my breath and hoping 4.7 HIT kit will fix the MPIO errors.
matt306
Influencer
Posts: 19
Liked: 11 times
Joined: Jan 06, 2014 2:29 pm
Full Name: Matthew Brown
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by matt306 »

7.0.2 is now available, which allegedly adds the VSS commands back in (although nothing is mentioned in the release notes). However it will not fix the deleting snapshots until HIT 4.7 is released.
estark
Lurker
Posts: 2
Liked: never
Joined: Feb 25, 2013 9:08 pm
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by estark »

It seems like this is a problem with Dell Equallogic HIT Kit 4.6. After working for a day with Dell, it seems like they have two documented bugs regarding the removal of snapshots with HIT Kit 4.6 which is required for PS Firmware 7.0.1. There is a problem with VEEAM and DPM (and I think others) where the programs are unable to set snapshots offline and delete them. There first documented this at the end of January and are planning to address this in the next HIT Kit scheduled to be released on February 28. Bugzilla has this bug reported as #60369 according to Dell.
matt306
Influencer
Posts: 19
Liked: 11 times
Joined: Jan 06, 2014 2:29 pm
Full Name: Matthew Brown
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by matt306 »

Even though Dell are saying it's HIT kit related I think it's only half the issue. When we were running 6.0.4 with HIT 4.6 on a 2012R2 Cluster it didn't have any problems removing the snapshots. The issue only presented itself once 7.0.1 was installed. I think they've added some of the code back in for 7.0.2 but are relying on the HIT to provide the correct syntax to the SAN, which is why we have to wait for 4.7
ryanpsulion
Lurker
Posts: 1
Liked: never
Joined: Feb 18, 2014 2:39 pm
Full Name: Ryan Levins
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by ryanpsulion »

So glad we went ahead and updated all of are arrays to 7.0.1 and HIT 4.6 two weekends ago in anticipation for Veeam deployment. To only learn we now have hundreds of online snapshots that are not being trimmed at all. Superb!
matt306
Influencer
Posts: 19
Liked: 11 times
Joined: Jan 06, 2014 2:29 pm
Full Name: Matthew Brown
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by matt306 »

Ryan - Other people are leaving their snapshots and configuring the Equallogic to remove the oldest one once the volume runs out of snapshot reserve. We are deleting ours manually as we had a number of CSV pauses/disconnects once the snapshot reserve was consumed. However we were still getting pause/disconnects outside of the backup window, just not as frequently (or cluster breaking). We turned ODX off and since doing so have not had any further CSV problems. I suspect that if I left the snapshots we wouldn't have a problem, but it's not the sort of environment to be testing that hypothesis!
matt306
Influencer
Posts: 19
Liked: 11 times
Joined: Jan 06, 2014 2:29 pm
Full Name: Matthew Brown
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by matt306 »

From a Dell Employee:
http://en.community.dell.com/techcenter ... 43118.aspx
4.7 is due out very soon. That should help.

There was a change made to 7.0.x firmware based on customer feedback. If a snapshot is online or online with connections then don't just delete it. There were cases when customers were running from a snapshot and a schedule max keep or to reclaim snapshot reserve would delete that active snapshot. When VSS goes to delete the snapshot, it's not putting it offline first, preventing the snapshot from being deleted.
Looking at the thread it also affects MS DPM users. It probably affects Symantec V-Ray customers too (if you can find any).
matt306
Influencer
Posts: 19
Liked: 11 times
Joined: Jan 06, 2014 2:29 pm
Full Name: Matthew Brown
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by matt306 »

HIT kit 4.7 EPA was released on 27/02:

https://eqlsupport.dell.com/support/dow ... px?id=2425

From the release notes:
Miscellaneous

• The EqualLogic VSS Hardware Provider now properly deletes snapshots created by third-party backup applications.

I would say that it's an EPA so probably best not to use in production, but given the recent quality in SAN firmware and HIT kit, it's perhaps no more risky than a GA release.
matt306
Influencer
Posts: 19
Liked: 11 times
Joined: Jan 06, 2014 2:29 pm
Full Name: Matthew Brown
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by matt306 »

We've just tested a single VM backup using hit 4.7 EPA with the SAN on 7.0.1 and it's removed the snapshot \o/
kennocha
Enthusiast
Posts: 25
Liked: 1 time
Joined: Aug 15, 2013 8:57 pm
Full Name: Stephen Carter
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by kennocha »

matt306 wrote:We've just tested a single VM backup using hit 4.7 EPA with the SAN on 7.0.1 and it's removed the snapshot \o/
Dell advised me the other day to roll to 4.7 EPA because of this. It is fully supported by Dell and the guy assured me 70 different ways it would work fine.

We shall see in tonight's job!
matt306
Influencer
Posts: 19
Liked: 11 times
Joined: Jan 06, 2014 2:29 pm
Full Name: Matthew Brown
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by matt306 » 1 person likes this post

We've been running 4.7 EPA for just under a week now on our production environment (as it is fully supported by Dell).

Pleased to say that snapshots are removed and the occasional blip where it wouldn't online a snapshot is also gone.

Result!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by foggy »

Thanks for taking your time to test and update us on the results.
bullerj
Lurker
Posts: 1
Liked: 1 time
Joined: Sep 23, 2014 2:19 pm
Full Name: Jordan Buller
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by bullerj » 1 person likes this post

I know this has already been solved, but wanted to share a solution I use after each job. It helps keep the snapshots cleaned up. I create this PS script and run it after the backup job finishes.(Post job activity)

Code: Select all

powershell.exe -Noninteractive -File "C:\PS Scripts\Delete_Snapshots.ps1"
****Script***
$date = Get-Date -format M.d.yyyy
$file = "c:\PS Scripts\logs\DeleteSnap_$date`.txt"
$GrpAddr = "192.168.1.1"
$VolNames = @("HyperVCluster-Vol1", "HyperVCluster-Vol2", "HyperVCluster-Vol3", "HyperVCluster-Vol4", "HyperVCluster-Vol5", "HyperVCluster-Vol6")

Import-Module "c:\program files\EqualLogic\bin\EqlPSTools.dll"

Connect-EqlGroup -GroupAddress $GrpAddr -UserName "Skippy" -Password "MyPassword" -ignoreSavedCredentials | out-file $file

foreach ($VolName in $VolNames) {
	Get-EqlSnapshot -GroupAddress $GrpAddr -VolumeName $VolName |Sort-Object CreationTimeStamp -descending | Select-Object SnapshotName |Set-EqlSnapshot -VolumeName $VolName -OnlineStatus offline| out-file -append $file
	Remove-EqlSnapshot -VolumeName $VolName | out-file -append $file
	}
****Script***
houstek
Novice
Posts: 5
Liked: 1 time
Joined: Dec 31, 2014 5:52 am
Contact:

Re: Equallogic On Latest Firmware-Volume Snapshots Not Delet

Post by houstek » 1 person likes this post

Correction :

Code: Select all

$date = Get-Date -format M.d.yyyy
$file = "c:\jobs\log\DeleteSnap_$date`.txt"
$GrpAddr = "192.168.1.xxx"
$VolNames = @("Lun-Prod-1", "Lun-Prod-2")

Import-Module "c:\program files\EqualLogic\bin\EqlPSTools.dll"

Connect-EqlGroup -GroupAddress $GrpAddr -UserName "toto" -Password "mypass" -ignoreSavedCredentials | out-file $file
foreach ($VolName in $VolNames) {
	Write-host $VolName
   Get-EqlSnapshot -GroupAddress $GrpAddr -VolumeName $VolName |Sort-Object CreationTimeStamp -descending | Select-Object SnapshotName |Set-EqlSnapshot -GroupAddress $GrpAddr -VolumeName $VolName -OnlineStatus offline| out-file -append $file
   Remove-EqlSnapshot -GroupAddress $GrpAddr -VolumeName $VolName | out-file -append $file
   }
 
Post Reply

Who is online

Users browsing this forum: No registered users and 18 guests