VEEAM Snapshot Hunter does not delete VEEAM snapshots

[ITP-Stan]

On a client server we had an issue with a VM crashing because the vmware datastore ran out of space. The VM had a "VEEAM TEMPORARY SNAPSHOT" on that grew to about 400GB before the datastore ran out of space.
This server sits offsite and has no Vcenter or other means of alerting yet.

Since this was VEEAM v8, I was stumped as to why the Snapshot Hunter feature did not work.
I had VEEAM support investigate the matter, only to get an answer stating that:

Snapshot Hunter isn't trying to consolidate VMs with snapshots titled "VEEAM BACKUP TEMPORARY SNAPSHOT".

So the Snapshot Hunter feature does not delete snapshot leftover by VEEAM itself, for instance when VEEAM was shutdown unexpectedly during a backup.
Is it just me or is that something that should be done, otherwise the snapshot hunter feature is pretty useless as it only solves human errors now.


This snapshot was left running on the VM for 3 months, without any warnings in the VEEAM logs.

support ID: 00980091

[cby]

Yikes! 3 month old snapshot.

I'm not familiar with Snapshot Hunter...

We have encountered a number of issues (with/without Veeam) relating to snapshots over the years. So our daily checks include running RVTOOLS to report on VCenter snapshots, running SQL scripts to report/delete on references to rogue snapshots in the Veeam database and a Powershell script to report direct on the presence of snapshots on the ESX hosts via VCenter. Orphaned/locked snapshots require different approaches, e.g. creating a tmp snap and deleting all snapshots, rebooting ESX host (not ideal in a production environment), and even cloning the VM -- answer to a lot of VM ills but a pain.

The odd snap sneaks through but not for long!

Bottom line, we don't rely on a single tool.

[Vitaliy S.]

Hi Stan,

Snapshot Hunter isn't trying to consolidate VMs with snapshots titled "VEEAM BACKUP TEMPORARY SNAPSHOT".

This statement is not fully correct. Snapshot Hunter does track all snapshots created on the VM, please take a look at this web page for more info on how it works > How Snapshot Hunter Works.

As regards why snapshot has not been deleted by this technology, then this should be investigated via our support team. Most likely there was a timeout for executing snapshot consolidation command to vCenter Server.

On a side note, if you're using Veeam ONE, then we have multiple alarms such as > orphaned VM snapshot, snapshot size and snapshot age, these alarms are enabled by default and allow you to proactively monitor your infrastructure and avoid situations like that. Moreover, Veeam ONE has buil-in reports for all active and orphaned VM snapshots as well.

Thanks!

[ITP-Stan]

Hi Vitaliy,

The quote comes from a VEEAM support engineer. Perhaps I misunderstood, but then he should express himself more clearly.

The fact remains that the snapshot, made by VEEAM B&R v8, was left hanging for 3 months, while the version used supposedly is safe from this happening with the Snapshot Hunter feature.

[Gostev]

Vitaly is wrong - the response from support is totally correct. Snapshot Hunter is designed strictly for hidden and stuck snapshots (those that cannot be seen in vSphere client at all, or those vSphere is aware of resulting in the corresponding VM having "Consolidation needed" status).

In this situation, we are clearly not looking at stuck or hidden snapshot, as OP can see the actual snapshot in vSphere. These "normal" snapshots are managed solely by Veeam jobs. Each job tracks the snapshot it creates in the configuration database, and each time the job starts the new or retry run - it will first delete any snapshots that it was not able to delete during any previous runs (normally, this only happens if the server crashes and the job simply never gets to that point of its execution when snapshots are normally deleted).

What needs to be investigated here by our support is why the job itself did not clean up this snapshot during all the job runs that took place in 3 months. For example, this snapshot might have been created by another job that got deleted before it got a chance to delete this snapshot, etc.

[Vitaliy S.]

According to your support engineer this VM has been renamed or some other actions have been performed to this VM and caused this behavior. Keep investigating it with our support team and take a look at Veeam ONE to avoid these issues in future.

[ITP-Stan]

The support engineer did not say anything about a rename or other changes on the VM.

Now there is a senior support engineer assigned. From what he says I understand that if VEAAM B&R did not remove the snapshot on the 15th of april 2015 for some reason, it will remain there and cosecutive job runs will not try to remove this snapshot.

I might misunderstand them again, but that's what I read in their replies.

I will supply all logs they request.

[Vitaliy S.]

Well... temporary VM snapshots should be removed by a subsequent job run.

I have just checked your case, and our support engineer stated that snapshot hunter will not try to remove this snapshot, however the next job run should do it. Currently it is not clear why this snapshot has not been removed, as logs do not contain information about that date and other job runs don't have it either. Let's check if new logs package has this data.

While we are waiting, I would recommend to remove it manually, so that it doesn't grow much and doesn't affect your production, then install free version of Veeam ONE to monitor VM snapshots in your infrastructure.

If you have the same situation in future, open a support case and post your case ID here.

[ITP-Stan]

The VEEAM logs from requested date have been uploaded a couple of hours ago. We will see what the engineer says.

[ITP-Stan]

The engineer has answered me hat because of a connection issue on April 15th the snapshot was not removed.

- Removing VM snapshot Details: An existing connection was forcibly closed by the remote host

Judging by the report, the issue has not returned

My suggestions for the Monitoring Products still stand

Gostev states:

it will first delete any snapshots that it was not able to delete during any previous runs

It seems this does not work as stated, since that snapshot removal had an error (see quote above) but the next run did not try to remove it again.

Will someone please make it clear for me?

[Gostev]

It seems this does not work as stated, since that snapshot removal had an error (see quote above) but the next run did not try to remove it again.

This would be a bug then, because this is a very old functionality from a few versions ago. However, I would expect this bug to generate a huge amount of issues in the field, which we are not seeing - so most likely, there are some other unique environmental dependencies here which cause it to trigger. Regardless, please request a fix for this, so that it gets escalated into R&D for deeper analysis of why the following job runs did not attempt to delete this snapshot.

[ITP-Stan]

It's easy to say that no-one else has had this problem, maybe they just didn't report it. Bigger companies will monitor the snapshot usage and will intervene when one gets stuck, I doubt they will open a case just for that.

Today I check VEEAM one free that I'm testing out for a customer, and what do I see? VM's with a VEEAM temporary snapshot still on (about 20GB in size), and yet the backup job was succesful last night.

So totally different environment, totally different customer, but same problem ... not so unique anymore it seems. This site has VEEAM 8.0.0.917

[Vitaliy S.]

Hi Stan, that's definitely unexpected, all Veeam temporary snapshots should be removed by a subsequent job run. Can you tell us your case ID when you open a ticket with our technical team? Meanwhile, please use Veeam ONE free to monitor VM snapshots.

[ITP-Stan]

As told support ID is 00980091 for the customer that had critical VM down issue because of a 3 month old snapshot.
I have not received any satisfactory response, but have today (as requested by engineer a couple of days ago) uploaded the entire Logs folder for the backup job from c:\programdata.
We will see if they find anything now, but I'm not hopefull.

The other customer I have removed the snapshots manually and will not waste my time by opening another case.

[Gostev]

It's easy to say that no-one else has had this problem, maybe they just didn't report it.

Extremely unlikely - as when this issue surfaced once a few years ago due to another fix, we had dozens of support cases and forum posts created very quickly. And this was back when we had 5 times less users than we do today. So, I suspect something environment-specific in this case.

It's actually a very simple 5 minutes long test to verify our left over snapshot management functionality: just kill the backup server when it is performing a backup, wait until it reboots and start the same job again - VM snapshot from the previous should be removed before the new one is created. Have you tried it?

This site has VEEAM 8.0.0.917

Wow, that's an old one. Please make sure they update.

[ITP-Stan]

I don't like killing B&R during a backup on production machines.

So it's environment specific, but I already have it on 2 seperate environments.
I must admit that for the 2nd environment the snapshots probably were not older than 1 or 2 days, so maybe VEEAM B&R would have eventually cleaned up after itself. But I don't want to take any risk.

Wow, that's an old one. Please make sure they update.

Old? It's Veeam B&R 8.0 Patch1, the only newer version is Patch2b which came out 2 months ago. Will update asap though.

[Gostev]

You're right, I confused 917 with 817 (GA version)

[ITP-Stan]

The escalation engineer has escalated the case to the next tier How many levels can I escalate ...

[foggy]

Until it eventually reaches R&D for a fix (typically, tier 3).

[ITP-Stan]

Let's hope so, but at this stage I'm not so confident as you are.

[ITP-Stan]

The case will be closing with a response that in future versions they will fix this:

After the long conversation with our developers we agreed that snapshot removal operations should not have "expiration time" and need to be carried in a special way in case of failures.
This behaviour is going to be changed in the future releases.

But no answer what future version.

[Vitaliy S.]

Based on the answer it seems like snapshot removal operation times out in the infrastructure you've deployed Veeam B&R to? Did I get it right? Do you see any timed out tasks in the vCenter Server?

[Kraken]

We ran into the same issue - temporary Veeam snapshot were not removed after some kind of job fails and the space ran out of space.

We use the v8.0.0.2084 Veeam B&R - according to http://www.veeam.com/kb1790 temporary snapshots should be removed in next job run:

In version 6.5.0.128 and later, Veeam Backup & Replication tracks all snapshots it creates and tries to remove those snapshots on later job runs.

Well, not really or not in all cases. I did some research: jobs fail from time to time, but not always leave the temporary snapshot behind. I found temporary snapshots that are not deleted, appearing when the job fails with the following error:

Error: An error occurred while communicating with the remote host.

Even if in retry the job pass, the temporary snapshot is left behind and is NOT deleted by next job run (i found these snapshots that were old a few months and the jobs run every day). So if you have a mail notification sent in case of fail just after last job retry, this might be a ticking bomb in your environment.

[veremin]

Did you open a case with our support team about the experienced behaviour? Thanks.

[Kraken]

Not yet - I found out yesterday. First, I wanted to ask here if it's not already known or by design in this case.

But will do today.

[Gostev]

No, definitely not by design or known issue.

[ITP-Stan]

Yes this is a known issue, contrary to what Veeam representatives say. I had a case open for weeks and the responses you can read a few posts up.

[Gostev]

Stan, YOUR issue above is indeed known (and even fixed already in v9). However, the error Jaroslav is seeing does not match the description of the bug report connected to your support case. Both your issue, and the internally reproduced one came with a different error in the job log. So please, don't discourage him from opening a support case with your comments, to let us investigate his issue properly.

I am suspecting a different issue here because, at least according to the internal case notes, in your case vCenter Server went down during backup, and remained offline for an extended time after that, thus preventing us from removing the snapshot. However, above Jaroslav clearly states that job retry cycle completed successfully, which is obviously impossible when vCenter Server remains down.

Thanks!

[ccatlett1984]

Below is a script I run daily that emails out a report of vm's with open snapshots.

Code: Select all

#Load VMWare snap-ins
If (-not (Get-PSSnapin VMware.VimAutomation.Core))
{  Try { Add-PSSnapin VMware.VimAutomation.Core -ErrorAction Stop }
   Catch { Write-Host "Unable to load PowerCLI, is it installed?" -ForegroundColor Red; Break }
}

#Set your variables
$host = "host1"
$user = "readonly"
$password = "password"
$SMTPserver = "emailserver"
$SMTP_Send_to = "monitor@domain.com"
$SMTP_Send_from = "vmware.reports@domain.com"
$VMList = "C:\windows\temp\vm_open_snapshots.csv"

#Change nothing below this line.
$hostname = $env:COMPUTERNAME
#Connect to esxi
connect-viserver –server $host -User $user -Password $password -WarningAction SilentlyContinue
#pull vm's with open snapshots into a list, exclude vm's we dont care about.
$snapshotted_vm = Get-VM | Sort Name | Get-Snapshot | Where { $_.Name.Length -gt 0 -AND $_.Name -ne "XENAPPTEST"} | Select VM,Name,Created,SizeMB | Sort-Object SizeMB –Descending | export-csv -notypeinformation $VMList
#email a list of the vm's with snapshots.
IF($snapshotted_vm.count -ge 1)
{
send-mailmessage -SmtpServer $SMTPserver -to $SMTP_Send_to -from $SMTP_Send_from -attachments "$VMList" -subject "VMWare Report: VM's with open snapshots $date" -body "This email was generated by a powershell script running on $hostname";
}
Disconnect-VIServer * -Confirm:$false

[Gostev]

Hi, Chris - thanks for sharing! Please note that this script will not report open stuck or hidden snapshots though, as those can only be detected by analyzing VM files present on the datastore, which is exactly what Snapshot Hunter is doing.