VEEAM BACKUP TEMPORARY SNAPSHOT problem

[cparker4486]

Hello Gostev and team,

I just had a VM go offline because a temporary Veeam snapshot, created a few days ago, is now reported as being 656 GB in size. The server is now offline. I've been able to manually remove the snapshot but I have to say how disappointed I am that this is even possible. THIS SHOULD NEVER EVER HAPPEN.

Why doesn't B&R remove these snapshots itself when found? You've GOT to improve the aspect of B&R. It's hit me in the past and I've seen it hit other customers too.

This is not a fun day now all because a TEMPORARY snapshot has killed my server.

[Gostev]

Why doesn't B&R remove these snapshots itself when found?

Actually it does, so we need to review your debug logs to see why it did not.
For example, we can't remove a snapshot when vCenter is not responding

[cparker4486]

In that case I'll open a ticket but here's the problem: My backup logs report successful backups. Jobs should not be successful if there are unaccounted for snapshots. If an existing temp snapshot cannot be removed the job log should indicate so with a warning. Also, I find it hard to believe that just by chance, every time B&R went to remove this specific snapshot, vCenter was not responding yet worked perfectly at all other moments.

In any case, I'm sure you would agree this aspect of B&R needs to be more robust (because I know you want your software to work flawlessly).

[Gostev]

Here's the problem: My backup logs report successful backups. Jobs should not be successful if there are unaccounted for snapshots. If an existing temp snapshot cannot be removed the job log should indicate so with a warning.

Unless vCenter Server tells us that the snapshot was successfully removed, when in fact it was not... which unfortunately happens quite often.

Also, I find it hard to believe that just by chance, every time B&R went to remove this specific snapshot, vCenter was not responding yet worked perfectly at all other moments.

How do you know it worked perfectly at all other moments? Thing is, we had two cases in support with this behavior confirmed in the past few weeks alone.

In any case, I'm sure you would agree this aspect of B&R needs to be more robust (because I know you want your software to work flawlessly).

Absolutely, which is why we want to investigate every occurrence of stuck snapshots.

[cparker4486]

Unless vCenter server tells us that the snapshot was in fact successfully removed, when in fact it was not... which unfortunately happens quite often.

I don't understand this but I also don't know the inner workings so please explain. Can B&R not check for the existence of snapshots before it begins a new run? If it finds a snapshot called VEEAM BACKUP TEMPORARY SNAPSHOT doesn't that indicate a previous run's snapshot was not removed and might need the admin's intervention? "Hey, buddy. I found a lingering temp snapshot. You should try removing it manually. This is your first warning."

How do you know it worked perfectly at all other moments? Thing is, we had two cases in support with this behavior confirmed in the past few weeks alone.

That's easy. Because the job log shows me all green check marks and no warnings indicating anything went wrong.

[Gostev]

Can B&R not check for the existence of snapshots before it begins a new run? If it finds a snapshot called VEEAM BACKUP TEMPORARY SNAPSHOT doesn't that indicate a previous run's snapshot was not removed

It does so, and for many versions now. Except we do not match them by name as you are suggesting, but rather by unique ID. Actually, we used to match them by name in the early days - but this proven to be an unreliable approach due to users renaming open snapshots, or concurrent jobs processing the same VM deleting each other's snapshots. Anyway, every time the job starts, we do attempt to delete any snapshots that we created, but failed to delete before (they are tracked in the dedicated table).

Now, there are some known issues around snapshot handling when vCenter server goes unresponsive during snapshot deletion, and we saw this a few times recently, so I suggested this may be your case as well. It will be apparent as soon as our engineers are able to review your debug logs, because they document each occurence of vCenter becoming unresponsive.

That's easy. Because the job log shows me all green check marks and no warnings indicating anything went wrong.

Well, as I already explained above it's not uncommon that vCenter Server returns invalid information to us, and this is what we show you in the job log.

For some reason, many users (not just you) tend to treat Veeam B&R as a thing in itself - which is not the case as we rely on the hypervisor in many things. And unfortunately, majority of issues with Veeam are found to be caused by problems elsewhere. This is why monitoring is so critically important. In fact, we do provide a number of snapshot alarms, monitors and reports in Veeam ONE for this reason specifically. So, I highly recommend that you deploy some sort of snapshot monitoring solution in place (can be as easy as a scheduled PowerCLI script even). Otherwise, there is no guarantee you will not run into the same "production down" situation again due to some new VMware or Veeam bug around snapshot handling.

Rest assured however, we are trying to cover all the bases we can from our side around stuck snapshots. In fact, this resulted into the whole new framework added into v8 (aka Snapshot Hunter) that automatically detects and handles hidden snapshots. Unfortunately, judging on your topic name, it clearly does not apply to your specific case, since you can still see Veeam snapshot in the Snapshot Manager. Which makes me think your issue is similar to those two issues I referenced earlier.

Please post the support case ID here once you had a chance to open it.

[cparker4486]

Thank you, Gostev. The detailed response is much appreciated!

Veeam thinks my support contract has expired (but I think I know the cause) so now I'm waiting for sales to call me back to resolve the issue.

[Gostev]

Sounds good. If you are running into the issue I am thinking, then it is addressed in v9.
However, just to be sure, it would be best for us to confirm that through your debug logs.

[Vitaliy S.]

On a side note, even Veeam ONE Free edition allows snapshot monitoring and reporting, so you can give it a try.

[jftuga]

You can create an alarm in vCenter that will send an email if a snapshot exceeds a certain size.

Under 5.1:
Go to Home -> Inventory -> Datastores and Datastore Clusters
Alarms tab
View: Defintion
Right Click to create New Alarm

General
----------
Alarm Name: VM with snapshots
Monitor: Virtual Machines, Monitor for specific conditions

Triggers
----------
VM Snapshot Size(GB)
Condition: is above
Warning: 10
Alert:20
Trigger if any of the conditions are satisfied

Actions
---------
Send a notification email

[Vitaliy S.]

Alarms are good indeed, though I would also recommend to schedule VMs with active snapshots report (jut to be on the safe side).

[ccatlett1984]

Look at the script I posted previously:
http://forums.veeam.com/veeam-backup-re ... ml#p168453