Host-based backup of VMware vSphere VMs.
Post Reply
cparker4486
Expert
Posts: 231
Liked: 18 times
Joined: Dec 07, 2009 5:09 pm
Full Name: Chris
Contact:

VEEAM BACKUP TEMPORARY SNAPSHOT problem

Post by cparker4486 »

Hello Gostev and team,

I just had a VM go offline because a temporary Veeam snapshot, created a few days ago, is now reported as being 656 GB in size. The server is now offline. I've been able to manually remove the snapshot but I have to say how disappointed I am that this is even possible. THIS SHOULD NEVER EVER HAPPEN.

Why doesn't B&R remove these snapshots itself when found? You've GOT to improve the aspect of B&R. It's hit me in the past and I've seen it hit other customers too.

This is not a fun day now all because a TEMPORARY snapshot has killed my server.
-- Chris
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Post by Gostev »

cparker4486 wrote:Why doesn't B&R remove these snapshots itself when found?
Actually it does, so we need to review your debug logs to see why it did not.
For example, we can't remove a snapshot when vCenter is not responding :)
cparker4486
Expert
Posts: 231
Liked: 18 times
Joined: Dec 07, 2009 5:09 pm
Full Name: Chris
Contact:

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Post by cparker4486 »

In that case I'll open a ticket but here's the problem: My backup logs report successful backups. Jobs should not be successful if there are unaccounted for snapshots. If an existing temp snapshot cannot be removed the job log should indicate so with a warning. Also, I find it hard to believe that just by chance, every time B&R went to remove this specific snapshot, vCenter was not responding yet worked perfectly at all other moments.

In any case, I'm sure you would agree this aspect of B&R needs to be more robust (because I know you want your software to work flawlessly). :D
-- Chris
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Post by Gostev »

cparker4486 wrote:Here's the problem: My backup logs report successful backups. Jobs should not be successful if there are unaccounted for snapshots. If an existing temp snapshot cannot be removed the job log should indicate so with a warning.
Unless vCenter Server tells us that the snapshot was successfully removed, when in fact it was not... which unfortunately happens quite often.
cparker4486 wrote:Also, I find it hard to believe that just by chance, every time B&R went to remove this specific snapshot, vCenter was not responding yet worked perfectly at all other moments.
How do you know it worked perfectly at all other moments? Thing is, we had two cases in support with this behavior confirmed in the past few weeks alone.
cparker4486 wrote:In any case, I'm sure you would agree this aspect of B&R needs to be more robust (because I know you want your software to work flawlessly). :D
Absolutely, which is why we want to investigate every occurrence of stuck snapshots.
cparker4486
Expert
Posts: 231
Liked: 18 times
Joined: Dec 07, 2009 5:09 pm
Full Name: Chris
Contact:

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Post by cparker4486 »

Gostev wrote:Unless vCenter server tells us that the snapshot was in fact successfully removed, when in fact it was not... which unfortunately happens quite often.
I don't understand this but I also don't know the inner workings so please explain. Can B&R not check for the existence of snapshots before it begins a new run? If it finds a snapshot called VEEAM BACKUP TEMPORARY SNAPSHOT doesn't that indicate a previous run's snapshot was not removed and might need the admin's intervention? "Hey, buddy. I found a lingering temp snapshot. You should try removing it manually. This is your first warning."
Gostev wrote:How do you know it worked perfectly at all other moments? Thing is, we had two cases in support with this behavior confirmed in the past few weeks alone.
That's easy. Because the job log shows me all green check marks and no warnings indicating anything went wrong. :D
-- Chris
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Post by Gostev » 2 people like this post

cparker4486 wrote:Can B&R not check for the existence of snapshots before it begins a new run? If it finds a snapshot called VEEAM BACKUP TEMPORARY SNAPSHOT doesn't that indicate a previous run's snapshot was not removed
It does so, and for many versions now. Except we do not match them by name as you are suggesting, but rather by unique ID. Actually, we used to match them by name in the early days - but this proven to be an unreliable approach due to users renaming open snapshots, or concurrent jobs processing the same VM deleting each other's snapshots. Anyway, every time the job starts, we do attempt to delete any snapshots that we created, but failed to delete before (they are tracked in the dedicated table).

Now, there are some known issues around snapshot handling when vCenter server goes unresponsive during snapshot deletion, and we saw this a few times recently, so I suggested this may be your case as well. It will be apparent as soon as our engineers are able to review your debug logs, because they document each occurence of vCenter becoming unresponsive.
Gostev wrote:That's easy. Because the job log shows me all green check marks and no warnings indicating anything went wrong. :D
Well, as I already explained above it's not uncommon that vCenter Server returns invalid information to us, and this is what we show you in the job log.

For some reason, many users (not just you) tend to treat Veeam B&R as a thing in itself - which is not the case as we rely on the hypervisor in many things. And unfortunately, majority of issues with Veeam are found to be caused by problems elsewhere. This is why monitoring is so critically important. In fact, we do provide a number of snapshot alarms, monitors and reports in Veeam ONE for this reason specifically. So, I highly recommend that you deploy some sort of snapshot monitoring solution in place (can be as easy as a scheduled PowerCLI script even). Otherwise, there is no guarantee you will not run into the same "production down" situation again due to some new VMware or Veeam bug around snapshot handling.

Rest assured however, we are trying to cover all the bases we can from our side around stuck snapshots. In fact, this resulted into the whole new framework added into v8 (aka Snapshot Hunter) that automatically detects and handles hidden snapshots. Unfortunately, judging on your topic name, it clearly does not apply to your specific case, since you can still see Veeam snapshot in the Snapshot Manager. Which makes me think your issue is similar to those two issues I referenced earlier.

Please post the support case ID here once you had a chance to open it.
cparker4486
Expert
Posts: 231
Liked: 18 times
Joined: Dec 07, 2009 5:09 pm
Full Name: Chris
Contact:

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Post by cparker4486 »

Thank you, Gostev. The detailed response is much appreciated!

Veeam thinks my support contract has expired (but I think I know the cause) so now I'm waiting for sales to call me back to resolve the issue.
-- Chris
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Post by Gostev »

Sounds good. If you are running into the issue I am thinking, then it is addressed in v9.
However, just to be sure, it would be best for us to confirm that through your debug logs.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Post by Vitaliy S. »

On a side note, even Veeam ONE Free edition allows snapshot monitoring and reporting, so you can give it a try.
jftuga
Enthusiast
Posts: 26
Liked: 2 times
Joined: Oct 22, 2010 7:02 pm
Full Name: John Taylor
Location: Athens, GA
Contact:

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Post by jftuga »

You can create an alarm in vCenter that will send an email if a snapshot exceeds a certain size.

Under 5.1:
Go to Home -> Inventory -> Datastores and Datastore Clusters
Alarms tab
View: Defintion
Right Click to create New Alarm

General
----------
Alarm Name: VM with snapshots
Monitor: Virtual Machines, Monitor for specific conditions

Triggers
----------
VM Snapshot Size(GB)
Condition: is above
Warning: 10
Alert:20
Trigger if any of the conditions are satisfied

Actions
---------
Send a notification email
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Post by Vitaliy S. »

Alarms are good indeed, though I would also recommend to schedule VMs with active snapshots report (jut to be on the safe side).
ccatlett1984
Enthusiast
Posts: 83
Liked: 9 times
Joined: Oct 31, 2013 5:11 pm
Full Name: Chris Catlett
Contact:

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Post by ccatlett1984 »

Look at the script I posted previously:
http://forums.veeam.com/veeam-backup-re ... ml#p168453
Post Reply

Who is online

Users browsing this forum: FrankCl2 and 81 guests