VEEAM BACKUP TEMPORARY SNAPSHOT problem

VMware specific discussions

VEEAM BACKUP TEMPORARY SNAPSHOT problem

Veeam Logoby cparker4486 » Thu Dec 10, 2015 5:47 pm

Hello Gostev and team,

I just had a VM go offline because a temporary Veeam snapshot, created a few days ago, is now reported as being 656 GB in size. The server is now offline. I've been able to manually remove the snapshot but I have to say how disappointed I am that this is even possible. THIS SHOULD NEVER EVER HAPPEN.

Why doesn't B&R remove these snapshots itself when found? You've GOT to improve the aspect of B&R. It's hit me in the past and I've seen it hit other customers too.

This is not a fun day now all because a TEMPORARY snapshot has killed my server.
-- Chris
cparker4486
Expert
 
Posts: 222
Liked: 18 times
Joined: Mon Dec 07, 2009 5:09 pm
Full Name: Chris

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Veeam Logoby Gostev » Thu Dec 10, 2015 8:18 pm

cparker4486 wrote:Why doesn't B&R remove these snapshots itself when found?

Actually it does, so we need to review your debug logs to see why it did not.
For example, we can't remove a snapshot when vCenter is not responding :)
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Veeam Logoby cparker4486 » Thu Dec 10, 2015 9:48 pm

In that case I'll open a ticket but here's the problem: My backup logs report successful backups. Jobs should not be successful if there are unaccounted for snapshots. If an existing temp snapshot cannot be removed the job log should indicate so with a warning. Also, I find it hard to believe that just by chance, every time B&R went to remove this specific snapshot, vCenter was not responding yet worked perfectly at all other moments.

In any case, I'm sure you would agree this aspect of B&R needs to be more robust (because I know you want your software to work flawlessly). :D
-- Chris
cparker4486
Expert
 
Posts: 222
Liked: 18 times
Joined: Mon Dec 07, 2009 5:09 pm
Full Name: Chris

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Veeam Logoby Gostev » Thu Dec 10, 2015 10:02 pm

cparker4486 wrote:Here's the problem: My backup logs report successful backups. Jobs should not be successful if there are unaccounted for snapshots. If an existing temp snapshot cannot be removed the job log should indicate so with a warning.

Unless vCenter Server tells us that the snapshot was successfully removed, when in fact it was not... which unfortunately happens quite often.

cparker4486 wrote:Also, I find it hard to believe that just by chance, every time B&R went to remove this specific snapshot, vCenter was not responding yet worked perfectly at all other moments.

How do you know it worked perfectly at all other moments? Thing is, we had two cases in support with this behavior confirmed in the past few weeks alone.

cparker4486 wrote:In any case, I'm sure you would agree this aspect of B&R needs to be more robust (because I know you want your software to work flawlessly). :D

Absolutely, which is why we want to investigate every occurrence of stuck snapshots.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Veeam Logoby cparker4486 » Thu Dec 10, 2015 10:10 pm

Gostev wrote:Unless vCenter server tells us that the snapshot was in fact successfully removed, when in fact it was not... which unfortunately happens quite often.

I don't understand this but I also don't know the inner workings so please explain. Can B&R not check for the existence of snapshots before it begins a new run? If it finds a snapshot called VEEAM BACKUP TEMPORARY SNAPSHOT doesn't that indicate a previous run's snapshot was not removed and might need the admin's intervention? "Hey, buddy. I found a lingering temp snapshot. You should try removing it manually. This is your first warning."

Gostev wrote:How do you know it worked perfectly at all other moments? Thing is, we had two cases in support with this behavior confirmed in the past few weeks alone.

That's easy. Because the job log shows me all green check marks and no warnings indicating anything went wrong. :D
-- Chris
cparker4486
Expert
 
Posts: 222
Liked: 18 times
Joined: Mon Dec 07, 2009 5:09 pm
Full Name: Chris

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Veeam Logoby Gostev » Thu Dec 10, 2015 10:43 pm 2 people like this post

cparker4486 wrote:Can B&R not check for the existence of snapshots before it begins a new run? If it finds a snapshot called VEEAM BACKUP TEMPORARY SNAPSHOT doesn't that indicate a previous run's snapshot was not removed

It does so, and for many versions now. Except we do not match them by name as you are suggesting, but rather by unique ID. Actually, we used to match them by name in the early days - but this proven to be an unreliable approach due to users renaming open snapshots, or concurrent jobs processing the same VM deleting each other's snapshots. Anyway, every time the job starts, we do attempt to delete any snapshots that we created, but failed to delete before (they are tracked in the dedicated table).

Now, there are some known issues around snapshot handling when vCenter server goes unresponsive during snapshot deletion, and we saw this a few times recently, so I suggested this may be your case as well. It will be apparent as soon as our engineers are able to review your debug logs, because they document each occurence of vCenter becoming unresponsive.

Gostev wrote:That's easy. Because the job log shows me all green check marks and no warnings indicating anything went wrong. :D

Well, as I already explained above it's not uncommon that vCenter Server returns invalid information to us, and this is what we show you in the job log.

For some reason, many users (not just you) tend to treat Veeam B&R as a thing in itself - which is not the case as we rely on the hypervisor in many things. And unfortunately, majority of issues with Veeam are found to be caused by problems elsewhere. This is why monitoring is so critically important. In fact, we do provide a number of snapshot alarms, monitors and reports in Veeam ONE for this reason specifically. So, I highly recommend that you deploy some sort of snapshot monitoring solution in place (can be as easy as a scheduled PowerCLI script even). Otherwise, there is no guarantee you will not run into the same "production down" situation again due to some new VMware or Veeam bug around snapshot handling.

Rest assured however, we are trying to cover all the bases we can from our side around stuck snapshots. In fact, this resulted into the whole new framework added into v8 (aka Snapshot Hunter) that automatically detects and handles hidden snapshots. Unfortunately, judging on your topic name, it clearly does not apply to your specific case, since you can still see Veeam snapshot in the Snapshot Manager. Which makes me think your issue is similar to those two issues I referenced earlier.

Please post the support case ID here once you had a chance to open it.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Veeam Logoby cparker4486 » Fri Dec 11, 2015 12:36 am

Thank you, Gostev. The detailed response is much appreciated!

Veeam thinks my support contract has expired (but I think I know the cause) so now I'm waiting for sales to call me back to resolve the issue.
-- Chris
cparker4486
Expert
 
Posts: 222
Liked: 18 times
Joined: Mon Dec 07, 2009 5:09 pm
Full Name: Chris

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Veeam Logoby Gostev » Fri Dec 11, 2015 1:47 am

Sounds good. If you are running into the issue I am thinking, then it is addressed in v9.
However, just to be sure, it would be best for us to confirm that through your debug logs.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Veeam Logoby Vitaliy S. » Fri Dec 11, 2015 11:38 am

On a side note, even Veeam ONE Free edition allows snapshot monitoring and reporting, so you can give it a try.
Vitaliy S.
Veeam Software
 
Posts: 19575
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Veeam Logoby jftuga » Thu Dec 17, 2015 9:44 pm

You can create an alarm in vCenter that will send an email if a snapshot exceeds a certain size.

Under 5.1:
Go to Home -> Inventory -> Datastores and Datastore Clusters
Alarms tab
View: Defintion
Right Click to create New Alarm

General
----------
Alarm Name: VM with snapshots
Monitor: Virtual Machines, Monitor for specific conditions

Triggers
----------
VM Snapshot Size(GB)
Condition: is above
Warning: 10
Alert:20
Trigger if any of the conditions are satisfied

Actions
---------
Send a notification email
jftuga
Enthusiast
 
Posts: 26
Liked: 2 times
Joined: Fri Oct 22, 2010 7:02 pm
Location: Athens, GA
Full Name: John Taylor

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Veeam Logoby Vitaliy S. » Fri Dec 18, 2015 9:45 am

Alarms are good indeed, though I would also recommend to schedule VMs with active snapshots report (jut to be on the safe side).
Vitaliy S.
Veeam Software
 
Posts: 19575
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: VEEAM BACKUP TEMPORARY SNAPSHOT problem

Veeam Logoby ccatlett1984 » Fri Dec 18, 2015 10:35 pm

ccatlett1984
Enthusiast
 
Posts: 83
Liked: 9 times
Joined: Thu Oct 31, 2013 5:11 pm
Full Name: Chris Catlett


Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 23 guests