Veeam keeps creating/removing snapshots

VMware specific discussions

Veeam keeps creating/removing snapshots

Veeam Logoby myrdin » Tue Jan 13, 2015 1:53 am

Hi,

I have a case opened already for this number 00725141 but i'd like to know if anyone else had this problem.

I am on Veeam 8 patch 1.

we had a very weird issue: We had an Exchange box getting backed up (for months now) and customer called saying that CPU was 100%. I have waited to the job to finish (that runs every hour) but that didnt fix the issue, so ok Veeam was not the cause. After i have disabled the job, Veeam KEPT sending requests to the Vcenter to take and remove snapshots for that VM, continuosly. This literally killed the Exchange box that went unreachable. I had to forcefully power off the Veeam server to stop it from requesting to process the snapshots on that machine.

I am investigating with both Vmware/Veeam now, but i am wondering if you had any clues.

thanks
myrdin
Enthusiast
 
Posts: 57
Liked: 3 times
Joined: Tue Jul 02, 2013 4:17 am
Full Name: NIck

Re: Veeam keeps creating/removing snapshots

Veeam Logoby v.Eremin » Tue Jan 13, 2015 8:02 am

So, there are no jobs running against that VM and VB&R keeps instructing vCenter to create and consolidate snapshot for that VM? Do the created snapshot have corresponding name, like "Veeam Helper snapshot"? What happens if you shutdown a VB&R machine? Will the problem disappear? Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Veeam keeps creating/removing snapshots

Veeam Logoby myrdin » Tue Jan 13, 2015 8:52 am

No helper snapshot, snapshot manager was empty but inside the vm folder vm was running on snapshot number 67.

I had to kill the Veeam server (literally forced power off) that stopped vcenter to try to take snapshot. Again, we created an ad hoc user the veeam uses to talk to vcenter for these reason (so we can identify whos passing instructions to vcenter), and it was veeam server asking vcenter to keep taking snapshots and removig immediately after.
myrdin
Enthusiast
 
Posts: 57
Liked: 3 times
Joined: Tue Jul 02, 2013 4:17 am
Full Name: NIck

Re: Veeam keeps creating/removing snapshots

Veeam Logoby myrdin » Tue Jan 13, 2015 9:33 am

Apparently Veeam will always try to consolidate if it finds a vm with snapshots, if it cant, it will automatically create and remove a snapshot, and if it doesnt work it will loop on it...
myrdin
Enthusiast
 
Posts: 57
Liked: 3 times
Joined: Tue Jul 02, 2013 4:17 am
Full Name: NIck

Re: Veeam keeps creating/removing snapshots

Veeam Logoby dellock6 » Tue Jan 13, 2015 1:55 pm

That's the beheviour of Snapshot Hunter during "hard consolidation": if a snapshot removal doesn't work, the next step is to try to create and immediately remove a new snapshot to try and "unlock" the stuck snapshots.
however, the frequency you are describing is not expected as by default SH kicks in every 4 hours, so probably some more investigation via support is needed.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5047
Liked: 1330 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: Veeam keeps creating/removing snapshots

Veeam Logoby myrdin » Wed Jan 14, 2015 12:05 am

thanks.

After working with Vmware we discovered that that was a lock on the vmdk of the machine being backed up. This prevented Veeam to consolidate.

What i dont like is the fact that Veeam doesnt tell me that it cannot consolidate back, and the job says it was OK whereas it was keeping generating one snap over the other (up to 70 snapshots).

thanks
myrdin
Enthusiast
 
Posts: 57
Liked: 3 times
Joined: Tue Jul 02, 2013 4:17 am
Full Name: NIck

Re: Veeam keeps creating/removing snapshots

Veeam Logoby dellock6 » Wed Jan 14, 2015 12:13 am 1 person likes this post

Again, probably a support ticket should help: as I explained in the dedicated blog post about Snapshot Hunter after 3 retries over 12 hours, if you have configured mail notifications, you should receive an email with the warning that snapshot hunter was not able to remove the snapshotexactly because of a permanent lock. So, something seems to have not worked properly.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5047
Liked: 1330 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: Veeam keeps creating/removing snapshots

Veeam Logoby myrdin » Wed Jan 14, 2015 2:48 am

thanks! I still think thou that Veeam should put the job in warning state.

Notifications were disabled as we are still in deploying phase.
myrdin
Enthusiast
 
Posts: 57
Liked: 3 times
Joined: Tue Jul 02, 2013 4:17 am
Full Name: NIck

Re: Veeam keeps creating/removing snapshots

Veeam Logoby Vitaliy S. » Wed Jan 14, 2015 9:05 am

Hi Nick, in order to detect orphaned snapshots you can leverage Veeam ONE monitoring capabilities. In addition to this, we have a dedicated report template for all snapshots not visible in the snapshot manager of the vSphere Client. Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Veeam keeps creating/removing snapshots

Veeam Logoby resruss » Fri Jan 16, 2015 7:33 am

We're seeing this on 1 particular system running v8 Patch 1
Support gave me a registry key to disable snapshot hunter, which we've done and restarted Veeam services, but still seeing the occasional Create / Remove snapshot in vCenter
Even though we haven't seen anything snapshot-wise in the System History for Veeam since then

I've just rebooted the Veeam B&R host, is there anything else needed to totally disable Snapshot Hunter?
resruss
Service Provider
 
Posts: 55
Liked: 6 times
Joined: Mon Apr 30, 2012 2:04 am

Re: Veeam keeps creating/removing snapshots

Veeam Logoby foggy » Fri Jan 16, 2015 10:56 am

Please check with support engineer whether those snapshots are triggered by Veeam B&R.
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Veeam keeps creating/removing snapshots

Veeam Logoby jp34 » Wed Jan 06, 2016 8:53 am

If someone needs the registry key...
You can create following registry key on a Veeam server to disable snapshot hunter:DisableAutoSnapshotConsolidationType: REG_DWORDValue: 1
It should be created under HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
A Veeam Backup Service restart is required.
jp34
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Tue Dec 22, 2009 8:31 am
Full Name: JP

Re: Veeam keeps creating/removing snapshots

Veeam Logoby EricZBA » Thu Jan 28, 2016 4:45 pm

FYI the above registry key does not seem to work in Veeam v9. We've had a VM go through the snapshot hunter process (soft/hard without quiesce/hard with quiesce) three times over the past 48+ hours after a backup went awry on Monday. I have a support case opened with Veeam since we can't troubleshoot the issue with VMware until Veeam stops launching the snapshot creation/removal/disk consolidation process.
EricZBA
Novice
 
Posts: 9
Liked: 1 time
Joined: Wed Mar 05, 2014 7:23 pm
Location: Dallas, TX
Full Name: Eric Betancourt

Re: Veeam keeps creating/removing snapshots

Veeam Logoby foggy » Fri Jan 29, 2016 10:02 am

I'd check whether you're specifying the key correctly, since it's name could be misinterpreted from the previous post. The correct key name is DisableAutoSnapshotConsolidation.
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Veeam keeps creating/removing snapshots

Veeam Logoby EricZBA » Fri Jan 29, 2016 2:20 pm

Checked again and sent screenshot to support for verification. They suggested a whole server reboot in case the service restart didn't do the trick but no luck. Support case was escalated yesterday and hopefulyl I'll get an answer soon. I'll post again if the issue is found to be a bug or if I'm told the reg key was changed in v9 for future reference.

Image
EricZBA
Novice
 
Posts: 9
Liked: 1 time
Joined: Wed Mar 05, 2014 7:23 pm
Location: Dallas, TX
Full Name: Eric Betancourt

Next

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 15 guests