Host-based backup of VMware vSphere VMs.
Post Reply
myrdin
Enthusiast
Posts: 57
Liked: 3 times
Joined: Jul 02, 2013 4:17 am
Full Name: NIck
Contact:

Veeam keeps creating/removing snapshots

Post by myrdin »

Hi,

I have a case opened already for this number 00725141 but i'd like to know if anyone else had this problem.

I am on Veeam 8 patch 1.

we had a very weird issue: We had an Exchange box getting backed up (for months now) and customer called saying that CPU was 100%. I have waited to the job to finish (that runs every hour) but that didnt fix the issue, so ok Veeam was not the cause. After i have disabled the job, Veeam KEPT sending requests to the Vcenter to take and remove snapshots for that VM, continuosly. This literally killed the Exchange box that went unreachable. I had to forcefully power off the Veeam server to stop it from requesting to process the snapshots on that machine.

I am investigating with both Vmware/Veeam now, but i am wondering if you had any clues.

thanks
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam keeps creating/removing snapshots

Post by veremin »

So, there are no jobs running against that VM and VB&R keeps instructing vCenter to create and consolidate snapshot for that VM? Do the created snapshot have corresponding name, like "Veeam Helper snapshot"? What happens if you shutdown a VB&R machine? Will the problem disappear? Thanks.
myrdin
Enthusiast
Posts: 57
Liked: 3 times
Joined: Jul 02, 2013 4:17 am
Full Name: NIck
Contact:

Re: Veeam keeps creating/removing snapshots

Post by myrdin »

No helper snapshot, snapshot manager was empty but inside the vm folder vm was running on snapshot number 67.

I had to kill the Veeam server (literally forced power off) that stopped vcenter to try to take snapshot. Again, we created an ad hoc user the veeam uses to talk to vcenter for these reason (so we can identify whos passing instructions to vcenter), and it was veeam server asking vcenter to keep taking snapshots and removig immediately after.
myrdin
Enthusiast
Posts: 57
Liked: 3 times
Joined: Jul 02, 2013 4:17 am
Full Name: NIck
Contact:

Re: Veeam keeps creating/removing snapshots

Post by myrdin »

Apparently Veeam will always try to consolidate if it finds a vm with snapshots, if it cant, it will automatically create and remove a snapshot, and if it doesnt work it will loop on it...
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Veeam keeps creating/removing snapshots

Post by dellock6 »

That's the beheviour of Snapshot Hunter during "hard consolidation": if a snapshot removal doesn't work, the next step is to try to create and immediately remove a new snapshot to try and "unlock" the stuck snapshots.
however, the frequency you are describing is not expected as by default SH kicks in every 4 hours, so probably some more investigation via support is needed.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
myrdin
Enthusiast
Posts: 57
Liked: 3 times
Joined: Jul 02, 2013 4:17 am
Full Name: NIck
Contact:

Re: Veeam keeps creating/removing snapshots

Post by myrdin »

thanks.

After working with Vmware we discovered that that was a lock on the vmdk of the machine being backed up. This prevented Veeam to consolidate.

What i dont like is the fact that Veeam doesnt tell me that it cannot consolidate back, and the job says it was OK whereas it was keeping generating one snap over the other (up to 70 snapshots).

thanks
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Veeam keeps creating/removing snapshots

Post by dellock6 » 1 person likes this post

Again, probably a support ticket should help: as I explained in the dedicated blog post about Snapshot Hunter after 3 retries over 12 hours, if you have configured mail notifications, you should receive an email with the warning that snapshot hunter was not able to remove the snapshotexactly because of a permanent lock. So, something seems to have not worked properly.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
myrdin
Enthusiast
Posts: 57
Liked: 3 times
Joined: Jul 02, 2013 4:17 am
Full Name: NIck
Contact:

Re: Veeam keeps creating/removing snapshots

Post by myrdin »

thanks! I still think thou that Veeam should put the job in warning state.

Notifications were disabled as we are still in deploying phase.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veeam keeps creating/removing snapshots

Post by Vitaliy S. »

Hi Nick, in order to detect orphaned snapshots you can leverage Veeam ONE monitoring capabilities. In addition to this, we have a dedicated report template for all snapshots not visible in the snapshot manager of the vSphere Client. Thanks!
resruss
Service Provider
Posts: 56
Liked: 6 times
Joined: Apr 30, 2012 2:04 am
Contact:

Re: Veeam keeps creating/removing snapshots

Post by resruss »

We're seeing this on 1 particular system running v8 Patch 1
Support gave me a registry key to disable snapshot hunter, which we've done and restarted Veeam services, but still seeing the occasional Create / Remove snapshot in vCenter
Even though we haven't seen anything snapshot-wise in the System History for Veeam since then

I've just rebooted the Veeam B&R host, is there anything else needed to totally disable Snapshot Hunter?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam keeps creating/removing snapshots

Post by foggy »

Please check with support engineer whether those snapshots are triggered by Veeam B&R.
jp34
Enthusiast
Posts: 47
Liked: 1 time
Joined: Dec 22, 2009 8:31 am
Full Name: JP
Contact:

Re: Veeam keeps creating/removing snapshots

Post by jp34 »

If someone needs the registry key...
You can create following registry key on a Veeam server to disable snapshot hunter:DisableAutoSnapshotConsolidationType: REG_DWORDValue: 1
It should be created under HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
A Veeam Backup Service restart is required.
EricZBA
Novice
Posts: 9
Liked: 1 time
Joined: Mar 05, 2014 7:23 pm
Full Name: Eric Betancourt
Location: Dallas, TX
Contact:

Re: Veeam keeps creating/removing snapshots

Post by EricZBA »

FYI the above registry key does not seem to work in Veeam v9. We've had a VM go through the snapshot hunter process (soft/hard without quiesce/hard with quiesce) three times over the past 48+ hours after a backup went awry on Monday. I have a support case opened with Veeam since we can't troubleshoot the issue with VMware until Veeam stops launching the snapshot creation/removal/disk consolidation process.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam keeps creating/removing snapshots

Post by foggy »

I'd check whether you're specifying the key correctly, since it's name could be misinterpreted from the previous post. The correct key name is DisableAutoSnapshotConsolidation.
EricZBA
Novice
Posts: 9
Liked: 1 time
Joined: Mar 05, 2014 7:23 pm
Full Name: Eric Betancourt
Location: Dallas, TX
Contact:

Re: Veeam keeps creating/removing snapshots

Post by EricZBA »

Checked again and sent screenshot to support for verification. They suggested a whole server reboot in case the service restart didn't do the trick but no luck. Support case was escalated yesterday and hopefulyl I'll get an answer soon. I'll post again if the issue is found to be a bug or if I'm told the reg key was changed in v9 for future reference.

Image
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam keeps creating/removing snapshots

Post by veremin »

Based on the provided screenshot, the specified regkey should have taken effect indeed. So, the described behaviour looks unexpected. Kindly, keep working with our support team and update the topic once the resolution is found. Thanks.
EricZBA
Novice
Posts: 9
Liked: 1 time
Joined: Mar 05, 2014 7:23 pm
Full Name: Eric Betancourt
Location: Dallas, TX
Contact:

Re: Veeam keeps creating/removing snapshots

Post by EricZBA »

FYI for future reference, according to RnD and Support, the registry key has to be added in two places. Here is the correct information to disable Snapshot Hunter in v9

Name: DisableAutoSnapshotConsolidation
Type: REG_DWORD
Value: 1

Create the following in both of the following locations:
HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Veeam\Veeam Backup and Replication

Thanks and good luck!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam keeps creating/removing snapshots

Post by foggy »

Eric, what is your support case number? I'd like to know the details, since Wow6432Node should not be involved.
EricZBA
Novice
Posts: 9
Liked: 1 time
Joined: Mar 05, 2014 7:23 pm
Full Name: Eric Betancourt
Location: Dallas, TX
Contact:

Re: Veeam keeps creating/removing snapshots

Post by EricZBA »

Case # 01682148

If you want to look at any additional logs let me know and I'll be sure to send them your way.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam keeps creating/removing snapshots

Post by foggy »

I'm more interested in the fact why the Wow6432Node node took effect (if it did) as it actually should not, since this value is read by the 64-bit component. I will look into the case and also involve QA to verify this fact.
Post Reply

Who is online

Users browsing this forum: No registered users and 119 guests