Veeam keeps creating/removing snapshots

[myrdin]

Hi,

I have a case opened already for this number 00725141 but i'd like to know if anyone else had this problem.

I am on Veeam 8 patch 1.

we had a very weird issue: We had an Exchange box getting backed up (for months now) and customer called saying that CPU was 100%. I have waited to the job to finish (that runs every hour) but that didnt fix the issue, so ok Veeam was not the cause. After i have disabled the job, Veeam KEPT sending requests to the Vcenter to take and remove snapshots for that VM, continuosly. This literally killed the Exchange box that went unreachable. I had to forcefully power off the Veeam server to stop it from requesting to process the snapshots on that machine.

I am investigating with both Vmware/Veeam now, but i am wondering if you had any clues.

thanks

[veremin]

So, there are no jobs running against that VM and VB&R keeps instructing vCenter to create and consolidate snapshot for that VM? Do the created snapshot have corresponding name, like "Veeam Helper snapshot"? What happens if you shutdown a VB&R machine? Will the problem disappear? Thanks.

[myrdin]

No helper snapshot, snapshot manager was empty but inside the vm folder vm was running on snapshot number 67.

I had to kill the Veeam server (literally forced power off) that stopped vcenter to try to take snapshot. Again, we created an ad hoc user the veeam uses to talk to vcenter for these reason (so we can identify whos passing instructions to vcenter), and it was veeam server asking vcenter to keep taking snapshots and removig immediately after.

[myrdin]

Apparently Veeam will always try to consolidate if it finds a vm with snapshots, if it cant, it will automatically create and remove a snapshot, and if it doesnt work it will loop on it...

[dellock6]

That's the beheviour of Snapshot Hunter during "hard consolidation": if a snapshot removal doesn't work, the next step is to try to create and immediately remove a new snapshot to try and "unlock" the stuck snapshots.
however, the frequency you are describing is not expected as by default SH kicks in every 4 hours, so probably some more investigation via support is needed.

[myrdin]

thanks.

After working with Vmware we discovered that that was a lock on the vmdk of the machine being backed up. This prevented Veeam to consolidate.

What i dont like is the fact that Veeam doesnt tell me that it cannot consolidate back, and the job says it was OK whereas it was keeping generating one snap over the other (up to 70 snapshots).

thanks

[dellock6]

Again, probably a support ticket should help: as I explained in the dedicated blog post about Snapshot Hunter after 3 retries over 12 hours, if you have configured mail notifications, you should receive an email with the warning that snapshot hunter was not able to remove the snapshotexactly because of a permanent lock. So, something seems to have not worked properly.

[myrdin]

thanks! I still think thou that Veeam should put the job in warning state.

Notifications were disabled as we are still in deploying phase.

[Vitaliy S.]

Hi Nick, in order to detect orphaned snapshots you can leverage Veeam ONE monitoring capabilities. In addition to this, we have a dedicated report template for all snapshots not visible in the snapshot manager of the vSphere Client. Thanks!

[resruss]

We're seeing this on 1 particular system running v8 Patch 1
Support gave me a registry key to disable snapshot hunter, which we've done and restarted Veeam services, but still seeing the occasional Create / Remove snapshot in vCenter
Even though we haven't seen anything snapshot-wise in the System History for Veeam since then

I've just rebooted the Veeam B&R host, is there anything else needed to totally disable Snapshot Hunter?

[foggy]

Please check with support engineer whether those snapshots are triggered by Veeam B&R.

[jp34]

If someone needs the registry key...
You can create following registry key on a Veeam server to disable snapshot hunter:DisableAutoSnapshotConsolidationType: REG_DWORDValue: 1
It should be created under HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
A Veeam Backup Service restart is required.

[EricZBA]

FYI the above registry key does not seem to work in Veeam v9. We've had a VM go through the snapshot hunter process (soft/hard without quiesce/hard with quiesce) three times over the past 48+ hours after a backup went awry on Monday. I have a support case opened with Veeam since we can't troubleshoot the issue with VMware until Veeam stops launching the snapshot creation/removal/disk consolidation process.

[foggy]

I'd check whether you're specifying the key correctly, since it's name could be misinterpreted from the previous post. The correct key name is DisableAutoSnapshotConsolidation.

[EricZBA]

Checked again and sent screenshot to support for verification. They suggested a whole server reboot in case the service restart didn't do the trick but no luck. Support case was escalated yesterday and hopefulyl I'll get an answer soon. I'll post again if the issue is found to be a bug or if I'm told the reg key was changed in v9 for future reference.

[veremin]

Based on the provided screenshot, the specified regkey should have taken effect indeed. So, the described behaviour looks unexpected. Kindly, keep working with our support team and update the topic once the resolution is found. Thanks.

[EricZBA]

FYI for future reference, according to RnD and Support, the registry key has to be added in two places. Here is the correct information to disable Snapshot Hunter in v9

Name: DisableAutoSnapshotConsolidation
Type: REG_DWORD
Value: 1

Create the following in both of the following locations:
HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Veeam\Veeam Backup and Replication

Thanks and good luck!

[foggy]

Eric, what is your support case number? I'd like to know the details, since Wow6432Node should not be involved.

[EricZBA]

Case # 01682148

If you want to look at any additional logs let me know and I'll be sure to send them your way.

[foggy]

I'm more interested in the fact why the Wow6432Node node took effect (if it did) as it actually should not, since this value is read by the 64-bit component. I will look into the case and also involve QA to verify this fact.