"VM has stuck VM snapshot, will attempt to consolidate"

[szimmerman]

I was backing up our Virtual Center VM last night. Taking a lot longer then it normally does when doing a full backup (5 hours vs 10+ hours) so I had to stop the backup. I noticed the 'Removing VM snapshot' took 54 mins to remove! On top of that I got a message stating "vCenter has stuck VM snapshot, will attempt to consolidate periodically." I have never seen that before and naturally I am worried about rebooting the VM. Also, the VM ended backup normally with errors since I stopped it. Any help would be great!

[Vitaliy S.]

Hi Scott,

Please take a look at this description of the feature you're currently seeing "in action" > Snapshot Hunter. Since you've aborted the backup job, there could VM snapshot left on the vCenter Server VM.

Thanks!

[B.F.]

Greetings,

The last few backups of our file server VM, I've noticed this in the job details which didn't happen before:

<snipped>
2/27/2019 9:34:41 PM :: Removing VM snapshot
2/27/2019 9:35:37 PM :: <VM NAME> has stuck VM snapshot, will attempt to consolidate periodically
2/27/2019 9:35:39 PM :: Finalizing
<snipped>

There are green checks across the board and appears to be working. I'm not seeing any lingering snapshots in vSphere either. Just wanted to make sure there isn't something we need to be looking into or just let it roll.

Thanks!

[B.F.]

Moderators,

Not sure what solution is gained by this merge? Is Snapshot Hunter finding snapshots I'm not seeing in vSphere? Is there a way to look for these phantom snapshots? Snapshot manger in vSphere shows nothing.

Thanks!

[foggy]

Yes, Snapshot Hunter detects "phantom" snapshot files that exist on datastore but are not registered in vSphere.

[Vitaliy S.]

You can also use Veeam ONE to track all orphaned and stalled VM snapshots with our predefined alarms. Not only will these alarms spot these snapshots but they will also attempt to remove these for you via remediation actions set in the alarm.

[B.F.]

I'm browsing the datastore file structure on this VM. According to VMware docs the snapshot files "... uses the following syntax, S1vm-000001.vmdk where S1vm is the name of the virtual machine and the six-digit number, 000001, is based on the files that already exist in the directory."

I'm not seeing any 000001 vmdk files in the file structure. I created a snapshot manually and then I observed the presence of the 000001 files.

Snapshot Hunter false positive?

Thanks

[foggy]

Any chance it succeeded to consolidate it during the last attempt? If you still see this message during backup, feel free to contact support for a closer look.

[B.F.]

We may have stumbled on the cause. There was a little bit of overlap for this VM where the backup was spilling over into the replication job for that same VM. We've made adjustments and will report back if that was the issue.

Thanks

[foggy]

It generally shouldn't be the case if both backup and replication jobs are managed by the same backup server. But let's see.

[B.F.]

Problem solved. The backup and replication are actually on 2 different Veeam servers so the overlap was the cause.

Why 2 servers you ask? Backup jobs run locally on prem. Replication server is at the secondary site which pulls the data for replication. Main site blows up, we can recover quickly since Veeam is still alive at the secondary.

Thanks

[foggy]

Thanks for sharing. Your setup is totally reasonable. In some cases though a single Veeam B&R instance in DR is preferred (unless both sites have equal roles and you cross-replicate in both directions).

[B.F.]

In some cases though a single Veeam B&R instance in DR is preferred (unless both sites have equal roles and you cross-replicate in both directions).

Yes, the secondary site also is used on a smaller scale for it's own production work. Cross replication does occur and works well....as long as there is no backup overlapping.

[foggy]

Understood. At least now you know you should keep an eye on the schedule.