Feature Request: Flag a Job with a "Warning" when an Orphaned Snapshot is detected.

[cvanliere]

The "Task.XXXXX.vm-nnnn.log" will show the name of the .vmdk. If it has /XXXXX-nnnnn.vmdk (instead of the correct /XXXXX.vmdk) flag the Job with a the Warning below.

Normal (Green)
[28.10.2022 10:35:32] <97> Info Disk: label "Hard disk 1", path "[XXXXX_Servers_VMFS1] XXXXXX/XXXXXXX.vmdk", capacity 80 GB, backing "CFlatVirtualDiskV2", mode "persistent", thinProvisioned "False"

WARNING (Yellow) and reference the fix/correction document of "https://kb.vmware.com/s/article/2114949"
[28.10.2022 10:33:32] <128> Info Disk: label "Hard disk 1", path "[XXXX_Servers_VMFS1] YYYYYY/YYYYYYY-000001.vmdk", capacity 250 GB, backing "CFlatVirtualDiskV2", mode "persistent", thinProvisioned "False"

We opened Case #05696680. We had a long outage (10 minutes) during a VM backup that caused our SQL Server to drop connections. Normally Veeam did not cause this kind of outage. We found out the long way (collect and send Veeam our logs) that we had an Orphaned Snapshot and Support provided "https://kb.vmware.com/s/article/2114949". Following this document we fixed our issue.

It would be nice is the system could Warn that the Duration was longer than expected due to a known Orphaned Snapshot.

[HannesK]

Hello,
actually, there are two countermeasures in place, which should have prevented such issues. So it seems to be something specific to your environment, that it did not work. For whatever reason, we did not detect that orphaned snapshot.

Because otherwise, Snapshot Hunter should have cleaned it up.

Additionally, there is a Veeam ONE report, that covers orphaned snapshots.

Best regards,
Hannes

[cvanliere]

Hannes,

So, If "it seems to be something specific to your environment, that it did not work.", what do we need to do to get it "to work"?

I read "orphaned snapshots" Our system did not have multiple snapshots consuming space. The 00001.vmdk was the ONLY one, so no wasted space. BUT this did cause the backup to be very slow in the release process.

I read "snapshot hunter". How do enable this feature?

[HannesK]

Hello,

what do we need to do to get it "to work"?

support should check that. VMware support might also be needed.

I read "snapshot hunter". How do enable this feature?

it's enabled per default. No need to do anything. The link to the user guide above has some more details.

Best regards,
Hannes

[cvanliere]

Hannes,

"actually, there are two countermeasures in place, which should have prevented such issues." and "support should check that. VMware support might also be needed."
I have two Cases that were opened due to this issue. Neither one pointed me to "two countermeasures in place". Should I re-open those Cases and reference your post,or start a new Case?

If snapshot hunter is on by default, why did it not prevent our issue? Can I do anything to verify it is enabled and running properly?

I still think this Feature Request would be useful. Clearly what ever you think should of happened automatically, did not. This message would be a good indication that follow up is needed by someone.

[HannesK]

Hello,

I have two Cases that were opened due to this issue. Neither one pointed me to "two countermeasures in place". Should I re-open those Cases and reference your post,or start a new Case?

The job of support is to "fix the problem". It's not supports job to explain all details on how the software works, as long as they can fix the problem. It's a good practice to have one topic per support case and work on that topic until it's solved. I assume, that re-opening might be easier than creating a new case. But that's just an assumption. You can also refer to the other cases in a new case.

If snapshot hunter is on by default, why did it not prevent our issue?

that's something support has to check. I don't know, how anyone could guess the reason for that on a forum

Question: how many machines do you have and how often do issues like that happen to you?

Best regards,
Hannes

[cvanliere]

Hannes,

[The job of support is to "fix the problem".] Understood, BUT... As Support tries to fix the problem, did they know the issue should not occur due to these countermeasures? Since the issue did occur these countermeasures did not function, that during those tickets they would investigate why those countermeasures did not do their job. They did not. You post was the first time I heard about these countermeasures, and your concern that the issue should not have occurred in the first place. I was thinking I should go back to those Cases and post your concern and see what the response from that side of Support is.

I have open a new ticket on "Snapshot Hunter" - Case #05790716

[cvanliere]

Hannes,

"Snapshot Hunter" - Case #05790716 requested sending the Logs again as they were old enough cases your system had purged them. I just sent them under Case #05790716.

Back to this "Feature Request: Flag a Job with a "Warning" when an Orphaned Snapshot is detected."

Regardless of Countermeasures that "should" have triggered and corrected this... The Feature request still applies. IF the system finds an Orphaned Snapshot in the Logs as described ("000001.vmdk") the Message should be added to the results as I requested. This would add an extra layer of protection and allow the community to see if "https://kb.vmware.com/s/article/2114949" needs to be reviewed/processed. Clearly that Article is needed, so I assume others have had the Countermeasures fail too.

If no Orphaned Snapshots are found, no harm, no foul, no messages logged. The Snapshots all worked perfectly (added and removed on the fly) or the countermeasures did their job.

Please reconsider adding this Feature.

[c.evans]

@cvanliere ,

Are you by chance backing up your vCenter appliance(s) in any Veeam VMware Backup Job(s)?

[HannesK]

Hello,
I did some more research on your support cases and it looks like there were some investigation issues on our side initially. I can only apologize for that!

Snapshot Hunter is a feature that tries to fix "consolidation needed" issues for Veeam snapshots. It sometimes happens, that VMware has issues with snapshot handling. It's a relative rare situation, but it caused enough issues / support load, that we implemented that feature some years ago.

As far as I see, your snapshot was not a Veeam snapshot with "consolidation needed". So Snapshot Hunter is not applicable.

The VeeamONE report I mentioned above is a feature that is available to monitor such scenarios.

This report detects VM snapshots that reside on datastores but do not show up in the VMware Snapshot Manager.

Best regards,
Hannes

[cvanliere]

C.Evans,
"Are you by chance backing up your vCenter appliance(s) in any Veeam VMware Backup Job(s)?" How do I verify? I believe so. Shouldn't all the Logs we have attached to our tickets tell you?

Hannes,
"Snapshot Hunter" - Case #05790716 has some traction. They asked about a Registry setting "DisableAutoSnapshotConsolidation" that does not exist on our system. I just updated that ticket, we will see their response. (If the registry does not exist, I assume it is probably not running, so I think it is NOT "enabled per default" as you stated)

"The VeeamONE report I mentioned above is a feature that is available to monitor such scenarios." Why should another tool like VeeamONE be needed?

I still think the Feature request still applies! If you can easily trap for this (seeing a .0000X.vmdk) then show in the results my suggested;
"WARNING (Yellow) and reference the fix/correction document of "https://kb.vmware.com/s/article/2114949"
[28.10.2022 10:33:32] <128> Info Disk: label "Hard disk 1", path "[XXXX_Servers_VMFS1] YYYYYY/YYYYYYY-000001.vmdk", capacity 250 GB, backing "CFlatVirtualDiskV2", mode "persistent", thinProvisioned "False"

Can you link me to the VeeamONE download and its detailed documentation?

[HannesK]

They asked about a Registry setting "DisableAutoSnapshotConsolidation" that does not exist on our system.

that's fine. You want to enable it. So "disable" needs to be set to 0 or DisableAutoSnapshotConsolidation needs to be non-existent. As I said... snapshot hunter is running per default, but it's not applicable to your scenario.

Why should another tool like VeeamONE be needed?

because that tool is built for monitoring while Veeam Backup & Replication is built for backup and restore. Sure, one can argue which feature should be in which product. The general idea is, that a customer has both products as a bundle.

The documentation for the report is linked above and the download is available on veeam.com -> downloads (top menu)

Full documentation of every product is available on helpcenter.veeam.com

[cvanliere]

Hannes,

Downloaded the PDFs for Veeam ONE Documentation. Thank you.

It looks like it does a lot of Monitoring. We have Perpetual license for Veeam Backup & Replication Enterprise. Does this allow us to Download, Install, Configure and Run Veeam ONE? 'https://www.veeam.com/downloads.html' says it is a Free Trial.

So you will not consider this Feature request to allow Veeam Backup & Replication Enterprise to notify users of an Orphaned Snapshot?

[HannesK]

Hello,
about the VCenter question from my colleague above

How do I verify? I believe so.

if you have VCenter VMs in your backup jobs, then it's a bad idea in general. Because you create snapshots of the system that manages snapshots. This can lead to unwanted side-effects. File-based backup is recommended for VCenter. VM-based backups of VCenter is supported in vSphere 7, but not recommended. If you really back up VCenter with Veeam, then I would stop that and see how it goes with orphaned snapshots.

We have Perpetual license for Veeam Backup & Replication Enterprise. Does this allow us to Download, Install, Configure and Run Veeam ONE? '

only the trial. Backup & Replication can / should be bundled with Veeam ONE as "Veeam Availability Suite".

So you will not consider this Feature request to allow Veeam Backup & Replication Enterprise to notify users of an Orphaned Snapshot?

I put it on the list. Realistically, Veeam ONE will be the way to go for foreseeable future because the feature already exists there.

Best regards,
Hannes

[cvanliere]

Hannes,

Thank you for the update.

Rereading C.Evans post about the backing up your vCenter appliance(s), I now DO NOT believe we do this. All our Jobs are for the VMs that run on the VMWare vSphere, not the VMWare vSphere themselves. One, and only one, VM that was running had the Orphaned Snapshot.

It looks like Veeam ONE needs additional licensing/server costs. I still vote for putting the Notification in Veeam Backup & Replication Enterprise so users can see any Orphaned Snapshot. We already have Veeam Backup & Replication Enterprise and are not prepared to incur additional licensing/server costs just for Orphaned Snapshots. Worst case, we will Manually look.

[cvanliere]

Hannes,

My other Feature Request got shot down on the Veeam end.

Any update on this one?

[HannesK]

Hello,

I put it on the list.

that's the status and your vote is counted. Depending on how many other customers ask for it, the status might change eventually. As I said before... realistically VeeamONE is the way for foreseeable future.

Best regards,
Hannes