All Veeam jobs hanging at 0%

[cookiejc]

Hi, we had an issue on Sunday night with one of our Vcenters which hung during Veeam backups. The issue has been resolved but the knock on effect seems to have stopped veeam from backing up anything across all three vcenters. From troubleshooting so far, every VM I try to backup in test jobs shows the same thing at the end of the task log before hanging: [VimApi] RemoveSnapshot, type "VirtualMachineSnapshot", ref "snapshot-53781"

The reference number is the same even if I try to backup a different VM on a different Vcenter. There is a snapshot which is currently being removed in vcenter which I can only assume is the one the logs are referring to. The situation at the moment is that none of our production backups are running and I can only assume it's because Veeam is waiting for this snapshot task to be removed. Parallel proccessing is turned on so I am confused as to why the removal of a snapshot on one VM could be stopping backups of all VM's across Vcenters. This is only a theory but looking at the logs I can't come to any other conclusion.

Has anyone seen this kind of behaviour before? I have a call logged with Veeam but I am not getting very far at the moment

[Gostev]

Hi, please include support case ID with any technical issue as requested when you click New Topic, otherwise your post will be removed by the moderators. Thanks!

[foggy]

Support should definitely be involved here as particular reason should be clear from the job debug logs. What kind of VM it is and what is the progress for this snapshot removal?

[cookiejc]

My apologies, the case no. is Case # 00577079

I am starting to get a response from support, they already have the logs I sent yesterday

The VM is part of an email archiving solution and is subject to a high rate of change. A snapshot taking a long time to remove is not unexpected, I just wouldn't expect Veeam not be able to continue processing other VM's across Vcenters whilst this is taking place.

What I was finding is every time I ran a job to backup any VM across vcenters, a new task was added to the Vcenter to remove the snapshot for this same VM. At one point there were about 18 tasks in vcenter to remove a snapshot from the same VM. The removal finished an hour or so ago, all the subsequent removal requests failed (as the snapshot had already been removed) and suddenly veeam kicked into life and started the test job which was queued at 0%. This seems to confirm that Veeam was stuck and couldn't perform any tasks until this one snapshot had been removed.

[foggy]

Let's see what our support could suggest based on logs review.

[cookiejc]

For anyone with the same issue, it appears the behaviour is by design. Apparently a feature for proper handling of orphaned snapshots uses a shared DB table between all jobs. It checks this to verify there are no snapshots left by Veeam from previous runs which may have been interrupted. It was an unfortunate scenario that an outage of Vcenter combined with a very large snapshot can stop all our backups from working for 2 days. I understand the need for the safeguard within Veeam but hope there is a better way for this situation to be handled in future updates.

Thanks to Veeam for the support

[foggy]

Thanks for providing the update on your case, much appreciated.