Jobs stuck "pending" forever

[RobMiller86]

Case # 07224264

Anyone else see this issue where Hyper-V jobs start, the list of VMs is populated in job statistics, but each one of the VMs, if you click on it in the job statistics, the right side "action" column is blank? The VMs never start processing, VBR doesn't list anything in statistics about why it's pending, or what it's waiting on, and the job will sit like this forever. If you try to stop it, it will then sit on "stopping" forever. The only fix once a job enters this state is to restart the VBR services, or reboot the VBR. Then the same job works fine. When this does happen, it affects every VM in the job, but every other job runs fine. It's 1 job, and all VMs in the job, and the job that has the issue is random, other than always being a Hyper-V job.

It only happens to Hyper-V jobs, never to VMware jobs. We've been increasing resources to our DCs, updating and rebooting clusters, the problem always returns. As we add more jobs, this is becoming a large hindrance. I have a case open, but was wondering if anyone else has run across this issue.

Thanks.

[david.domask]

Hi Rob,

Thanks for sharing the case number; the behavior is a bit unclear at the moment, but the Support Team is working on a theory I see and should have an update for you soon, so let's wait for Support's update.

I did want to note regarding SCVMM as you asked about it in the case, I don't think the blog post you referenced was meaning to tell you _should_ rush away from an SCVMM configuration with Veeam, more that the blog post was hoping for some QOL improvements for SCVMM from Microsoft. The "easy" part is more about less complexity I think as you don't need to deal with SCVMM issues like problems with network connection or SCVMM going un-responsive). So I don't think there's a need to rush SCVMM out the door just yet, I would advise waiting for Support's analysis and _IF_ SCVMM is suspect, it's best we understand what's hapening in full before deciding to make the change.

[RobMiller86]

I'm still combatting this issue. We've completely removed SC VMM from Veeam to reduce variables. Veeam now hits the clusters directly. I've since seen this happen once to a VMware job, and all other times it's been Hyper-V Jobs. I have one stuck right now. Job started 5 hours ago, all VMs showing "pending", but if you click them, they don't say what they are waiting on. During this time, there have been many times where this job is the only job running. There couldn't be anything else holding it up. Other jobs start and finish for VMs on the same cluster, going to the same repo, and this one just keeps sitting "pending". I still don't understand why Veeam allows a condition like this to continue, broken, forever, with no timeout.

Only way out is to kill the Veeam Backup Server service and restart all the Veeam services. I now have a new case: 07306709

[RobMiller86]

One thing I find odd and feel VBR could do better, is the handling of jobs and "stopping". I just worked with the Engineer so he could fully document this as it may need to be escalated to advanced support. But I don't understand why the VBR UI can't handle things more gracefully.

Example:
Job gets stuck with all VMs in "pending" with no explanation as to why. No other jobs running, no contention. Just stuck at "pending" forever. It was at 28 hours today when working with the Engineer. Why doesn't VBR have some type of timeout, even if lengthy, say 4 hours, where it will kill a job that clearly has no forward movement, with nothing in the logs as to why?

Additionally, if I tell the hung job to stop, either gracefully or immediately, it never stops. It sits on "stopping" forever. Now I can go to C:\programdata\veeam\backup\job_name\job.*.log and search for the PID, and then kill this PID in task manager which will immediately stop the job. Why the need to jump through hoops and search through logs? Can't VBR do this for us? If I pick "stop - immediately", doesn't VBR know the PID and can kill that?

I understand there is some bug somewhere causing this issue, but it also seems that various issues like this could be handled better. It would be great if VBR automatically handled this in some way after a period of time, but at a minimum "stop - immediately" should actually stop the job immediately, without me needing to review logs and find PIDs and kill that.

[david.domask]

Hi Rob,

Thanks for the update and sorry to hear the trouble persists; appreciate you sharing the new case number, I can see that right now Support has a few tests they want to do to isolate further, so let's see the results of those.

As for your question on the stopping, in essence the difference between gracefully and immediately is mostly just about "will Veeam wait for currently processing tasks to finish or stop and clean up right away?" It's not quite the same as just terminating the PID, as depending on when you stop the job, there may be additional cleanup required in the environment (disk shadows, hypervisor snapshots, incomplete backup files to be cleared, updates to the database to reflect the current state, etc). Basically, while I get the semantic confusion on "immediately", these housekeeping tasks are important outside of just tidying up the Veeam environment, and these processes can take some time. While just popping the PID naturally will return control faster in your situation, in most situations there will be a lot of housekeeping that needs to be done before the process should end.

Your patience on this issue is much appreciated, let's continue with Support and see what the result of their investigation is.

[RobMiller86]

It's too early to say for certain, but after applying the private fix provided by support regarding WMI changes, the issue hasn't returned. Backups are also quicker to complete as they don't appear to wait as long calculating various task slots as they were doing in the past. I've seen it go for a week before a single job becomes unrecoverable, so I can't say for certain we are out of the woods, but it appears promising. Hopefully this is the fix!

[david.domask]

Hi Rob,

Thank you for the update, and indeed sounds promising -- may I ask, was the hotfix provided for this case or a different one? Just want to follow up internally to try to understand how this may have helped with the "pending" jobs status.

[RobMiller86]

It was provided for this case, but Veeam support was not aware of it. I had them reference the cases in this thread that alerted me to a potential fix.

microsoft-hyper-v-f25/our-known-v12-1-p ... 93634.html

I asked for it and they provided it. So far so good, but not enough time has passed to be certain.

[RobMiller86]

I would like to confirm that this fix worked. Not only do jobs no longer hang, but the processing portion of each job runs much faster. Before it would often hang for longer periods of time determining proxy task slots, etc. Everything is much faster, and never hangs. This fix should definitely make it into the next VBR update for everyone. It's a huge improvement! Thanks.

[tbowness33]

Hello, I am having the same issues where once a week we have a Hyperv VM backups stuck at pending with no detail on right side just shows blank. All other jobs run beside it but usually one job will be stuck at 0% forever until I kill the job and have to restart services or server to be able to run it again. How can I get the fix that was applied to Rob's system to fix it.

[david.domask]

Hi Thomas, welcome to the forums.

Fixes are distributed via Veeam Support, so if you suspect you have a similar issue, open a Support Case and allow Veeam Support time to review the situation. You can refer to the case numbers here and Support review the situation and determine if it's the same issue the fix assists with.

Be sure to include logs for the affected jobs for Support to review; use the first radio button and select one of the affected jobs, it should be enough to start the investigation.

[tbowness33]

Thanks, I did open support case, but they are still looking for the fix.

[RobMiller86]

Can anyone confirm if this fix is now included in 12.2?

[RobMiller86]

Hi Rob,

Thank you for the update, and indeed sounds promising -- may I ask, was the hotfix provided for this case or a different one? Just want to follow up internally to try to understand how this may have helped with the "pending" jobs status.

David can you please confirm if this fix is now included in the 12.2 release? Thanks.

[david.domask]

Hi all!

Issue is 712608, and it currently is marked as resolved in 12.2, but there is available hotfix for 12.1.2.172 if you're not in a position to upgrade as of yet to 12.2, though 12.2 is a best bet at your earliest convenience.