Jobs start but do nothing

[Bjoern_ch]

Hi veeam team,

since two weeks or so I noticed an annoying behaviour.

Job starts, but then nothing happens and the job is running forever. Logs are unfortunately empty...
If we have to check the running jobs as well and not just the finished jobs it is quite some extra work here.

Cancelling/stopping the job does not work. Restarting Proxy service shuts down/interrupts the job. If I restart the job afterwards it is running successful. It happened to at least 50 different jobs (with different repos/proxies) and I see this on daily basis now.
Is anyone else experiencing this? Does anyone have any recommendations how to fix this?
Case #06146202
Any help is appreciated.
Best regards,
Bjoern

[mats.jansson]

Hi
We have the same behaviour, stopping the job does not work, we also need to restart the proxy to stop the job.
We haven´t hade the time to open a case yet.
/Mats

[Mike Resseler]

Hi Bjoern,

Thanks for creating a case. Please let me know what the follow-up is. Any change you are seeing memory swapping on that proxy? (Where the memory will be "clean" again after restart)

[mats.jansson]

Hi
We have the same behaviour, stopping the job does not work, we also need to restart the proxy to stop the job.
We haven´t hade the time to open a case yet.
/Mats
Case 06155714

[michael.garcia]

Support should be able to help it looks like the recent MS certificate issue..

[Mike Resseler]

Correct. I have been informed last night that many of these cases have to do with a certain .Net issue. Check this KB: https://www.veeam.com/kb4467

[Bjoern_ch]

Ok, support is telling me the same, but for me this issue appeared before patching my OS...how can this be the cause then?

[Mike Resseler]

I don't know the exact details but it has something to do with a certificate or something that expired. But as I said, I don't have the details

[Bjoern_ch]

I installed the windows update which should caused this June 30th. I opened the case June 29th because the problem was already there for some time. From a logical point of view it seems impossible that the windows update (.net framework/certificate issue) could have caused this. That is my point here.

[PeterC]

Glad i found this topic, we have exactly the same issue. after june 25th (where Windows updates were performed) we have several VBO's with job that start and never finish or show no information.
Logs are empty and the gui just says "bottleneck: detecting" or "Resolving objects".

I have created #06158093. Because I am helping out at this company I don't have access to the paid support so I used MS365 community edition.

[ChrisNaisbitt]

+1 from me.

I experienced this issue last week on a brand new environment that isn't in production yet. The behavior was exactly as described by the OP, job doesn't start and sits starting for days, won't cancel and restarting the proxy service resolves the issue. I initially wrote it off as a something I'd caused while messing around but the symptoms are too close a match.

My environment is running Server 2022 and has not had KB5027124 installed:

PS C:\Windows\system32> Get-HotFix -Id KB5027124
Get-HotFix : Cannot find the requested hotfix on the 'localhost' computer. Verify the input and run the command again.
At line:1 char:1
+ Get-HotFix -Id KB5027124
+ ~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (:) [Get-HotFix], ArgumentException
+ FullyQualifiedErrorId : GetHotFixNoEntriesFound,Microsoft.PowerShell.Commands.GetHotFixCommand

PS C:\Windows\system32> Get-HotFix

Source Description HotFixID InstalledBy InstalledOn
------ ----------- -------- ----------- -----------
XXXXXXXX... Update KB5027127 NT AUTHORITY\SYSTEM 19/06/2023 00:00:00
XXXXXXXX... Security Update KB5012170 XXXXXXXXXXXX\Ad... 18/04/2023 00:00:00
XXXXXXXX... Security Update KB5027225 NT AUTHORITY\SYSTEM 19/06/2023 00:00:00
XXXXXXXX... Update KB5026493 NT AUTHORITY\SYSTEM 12/06/2023 00:00:00
XXXXXXXX... Update KB5027396 NT AUTHORITY\SYSTEM 19/06/2023 00:00:00

[DaStivi]

i had this issue too on multiple proxys servers... after installing the .net Hotfix it's working again...
the issue didn't happen instantly... sometime it run for days without issues... i had around 30 Jobs running all around the clock starting in 4h intervalls.. and sometimes they started normally, and sometimes one of these jobs just hung... only restarting of the proxy services helped then..

installing hotfix and reboot fixed it

[PeterC]

I just installed KB5028581 (Server 2019) on 1 VBO and the proxy servers. The jobs now seem to be running again.
I just need to see if they complete successfull to see if this is indeed the solution.

[Polina]

Hi All,

Our support team sees that in many similar cases, the solution suggested in the KB4467 is helpful.

[Bjoern_ch]

Well, it seems that support was right all time.
Problem occured on my remote proxies which were indeed patched before the issue appeared. Management server got patched after issue was already there. Afterwards issue also appeared on local proxy. So it does make sense

[Bjoern_ch]

Mmh, there I was too quick
The first proxy I checked had .net framework installed and windows update was installed shortly after release. BUT my other 26 remote proxies does not have .net framework installed at all. But the error "PKCS12 (PFX) without a supplied password has exceeded maximum allowed iterations." is always in proxylog when a job does not start.
Unfortunately, veeam support does not know an answer and is asking me to open a case with MS now.