Maintain control of your Microsoft 365 data
t7MevELx0
Service Provider
Posts: 87
Liked: 12 times
Joined: Feb 06, 2024 6:55 pm
Contact:

Re: Stalling Jobs

Post by t7MevELx0 »

@mjr.epicfail

Case #07583409


I just fixed the indexing issue on the two repos (you'll see that issue in that ticket too...). The copy jobs just sit there, transferring very little, or absolutely nothing.

Things were working great on 8.05. These copy jobs haven't worked since upgrading to 8.1 in January.
mjr.epicfail
Veeam Legend
Posts: 531
Liked: 146 times
Joined: Apr 22, 2022 12:14 pm
Full Name: Danny de Heer
Contact:

Re: Stalling Jobs

Post by mjr.epicfail »

Hi t7MevELx0,

I dont have access to the support ticket, im not a Veeam Employee.
I asked because the Veeam members on the forum can look it up and maybe shed some light on the issue, or maybe that R&D is working (actively) on this issue.
VMCE / Veeam Legend 2*
t7MevELx0
Service Provider
Posts: 87
Liked: 12 times
Joined: Feb 06, 2024 6:55 pm
Contact:

Re: Stalling Jobs

Post by t7MevELx0 »

Yeah, I usually share the ticket number here for that reason. I'm still seeing jobs stall on version 8.1.1. It happens randomly, and I have to use a quirky workaround to get things running again. Just clearing the NATS JetStream isn't enough. I typically have to place each proxy in the pool into maintenance mode and then bring them back out. In some cases, I need to remove the proxies from the pool and re-add them to get things unstuck.
pat_ren
Service Provider
Posts: 126
Liked: 27 times
Joined: Jan 02, 2024 9:13 am
Full Name: Pat
Contact:

Re: Stalling Jobs

Post by pat_ren »

Have been continuing to have this issue intermittently, even on 8.1.1.159 and on my smallest servers and smallest jobs that usually take 1-2 mins max to run. Have received a fix from support yesterday that they say should address the issue.
My case is 07708238, as it's intermittent can't confirm if it's resolved anything yet, but for anyone waiting on a fix there may be some progress now.
pat_ren
Service Provider
Posts: 126
Liked: 27 times
Joined: Jan 02, 2024 9:13 am
Full Name: Pat
Contact:

Re: Stalling Jobs

Post by pat_ren »

Fix from support did not help so this is still an ongoing issue, every couple of days a job is getting stuck and running indefinitely until manual intervention occurs.
Case is still open
MarkBoothmaa
Veeam Legend
Posts: 225
Liked: 71 times
Joined: Mar 22, 2017 11:10 am
Full Name: Mark Boothman
Location: Darlington, United Kingdom
Contact:

Re: Stalling Jobs

Post by MarkBoothmaa »

It looks like our issue (the original post on the thread) has been fixed by an hotfix. We are currently monitoring progress but results have been positive for the last week.
pat_ren
Service Provider
Posts: 126
Liked: 27 times
Joined: Jan 02, 2024 9:13 am
Full Name: Pat
Contact:

Re: Stalling Jobs

Post by pat_ren »

Got some more progress on this recently, confirmed it is still an issue and a known issue being worked on by R&D.
From what I have been able to determine, the issue is related to NATS streams getting stuck with an object they don't know where to put and the job just stalls.

If you see something like this in the log, especially with repo id 00000000-0000-0000-0000-000000000000 then it means the nats is confused and your job is stuck.

[10.06.2025 05:10:41.165] 466 (1) [Info] Adding object to job report (session ID: xxx-xxx-xxx-xxx)...
[10.06.2025 05:10:41.165] 466 (1) [Info] Object was successfully unlocked: xxx-xxx-xxx-xxx (repository: 00000000-0000-0000-0000-000000000000, session: xxx-xxx-xxx-xxx)

No fix yet but it does seem like the issue is understood by R&D but they don't know how to fix it yet.

My case is still open.
tkutil
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 22, 2019 4:10 pm
Full Name: Troy Kutil
Contact:

Re: Stalling Jobs

Post by tkutil »

I believe we have the same issue, but only when it comes to our mail job. We have the users broken into 4 groups all in the same job and this seems to work, but if we try to switch from using groups to using the organization the job never seems to finish. I have even tried a combination of user groups and the organization, but that also does not work. For those having the problem are you also trying to backup the organization?
pat_ren
Service Provider
Posts: 126
Liked: 27 times
Joined: Jan 02, 2024 9:13 am
Full Name: Pat
Contact:

Re: Stalling Jobs

Post by pat_ren » 1 person likes this post

tkutil wrote: Jun 13, 2025 6:58 pm I believe we have the same issue, but only when it comes to our mail job. We have the users broken into 4 groups all in the same job and this seems to work, but if we try to switch from using groups to using the organization the job never seems to finish. I have even tried a combination of user groups and the organization, but that also does not work. For those having the problem are you also trying to backup the organization?
I am running approx 980 jobs, all backing up 'entire org' and then split into mail, one drive, sharepoint & teams so they can run as 3 separate jobs. Some of these jobs are up to 5000 objects and mostly this has been working fine for years, other than the occasional job stalling as described in this thread.
tkutil
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 22, 2019 4:10 pm
Full Name: Troy Kutil
Contact:

Re: Stalling Jobs

Post by tkutil »

Opened Case # 07733395
t7MevELx0
Service Provider
Posts: 87
Liked: 12 times
Joined: Feb 06, 2024 6:55 pm
Contact:

Re: Stalling Jobs

Post by t7MevELx0 »

Case #07674412

It’s now June 18th, and this issue remains unresolved.

Since April 16th, every single job has been consistently hanging and getting stuck in a queued state every night between 8–9 PM ET. This has been happening without exception for over two months.
pat_ren
Service Provider
Posts: 126
Liked: 27 times
Joined: Jan 02, 2024 9:13 am
Full Name: Pat
Contact:

Re: Stalling Jobs

Post by pat_ren »

anecdotal evidence but I have found the issue is worse on the more recent versions than on non 8.1.1 versions.
I'm only having a few jobs a week with this problem but it's still a few too many given how long it's been since v8 released.

My case is still open too - Case #07708238
Polina
Veeam Software
Posts: 3551
Liked: 844 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Stalling Jobs

Post by Polina »

Hi All,

From what I see, the following cases have already received fixes, and I'd appreciate if you confirm if it finally helped to resolve the issue:
07674412
07708238

Case #07733395 is still in investigation.

Thanks!
t7MevELx0
Service Provider
Posts: 87
Liked: 12 times
Joined: Feb 06, 2024 6:55 pm
Contact:

Re: Stalling Jobs

Post by t7MevELx0 »

Case 07674412 is not fixed. That hotfix did nothing. I'm still having outages each night.
karsten123
Service Provider
Posts: 605
Liked: 155 times
Joined: Apr 03, 2019 6:53 am
Full Name: Karsten Meja
Contact:

Re: Stalling Jobs

Post by karsten123 »

where is v8.1.2 and is it fixing all the problems?
pat_ren
Service Provider
Posts: 126
Liked: 27 times
Joined: Jan 02, 2024 9:13 am
Full Name: Pat
Contact:

Re: Stalling Jobs

Post by pat_ren »

Polina wrote: Jun 19, 2025 4:59 pm Hi All,

From what I see, the following cases have already received fixes, and I'd appreciate if you confirm if it finally helped to resolve the issue:
07674412
07708238

Case #07733395 is still in investigation.

Thanks!
i was given a 'fix' that had nothing to do with the issue that's actually happening, just a random guess from support, my case is still with RnD team
pat_ren
Service Provider
Posts: 126
Liked: 27 times
Joined: Jan 02, 2024 9:13 am
Full Name: Pat
Contact:

Re: Stalling Jobs

Post by pat_ren »

Received another new fix from support this week, have installed it so will see if this helps or not and report back.
ReKe
Service Provider
Posts: 38
Liked: 4 times
Joined: Apr 16, 2023 6:16 pm
Contact:

Re: Stalling Jobs

Post by ReKe »

Happend to us even with version 8.1.2. Case # 07754730
mjr.epicfail
Veeam Legend
Posts: 531
Liked: 146 times
Joined: Apr 22, 2022 12:14 pm
Full Name: Danny de Heer
Contact:

Re: Stalling Jobs

Post by mjr.epicfail »

had an ESX host failure, HA rebooted everything ofcourse, however VBM was $^##$% and stalled all jobs.
While troubleshooting I noticed that in the nats server clients overview (Im's using the nats server tool to check traffic and client) some proxies were still online (When all proxies were offline for more than 10 minutes).
Maybe that info helps for the R&D folks.

Also the fix was shutdown all proxies and archiver server, stop nats-server and delete the jetstream cache.
This environment was scalable as in 1 server for Archiver service (Windows), 1 server for PSQL (linux), 1 server for NATS (linux), several linux proxies.
VMCE / Veeam Legend 2*
t7MevELx0
Service Provider
Posts: 87
Liked: 12 times
Joined: Feb 06, 2024 6:55 pm
Contact:

Re: Stalling Jobs

Post by t7MevELx0 »

Yes, this is continuing to happen to us as well on 8.1.2. This is an extremely serious issue with the software.
Polina
Veeam Software
Posts: 3551
Liked: 844 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Stalling Jobs

Post by Polina »

@mjr.epicfail
Are you experiencing this with 8.1.2? If so, can you provide me with your support case ID?

@t7MevELx0
Have you already reported it to our support? May I know your case ID?
mjr.epicfail
Veeam Legend
Posts: 531
Liked: 146 times
Joined: Apr 22, 2022 12:14 pm
Full Name: Danny de Heer
Contact:

Re: Stalling Jobs

Post by mjr.epicfail »

Hi @Polina, (too bad that doesn't work for non Veeam employees)

Both clusters were running 8.1.1.159 P20250328.
NATS version: NATS server version 2.10.18

I didn't create a case for this as I knew what was happening and how to fix it.

But to be honest, in this case it did look like a partial NATS issue, as the NATS server still thought there were clients online while the servers themselves were powered off.
VMCE / Veeam Legend 2*
t7MevELx0
Service Provider
Posts: 87
Liked: 12 times
Joined: Feb 06, 2024 6:55 pm
Contact:

Re: Stalling Jobs

Post by t7MevELx0 » 1 person likes this post

Polina wrote: Jul 14, 2025 1:21 pm @mjr.epicfail
Are you experiencing this with 8.1.2? If so, can you provide me with your support case ID?

@t7MevELx0
Have you already reported it to our support? May I know your case ID?
Case #07762356
Polina
Veeam Software
Posts: 3551
Liked: 844 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Stalling Jobs

Post by Polina »

@mjr.epicfail

I'd recommend you first upgrading your Veeam installation to the latest 8.1.2.180, and NATS to 2.10.28 (see KB4742).
If the issue persists, the best would be open a case with our support team, and notify me as well.

Thanks!
mjr.epicfail
Veeam Legend
Posts: 531
Liked: 146 times
Joined: Apr 22, 2022 12:14 pm
Full Name: Danny de Heer
Contact:

Re: Stalling Jobs

Post by mjr.epicfail » 1 person likes this post

Hi Polina,

Ill plan this for the next maintenance cycle.
Thanks!
VMCE / Veeam Legend 2*
mjr.epicfail
Veeam Legend
Posts: 531
Liked: 146 times
Joined: Apr 22, 2022 12:14 pm
Full Name: Danny de Heer
Contact:

Re: Stalling Jobs

Post by mjr.epicfail »

Hi Polina,

I just reviewed the release notes for NATS Server version 2.10.28. It mentions that a regression was introduced in that version, which was subsequently fixed in 2.10.29. The recommendation is to install 2.10.29 instead.

Additionally, I noticed that version 2.11.3 is now available.

Is there a specific reason why we're installing version 2.10.28?

Best regards!
VMCE / Veeam Legend 2*
Polina
Veeam Software
Posts: 3551
Liked: 844 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Stalling Jobs

Post by Polina »

Hi Danny,

The issue found in 2.10.28 applies to a specific configuration which is not used in VB365 and has no impact on our product. This version remains the latest tested and approved by our QA.

Thanks!
t7MevELx0
Service Provider
Posts: 87
Liked: 12 times
Joined: Feb 06, 2024 6:55 pm
Contact:

Re: Stalling Jobs

Post by t7MevELx0 »

It doesn't matter - NATS 2.10.28 is installed and our 8.1.2 environment still stalls and then completely hangs.

8.1.1.2200 was a lot more stable. I regret upgrading.
Polina
Veeam Software
Posts: 3551
Liked: 844 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Stalling Jobs

Post by Polina »

Our tests show it differently, and that's why we would like to hear from you and investigate should issues continue on this newer version.
t7MevELx0
Service Provider
Posts: 87
Liked: 12 times
Joined: Feb 06, 2024 6:55 pm
Contact:

Re: Stalling Jobs

Post by t7MevELx0 »

This isn't a test environment. It's a massive production environment with real data. Your testing must be extremely limited.
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 4 guests