Can't start any job since 8.0.2.200 patch

[kristofpoppe]

After we've applied the latest patch (which should fix issues with datastore upgrade) we can't start any job since then.
Saw this due to the fact last night no job had been executed. we're running 8.0.2.200 P20240910)

The job is started, but then after a while we get the message "Job was unable to start within the configured timeout"

https://imgur.com/oC2sJ62


https://imgur.com/eNr64l7


All jobs then have the status 'Queued'

I assume this is related to a job which is stalled for 300h, Case #07410790

The Veeam.Archive.Proxy log gives the following entry, which I presume hold the other jobs to be started / queued...

Code: Select all

18.09.2024 10:48:09.728]    350 (1) Warning: Double message processing was detected. Message stream sequence: 93183 (key: UserMailboxResolvedEvent_17d15cf3-af0d-4f1f-8db6-518949af29c7_ab5afe5d-a68d-4d75-9e79-10d5b0b45491). Processed sequence top: 93187. Processing skipped
[18.09.2024 10:48:09.728]    350 (1) Warning: Double message processing was detected. Message stream sequence: 93184 (key: UserMailboxResolvedEvent_17d15cf3-af0d-4f1f-8db6-518949af29c7_2b66ff2b-507e-4125-81be-d5290dd795ab). Processed sequence top: 93187. Processing skipped
[18.09.2024 10:48:09.728]    350 (1) Warning: Double message processing was detected. Message stream sequence: 93185 (key: UserMailboxResolvedEvent_17d15cf3-af0d-4f1f-8db6-518949af29c7_ba914a6a-d577-41ea-8854-e79a32488505). Processed sequence top: 93187. Processing skipped

[Mildur]

Hi Kristof

I'm sorry to hear about the issues.
I can see that your case, along with another similar case, has been escalated to our QA team.
Please allow our R&D team some time to review the case.

We will get back to you when we have more information to share.

Best,
Fabian

PS: I edited your post to display the pictures directly.

[kristofpoppe]

Hi Fabian, thanks for the reply. I'm available for further logs and remote sessions.
Also a big thank you to show the images. I've learned something today.

[Mildur]

Hi Kristof

Thank you.
I just reached out to QA for an update.

Best,
Fabian

[Mildur]

Hi Kristof

QA has confirmed that research is in progress and that a fix will be developed. Our support team will inform you once it's ready.
I will also try to keep you updated through this topic if I receive any additional information.

Best,
Fabian

[kristofpoppe]

Thanks for the update Fabian, much appreciated !

[Mildur]

Hi Kristof

I can see that our support engineer reached out to you with a workaround until the fix is ready.
Please let us know if that helps in your case as well.

Best,
Fabian

[Erwin Linker]

Hi,

What is the work around? I have some jobs with the state Queued and can't stop the job trough the GUI.

[Mildur]

Hi Erwin

Please reach out to our customer support team. Our support team will share with you the workaround (if it's applicable to your situation) and also put you on the list for the hot fix.

Don't forget to share the case number with me.

Thank you,
Fabian

[Erwin Linker]

Hi Fabian,

I created a case:
Case # 07430421 - Jobs looks to be stuck

[Mildur]

Thank you, Erwin

Best,
Fabian

[kristofpoppe]

I can confirm the workaround worked until now.

If you're facing issues, open a support ticket as mentioned by Fabian to let them check if the workaround applies to your case.

We've gradually added jobs to control load on NATS server and now all jobs were processed.

[Mildur]

Thank you for the update, Kristof.
I'm glad the workaround solved the situation.

Best,
Fabian

[PhoenixMan]

Hello,

I am also experiencing this issue and have created a ticket : Case #07422717. Is there a way I can get the workaround?

[mjr.epicfail]

Hello,

I am also experiencing this issue and have created a ticket : Case #07422717. Is there a way I can get the workaround?

All workarounds / hotfixes are supplied by support to check if you have the same root cause.

[Mildur]

Hi Drake

Yes, Danny is right.
Our support team will provide you with the workaround and later with the fix if they can confirm that you are affected by the same situation.

Best,
Fabian

[lungtendo]

I upgrade to 8.0.2.200 yesterday and found this morning that this happened to the jobs that are assigned to a proxy. My workaround (not from Veeam) was to disable the firewall. I then logged a case asking if the port requirements have changed, but I was pointed to the documentation. Also, I just found out that those jobs have lost access to the backups as I can't get the explore option on right click. I've logged another case and I'm waiting for a response.

[david.domask]

Hi Lungten, welcome to the forums and sorry to hear about the challenges. Can you please share your case number for the current case? Thanks

[lungtendo]

Hi David, thanks. I've logged two cases, one for network port related issue (07436087) and the other for losing access to previous backups (07436260).

[Mildur]

Hello Lungten

Your cases are not related to the issue discussed in this topic. If you want to discuss those issues, I can move your comment to a dedicated topic.
If you see jobs hanging and not being able to start, please open a new case for this specific issue.

Best,
Fabian

[lungtendo]

Hi Fabian, thanks, but no need to create a new discussion. I only commented becuase I had the same "Job was unable to start within the configured timeout" error.

[Mildur]

Hi guys

We have released our first cumulative patch (P20241008 - 8.0.3.1044), which should solve the issue discussed in this topic.

Veeam Backup for Microsoft 365 v8 - P20241008 (8.0.3.1044)
https://www.veeam.com/kb4656

Resolved Issues
Under certain circumstances, a job will not stop, even after a reboot of the services or server.

Best,
Fabian

[lungtendo]

Hi Fabian, I was provided a special patch and my VBO version now is 8.0.2.501 P20240919. Should I be able to apply the latest cumulative patch on top of it? My case has been closed since the problem was resolved and I don't want to log another case just to ask this. Thank you.

[Mildur]

Hello Lungten

Yes, updating to the most recent patch is possible from your version.
The fix for your case 07436260 (Restore Points not visible) is included in version 8.0.3.1044 (P20241008).

Best,
Fabian

[oflores]

Hello, I am currently having the same problem in version 8.0.2.20 P20241111
When I try to run backup tasks, it throws the error in the image and then does nothing, not to mention some other errors in the repositories added to the case
Case #07526794

https://imgur.com/a/HTYieqi

[mjr.epicfail]

You are 3 versions behind, try update 8.0.5

[Mildur]

Hi Oscar

Danny is correct. Please update to the newest build of Veeam Backup for Microsoft 365 v8.0: https://www.veeam.com/kb4106

Best,
Fabian

[scojohn]

Hi,

I am seeing these same symptoms. Started in 8.0.5.20_20241112 and an upgrade to 8.1.0.305 did not resolve. I have Case #07581552 open however looking for the QA recommended workaround to clear up the issue.

Thanks in advance.
Scott

[Polina]

Hi Scott,

Did you have a chance to clean up NATS streams as support engineers suggested?

Thanks!

[scojohn]

Hi Polina,

Thanks so much!

That did the trick, I am now able to perform backups and restores.

Thanks.
Scott