Stalling Jobs

[keironbell]

@pat_ren do you have an example of your custom script? We are still having this issue every few days and I'd been keen to give it a shot to see if we can see some improvements too.

Its becoming a full time job just watching this one server at the minute!

[Polina]

@pat_ren

I see that your case 07619220 is closed now. Is your issue resolved? or have you opened another case?

[aeph]

@All I got a private hotfix that works now!

My version is 8.1.0.4132

[keironbell]

We have just received this hotfix too, we will be applying it this morning so hopefully we will see some improvement too!

[raftbone]

Hopefully this fix will be released officially soon. Due to our licensing model I always have to opentickets with our partnerinstead of directly with Veeam.

This always takes time to get such fixes

[aeph]

Update:

Since two hours i am getting a lot of job failures.

Jobs are failing with:

Failed - nothing to process (unable to cancel retention session for the repository)

I don't know why, I've never seen that error before. I hope that this error is no longer related to the hotfix.

I opened a support case (07655307).

[keironbell]

I did see similar failures, annoying had to do the normal stop services, stop nats, recreate streams folder - then a couple of reboots on top of that, then things started run successfully. I only done a small few jobs at a time. I'll double check what it's like tomorrow morning after scheduled jobs kick in tonight.

[pat_ren]

@pat_ren

I see that your case 07619220 is closed now. Is your issue resolved? or have you opened another case?

Hi Polina, my original case got closed but I made another one now, updated case is Case #07632207
I'll ask about the private fix others have received and see if it's appropriate for me, thanks

[keironbell]

I don't want to tempt fate just yet, but after applying the fix, I came in this morning to what would normally be 140+ jobs stalled, I have actually come in to 30 actively running jobs with a lot of successful backups during the night. This is after applying the HotFix (also with the additional reboots/stopping and starting of services).

Looking good so far, but we will continue to monitor.

[aeph]

@keironbell can confirm this one. As posted in my previous comment i had some issues yesterday but i was able to fix them by stoping/ deleting nats folder and rebooting. My large jobs are running / performing. Hopefully this was just a one-time thing.

[raftbone]

Thanks for keeping us all up to date

Hopefully this will be released soon public.

[pat_ren]

@keironbell can confirm this one. As posted in my previous comment i had some issues yesterday but i was able to fix them by stoping/ deleting nats folder and rebooting. My large jobs are running / performing. Hopefully this was just a one-time thing.

I would say any jobs getting stuck that require deleting the nats folder there's still an issue, even if performance is improved that's a positive change, but something fundamental is broken when the fix to so many issues is to stop every service and delete that nats folder.

[raftbone]

Just installed the new version and after deleting the jetstream folder and rebooting the linux proxies after the upgrade, all jobs seem to run fine so far.

What I can see is a massive speed improvement, especially on copy jobs. Those copy jobs where sometimes only running with 1 - 10 MB/s, now they are constantly running at 100 MB/s and are really using our 10 Gbit internet connection

[t7MevELx0]

@raftbone Have you tested restore speeds? We are seeing terrible restore performance on 8.1 and 8.1.1. I am currently running a backup on a 7.1 test system to see how far back the issue goes. I can duplicate the extremely poor restore speeds using any storage.

I'm glad the backup jobs and copy jobs are working. I'm still testing 8.1.1 before I roll it out into production.

[Polina]

@t7MevELx0

Are you experiencing any issues with restores, other than a mailbox export to PST?

[t7MevELx0]

@Polina

Yes, restore speeds have been consistently poor across every version of the product I’ve tested. I’m surprised this hasn’t been addressed sooner. We started looking into this deeply when a client needed to download a large amount of Exchange data and couldn’t do so effectively. Even using the Explorer to save OneDrive data in version 7.1, I’m only seeing speeds around 50 Mbps.

I've been testing older versions to see how long this issue has existed, and it appears the product has never delivered strong restore performance. In fact, downloading the same data directly from storage is about 12 times faster than restoring it through Veeam Backup for Microsoft 365.

I know the Explorer for Microsoft Exchange uses Outlook DLLs to export PST files. Do the other explorers also rely on Office components when exporting or restoring to local .ZIP files?

I started that other thread for this issue; someone else mentioned restoring from an NVME based storage system was showing poor performance as well.

[Polina]

It's only export to PST which requires Outlook, save/export for other workloads are performed differently.

[t7MevELx0]

@Polina

So far, restore performance has been poor for both Exchange and OneDrive. I'm currently testing version 7.1 to determine if this has always been the case or if the issue started with 8.x. It appears the restore performance has consistently been poor.

I have another thread and a support ticket open, and I’m hoping internal testing is underway on your end. The issue is easy to reproduce.

I'm still planning to test a mailbox restore versus a PST export to see if both experience the same transfer issue (start/stop/repeat).

[nicolassimondiddi]

Same here with the latest update, it gets slower and slower.
3 proxy with 3 public ip, rate limit at 1mbps and yet, it stalls with 503 error from microsoft.

It was just fine before, without any rate limit.

I will open a ticket and see I'm more lucky.

[Polina]

@nicolassimondiddi, please share you ticket ID here for the followup.

[juerg.hirschi]

Ticket No 07660767

we have a dead system since upgrading to 8.1.1.159 from 8.0.latest + private Hotfix for slow speed.

our jobs dont start doing anything anymore. the status shows job started, but it just sits there not doing anything at all.
can not stop the jobs. i have to kill the proxy task. stop/start or restart proxy service also doesnt work, timeout error service can not be stopped/restarted.
no backups for our customers anymore since last week.

waiting for help from support what went wrong with our update.

Jürg

[troll]

Hello Jürg.
Stop and disabled any other running jobs (if possible), stop the following services:
Veeam Backup for Microsoft 365
NATS Server

Clear the contents of the streams folder
%ProgamData%\Veeam\Backup365\nats\Jetstream\$G

Start the server(s) again.

[juerg.hirschi]

Hi Sascha
thanks, first thing we did, but this only clears the hanging jobs. the problem is back afterwards when the jobs start again.
a helpful engineer from veeam was on our system yesterday, we did plenty of tests and the case is now being analyzed.
waiting for new feedback.
thx
Jürg

[Erwin Linker]

Hi Jurg,

Are you using a proxy pool or dedicated nodes?
We had a similar problem and placed the jobs on a dedicated proxy node and the jobs start running again.

Regards,
Erwin

[juerg.hirschi]

Hi Erwin

we are using dedicated nodes. what we found out in the session with the Veeam engineer:
the job status shows "job started <date time>" then nothing happens. it does not start looking for available objects to backup.
the job log shows as last entry "payload found...." then nothing.

we are using blob storage on Azure for the repos.

when we changed the repo to the local disk of our veeam server the same job that is stuck on Blob storage worked just fine.
back to the Azure storage, job is stuck again.

no changes to the azure storage or veeam server except the upgrade to v8.1.1

Veeam engineering is looking deeper into it now.

cheers
Jürg

[bryanvaneeden]

We are in the same boat.

Eventhough we have a sub-optimal bucket system behind our environment. We have over 3 thousand jobs on the environment. Every now and then the environment hangs, jobs don't compleet or just simply hang at all with as a result hundreds of jobs running and hanging and doing nothing.

So far the only fix was to clear the NATS server cache and restart everything. On a environment with thousands of running (albeit stuck, or actually running) jobs this is simply unacceptable. We should be able to simply kill a certain message stream in the NATS server to have a more granular control mechanism on the environment. Essentially wiping the environment state is simply unacceptable in an environment as large as ours.

We are running the latest release (8.1.1.139 - pre GA release I suppose), using single proxies and using proxy pools.

[mjr.epicfail]

A lot of issues have been addressed in the latest version: 8.1.1.159
https://www.veeam.com/kb4711

[juerg.hirschi]

our issues described started only after we installed 8.1.1.159. before on 8.0.latest + 1 Hotfix we were fine.

[t7MevELx0]

I have copy jobs that still won't move data and it's been that way since January. This started with 8.1 and the issue is still with us in 8.1.1.

[mjr.epicfail]

Hi t7MevELx0,

Did you open a case for this? if so, whats the ID?