Jobs stalling with VBO365 8.0.2.200

[Seve CH]

Hello

We put into production a new VBO365 installation with the version 8.0.2.159 two weeks ago. We were experiencing jobs stalling: They just stopped transferring items (no CPU usage, no disk, no network usage), but the jobs were shown as running.
A reboot fixed the problem for some time (backup resumed), until the jobs got stuck again.

We patched to the 8.0.2.200 version and hoped that the new patch would fix that problem, but it seems that the jobs still stalls after some time running.

Now we can see a new phenomenon: Copy jobs (Mail items) fail with "Unable to cancel retention session for the repository (repository ID: a63f2843-523a-41f3-9ee7-f16d1ec5bc34, session ID: beab524a-8e82-4f23-84d7-1afbab00ce7e)." and "Nothing to process".
The repo ID is the one used as target by the backup copy job (S3-Compatible, not versioned). The session ID is not a backup Job or something I can find via PowerShell. I assume that ID is the stuck retention job's ID.

Is that a known problem? Is it related with this one? veeam-backup-for-microsoft-365-f47/can- ... 95551.html

My colleague will open a support ticket during the day. I will put the ticket Nr. here as soon as I have it.

Best regards
Seve

[Mildur]

Hi Seve

This forum is not intended as an alternative place to get support for technical issues (as stated in the red letters when you created this topic). It's best to start your investigation with our customer support team.
Please provide us with the case number after your colleague has opened the case.

I cannot confirm if it's related to the other issue, logs are required for confirmation. But you can ask your support engineer to check within the logs if it's related to the case in the other topic.

Best,
Fabian

[Seve CH]

Hi Fabian

Support Case Nr: #07432392

Best regards
Seve

[Mildur]

Hi Seve

Thank you.
I have replaced your contract number with the case id 07432392.
Let's see what our support engineer will find out from the logs.


Best,
Fabian

[Seve CH]

Thanks Fabian. I've put the information I've got from my colleague and didn't realize that it was our contract's number.

While trying to debug the problem this morning, I was unable to find via PowerShell the involved job based on the session ID in the error message. I assume that's the retention job's session ID. In VBR it is possible to see the retention jobs, health checks, etc. which helps to debug possible problems or plan resources for them. Is it possible to see these infrastructure/maintenance/health-check jobs in VBO via PowerShell or any other method?

Best regards
Seve

[andy999]

Have had issues with jobs stalling as well. I submitted a case a week ago but they haven't been very responsive. Case# 07423035

[pat_ren]

Not sure if this is the same issue or not, but we've also been having a lot of problems with jobs since upgrading to v8.
This seems to randomly affect different jobs each time the jobs run, some work and some don't, and next time the jobs run the dice get rolled again.

The error is:
Nothing to process.
Timeout has expired: 900000 milliseconds.
Job failed

I've also logged a support case - case# 07436922

[aeph]

We have the same problem. In our case, it only affects copy jobs. Copy jobs are started and run for 20 hours, but only process approx. 100 MB, for example. The jobs do not report any real errors, they simply do nothing more.

We are in contact with Veeam support, except for emptying folders (temp appdata folder for nats) no solution has been presented to us yet. However, it is probably a problem that has been reported by several customers.

Unfortunately, this is very inconvenient for us because we currently have to disable the copy jobs. We are urgently looking for a solution! Case id: 07438124

[Seve CH]

Hi

We managed to get the copy jobs done increasing the proxies' pool size of PostgreSQL connections from 20 (default) to 300 connections in the %programdata%\Veeam\Backup365\proxy.xml file (maxpoolsize=300 instead of maxpoolsize=20).

This is a temporary workaround, I've been told by my colleagues that the value is reset back to 20 after some time or service restart, but at least it seemed to help until it finished processing the backlog: No more errors about pool exhausted (20 connections) and it pumped several TBs between S3-compatible repos for hours with constant speed (several Gbit/s) instead of peaks every 10 minutes (the time the connections require to time out). So we expect that, once the pool exhaustion problem is definitely fixed, the performance of VBO365 copy jobs should be quite good.

We had another problem with a job stuck in queued state, but I'm not sharing the fix (NATS) because that one requires support to oversee it. That job is currently running and has transferred 110GB of mails in 2 hours so far, which is what we can expect for that job based on the number of users (low parallelism) and MS Exchange Online throttling policies in place.

Best regards
Seve

[edh]

Job stale is a current bug no?? Are there a private fix for that? We done the workarround of delete G$ of nat server due work stuck.. after open a ticket support suggested that solution...

Anyone on Veeam can tell us about new hotfix ETA?

[Mildur]

Hi all

07436922 from pr_osit - Case was closed with "Issue resolved by itself", assuming Antivirus was interfering with the job sessions
07423035 from andy999 - Case was closed by the customer
07432392 from Seve CH - The jobs could be stopped by the workaround (deleting NATS cache shared by our customer support team)

@edh
I am not certain if you are affected by the same issue as in case 07432392. I will leave that for our customer support team to confirm. If you are, there is a workaround that can be obtained from them.
Our R&D team is currently working on a fix, and we aim to include it in one of the upcoming cumulative patches. To get earlier access to the private fix once it's created, a support case will be required.

Best,
Fabian

[Seve CH]

Follow-up:
Since our full backup finished and the system now looks stable and/or with enough performance to cope with our deltas, we closed all support tickets. We are staying with 8.0.2.200 still some time until we are sure the new released patch is stable.