Using object storage as a backup target
Post Reply
mcz
Veeam Legend
Posts: 543
Liked: 92 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

offload job is stuck - and blocks offloading

Post by mcz »

Hi everyone,

this is the second time within some months that my offload job was stuck since many hours and that's why the whole offloading got blocked. I remember the first time I only realized it after a vacation, so the offload/copy job didn't run for one week which is quite bad! Today I observed the same situation again, a "stuck" offload job since 19 hours waiting for the index to be prepared. I mean the solution is quite simple: Kill the associated process and the next run will be successful, but I wonder why there's no timeout set so that it would fail by 'itself'.

Maybe there's a timeout but it takes much longer to get hit or should it fail after a while but doesn't due to a bug?

Thanks for the clarification!

veremin
Product Manager
Posts: 18968
Liked: 1921 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: offload job is stuck - and blocks offloading

Post by veremin »

The described situation does not look expected. So may I ask you, Michael, to create a support ticket and provide its number here? This way I can ask Q&A team to take a closer look at issue experienced. Thanks!

mcz
Veeam Legend
Posts: 543
Liked: 92 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload job is stuck - and blocks offloading

Post by mcz »

Hi Vladimir,

case #04793305

let's hunt this bug!

veremin
Product Manager
Posts: 18968
Liked: 1921 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: offload job is stuck - and blocks offloading

Post by veremin »

Thanks, passed the information to the QA team. Will let you know, once I have more information.

mcz
Veeam Legend
Posts: 543
Liked: 92 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload job is stuck - and blocks offloading

Post by mcz » 1 person likes this post

Oh wow, thanks for this very quick reaction, thanks Vladimir!

veremin
Product Manager
Posts: 18968
Liked: 1921 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: offload job is stuck - and blocks offloading

Post by veremin »

You're welcome. The case is already in QA team hands, so let's wait until they spot the root cause. Thanks!

veremin
Product Manager
Posts: 18968
Liked: 1921 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: offload job is stuck - and blocks offloading

Post by veremin »

Michael, we have a further question for you - can you reproduce the stuck process issue, collect dump of VeeamAgent.exe process (or ask support engineer to do so) and attach it to the case? Thanks!

mcz
Veeam Legend
Posts: 543
Liked: 92 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload job is stuck - and blocks offloading

Post by mcz »

Vladimir, I would if I could. I don't know when it gets stuck and it could take a while until we'll see it again... So we'd have to wait. Any other ideas how we could dig deeper?

veremin
Product Manager
Posts: 18968
Liked: 1921 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: offload job is stuck - and blocks offloading

Post by veremin »

Unfortunately, the debug logs haven't shown us much - just some strange freeze during certificate retrieval operation that happens only for a particular VM.

We will double check the source code to see what might be going wrong with the command, but the chances here are low.

So, it will be helpful, if next time when the issue shows up you collect the process dump and provide it within support ticket.

Thanks!

mcz
Veeam Legend
Posts: 543
Liked: 92 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload job is stuck - and blocks offloading

Post by mcz »

Ok Vladimir, I'll be able to catch it one day, I'm sure. But please tell me some details on collection the dump: Is it enough to create the dump from the taskmanager (on the details tab)? And why dumps of the VeeamAgent and not on the veeam.backup.manager process? There might be many veeamagent-processes running, so which one would be the right one?

Thanks!

veremin
Product Manager
Posts: 18968
Liked: 1921 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: offload job is stuck - and blocks offloading

Post by veremin » 1 person likes this post

Is it enough to create the dump from the taskmanager (on the details tab)?
Correct.
There might be many veeamagent-processes running, so which one would be the right one?
- identify the stuck offload task in backup server
- identify its name
- open machine running backup server (or machine running gateway server, if specified in object storage repository settings)
- find debug log:

Code: Select all

%programdata%\Veeam\Backup\SOBR_Offload_%SOBR_NAME%\%TASK_NAME%\%TASK_NAME%.Repair.Gate.log
- parse it to catch similar string:

Code: Select all

[04.05.2021 17:34:21.221] <   504> cli      | Network.RetrieveSslCertificate
- after it there should be multiple strings like:

Code: Select all

[04.05.2021 17:34:21.221] <   504> net      | Retrieving certificate for eu-central-1.wasabisys.com:443
[04.05.2021 17:44:18.827] < 23980> cli      | Number of sessions: 3. Interval: 599 sec.
- scroll up or down for the first string that has PID:

Code: Select all

[04.05.2021 17:34:18.799] < 25628>          |   PID: 22796
- go to task manager and create a dump for VeeamAgent process with the same PID

If this sounds complex, then just open a ticket with our support team as soon as the offload process gets stuck - everything else will be done by support engineer.

Thanks!

mcz
Veeam Legend
Posts: 543
Liked: 92 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload job is stuck - and blocks offloading

Post by mcz » 1 person likes this post

Thanks Vladimir, a very good cookbook, I'll create the dump when the time comes ;)

mcz
Veeam Legend
Posts: 543
Liked: 92 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload job is stuck - and blocks offloading

Post by mcz » 2 people like this post

Hi Vladimir,

just for your information, this bug didn't occur again. Maybe it was due to a 3rd party (like antivirus or something like that), who knows...

veremin
Product Manager
Posts: 18968
Liked: 1921 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: offload job is stuck - and blocks offloading

Post by veremin » 1 person likes this post

Thank you, Michael, for finding the time to update the thread. Good to know you are up and running now, but if something happens, you can always ask for assistance here ;)

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests