offload job is stuck - and blocks offloading

[mcz]

Hi everyone,

this is the second time within some months that my offload job was stuck since many hours and that's why the whole offloading got blocked. I remember the first time I only realized it after a vacation, so the offload/copy job didn't run for one week which is quite bad! Today I observed the same situation again, a "stuck" offload job since 19 hours waiting for the index to be prepared. I mean the solution is quite simple: Kill the associated process and the next run will be successful, but I wonder why there's no timeout set so that it would fail by 'itself'.

Maybe there's a timeout but it takes much longer to get hit or should it fail after a while but doesn't due to a bug?

Thanks for the clarification!

[veremin]

The described situation does not look expected. So may I ask you, Michael, to create a support ticket and provide its number here? This way I can ask Q&A team to take a closer look at issue experienced. Thanks!

[mcz]

Hi Vladimir,

case #04793305

let's hunt this bug!

[veremin]

Thanks, passed the information to the QA team. Will let you know, once I have more information.

[mcz]

Oh wow, thanks for this very quick reaction, thanks Vladimir!

[veremin]

You're welcome. The case is already in QA team hands, so let's wait until they spot the root cause. Thanks!

[veremin]

Michael, we have a further question for you - can you reproduce the stuck process issue, collect dump of VeeamAgent.exe process (or ask support engineer to do so) and attach it to the case? Thanks!

[mcz]

Vladimir, I would if I could. I don't know when it gets stuck and it could take a while until we'll see it again... So we'd have to wait. Any other ideas how we could dig deeper?

[veremin]

Unfortunately, the debug logs haven't shown us much - just some strange freeze during certificate retrieval operation that happens only for a particular VM.

We will double check the source code to see what might be going wrong with the command, but the chances here are low.

So, it will be helpful, if next time when the issue shows up you collect the process dump and provide it within support ticket.

Thanks!

[mcz]

Ok Vladimir, I'll be able to catch it one day, I'm sure. But please tell me some details on collection the dump: Is it enough to create the dump from the taskmanager (on the details tab)? And why dumps of the VeeamAgent and not on the veeam.backup.manager process? There might be many veeamagent-processes running, so which one would be the right one?

Thanks!

[veremin]

Is it enough to create the dump from the taskmanager (on the details tab)?

Correct.

There might be many veeamagent-processes running, so which one would be the right one?

- identify the stuck offload task in backup server
- identify its name
- open machine running backup server (or machine running gateway server, if specified in object storage repository settings)
- find debug log:

Code: Select all

%programdata%\Veeam\Backup\SOBR_Offload_%SOBR_NAME%\%TASK_NAME%\%TASK_NAME%.Repair.Gate.log

- parse it to catch similar string:

Code: Select all

[04.05.2021 17:34:21.221] <   504> cli      | Network.RetrieveSslCertificate

- after it there should be multiple strings like:

Code: Select all

[04.05.2021 17:34:21.221] <   504> net      | Retrieving certificate for eu-central-1.wasabisys.com:443
[04.05.2021 17:44:18.827] < 23980> cli      | Number of sessions: 3. Interval: 599 sec.

- scroll up or down for the first string that has PID:

Code: Select all

[04.05.2021 17:34:18.799] < 25628>          |   PID: 22796

- go to task manager and create a dump for VeeamAgent process with the same PID

If this sounds complex, then just open a ticket with our support team as soon as the offload process gets stuck - everything else will be done by support engineer.

Thanks!

[mcz]

Thanks Vladimir, a very good cookbook, I'll create the dump when the time comes

[mcz]

Hi Vladimir,

just for your information, this bug didn't occur again. Maybe it was due to a 3rd party (like antivirus or something like that), who knows...

[veremin]

Thank you, Michael, for finding the time to update the thread. Good to know you are up and running now, but if something happens, you can always ask for assistance here