Job Hangs before 'Initializing Storage' Without Timeout - Bug?

[sykerzner]

Hi,

One of our clients has a VBR server, with backups Copy jobs across MetroEth to another site. On that other site we have a NAS and a PC which acts as Gateway for it.

Every so often, we find that one of the jobs will throw an error like this

11/6/2022 6:01:44 PM :: Processing XXXXX Error: Reconnectable protocol device was closed.
Failed to upload disk. Skipped arguments: [XXXX-flat.vmdk];
Agent failed to process method {DataTransfer.SyncDisk}.

(Because the Copy is hung, local backups don't work either, because the first backup file is locked)(It usually the same job, but not always).

I was working with support Ticket 05661863, and they had some advice, so that's not my real concern.

My real concern, is then when the jobs gets interrupted, it looks like it then re-tries, and the GUI shows that its running, but nothing is actually happening. It kind of sits in Retry (2 of 5) at 'VM size: XX TB', and it does not get to 'Initializing Storage'. We’ve left it for days with no progress happening.

To me it sounds like there is some bug in operation on either side. I get that a job may fail, and retry but some process in Veeam seems to be stuck waiting, and can't initiate the next step or reply to the next step.

[HannesK]

Hello,

We’ve left it for days with no progress happening.

if it hangs longer than OrphanedAgentKillTimeout / HangedAgentKillTimeout, that sounds suspicious, yes... support should investigate further. I will talk to support.

Best regards,
Hannes

[HannesK]

Hello,
according to the logs, it seems to be a unique situation. We set up a monitoring for a specific error in your log files and will see, whether we see more cases.

If you see the same issue again, please open a new case and post the case number also here, so that we can continue investigations.

Best regards,
Hannes

[sykerzner]

Thank you Hannes, I will.

[sykerzner]

Hi Hannes,

The issue happened again yesterday and today. Same symptoms

- The remote job gets stuck on the first retry right before Initializing Storage.
- (As a side effect, the local job can't run because the points are locked).
- We can't cancel the remote job, but rebooting the Mount server on the remote side kills the job.

Here is the Veeam ticket:
Case #05776698
Remote backup job hangs without failing

[HannesK]

Hello,
thank you. I talked to support and will see, what we can do here.

Best regards,
Hannes