Job "hangs" for 40 minutes at 0%

[Wayno]

We have a scenario where 1 job (out of about 10 scheduled) will hang for 40 minutes at "Processing <VM name>". Then after 40 minutes, the job starts working - snapshot, perform backup, remove snapshot, indexing all processes in about 4 minutes. This happens every single time. Other jobs on the same Veeam server backing up the same infrastructure do not do this. During the time we have been working on this issue, we have upgraded Veeam, vCenter and ESXi to the latest versions, so we are fully patched. There are no tasks running against the VM or host during this pause.

Veeam support case: 2080676

The one "quirk" of the VM being backed up is that it has 16 VMDKs attached to it. While I don't think this is good practice, I haven't seen anything anywhere to suggest this is the issue. We are using NBD for transport anyway. Trawling the Veeam logs, the only thing I can see happening during the pause is this (in the Agent log):

[16.03.2017 13:11:05] < 2756> cli| Number of sessions: 5. Interval: 299 sec.
[16.03.2017 13:16:05] < 2756> cli| Number of sessions: 5. Interval: 599 sec.
[16.03.2017 13:21:05] < 2756> cli| Number of sessions: 5. Interval: 899 sec.
[16.03.2017 13:25:59] < 8560> cli| WARN|MTA invoke thread : timed out for backup events. Wait cycle will be resumed for '336' hours.
[16.03.2017 13:26:05] < 2756> cli| Number of sessions: 5. Interval: 1199 sec.
[16.03.2017 13:31:05] < 2756> cli| Number of sessions: 5. Interval: 1499 sec.
[16.03.2017 13:36:05] < 2756> cli| Number of sessions: 5. Interval: 1799 sec.
[16.03.2017 13:41:05] < 2756> cli| Number of sessions: 5. Interval: 2099 sec.
[16.03.2017 13:46:05] < 2756> cli| Number of sessions: 5. Interval: 2399 sec.
[16.03.2017 13:47:10] < 8372> srv| retrieved command: 127 (Invoke(127))

2399 seconds is 40 minutes, so this tallies but I don't know what it means or what it is counting against. There are no other jobs running (apart from a VM copy job which is currently waiting for its next window). The only thing Veeam support has come up with so far is they suggested that the Veeam proxy couldn't communicate with the VM over RPC but we tested and found that not to be the case. If this were the case then wouldn't the job just fail anyway?

Any suggestions?

[Mike Resseler]

Wayne,

First, welcome to the forums.

Unfortunately I don't have suggestions for you on this one, maybe others will chime in. But please keep working further with support in troubleshooting this. In case you feel it is not going anywhere, don't forget you can ask to escalate it

Cheers
Mike

[Wayno]

Thanks Mike. I've spoken to support and done some more troubleshooting. Support thought that maybe in-guest VSS processing was the issue. Unfortunately this is a customer VM and I don't have in-guest access right now. So I experimented turning off app aware and it made no difference. So I turned off indexing and it made no difference.

However when I turned them both off, the job completed in 5 minutes. I have repeated this to confirm it. So I've sent some more logs off to support to see if they can pin it down.

[mkretzer]

While it is "hanging" is you Veeam / Repository CPU load high? Sounds like our RPC issue which was fixed by a registry setting.

[foggy]

We have a scenario where 1 job (out of about 10 scheduled) will hang for 40 minutes at "Processing <VM name>".

What particular operation it is stuck at (you can see it if you select the particular VM in the list to the left)?

[Delo123]

What registry setting would that be? We have seen this a couple of times but could never really fix it with veeam support, rebooting veeamserver/vcenter was workaround...

[bolish]

Just reaching out to find out if anyone else has a solution to this topic. I know it's an old thread, but there was no resolution, and I am getting the same problem right now.

[Gostev]

Whatever the solution was in 2017, it is going to be irrelevant now as way too many things have changes in the job management architecture. So kindly open a support case to investigate your current issue. Thanks