Comprehensive data protection for all workloads
Post Reply
Wayno
Lurker
Posts: 2
Liked: never
Joined: Jul 25, 2014 2:25 pm
Full Name: Wayne Moore
Contact:

Job "hangs" for 40 minutes at 0%

Post by Wayno » Mar 16, 2017 4:37 am

We have a scenario where 1 job (out of about 10 scheduled) will hang for 40 minutes at "Processing <VM name>". Then after 40 minutes, the job starts working - snapshot, perform backup, remove snapshot, indexing all processes in about 4 minutes. This happens every single time. Other jobs on the same Veeam server backing up the same infrastructure do not do this. During the time we have been working on this issue, we have upgraded Veeam, vCenter and ESXi to the latest versions, so we are fully patched. There are no tasks running against the VM or host during this pause.

Veeam support case: 2080676

The one "quirk" of the VM being backed up is that it has 16 VMDKs attached to it. While I don't think this is good practice, I haven't seen anything anywhere to suggest this is the issue. We are using NBD for transport anyway. Trawling the Veeam logs, the only thing I can see happening during the pause is this (in the Agent log):

[16.03.2017 13:11:05] < 2756> cli| Number of sessions: 5. Interval: 299 sec.
[16.03.2017 13:16:05] < 2756> cli| Number of sessions: 5. Interval: 599 sec.
[16.03.2017 13:21:05] < 2756> cli| Number of sessions: 5. Interval: 899 sec.
[16.03.2017 13:25:59] < 8560> cli| WARN|MTA invoke thread : timed out for backup events. Wait cycle will be resumed for '336' hours.
[16.03.2017 13:26:05] < 2756> cli| Number of sessions: 5. Interval: 1199 sec.
[16.03.2017 13:31:05] < 2756> cli| Number of sessions: 5. Interval: 1499 sec.
[16.03.2017 13:36:05] < 2756> cli| Number of sessions: 5. Interval: 1799 sec.
[16.03.2017 13:41:05] < 2756> cli| Number of sessions: 5. Interval: 2099 sec.
[16.03.2017 13:46:05] < 2756> cli| Number of sessions: 5. Interval: 2399 sec.
[16.03.2017 13:47:10] < 8372> srv| retrieved command: 127 (Invoke(127))

2399 seconds is 40 minutes, so this tallies but I don't know what it means or what it is counting against. There are no other jobs running (apart from a VM copy job which is currently waiting for its next window). The only thing Veeam support has come up with so far is they suggested that the Veeam proxy couldn't communicate with the VM over RPC but we tested and found that not to be the case. If this were the case then wouldn't the job just fail anyway?

Any suggestions?

Mike Resseler
Product Manager
Posts: 5740
Liked: 612 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Job "hangs" for 40 minutes at 0%

Post by Mike Resseler » Mar 16, 2017 5:54 am

Wayne,

First, welcome to the forums.

Unfortunately I don't have suggestions for you on this one, maybe others will chime in. But please keep working further with support in troubleshooting this. In case you feel it is not going anywhere, don't forget you can ask to escalate it

Cheers
Mike

Wayno
Lurker
Posts: 2
Liked: never
Joined: Jul 25, 2014 2:25 pm
Full Name: Wayne Moore
Contact:

Re: Job "hangs" for 40 minutes at 0%

Post by Wayno » Mar 16, 2017 6:00 am

Thanks Mike. I've spoken to support and done some more troubleshooting. Support thought that maybe in-guest VSS processing was the issue. Unfortunately this is a customer VM and I don't have in-guest access right now. So I experimented turning off app aware and it made no difference. So I turned off indexing and it made no difference.

However when I turned them both off, the job completed in 5 minutes. I have repeated this to confirm it. So I've sent some more logs off to support to see if they can pin it down.

mkretzer
Expert
Posts: 555
Liked: 124 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Job "hangs" for 40 minutes at 0%

Post by mkretzer » Mar 16, 2017 6:31 am

While it is "hanging" is you Veeam / Repository CPU load high? Sounds like our RPC issue which was fixed by a registry setting.

foggy
Veeam Software
Posts: 18278
Liked: 1565 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Job "hangs" for 40 minutes at 0%

Post by foggy » Mar 16, 2017 3:00 pm

Wayno wrote:We have a scenario where 1 job (out of about 10 scheduled) will hang for 40 minutes at "Processing <VM name>".
What particular operation it is stuck at (you can see it if you select the particular VM in the list to the left)?

Delo123
Expert
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Job "hangs" for 40 minutes at 0%

Post by Delo123 » Mar 16, 2017 3:38 pm

What registry setting would that be? We have seen this a couple of times but could never really fix it with veeam support, rebooting veeamserver/vcenter was workaround...

Post Reply

Who is online

Users browsing this forum: Erwin Linker, totoche and 53 guests