Job "hangs" for 40 minutes at 0%

Availability for the Always-On Enterprise

Job "hangs" for 40 minutes at 0%

Veeam Logoby Wayno » Thu Mar 16, 2017 4:37 am

We have a scenario where 1 job (out of about 10 scheduled) will hang for 40 minutes at "Processing <VM name>". Then after 40 minutes, the job starts working - snapshot, perform backup, remove snapshot, indexing all processes in about 4 minutes. This happens every single time. Other jobs on the same Veeam server backing up the same infrastructure do not do this. During the time we have been working on this issue, we have upgraded Veeam, vCenter and ESXi to the latest versions, so we are fully patched. There are no tasks running against the VM or host during this pause.

Veeam support case: 2080676

The one "quirk" of the VM being backed up is that it has 16 VMDKs attached to it. While I don't think this is good practice, I haven't seen anything anywhere to suggest this is the issue. We are using NBD for transport anyway. Trawling the Veeam logs, the only thing I can see happening during the pause is this (in the Agent log):

[16.03.2017 13:11:05] < 2756> cli| Number of sessions: 5. Interval: 299 sec.
[16.03.2017 13:16:05] < 2756> cli| Number of sessions: 5. Interval: 599 sec.
[16.03.2017 13:21:05] < 2756> cli| Number of sessions: 5. Interval: 899 sec.
[16.03.2017 13:25:59] < 8560> cli| WARN|MTA invoke thread : timed out for backup events. Wait cycle will be resumed for '336' hours.
[16.03.2017 13:26:05] < 2756> cli| Number of sessions: 5. Interval: 1199 sec.
[16.03.2017 13:31:05] < 2756> cli| Number of sessions: 5. Interval: 1499 sec.
[16.03.2017 13:36:05] < 2756> cli| Number of sessions: 5. Interval: 1799 sec.
[16.03.2017 13:41:05] < 2756> cli| Number of sessions: 5. Interval: 2099 sec.
[16.03.2017 13:46:05] < 2756> cli| Number of sessions: 5. Interval: 2399 sec.
[16.03.2017 13:47:10] < 8372> srv| retrieved command: 127 (Invoke(127))

2399 seconds is 40 minutes, so this tallies but I don't know what it means or what it is counting against. There are no other jobs running (apart from a VM copy job which is currently waiting for its next window). The only thing Veeam support has come up with so far is they suggested that the Veeam proxy couldn't communicate with the VM over RPC but we tested and found that not to be the case. If this were the case then wouldn't the job just fail anyway?

Any suggestions?
Wayno
Lurker
 
Posts: 2
Liked: never
Joined: Fri Jul 25, 2014 2:25 pm
Full Name: Wayne Moore

Re: Job "hangs" for 40 minutes at 0%

Veeam Logoby Mike Resseler » Thu Mar 16, 2017 5:54 am

Wayne,

First, welcome to the forums.

Unfortunately I don't have suggestions for you on this one, maybe others will chime in. But please keep working further with support in troubleshooting this. In case you feel it is not going anywhere, don't forget you can ask to escalate it

Cheers
Mike
Mike Resseler
Veeam Software
 
Posts: 3098
Liked: 359 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Job "hangs" for 40 minutes at 0%

Veeam Logoby Wayno » Thu Mar 16, 2017 6:00 am

Thanks Mike. I've spoken to support and done some more troubleshooting. Support thought that maybe in-guest VSS processing was the issue. Unfortunately this is a customer VM and I don't have in-guest access right now. So I experimented turning off app aware and it made no difference. So I turned off indexing and it made no difference.

However when I turned them both off, the job completed in 5 minutes. I have repeated this to confirm it. So I've sent some more logs off to support to see if they can pin it down.
Wayno
Lurker
 
Posts: 2
Liked: never
Joined: Fri Jul 25, 2014 2:25 pm
Full Name: Wayne Moore

Re: Job "hangs" for 40 minutes at 0%

Veeam Logoby mkretzer » Thu Mar 16, 2017 6:31 am

While it is "hanging" is you Veeam / Repository CPU load high? Sounds like our RPC issue which was fixed by a registry setting.
mkretzer
Expert
 
Posts: 304
Liked: 67 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Job "hangs" for 40 minutes at 0%

Veeam Logoby foggy » Thu Mar 16, 2017 3:00 pm

Wayno wrote:We have a scenario where 1 job (out of about 10 scheduled) will hang for 40 minutes at "Processing <VM name>".

What particular operation it is stuck at (you can see it if you select the particular VM in the list to the left)?
foggy
Veeam Software
 
Posts: 14560
Liked: 1060 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Job "hangs" for 40 minutes at 0%

Veeam Logoby Delo123 » Thu Mar 16, 2017 3:38 pm

What registry setting would that be? We have seen this a couple of times but could never really fix it with veeam support, rebooting veeamserver/vcenter was workaround...
Delo123
Expert
 
Posts: 340
Liked: 94 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], JimmyO, jslic, UT2015 and 54 guests