Comprehensive data protection for all workloads
Post Reply
mplep
Service Provider
Posts: 62
Liked: never
Joined: Sep 16, 2009 7:50 pm
Contact:

v6. Killing a stuck job.

Post by mplep »

Hi,

I was going to contact support as I could not stop a replication job which had apparently frozen and I really needed to retry it ASAP. I know support will be VERY busy so I looked through the forums for assistance. I saw threads about killing processes on the Veeam server and restarting Veeam, but I came up with something else which overcame my issue and I thought it worth posting in case it helps anybody.

I had a replication set (new, not legacy) which appeared to have stopped in its tracks whilst all other jobs were running OK. This job started OK, but stopped on processing "Hard Disk 1" and didn't go any further after being left for a long while. It had processed about 19.7GB on the 30GB disk, but the processed and read values were not increasing after three hours. The Duration timer next to "Hard Disk 1" also wasn't increasing as you'd expect either, it had stopped at 2mins where usually it increases at regular increments. I could also see no further changes in Data fields occuring. The job in total had a Duration timer of 3 hours and this timer was increasing however. This was just an incremental run which usually completes in minutes and I didn't expect many changes to the CBT data. I had clicked "Stop Job" a few times and left that for a couple of hours without any change being noticed and the job not stopping.

So I disabled all other replications jobs (three) from the source site and waiting for running jobs to finish. I then restarted the "Veeam Backup Proxy Service" upon the Windows Proxy server at the source site (we currently have a single VM proxy with lots of resources). The stuck job then immediately changed to "completed with errors" and the action list showed "job has been terminated". I was then able to restart the affected replication job and it worked perfectly and quickly as I'd expect. All jobs are again enabled and running together again quite happily.

Mark
tgiphil
Influencer
Posts: 17
Liked: never
Joined: Nov 15, 2011 11:46 pm
Full Name: Phil Garcia
Contact:

Re: v6. Killing a stuck job.

Post by tgiphil »

Note: I had a similar issue with a backup job "freezing" middle way through (using CBT). Support basically had me stop and restart all the services to get the stop to stop. Hopefully, this is a super rare event until it can be fixed.
mplep
Service Provider
Posts: 62
Liked: never
Joined: Sep 16, 2009 7:50 pm
Contact:

Re: v6. Killing a stuck job.

Post by mplep »

Got the same thing again, but this is a seperate job with different source proxy and different source ESXi hosts. I've logged a call with support (5159867) and have left the job running (well stuck) until they advise. What I do notice is that although the last item in the Action list is "Hard Disk 1" with a low Duration (minutes whilst the total is hours), that last Action item is ticked (completed) but nothing new appears below it. The job stops processing on that VM and goes no further. Restarting the source proxy might fix (fail) it but I need a solution as its happened three times now. I only notice this problem when I review the running jobs, otherwise it just stays stuck forever and nobody knows.

Mark
chad_c
Lurker
Posts: 2
Liked: never
Joined: Nov 23, 2011 4:18 pm
Contact:

Re: v6. Killing a stuck job.

Post by chad_c »

I tried stopping and restarting all services, but I have two jobs (one Backup, one Replication) that continue to 'run'. I suppose I should phone in a support case as well?
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v6. Killing a stuck job.

Post by Gostev »

If the above does not help then yes.
cbachel
Novice
Posts: 4
Liked: never
Joined: May 06, 2011 3:11 pm
Contact:

Re: v6. Killing a stuck job.

Post by cbachel »

I've had the same problem of a replication job getting stuck every 2-3 days. The job runs continuously and successfully for 2 or 3 days then just stops processing. I've been dealing with this issue for over a month now since upgrading VBR 5.0 to version 6.0 and was wondering if anyone else came up with a solution to this? I have been working with support for the whole time, but they determined that the issue is environmental. Anyone have any ideas?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: v6. Killing a stuck job.

Post by Vitaliy S. »

I've seen similar cases where this issue was caused by misbehaving SQL Server. Does this issue occur periodically? What about other jobs? Are they also affected?
cbachel
Novice
Posts: 4
Liked: never
Joined: May 06, 2011 3:11 pm
Contact:

Re: v6. Killing a stuck job.

Post by cbachel »

I only have one replication job that is affected by this every 2-3 days. What fixed the issue in the other cases?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: v6. Killing a stuck job.

Post by Vitaliy S. »

Moving Veeam database to another SQL Server fixed the issue, though I'm not sure It is applicable to your case since only one job is affected by this.
cbachel
Novice
Posts: 4
Liked: never
Joined: May 06, 2011 3:11 pm
Contact:

Re: v6. Killing a stuck job.

Post by cbachel »

Do you have any advice to troubleshoot this issue? I have to constantly watch this job and stop the Veeam services every 2-3 days to keep VBR functioning right now.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: v6. Killing a stuck job.

Post by Vitaliy S. »

Did your support engineer tell you on what step your replication job stalls every time (based on the information from the debug logs)?
Amer
Novice
Posts: 6
Liked: never
Joined: Feb 02, 2011 8:15 pm
Full Name: Amir Abid
Contact:

VEEAM JOB Hang in Stopping status

Post by Amer »

[merged]

Hi there:
I had this issue with two jobs (Replica and backup), the state og jon on stopping for ever.
How can I stop that ? it seems more like SQL issue rather VEEAM to me, I can run other jobs no problem there.
It is getting realu urgent as I need to have successful backup for this machine.
Regards
kurt
Enthusiast
Posts: 71
Liked: 2 times
Joined: Jul 07, 2010 9:03 pm
Full Name: Robert
Contact:

Re: v6. Killing a stuck job.

Post by kurt »

I have the same issue with veeam 6.5.0.106. The backup job was successful except for 1 VM:

Unable to release guest. Error: Unfreeze error: [Freeze job already stopped.]
Error: Unfreeze error: [Freeze job already stopped.]

Normally a retry is successful but it was stuck at 0% on retry for hours. I tried to stop the job and now it's stuck at stopping. Tried restarting veeam services and it's still stuck at stopping.
ZachW
Enthusiast
Posts: 68
Liked: 10 times
Joined: Aug 02, 2011 6:09 pm
Full Name: Zach Weed
Contact:

Re: v6. Killing a stuck job.

Post by ZachW »

My recommendation would be to open a ticket with support and as Vitaliy said above, provide support with the logs from the most recent job run as this can point us down a valid route of troubleshooting.. Different tasks that we are stuck at can mean different things.
kurt
Enthusiast
Posts: 71
Liked: 2 times
Joined: Jul 07, 2010 9:03 pm
Full Name: Robert
Contact:

Re: v6. Killing a stuck job.

Post by kurt »

Well, I rebooted veeam and the problem vm. Job was no longer in stopping status. So I click retry now it's stuck at 0% again. Need to get this job going and don't have much time to contact support. Backup has been working fine on this VM for weeks until now.
ZachW
Enthusiast
Posts: 68
Liked: 10 times
Joined: Aug 02, 2011 6:09 pm
Full Name: Zach Weed
Contact:

Re: v6. Killing a stuck job.

Post by ZachW » 2 people like this post

Well one thing I can say is that while you don't have time to contact support it would probably be the most efficient and productive route to go down in solving this issue. In the case of issues that require troubleshooting it is very likely that you will be unable to find the answers that you are looking for within the forums, as this is an issue that will require log investigation. Just knowing that a job is stuck at 0% doesn't really say much as the next steps to go down depend entirely on what task the job is stuck on.
kurt
Enthusiast
Posts: 71
Liked: 2 times
Joined: Jul 07, 2010 9:03 pm
Full Name: Robert
Contact:

Re: v6. Killing a stuck job.

Post by kurt » 1 person likes this post

Looks like the repository was causing the issues. I rebooted my NAS repo and hit retry. The job is now running normally. Thanks
Post Reply

Who is online

Users browsing this forum: Mildur and 142 guests