Comprehensive data protection for all workloads
xray6969
Novice
Posts: 8
Liked: never
Joined: Mar 07, 2011 3:20 pm
Full Name: Raimonds Martinovs
Location: Belfast, United Kingdom

What is the right way to kill stalled backup job?

Post by xray6969 »

I have a backup job that has been running for a weekend and I wanted to stop that job. I right click it and select "Stop job". It gives a message "That this will leave backup in inconsistent state (not exact wording)" (I do not really understand - what exactly that means? Am I loosing backup of VM which was running at a time or whole session?). Job has been "running" in STOPPING state for last 12 hours. Now the question is how do you properly kill such job in a way that backup files are usable - that is next time I can re-run job (OK, I might loose whole last "killed" backup session). Or do I have to run Full backup after such kill?

Vitaliy S.
Product Manager
Posts: 25114
Liked: 2153 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: What is the right way to kill stalled backup job?

Post by Vitaliy S. »

Hello Raimonds,

You should open Task Manager on the backup server and end all VeeamAgent processes, this will stop your job. The message you see while stopping backup job means that the last restore point will not be consistent and you'll loose the latest backup for a VM running at that time, not the whole session.

There is no need to re-create backup job and trigger full backup pass, as subsequent job run will fix the previous state and will continue that backup chain.

Hope this helps.

xray6969
Novice
Posts: 8
Liked: never
Joined: Mar 07, 2011 3:20 pm
Full Name: Raimonds Martinovs
Location: Belfast, United Kingdom

Re: What is the right way to kill stalled backup job?

Post by xray6969 »

Killed all VeeamAgent.exe, but job did not stop (still in STOPPING state).

Vitaliy S.
Product Manager
Posts: 25114
Liked: 2153 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: What is the right way to kill stalled backup job?

Post by Vitaliy S. »

Trying re-opening Veeam backup console, it should update the job status. If it doesn't help, please contact our technical support team.

cby
Expert
Posts: 109
Liked: 6 times
Joined: Feb 24, 2009 5:02 pm
Contact:

Re: What is the right way to kill stalled backup job?

Post by cby »

While Veeam reports 'stopping' check the VC for snapshot deletion messages appearing to hang at 95%. Give it time to complete, the deletion can take a long time depending on VM size!

Also, for failed jobs check the snapshot manager for 'Consolidated Helper' snapshots. You can remove these and retrieve storage on your datastore. Quite often these types of snapshots are the reason for inconsistent looking storage capacities.

xray6969
Novice
Posts: 8
Liked: never
Joined: Mar 07, 2011 3:20 pm
Full Name: Raimonds Martinovs
Location: Belfast, United Kingdom

Re: What is the right way to kill stalled backup job?

Post by xray6969 »

No snapshot tasks even running in VC. Like I said it has been 12 hours on system not being stressed. And I am not concerned about failed jobs and snapshots - can handle those. All I wanted is safe way to kill veeam backup job no matter what is going wrong on my NAS, ESX, VC etc. If I just reboot/kill veeam backup server - will that be OK (cause my previous experience showed - if you do that backup files become locked and I could not restart backing up? So, it is not actually support question, it is veeam question in general - what happens if backup job crashes (being it - servers reboot, ESX server had memory fault, job was killed, killed by task manager, killed any other way, Windows BlueScreen etc.) What I am getting at - veeam have tested various scenarios when backup is OK (in their labs). But, there is no FAQ on what happens when backup hangs for instance. Do not get me wrong - I love veeam backup!

cby
Expert
Posts: 109
Liked: 6 times
Joined: Feb 24, 2009 5:02 pm
Contact:

Re: What is the right way to kill stalled backup job?

Post by cby »

Veeam support provided a SQL script and instructions to reset a job. I'm not sure if it only applies to FileLevelRestore locked state (i.e.the value of 'state' and/or 'overall_status')

JobsReset.sql:

Code: Select all

update [BSessions] set overall_status = 0 where overall_status <> 2
update [BJobs] set control = 0
update [ObjectsInBackups] set [state] = 0 where [state]=2 or [state]=3
1. Save JobsReset.sql file to local disk on Veeam server.
2. Make sure that there are no jobs running to avoid resetting their status accidentally.
3. Run the following command: sqlcmd -S COMPUTER\INSTANCE -d VeeamBackup -i PATHTOSCRIPT\JobsReset.sql
where COMPUTER is the name of computer with database, INSTANCE is the name of instance (VEEAM for default installation), PATHTOSCRIPT is the path to the JobsReset.sql script.

After running this, restart both the SQL Service for Veeam (VEEAM), and the Veeam Backup Scheduling Service. (in that order)

One of the Veeam product managers will put me right if this a bum steer!

Vitaliy S.
Product Manager
Posts: 25114
Liked: 2153 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: What is the right way to kill stalled backup job?

Post by Vitaliy S. »

If anything happens to the backup server while backup job is running, nothing should become unrecoverable. It's true that some files might be locked, but each and every situation has it's own timeout and fixing engine, which shouldn't prevent you from re-running backup/replication jobs.

Also I suggest that those kind of situations should be handled with our support team assistance, because you definitely want to get everything up and running as soon as possible.

As for this particular situation with stalled jobs, if killing VeeamAgents didn't help, then a script mentioned above should be a way to go.

MrTwistoff
Novice
Posts: 7
Liked: never
Joined: Mar 10, 2011 7:48 pm
Full Name: Brian L Shunamon
Contact:

Re: What is the right way to kill stalled backup job?

Post by MrTwistoff »

Having this issue on Veeam 5...and ran the script against the (remote database) and get the following errors:

Msg 207, Level 16, State 1, Line 2

Invalid column name 'overall_status'.

Msg 207, Level 16, State 1, Line 3

Invalid column name 'control'.


Any updates to that script that might be needed for it to work on 5?

Gostev
SVP, Product Management
Posts: 28654
Liked: 5183 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: What is the right way to kill stalled backup job?

Post by Gostev »

I believe that script is for v5. Please work with our support if you are having difficulties running it. Thanks.

Olaf
Lurker
Posts: 1
Liked: never
Joined: Dec 22, 2010 2:13 pm
Contact:

Re: What is the right way to kill stalled backup job?

Post by Olaf »

a few months ago I've had the problem that a VEEAM Backup job crashed. Every retry of that job fails (errormessage I can't remember exactly --> something with snapshot). No snapshots shown in VC
I found this link :
http://www.kendrickcoleman.com/index.ph ... qus_thread
The cold cloning trick worked fine for me --> look --> How to remove all these snapshots.

MrTwistoff
Novice
Posts: 7
Liked: never
Joined: Mar 10, 2011 7:48 pm
Full Name: Brian L Shunamon
Contact:

Re: What is the right way to kill stalled backup job?

Post by MrTwistoff »

Just an FYI on this thread. I did open a case with Support on it. One of the database tables has a field that tells the backup engine the state of the job. As long as that field is set to "Starting"..."Stopping" etc...(in some binary form) then the database engine will read that state. We did a WebEx and support perused the tables to find the field (comparing to other jobs not running), once that field got reset for that job it cleared the job state and everything functioned normally again.

antspants77
Influencer
Posts: 20
Liked: 5 times
Joined: Mar 16, 2011 3:15 pm
Full Name: Tony Spencer
Contact:

Re: What is the right way to kill stalled backup job?

Post by antspants77 »

With regards to Olaf's comments about cold cloning.
If it's a big VM with disks on different LUNs this can be a tricky process.
We had an issue where the Virtual Centre was rebooted during a replication job and snapshots left on the original VM.
These did not show in snapshot manager, but could be seen within the VM properties as the disks showed E.g. server-000001.vmdk.

The fix was to shutdown the Veeam server and remove the disks that belonged to to the above VM (These are attached in Independent, readonly mode)
Restart the Veeam VM, then in Virtual Centre, create a new snapshot and after it completes (could be a while), then go to snapshot manger and select "Delete All"
Make sure you backup your data first.

Satanek
Novice
Posts: 6
Liked: never
Joined: Oct 25, 2010 4:10 pm
Full Name: Jakub Satanek
Contact:

How to safely stop the job?

Post by Satanek »

[merged]

Hello,
How to safely stop the job?
For more than 12 hours I was running a job. Job can not manually stop. Log In appeared the following. On the machine where it froze, I rebooted windows service VeeamVssSupport
Jakub

TaylorB
Enthusiast
Posts: 70
Liked: 10 times
Joined: Jan 28, 2011 4:40 pm
Full Name: Taylor B.
Contact:

How to kill a File copy job?

Post by TaylorB »

[merged]

Anyone know how to kill a file copy job? I scheduled a large file copy with Veeam that has now been running for 2 days and is only half done. I may run out of space and I need to cancel it but the options for doing so are greyed out.

ThomasMc
Expert
Posts: 293
Liked: 18 times
Joined: Apr 13, 2011 12:45 pm
Full Name: Thomas McConnell
Contact:

Re: How to kill a File copy job?

Post by ThomasMc »

TaylorB wrote:Anyone know how to kill a file copy job? I scheduled a large file copy with Veeam that has now been running for 2 days and is only half done. I may run out of space and I need to cancel it but the options for doing so are greyed out.
I would always try the shell first :)

Code: Select all

Get-VBRJob | ?{$_.Name -eq "FCJ1"} | Stop-VBRJob

TaylorB
Enthusiast
Posts: 70
Liked: 10 times
Joined: Jan 28, 2011 4:40 pm
Full Name: Taylor B.
Contact:

Re: What is the right way to kill stalled backup job?

Post by TaylorB »

That seems to be working but now it just has said "Stopping" for about 30 mins.... :)

ThomasMc
Expert
Posts: 293
Liked: 18 times
Joined: Apr 13, 2011 12:45 pm
Full Name: Thomas McConnell
Contact:

Re: What is the right way to kill stalled backup job?

Post by ThomasMc »

Always worth a try first :) most likely you'll need to kill the service as above

Sethbartlett
Expert
Posts: 282
Liked: 26 times
Joined: Nov 10, 2010 6:51 pm
Full Name: Seth Bartlett
Contact:

Re: What is the right way to kill stalled backup job?

Post by Sethbartlett »

If it's stuck in stopping, your best bet is to stop/start the services. You will want to make sure no other jobs are running and do the following:

Code: Select all

Get-Process | ?{$_.name -like "*veeam*"} | Stop-Process
Get-Service | ?{$_.name -like "*veeam*"} | Stop-Service
Get-Service | ?{$_.name -like "*veeam*"} | Start-Service
From here, we may need to fix the DB too:

Code: Select all

delete [Backup.TrackedActions.Leases]
delete [Backup.TrackedActions.LockItems]
delete [Backup.TrackedActions.Locks]
Skype: Sethbartlett88 - Make sure to label who you are and why you want to add me ;)
Twitter: @sethbartlett
If my post was helpful, please like it. Sometimes twitter is quicker to hit me up if you need me.

Sethbartlett
Expert
Posts: 282
Liked: 26 times
Joined: Nov 10, 2010 6:51 pm
Full Name: Seth Bartlett
Contact:

Re: What is the right way to kill stalled backup job?

Post by Sethbartlett »

An added note, please keep in mind, the above SQL code is for version 5. I'm not sure if it will work in version 4 or 6 :)
Skype: Sethbartlett88 - Make sure to label who you are and why you want to add me ;)
Twitter: @sethbartlett
If my post was helpful, please like it. Sometimes twitter is quicker to hit me up if you need me.

TaylorB
Enthusiast
Posts: 70
Liked: 10 times
Joined: Jan 28, 2011 4:40 pm
Full Name: Taylor B.
Contact:

Re: What is the right way to kill stalled backup job?

Post by TaylorB »

It finally stopped after it finished the one 1.5TB file it was copying (2 days later at 9MB/s)

I think there is something in the code that doesn't let a file copy job stop mid file.

needhelp
Lurker
Posts: 1
Liked: never
Joined: Dec 21, 2012 6:02 am
Full Name: norman
Contact:

[MERGED] veeam backup v6

Post by needhelp »

hi,

Does anyone know how to kill the backup job? I have 2 jobs stuck(for 6 hours), one is stopping & another 50% completed. Thanks! :)

Vitaliy S.
Product Manager
Posts: 25114
Liked: 2153 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: veeam backup v6

Post by Vitaliy S. »

Hi Norman,

If you cannot stop your backup jobs via GUI, then please open Task Manager on your backup server and end all processes that start with Veeam*.

Thanks!

dapidap
Novice
Posts: 4
Liked: never
Joined: Aug 01, 2012 3:35 am
Full Name: Dapearl SJ
Contact:

Re: What is the right way to kill stalled backup job?

Post by dapidap »

Hi, I am so unhappy.

My simple file copy jobs doesn't execute and I cant seem to delete the job or stop it, its in "Stopping" state since even when disabled. How do you exactly end a job and delete in veeam?

dapidap
Novice
Posts: 4
Liked: never
Joined: Aug 01, 2012 3:35 am
Full Name: Dapearl SJ
Contact:

Re: What is the right way to kill stalled backup job?

Post by dapidap »

Hi Vitaliy,

All is well, i have deleted the job by killing all processes related to veeam and restarting the veeam server.

Thanks!

antman157
Influencer
Posts: 16
Liked: 2 times
Joined: Apr 03, 2013 2:53 pm
Full Name: Anthony Jones
Contact:

[MERGED] : 3 instant recovery jobs stuck

Post by antman157 »

Ive got 3 instant recovery jobs stuck in the status "Stopping"

Is there a way to remove them? If I right click the Job Name, "Stop Session" is grayed out.

Veeam 6.5.0.128 (64 bit)
Windows 2008 Server with SQL server mgmt studio express

krismcewan
Influencer
Posts: 14
Liked: never
Joined: Jan 21, 2010 4:40 pm
Full Name: Chris mcewan
Contact:

Re: What is the right way to kill stalled backup job?

Post by krismcewan »

ok how about this for a conundrum


Backup Server is backup 5 on win 2003 sp2

Powershell does not work as the tools are not compatible with powershell v1

I cant seem to copy sql mgmt studio onto there so i cant check the SQL tables.

and i cant rebuild as there is no documentation on how it was implemented in the first place.

Vitaliy S.
Product Manager
Posts: 25114
Liked: 2153 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: What is the right way to kill stalled backup job?

Post by Vitaliy S. »

Do you have any Veeam* processes running? Have you tried to end them? If this doesn't help, then please contact our technical team directly. Thanks!

pkelly_sts
Expert
Posts: 600
Liked: 66 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

[MERGED] What's the cleanest way to kill a hung replication

Post by pkelly_sts »

I seem to be getting regular job-hangs at the moment, possibly because I'm trying to sort our v7 migration and I'm running multiple jobs at various times, but I'd like to know the cleanest way to restart Veeam without upsetting it too much if a job is in the hung state.

Reason I ask is because yesterday I tried restarting some of the services which wasn't working so I killed the processes and managed to get everything up & running again, but when I tried to start a job it just sat there saying it was waiting for the repository to come available (or something similar).

I want to start a fresh copy job for a seeded backup at a remote site (i.e. to create the single .vbk that the copy job requires) but I want to get rid of this hung job before I start that one.

(I'd log a call but it would be hard to justify a Sev 1 and could really do with some idea before this evening)

Paul

Vitaliy S.
Product Manager
Posts: 25114
Liked: 2153 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: What is the right way to kill stalled backup job?

Post by Vitaliy S. »

Hi Paul,

Ending all Veeam process on the backup server should be all you need to do, however I would strongly recommend opening a support case to investigate the initial reason of this issue.

Thanks!

Post Reply

Who is online

Users browsing this forum: Google [Bot], ifb and 62 guests