What is the right way to kill stalled backup job?

Availability for the Always-On Enterprise

What is the right way to kill stalled backup job?

Veeam Logoby xray6969 » Tue Aug 16, 2011 6:31 am

I have a backup job that has been running for a weekend and I wanted to stop that job. I right click it and select "Stop job". It gives a message "That this will leave backup in inconsistent state (not exact wording)" (I do not really understand - what exactly that means? Am I loosing backup of VM which was running at a time or whole session?). Job has been "running" in STOPPING state for last 12 hours. Now the question is how do you properly kill such job in a way that backup files are usable - that is next time I can re-run job (OK, I might loose whole last "killed" backup session). Or do I have to run Full backup after such kill?
xray6969
Novice
 
Posts: 8
Liked: never
Joined: Mon Mar 07, 2011 3:20 pm
Location: Belfast, United Kingdom
Full Name: Raimonds Martinovs

Re: What is the right way to kill stalled backup job?

Veeam Logoby Vitaliy S. » Tue Aug 16, 2011 8:33 am

Hello Raimonds,

You should open Task Manager on the backup server and end all VeeamAgent processes, this will stop your job. The message you see while stopping backup job means that the last restore point will not be consistent and you'll loose the latest backup for a VM running at that time, not the whole session.

There is no need to re-create backup job and trigger full backup pass, as subsequent job run will fix the previous state and will continue that backup chain.

Hope this helps.
Vitaliy S.
Veeam Software
 
Posts: 19974
Liked: 1145 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: What is the right way to kill stalled backup job?

Veeam Logoby xray6969 » Tue Aug 16, 2011 8:41 am

Killed all VeeamAgent.exe, but job did not stop (still in STOPPING state).
xray6969
Novice
 
Posts: 8
Liked: never
Joined: Mon Mar 07, 2011 3:20 pm
Location: Belfast, United Kingdom
Full Name: Raimonds Martinovs

Re: What is the right way to kill stalled backup job?

Veeam Logoby Vitaliy S. » Tue Aug 16, 2011 8:54 am

Trying re-opening Veeam backup console, it should update the job status. If it doesn't help, please contact our technical support team.
Vitaliy S.
Veeam Software
 
Posts: 19974
Liked: 1145 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: What is the right way to kill stalled backup job?

Veeam Logoby cby » Tue Aug 16, 2011 10:14 am

While Veeam reports 'stopping' check the VC for snapshot deletion messages appearing to hang at 95%. Give it time to complete, the deletion can take a long time depending on VM size!

Also, for failed jobs check the snapshot manager for 'Consolidated Helper' snapshots. You can remove these and retrieve storage on your datastore. Quite often these types of snapshots are the reason for inconsistent looking storage capacities.
cby
Expert
 
Posts: 106
Liked: 6 times
Joined: Tue Feb 24, 2009 5:02 pm

Re: What is the right way to kill stalled backup job?

Veeam Logoby xray6969 » Tue Aug 16, 2011 10:32 am

No snapshot tasks even running in VC. Like I said it has been 12 hours on system not being stressed. And I am not concerned about failed jobs and snapshots - can handle those. All I wanted is safe way to kill veeam backup job no matter what is going wrong on my NAS, ESX, VC etc. If I just reboot/kill veeam backup server - will that be OK (cause my previous experience showed - if you do that backup files become locked and I could not restart backing up? So, it is not actually support question, it is veeam question in general - what happens if backup job crashes (being it - servers reboot, ESX server had memory fault, job was killed, killed by task manager, killed any other way, Windows BlueScreen etc.) What I am getting at - veeam have tested various scenarios when backup is OK (in their labs). But, there is no FAQ on what happens when backup hangs for instance. Do not get me wrong - I love veeam backup!
xray6969
Novice
 
Posts: 8
Liked: never
Joined: Mon Mar 07, 2011 3:20 pm
Location: Belfast, United Kingdom
Full Name: Raimonds Martinovs

Re: What is the right way to kill stalled backup job?

Veeam Logoby cby » Tue Aug 16, 2011 11:21 am

Veeam support provided a SQL script and instructions to reset a job. I'm not sure if it only applies to FileLevelRestore locked state (i.e.the value of 'state' and/or 'overall_status')

JobsReset.sql:
Code: Select all
update [BSessions] set overall_status = 0 where overall_status <> 2
update [BJobs] set control = 0
update [ObjectsInBackups] set [state] = 0 where [state]=2 or [state]=3


1. Save JobsReset.sql file to local disk on Veeam server.
2. Make sure that there are no jobs running to avoid resetting their status accidentally.
3. Run the following command: sqlcmd -S COMPUTER\INSTANCE -d VeeamBackup -i PATHTOSCRIPT\JobsReset.sql
where COMPUTER is the name of computer with database, INSTANCE is the name of instance (VEEAM for default installation), PATHTOSCRIPT is the path to the JobsReset.sql script.

After running this, restart both the SQL Service for Veeam (VEEAM), and the Veeam Backup Scheduling Service. (in that order)

One of the Veeam product managers will put me right if this a bum steer!
cby
Expert
 
Posts: 106
Liked: 6 times
Joined: Tue Feb 24, 2009 5:02 pm

Re: What is the right way to kill stalled backup job?

Veeam Logoby Vitaliy S. » Tue Aug 16, 2011 11:26 am

If anything happens to the backup server while backup job is running, nothing should become unrecoverable. It's true that some files might be locked, but each and every situation has it's own timeout and fixing engine, which shouldn't prevent you from re-running backup/replication jobs.

Also I suggest that those kind of situations should be handled with our support team assistance, because you definitely want to get everything up and running as soon as possible.

As for this particular situation with stalled jobs, if killing VeeamAgents didn't help, then a script mentioned above should be a way to go.
Vitaliy S.
Veeam Software
 
Posts: 19974
Liked: 1145 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: What is the right way to kill stalled backup job?

Veeam Logoby MrTwistoff » Mon Sep 19, 2011 4:50 pm

Having this issue on Veeam 5...and ran the script against the (remote database) and get the following errors:

Msg 207, Level 16, State 1, Line 2

Invalid column name 'overall_status'.

Msg 207, Level 16, State 1, Line 3

Invalid column name 'control'.


Any updates to that script that might be needed for it to work on 5?
MrTwistoff
Novice
 
Posts: 7
Liked: never
Joined: Thu Mar 10, 2011 7:48 pm
Full Name: Brian L Shunamon

Re: What is the right way to kill stalled backup job?

Veeam Logoby Gostev » Mon Sep 19, 2011 5:34 pm

I believe that script is for v5. Please work with our support if you are having difficulties running it. Thanks.
Gostev
Veeam Software
 
Posts: 21622
Liked: 2411 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: What is the right way to kill stalled backup job?

Veeam Logoby Olaf » Wed Sep 28, 2011 2:13 pm

a few months ago I've had the problem that a VEEAM Backup job crashed. Every retry of that job fails (errormessage I can't remember exactly --> something with snapshot). No snapshots shown in VC
I found this link :
http://www.kendrickcoleman.com/index.ph ... qus_thread
The cold cloning trick worked fine for me --> look --> How to remove all these snapshots.
Olaf
Lurker
 
Posts: 1
Liked: never
Joined: Wed Dec 22, 2010 2:13 pm

Re: What is the right way to kill stalled backup job?

Veeam Logoby MrTwistoff » Mon Oct 03, 2011 2:06 pm

Just an FYI on this thread. I did open a case with Support on it. One of the database tables has a field that tells the backup engine the state of the job. As long as that field is set to "Starting"..."Stopping" etc...(in some binary form) then the database engine will read that state. We did a WebEx and support perused the tables to find the field (comparing to other jobs not running), once that field got reset for that job it cleared the job state and everything functioned normally again.
MrTwistoff
Novice
 
Posts: 7
Liked: never
Joined: Thu Mar 10, 2011 7:48 pm
Full Name: Brian L Shunamon

Re: What is the right way to kill stalled backup job?

Veeam Logoby antspants77 » Tue Oct 04, 2011 7:57 am

With regards to Olaf's comments about cold cloning.
If it's a big VM with disks on different LUNs this can be a tricky process.
We had an issue where the Virtual Centre was rebooted during a replication job and snapshots left on the original VM.
These did not show in snapshot manager, but could be seen within the VM properties as the disks showed E.g. server-000001.vmdk.

The fix was to shutdown the Veeam server and remove the disks that belonged to to the above VM (These are attached in Independent, readonly mode)
Restart the Veeam VM, then in Virtual Centre, create a new snapshot and after it completes (could be a while), then go to snapshot manger and select "Delete All"
Make sure you backup your data first.
antspants77
Influencer
 
Posts: 20
Liked: 5 times
Joined: Wed Mar 16, 2011 3:15 pm
Full Name: Tony Spencer

How to safely stop the job?

Veeam Logoby Satanek » Thu Nov 03, 2011 10:13 am

[merged]

Hello,
How to safely stop the job?
For more than 12 hours I was running a job. Job can not manually stop. Log In appeared the following. On the machine where it froze, I rebooted windows service VeeamVssSupport
Jakub
Satanek
Novice
 
Posts: 6
Liked: never
Joined: Mon Oct 25, 2010 4:10 pm
Full Name: Jakub Satanek

How to kill a File copy job?

Veeam Logoby TaylorB » Fri Nov 18, 2011 6:07 pm

[merged]

Anyone know how to kill a file copy job? I scheduled a large file copy with Veeam that has now been running for 2 days and is only half done. I may run out of space and I need to cancel it but the options for doing so are greyed out.
TaylorB
Enthusiast
 
Posts: 42
Liked: 2 times
Joined: Fri Jan 28, 2011 4:40 pm
Full Name: Taylor B.

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: baber, Bing [Bot] and 1 guest