What is the right way to kill stalled backup job?

[xray6969]

I have a backup job that has been running for a weekend and I wanted to stop that job. I right click it and select "Stop job". It gives a message "That this will leave backup in inconsistent state (not exact wording)" (I do not really understand - what exactly that means? Am I loosing backup of VM which was running at a time or whole session?). Job has been "running" in STOPPING state for last 12 hours. Now the question is how do you properly kill such job in a way that backup files are usable - that is next time I can re-run job (OK, I might loose whole last "killed" backup session). Or do I have to run Full backup after such kill?

[Vitaliy S.]

Hello Raimonds,

You should open Task Manager on the backup server and end all VeeamAgent processes, this will stop your job. The message you see while stopping backup job means that the last restore point will not be consistent and you'll loose the latest backup for a VM running at that time, not the whole session.

There is no need to re-create backup job and trigger full backup pass, as subsequent job run will fix the previous state and will continue that backup chain.

Hope this helps.

[xray6969]

Killed all VeeamAgent.exe, but job did not stop (still in STOPPING state).

[Vitaliy S.]

Trying re-opening Veeam backup console, it should update the job status. If it doesn't help, please contact our technical support team.

[cby]

While Veeam reports 'stopping' check the VC for snapshot deletion messages appearing to hang at 95%. Give it time to complete, the deletion can take a long time depending on VM size!

Also, for failed jobs check the snapshot manager for 'Consolidated Helper' snapshots. You can remove these and retrieve storage on your datastore. Quite often these types of snapshots are the reason for inconsistent looking storage capacities.

[xray6969]

No snapshot tasks even running in VC. Like I said it has been 12 hours on system not being stressed. And I am not concerned about failed jobs and snapshots - can handle those. All I wanted is safe way to kill veeam backup job no matter what is going wrong on my NAS, ESX, VC etc. If I just reboot/kill veeam backup server - will that be OK (cause my previous experience showed - if you do that backup files become locked and I could not restart backing up? So, it is not actually support question, it is veeam question in general - what happens if backup job crashes (being it - servers reboot, ESX server had memory fault, job was killed, killed by task manager, killed any other way, Windows BlueScreen etc.) What I am getting at - veeam have tested various scenarios when backup is OK (in their labs). But, there is no FAQ on what happens when backup hangs for instance. Do not get me wrong - I love veeam backup!

[cby]

Veeam support provided a SQL script and instructions to reset a job. I'm not sure if it only applies to FileLevelRestore locked state (i.e.the value of 'state' and/or 'overall_status')

JobsReset.sql:

Code: Select all

update [BSessions] set overall_status = 0 where overall_status <> 2
update [BJobs] set control = 0
update [ObjectsInBackups] set [state] = 0 where [state]=2 or [state]=3

1. Save JobsReset.sql file to local disk on Veeam server.
2. Make sure that there are no jobs running to avoid resetting their status accidentally.
3. Run the following command: sqlcmd -S COMPUTER\INSTANCE -d VeeamBackup -i PATHTOSCRIPT\JobsReset.sql
where COMPUTER is the name of computer with database, INSTANCE is the name of instance (VEEAM for default installation), PATHTOSCRIPT is the path to the JobsReset.sql script.

After running this, restart both the SQL Service for Veeam (VEEAM), and the Veeam Backup Scheduling Service. (in that order)

One of the Veeam product managers will put me right if this a bum steer!

[Vitaliy S.]

If anything happens to the backup server while backup job is running, nothing should become unrecoverable. It's true that some files might be locked, but each and every situation has it's own timeout and fixing engine, which shouldn't prevent you from re-running backup/replication jobs.

Also I suggest that those kind of situations should be handled with our support team assistance, because you definitely want to get everything up and running as soon as possible.

As for this particular situation with stalled jobs, if killing VeeamAgents didn't help, then a script mentioned above should be a way to go.

[MrTwistoff]

Having this issue on Veeam 5...and ran the script against the (remote database) and get the following errors:

Msg 207, Level 16, State 1, Line 2

Invalid column name 'overall_status'.

Msg 207, Level 16, State 1, Line 3

Invalid column name 'control'.


Any updates to that script that might be needed for it to work on 5?

[Gostev]

I believe that script is for v5. Please work with our support if you are having difficulties running it. Thanks.

[Olaf]

a few months ago I've had the problem that a VEEAM Backup job crashed. Every retry of that job fails (errormessage I can't remember exactly --> something with snapshot). No snapshots shown in VC
I found this link :
http://www.kendrickcoleman.com/index.ph ... qus_thread
The cold cloning trick worked fine for me --> look --> How to remove all these snapshots.

[MrTwistoff]

Just an FYI on this thread. I did open a case with Support on it. One of the database tables has a field that tells the backup engine the state of the job. As long as that field is set to "Starting"..."Stopping" etc...(in some binary form) then the database engine will read that state. We did a WebEx and support perused the tables to find the field (comparing to other jobs not running), once that field got reset for that job it cleared the job state and everything functioned normally again.

[antspants77]

With regards to Olaf's comments about cold cloning.
If it's a big VM with disks on different LUNs this can be a tricky process.
We had an issue where the Virtual Centre was rebooted during a replication job and snapshots left on the original VM.
These did not show in snapshot manager, but could be seen within the VM properties as the disks showed E.g. server-000001.vmdk.

The fix was to shutdown the Veeam server and remove the disks that belonged to to the above VM (These are attached in Independent, readonly mode)
Restart the Veeam VM, then in Virtual Centre, create a new snapshot and after it completes (could be a while), then go to snapshot manger and select "Delete All"
Make sure you backup your data first.

[Satanek]

[merged]

Hello,
How to safely stop the job?
For more than 12 hours I was running a job. Job can not manually stop. Log In appeared the following. On the machine where it froze, I rebooted windows service VeeamVssSupport
Jakub

[TaylorB]

[merged]

Anyone know how to kill a file copy job? I scheduled a large file copy with Veeam that has now been running for 2 days and is only half done. I may run out of space and I need to cancel it but the options for doing so are greyed out.

[ThomasMc]

Anyone know how to kill a file copy job? I scheduled a large file copy with Veeam that has now been running for 2 days and is only half done. I may run out of space and I need to cancel it but the options for doing so are greyed out.

I would always try the shell first

Code: Select all

Get-VBRJob | ?{$_.Name -eq "FCJ1"} | Stop-VBRJob

[TaylorB]

That seems to be working but now it just has said "Stopping" for about 30 mins....

[ThomasMc]

Always worth a try first most likely you'll need to kill the service as above

[Sethbartlett]

If it's stuck in stopping, your best bet is to stop/start the services. You will want to make sure no other jobs are running and do the following:

Code: Select all

Get-Process | ?{$_.name -like "*veeam*"} | Stop-Process
Get-Service | ?{$_.name -like "*veeam*"} | Stop-Service
Get-Service | ?{$_.name -like "*veeam*"} | Start-Service

From here, we may need to fix the DB too:

Code: Select all

delete [Backup.TrackedActions.Leases]
delete [Backup.TrackedActions.LockItems]
delete [Backup.TrackedActions.Locks]

[Sethbartlett]

An added note, please keep in mind, the above SQL code is for version 5. I'm not sure if it will work in version 4 or 6

[TaylorB]

It finally stopped after it finished the one 1.5TB file it was copying (2 days later at 9MB/s)

I think there is something in the code that doesn't let a file copy job stop mid file.

[needhelp]

hi,

Does anyone know how to kill the backup job? I have 2 jobs stuck(for 6 hours), one is stopping & another 50% completed. Thanks!

[Vitaliy S.]

Hi Norman,

If you cannot stop your backup jobs via GUI, then please open Task Manager on your backup server and end all processes that start with Veeam*.

Thanks!

[dapidap]

Hi, I am so unhappy.

My simple file copy jobs doesn't execute and I cant seem to delete the job or stop it, its in "Stopping" state since even when disabled. How do you exactly end a job and delete in veeam?

[dapidap]

Hi Vitaliy,

All is well, i have deleted the job by killing all processes related to veeam and restarting the veeam server.

Thanks!

[antman157]

Ive got 3 instant recovery jobs stuck in the status "Stopping"

Is there a way to remove them? If I right click the Job Name, "Stop Session" is grayed out.

Veeam 6.5.0.128 (64 bit)
Windows 2008 Server with SQL server mgmt studio express

[krismcewan]

ok how about this for a conundrum


Backup Server is backup 5 on win 2003 sp2

Powershell does not work as the tools are not compatible with powershell v1

I cant seem to copy sql mgmt studio onto there so i cant check the SQL tables.

and i cant rebuild as there is no documentation on how it was implemented in the first place.

[Vitaliy S.]

Do you have any Veeam* processes running? Have you tried to end them? If this doesn't help, then please contact our technical team directly. Thanks!

[pkelly_sts]

I seem to be getting regular job-hangs at the moment, possibly because I'm trying to sort our v7 migration and I'm running multiple jobs at various times, but I'd like to know the cleanest way to restart Veeam without upsetting it too much if a job is in the hung state.

Reason I ask is because yesterday I tried restarting some of the services which wasn't working so I killed the processes and managed to get everything up & running again, but when I tried to start a job it just sat there saying it was waiting for the repository to come available (or something similar).

I want to start a fresh copy job for a seeded backup at a remote site (i.e. to create the single .vbk that the copy job requires) but I want to get rid of this hung job before I start that one.

(I'd log a call but it would be hard to justify a Sev 1 and could really do with some idea before this evening)

Paul

[Vitaliy S.]

Hi Paul,

Ending all Veeam process on the backup server should be all you need to do, however I would strongly recommend opening a support case to investigate the initial reason of this issue.

Thanks!