-
- Novice
- Posts: 8
- Liked: never
- Joined: Mar 07, 2011 3:20 pm
- Full Name: Raimonds Martinovs
- Location: Belfast, United Kingdom
What is the right way to kill stalled backup job?
I have a backup job that has been running for a weekend and I wanted to stop that job. I right click it and select "Stop job". It gives a message "That this will leave backup in inconsistent state (not exact wording)" (I do not really understand - what exactly that means? Am I loosing backup of VM which was running at a time or whole session?). Job has been "running" in STOPPING state for last 12 hours. Now the question is how do you properly kill such job in a way that backup files are usable - that is next time I can re-run job (OK, I might loose whole last "killed" backup session). Or do I have to run Full backup after such kill?
-
- VP, Product Management
- Posts: 27347
- Liked: 2785 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: What is the right way to kill stalled backup job?
Hello Raimonds,
You should open Task Manager on the backup server and end all VeeamAgent processes, this will stop your job. The message you see while stopping backup job means that the last restore point will not be consistent and you'll loose the latest backup for a VM running at that time, not the whole session.
There is no need to re-create backup job and trigger full backup pass, as subsequent job run will fix the previous state and will continue that backup chain.
Hope this helps.
You should open Task Manager on the backup server and end all VeeamAgent processes, this will stop your job. The message you see while stopping backup job means that the last restore point will not be consistent and you'll loose the latest backup for a VM running at that time, not the whole session.
There is no need to re-create backup job and trigger full backup pass, as subsequent job run will fix the previous state and will continue that backup chain.
Hope this helps.
-
- Novice
- Posts: 8
- Liked: never
- Joined: Mar 07, 2011 3:20 pm
- Full Name: Raimonds Martinovs
- Location: Belfast, United Kingdom
Re: What is the right way to kill stalled backup job?
Killed all VeeamAgent.exe, but job did not stop (still in STOPPING state).
-
- VP, Product Management
- Posts: 27347
- Liked: 2785 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: What is the right way to kill stalled backup job?
Trying re-opening Veeam backup console, it should update the job status. If it doesn't help, please contact our technical support team.
-
- Enthusiast
- Posts: 97
- Liked: 6 times
- Joined: Feb 24, 2009 5:02 pm
- Contact:
Re: What is the right way to kill stalled backup job?
While Veeam reports 'stopping' check the VC for snapshot deletion messages appearing to hang at 95%. Give it time to complete, the deletion can take a long time depending on VM size!
Also, for failed jobs check the snapshot manager for 'Consolidated Helper' snapshots. You can remove these and retrieve storage on your datastore. Quite often these types of snapshots are the reason for inconsistent looking storage capacities.
Also, for failed jobs check the snapshot manager for 'Consolidated Helper' snapshots. You can remove these and retrieve storage on your datastore. Quite often these types of snapshots are the reason for inconsistent looking storage capacities.
-
- Novice
- Posts: 8
- Liked: never
- Joined: Mar 07, 2011 3:20 pm
- Full Name: Raimonds Martinovs
- Location: Belfast, United Kingdom
Re: What is the right way to kill stalled backup job?
No snapshot tasks even running in VC. Like I said it has been 12 hours on system not being stressed. And I am not concerned about failed jobs and snapshots - can handle those. All I wanted is safe way to kill veeam backup job no matter what is going wrong on my NAS, ESX, VC etc. If I just reboot/kill veeam backup server - will that be OK (cause my previous experience showed - if you do that backup files become locked and I could not restart backing up? So, it is not actually support question, it is veeam question in general - what happens if backup job crashes (being it - servers reboot, ESX server had memory fault, job was killed, killed by task manager, killed any other way, Windows BlueScreen etc.) What I am getting at - veeam have tested various scenarios when backup is OK (in their labs). But, there is no FAQ on what happens when backup hangs for instance. Do not get me wrong - I love veeam backup!
-
- Enthusiast
- Posts: 97
- Liked: 6 times
- Joined: Feb 24, 2009 5:02 pm
- Contact:
Re: What is the right way to kill stalled backup job?
Veeam support provided a SQL script and instructions to reset a job. I'm not sure if it only applies to FileLevelRestore locked state (i.e.the value of 'state' and/or 'overall_status')
JobsReset.sql:
1. Save JobsReset.sql file to local disk on Veeam server.
2. Make sure that there are no jobs running to avoid resetting their status accidentally.
3. Run the following command: sqlcmd -S COMPUTER\INSTANCE -d VeeamBackup -i PATHTOSCRIPT\JobsReset.sql
where COMPUTER is the name of computer with database, INSTANCE is the name of instance (VEEAM for default installation), PATHTOSCRIPT is the path to the JobsReset.sql script.
After running this, restart both the SQL Service for Veeam (VEEAM), and the Veeam Backup Scheduling Service. (in that order)
One of the Veeam product managers will put me right if this a bum steer!
JobsReset.sql:
Code: Select all
update [BSessions] set overall_status = 0 where overall_status <> 2
update [BJobs] set control = 0
update [ObjectsInBackups] set [state] = 0 where [state]=2 or [state]=3
2. Make sure that there are no jobs running to avoid resetting their status accidentally.
3. Run the following command: sqlcmd -S COMPUTER\INSTANCE -d VeeamBackup -i PATHTOSCRIPT\JobsReset.sql
where COMPUTER is the name of computer with database, INSTANCE is the name of instance (VEEAM for default installation), PATHTOSCRIPT is the path to the JobsReset.sql script.
After running this, restart both the SQL Service for Veeam (VEEAM), and the Veeam Backup Scheduling Service. (in that order)
One of the Veeam product managers will put me right if this a bum steer!
-
- VP, Product Management
- Posts: 27347
- Liked: 2785 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: What is the right way to kill stalled backup job?
If anything happens to the backup server while backup job is running, nothing should become unrecoverable. It's true that some files might be locked, but each and every situation has it's own timeout and fixing engine, which shouldn't prevent you from re-running backup/replication jobs.
Also I suggest that those kind of situations should be handled with our support team assistance, because you definitely want to get everything up and running as soon as possible.
As for this particular situation with stalled jobs, if killing VeeamAgents didn't help, then a script mentioned above should be a way to go.
Also I suggest that those kind of situations should be handled with our support team assistance, because you definitely want to get everything up and running as soon as possible.
As for this particular situation with stalled jobs, if killing VeeamAgents didn't help, then a script mentioned above should be a way to go.
-
- Novice
- Posts: 7
- Liked: never
- Joined: Mar 10, 2011 7:48 pm
- Full Name: Brian L Shunamon
- Contact:
Re: What is the right way to kill stalled backup job?
Having this issue on Veeam 5...and ran the script against the (remote database) and get the following errors:
Msg 207, Level 16, State 1, Line 2
Invalid column name 'overall_status'.
Msg 207, Level 16, State 1, Line 3
Invalid column name 'control'.
Any updates to that script that might be needed for it to work on 5?
Msg 207, Level 16, State 1, Line 2
Invalid column name 'overall_status'.
Msg 207, Level 16, State 1, Line 3
Invalid column name 'control'.
Any updates to that script that might be needed for it to work on 5?
-
- Chief Product Officer
- Posts: 31754
- Liked: 7259 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: What is the right way to kill stalled backup job?
I believe that script is for v5. Please work with our support if you are having difficulties running it. Thanks.
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Dec 22, 2010 2:13 pm
- Contact:
Re: What is the right way to kill stalled backup job?
a few months ago I've had the problem that a VEEAM Backup job crashed. Every retry of that job fails (errormessage I can't remember exactly --> something with snapshot). No snapshots shown in VC
I found this link :
http://www.kendrickcoleman.com/index.ph ... qus_thread
The cold cloning trick worked fine for me --> look --> How to remove all these snapshots.
I found this link :
http://www.kendrickcoleman.com/index.ph ... qus_thread
The cold cloning trick worked fine for me --> look --> How to remove all these snapshots.
-
- Novice
- Posts: 7
- Liked: never
- Joined: Mar 10, 2011 7:48 pm
- Full Name: Brian L Shunamon
- Contact:
Re: What is the right way to kill stalled backup job?
Just an FYI on this thread. I did open a case with Support on it. One of the database tables has a field that tells the backup engine the state of the job. As long as that field is set to "Starting"..."Stopping" etc...(in some binary form) then the database engine will read that state. We did a WebEx and support perused the tables to find the field (comparing to other jobs not running), once that field got reset for that job it cleared the job state and everything functioned normally again.
-
- Influencer
- Posts: 20
- Liked: 5 times
- Joined: Mar 16, 2011 3:15 pm
- Full Name: Tony Spencer
- Contact:
Re: What is the right way to kill stalled backup job?
With regards to Olaf's comments about cold cloning.
If it's a big VM with disks on different LUNs this can be a tricky process.
We had an issue where the Virtual Centre was rebooted during a replication job and snapshots left on the original VM.
These did not show in snapshot manager, but could be seen within the VM properties as the disks showed E.g. server-000001.vmdk.
The fix was to shutdown the Veeam server and remove the disks that belonged to to the above VM (These are attached in Independent, readonly mode)
Restart the Veeam VM, then in Virtual Centre, create a new snapshot and after it completes (could be a while), then go to snapshot manger and select "Delete All"
Make sure you backup your data first.
If it's a big VM with disks on different LUNs this can be a tricky process.
We had an issue where the Virtual Centre was rebooted during a replication job and snapshots left on the original VM.
These did not show in snapshot manager, but could be seen within the VM properties as the disks showed E.g. server-000001.vmdk.
The fix was to shutdown the Veeam server and remove the disks that belonged to to the above VM (These are attached in Independent, readonly mode)
Restart the Veeam VM, then in Virtual Centre, create a new snapshot and after it completes (could be a while), then go to snapshot manger and select "Delete All"
Make sure you backup your data first.
-
- Novice
- Posts: 6
- Liked: never
- Joined: Oct 25, 2010 4:10 pm
- Full Name: Jakub Satanek
- Contact:
How to safely stop the job?
[merged]
Hello,
How to safely stop the job?
For more than 12 hours I was running a job. Job can not manually stop. Log In appeared the following. On the machine where it froze, I rebooted windows service VeeamVssSupport
Jakub
Hello,
How to safely stop the job?
For more than 12 hours I was running a job. Job can not manually stop. Log In appeared the following. On the machine where it froze, I rebooted windows service VeeamVssSupport
Jakub
-
- Enthusiast
- Posts: 92
- Liked: 14 times
- Joined: Jan 28, 2011 4:40 pm
- Full Name: Taylor B.
- Contact:
How to kill a File copy job?
[merged]
Anyone know how to kill a file copy job? I scheduled a large file copy with Veeam that has now been running for 2 days and is only half done. I may run out of space and I need to cancel it but the options for doing so are greyed out.
Anyone know how to kill a file copy job? I scheduled a large file copy with Veeam that has now been running for 2 days and is only half done. I may run out of space and I need to cancel it but the options for doing so are greyed out.
-
- Veteran
- Posts: 293
- Liked: 19 times
- Joined: Apr 13, 2011 12:45 pm
- Full Name: Thomas McConnell
- Contact:
Re: How to kill a File copy job?
I would always try the shell firstTaylorB wrote:Anyone know how to kill a file copy job? I scheduled a large file copy with Veeam that has now been running for 2 days and is only half done. I may run out of space and I need to cancel it but the options for doing so are greyed out.
Code: Select all
Get-VBRJob | ?{$_.Name -eq "FCJ1"} | Stop-VBRJob
-
- Enthusiast
- Posts: 92
- Liked: 14 times
- Joined: Jan 28, 2011 4:40 pm
- Full Name: Taylor B.
- Contact:
Re: What is the right way to kill stalled backup job?
That seems to be working but now it just has said "Stopping" for about 30 mins....
-
- Veteran
- Posts: 293
- Liked: 19 times
- Joined: Apr 13, 2011 12:45 pm
- Full Name: Thomas McConnell
- Contact:
Re: What is the right way to kill stalled backup job?
Always worth a try first most likely you'll need to kill the service as above
-
- Veteran
- Posts: 282
- Liked: 26 times
- Joined: Nov 10, 2010 6:51 pm
- Full Name: Seth Bartlett
- Contact:
Re: What is the right way to kill stalled backup job?
If it's stuck in stopping, your best bet is to stop/start the services. You will want to make sure no other jobs are running and do the following:
From here, we may need to fix the DB too:
Code: Select all
Get-Process | ?{$_.name -like "*veeam*"} | Stop-Process
Get-Service | ?{$_.name -like "*veeam*"} | Stop-Service
Get-Service | ?{$_.name -like "*veeam*"} | Start-Service
Code: Select all
delete [Backup.TrackedActions.Leases]
delete [Backup.TrackedActions.LockItems]
delete [Backup.TrackedActions.Locks]
Skype: Sethbartlett88 - Make sure to label who you are and why you want to add me
Twitter: @sethbartlett
If my post was helpful, please like it. Sometimes twitter is quicker to hit me up if you need me.
Twitter: @sethbartlett
If my post was helpful, please like it. Sometimes twitter is quicker to hit me up if you need me.
-
- Veteran
- Posts: 282
- Liked: 26 times
- Joined: Nov 10, 2010 6:51 pm
- Full Name: Seth Bartlett
- Contact:
Re: What is the right way to kill stalled backup job?
An added note, please keep in mind, the above SQL code is for version 5. I'm not sure if it will work in version 4 or 6
Skype: Sethbartlett88 - Make sure to label who you are and why you want to add me
Twitter: @sethbartlett
If my post was helpful, please like it. Sometimes twitter is quicker to hit me up if you need me.
Twitter: @sethbartlett
If my post was helpful, please like it. Sometimes twitter is quicker to hit me up if you need me.
-
- Enthusiast
- Posts: 92
- Liked: 14 times
- Joined: Jan 28, 2011 4:40 pm
- Full Name: Taylor B.
- Contact:
Re: What is the right way to kill stalled backup job?
It finally stopped after it finished the one 1.5TB file it was copying (2 days later at 9MB/s)
I think there is something in the code that doesn't let a file copy job stop mid file.
I think there is something in the code that doesn't let a file copy job stop mid file.
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Dec 21, 2012 6:02 am
- Full Name: norman
- Contact:
[MERGED] veeam backup v6
hi,
Does anyone know how to kill the backup job? I have 2 jobs stuck(for 6 hours), one is stopping & another 50% completed. Thanks!
Does anyone know how to kill the backup job? I have 2 jobs stuck(for 6 hours), one is stopping & another 50% completed. Thanks!
-
- VP, Product Management
- Posts: 27347
- Liked: 2785 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: veeam backup v6
Hi Norman,
If you cannot stop your backup jobs via GUI, then please open Task Manager on your backup server and end all processes that start with Veeam*.
Thanks!
If you cannot stop your backup jobs via GUI, then please open Task Manager on your backup server and end all processes that start with Veeam*.
Thanks!
-
- Novice
- Posts: 4
- Liked: never
- Joined: Aug 01, 2012 3:35 am
- Full Name: Dapearl SJ
- Contact:
Re: What is the right way to kill stalled backup job?
Hi, I am so unhappy.
My simple file copy jobs doesn't execute and I cant seem to delete the job or stop it, its in "Stopping" state since even when disabled. How do you exactly end a job and delete in veeam?
My simple file copy jobs doesn't execute and I cant seem to delete the job or stop it, its in "Stopping" state since even when disabled. How do you exactly end a job and delete in veeam?
-
- Novice
- Posts: 4
- Liked: never
- Joined: Aug 01, 2012 3:35 am
- Full Name: Dapearl SJ
- Contact:
Re: What is the right way to kill stalled backup job?
Hi Vitaliy,
All is well, i have deleted the job by killing all processes related to veeam and restarting the veeam server.
Thanks!
All is well, i have deleted the job by killing all processes related to veeam and restarting the veeam server.
Thanks!
-
- Influencer
- Posts: 17
- Liked: 2 times
- Joined: Apr 03, 2013 2:53 pm
- Full Name: Anthony Jones
- Contact:
[MERGED] : 3 instant recovery jobs stuck
Ive got 3 instant recovery jobs stuck in the status "Stopping"
Is there a way to remove them? If I right click the Job Name, "Stop Session" is grayed out.
Veeam 6.5.0.128 (64 bit)
Windows 2008 Server with SQL server mgmt studio express
Is there a way to remove them? If I right click the Job Name, "Stop Session" is grayed out.
Veeam 6.5.0.128 (64 bit)
Windows 2008 Server with SQL server mgmt studio express
-
- Influencer
- Posts: 14
- Liked: never
- Joined: Jan 21, 2010 4:40 pm
- Full Name: Chris mcewan
- Contact:
Re: What is the right way to kill stalled backup job?
ok how about this for a conundrum
Backup Server is backup 5 on win 2003 sp2
Powershell does not work as the tools are not compatible with powershell v1
I cant seem to copy sql mgmt studio onto there so i cant check the SQL tables.
and i cant rebuild as there is no documentation on how it was implemented in the first place.
Backup Server is backup 5 on win 2003 sp2
Powershell does not work as the tools are not compatible with powershell v1
I cant seem to copy sql mgmt studio onto there so i cant check the SQL tables.
and i cant rebuild as there is no documentation on how it was implemented in the first place.
-
- VP, Product Management
- Posts: 27347
- Liked: 2785 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: What is the right way to kill stalled backup job?
Do you have any Veeam* processes running? Have you tried to end them? If this doesn't help, then please contact our technical team directly. Thanks!
-
- Veteran
- Posts: 600
- Liked: 66 times
- Joined: Jun 13, 2013 10:08 am
- Full Name: Paul Kelly
- Contact:
[MERGED] What's the cleanest way to kill a hung replication
I seem to be getting regular job-hangs at the moment, possibly because I'm trying to sort our v7 migration and I'm running multiple jobs at various times, but I'd like to know the cleanest way to restart Veeam without upsetting it too much if a job is in the hung state.
Reason I ask is because yesterday I tried restarting some of the services which wasn't working so I killed the processes and managed to get everything up & running again, but when I tried to start a job it just sat there saying it was waiting for the repository to come available (or something similar).
I want to start a fresh copy job for a seeded backup at a remote site (i.e. to create the single .vbk that the copy job requires) but I want to get rid of this hung job before I start that one.
(I'd log a call but it would be hard to justify a Sev 1 and could really do with some idea before this evening)
Paul
Reason I ask is because yesterday I tried restarting some of the services which wasn't working so I killed the processes and managed to get everything up & running again, but when I tried to start a job it just sat there saying it was waiting for the repository to come available (or something similar).
I want to start a fresh copy job for a seeded backup at a remote site (i.e. to create the single .vbk that the copy job requires) but I want to get rid of this hung job before I start that one.
(I'd log a call but it would be hard to justify a Sev 1 and could really do with some idea before this evening)
Paul
-
- VP, Product Management
- Posts: 27347
- Liked: 2785 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: What is the right way to kill stalled backup job?
Hi Paul,
Ending all Veeam process on the backup server should be all you need to do, however I would strongly recommend opening a support case to investigate the initial reason of this issue.
Thanks!
Ending all Veeam process on the backup server should be all you need to do, however I would strongly recommend opening a support case to investigate the initial reason of this issue.
Thanks!
Who is online
Users browsing this forum: Egor Yakovlev and 53 guests