v8 post-job scripting changes "post-job script timed out"

[chjones]

This isn't a question, just thought I'd post this incase anyone else experiences this too.

I have upgraded to version 8 and have seen warnings on a few of my backup jobs that I never saw in version 7. The warning is "Post-job script timed out".

Our backup jobs call a post-job powershell script that detects the status of the job (success, warning, failure) and calls a backup job in HP Data Protector to backup the resulting VIB or VBK. In version 7 these scripts ran without any warnings, but in v8 this has changed. The script does run as I can see the job being called in Data Protector and completing.

However, the powershell calls the "omnib.exe" command which is used to start a Data Protector job, and the default of this executable is to run the backup job and display the status and output of the job in the command window. The window stays running for the entire duration of the backup. If the backup takes longer than 15 minutes it produces the warning in Veeam as the post-job script has yet to complete. I had a few jobs run last night where the Data Protector job only took a couple of minutes and there was no warning in Veeam.

The solution to this has been to add the "-no_monitor" argument to the Data Protector command-line, which instructs omnib.exe to start the job and not to monitor, so the post-job script from Veeam completes in a couple of seconds as opposed to waiting for the Data Protector job to complete.

This wasn't an issue in version 7, only in version 8, so something has changed from the Veeam side, but it's still more efficient to not monitor the data protector job anyway so at least this warning has helped improve my code

[sacman]

Thanks for this. I'm experiencing nearly the exact same issue. My script, which is just a run-of-the-mill batch file, copies my Veeam backups to a portable hard drive.

It worked great in 7, and doesn't work at all in 8. And I do get the "Post-job script timed out" error. The backups themselves are fine. With 7, the batch file would launch in a shell window. With 8, Veeam just sits there for 15 minutes and then times the job out. Based on the nature of the script, I can tell Veeam isn't even attempting to execute it - it just sits there for 15 minutes then gives up.

I'm wondering if you might clarify what you did to instruct Veeam to not monitor the script. Is the "-no_monitor" argument something you put in your script itself, is it an argument you gave Veeam, or something I'm missing? Google is failing me on this one - and executing that script by hand is getting old, fast.

Thanks!

-sacman

[Gostev]

Default timeouts of 900 seconds (15 minutes) is customizable for both pre- and post-job scripts via the following registry values:

Code: Select all

PreJobScriptTimeoutSec (DWORD)
PostJobScriptTimeoutSec (DWORD)

Unlike with previous versions, in v8 we are executing these scripts as a part of the job, instead of just launching them in a separate shell window as before. This allows us to reflect script execution results (depending on the exit code) in the job status, so that the job notification email tells you everything you need to know. Without this in place, our customers had to monitor additional activities spawned by post-job script separately.

I hope you will find this useful and yes, this is one feature that I forgot to include in the What's New document.

[veremin]

My script, which is just a run-of-the-mill batch file, copies my Veeam backups to a portable hard drive.

Just out of curiosity - may I ask you why backup copy job isn't an option for you? You can add portable hard drive as a rotated repository, and point backup copy job to it. This way, everything will be done thorough single point of glass, and you won't have to deal with custom scripts any longer. Thanks.

[sacman]

Gostev,

Thanks for this. It appears that's the explanation. However, I cannot find either of the registry values you stated. I'm searching the registry using regedit.exe launched from an elevated command prompt. Are these values I should add myself, and if so, can you tell me the exact key?

As for why I'm not using backup copy...the point of this script is to copy the backup files from an internal hard drive (backup repository) to a portable external hard drive. This is the hard drive I take home every night. Last time I tried this, Veeam was unable to account for the possibility of a drive letter changing. Therefore I wrote a script that finds the portable hard drive based upon the volume name, formats the drive, then copies over the entire repository contents.

Am I missing something? I'd be more than happy to drop the script if Veeam can be agile about the copy job target.

Thanks!

-sacman

[foggy]

However, I cannot find either of the registry values you stated. I'm searching the registry using regedit.exe launched from an elevated command prompt. Are these values I should add myself, and if so, can you tell me the exact key?

Yes, you should create the values manually under the standard HKLM\SOFTWARE\VeeaM\Veeam Backup and Replication hive.

Last time I tried this, Veeam was unable to account for the possibility of a drive letter changing.

Do you mean that each time the drive is mounted, it is assigned a different letter?

[sacman]

Thanks, I'll create the values manually and see what happens.

I'm not saying the drive letter changes each time it's mounted. But I've seen it happen enough times on various workstations that it's a concern for me - hence the script.

That said, I've got a box dedicated to Veeam in my server rack, and I do plug the external drive into the same USB port every day. Perhaps I'll try a backup copy job, and just cross my fingers that the drive letter stays static.

-sacman

[veremin]

Just to be sure - have you tried to assign repository role to the said drive, enabling the new "rotated" option, and point backup copy job to it? Will the said issue show up in that case? Thanks.

[sacman]

Nope, have not tried that. But I'll give it a go over the next few days and report my findings.

Thanks again for all your suggestions and help!

-sacman

[zone]

Hi guys,

I know that v8 has a 15min default timeout period for running a post-job script. What I was wondering was if it was possible to disable the post-job script check.

Currently we have customers running post-job scripts for the purpose of usb copies and it can take hours to complete. Would the only way to suppress the warnings be blowing out the timeouts via registry?

[veremin]

What commands are you using as post-job activity? Don't they happen to have -no_monitor argument analogue? Thanks.

[Gostev]

Last time I tried this, Veeam was unable to account for the possibility of a drive letter changing.

This is actually resolved in v8, as the backup job will "follow" known external disks even as they change drive letters. See What's New in v8 document for this and other enhancements around rotated media support.

Would the only way to suppress the warnings be blowing out the timeouts via registry?

No, you can also create the script in a way that it initiates the copy operation, and exists immediately.

[zone]

This is actually resolved in v8, as the backup job will "follow" known external disks even as they change drive letters. See What's New in v8 document for this and other enhancements around rotated media support.

Awesome, this was the entire reason for such a script. I will configure a new backup copy job and see how we go. Thanks

[veremin]

Just don't forget to enable the corresponding option in the repository settings. Thanks.

[chjones]

Thanks for this. I'm experiencing nearly the exact same issue. My script, which is just a run-of-the-mill batch file, copies my Veeam backups to a portable hard drive.

It worked great in 7, and doesn't work at all in 8. And I do get the "Post-job script timed out" error. The backups themselves are fine. With 7, the batch file would launch in a shell window. With 8, Veeam just sits there for 15 minutes and then times the job out. Based on the nature of the script, I can tell Veeam isn't even attempting to execute it - it just sits there for 15 minutes then gives up.

I'm wondering if you might clarify what you did to instruct Veeam to not monitor the script. Is the "-no_monitor" argument something you put in your script itself, is it an argument you gave Veeam, or something I'm missing? Google is failing me on this one - and executing that script by hand is getting old, fast.

Thanks!

-sacman

The "-no_monitor" option is an argument that the HP Data Protector executable supports. It's not something native to powershell. Thankfully HP included this as an option for their executable so we could get around this issue.

The other option I was investigating before I came across the above argument was having the post-job powershell script to launch whatever action you want it to take in a new process. I'd started looking into the "Start-Process" powershell command thinking that may do it but didn't get too far until I found the argument for the HP executable.

In the days of batch files you could get a batch file to launch several tasks and none of them be dependant on each other and the original batch file does not wait for any of them to complete. You achieved this by using "start" before your command the batch file, such as "start copy test.txt test2.txt" ... the file copy would start in a new batch file process and the original batch file would finish immediately. Perhaps this is a way to get around this?

[Gostev]

In the days of batch files you could get a batch file to launch several tasks and none of them be dependant on each other and the original batch file does not wait for any of them to complete. You achieved this by using "start" before your command the batch file, such as "start copy test.txt test2.txt" ... the file copy would start in a new batch file process and the original batch file would finish immediately. Perhaps this is a way to get around this?

That's exactly right.

[chjones]

This is more of a rant than anything else, but hopefully it helps anyone else who has the same problem.

We have a number of backup jobs and they all run Post-Job Scripts, which are Powershell scripts that perform actions based on various outputs, the most critical being whether the latest run of the Veeam Backup Job was successful. We use the GetLastResult method to return the status of the backup, which is Success, Warning or Failure. Since v8 we've noticed that every job that runs returns the last backup result as being "none".

I spent about 3 hours this morning trying to work out what was going on as every time I run the scripts manually the job result is returned correctly. I have since worked out that due to the change mentioned by Gostev in one of my other topics, http://forums.veeam.com/veeam-backup-re ... 24354.html, this is because the Veeam Server hasn't updated the Last Result of the Backup Job yet when it runs the Post-Job Script.

Since v8 now includes the status of running the Post-Job Script in its results, Veeam obviously can't update the job result until after the script has run. This renders checking the Last Result of the backup in a Post-Job Script useless.

We've added a workaround which is to have Veeam run a batch file which uses the "start" command to run the powershell, and then pause the powershell for 60 seconds. The Start command launches powershell in a new process and allows the batch file to complete so Veeam can continue and update the Last Result. The Powershell sleeps for 60 seconds and then continues, and can now use LastResult.

Below are the steps I had to take to get my scripts working again:

1. Change the Backup Job to run a batch file instead of the powershell directly, e.g. "E:\Batches\TestBatchFile.bat"
2. Inside the batch file, use the "start" command to launch the powershell, e.g. Start powershell.exe -Noninteractive -File E:\Batches\TestPowershell.ps1
3. Inside the Powershell, pause the script for 60 seconds e.g. Start-Sleep -s 60

I can see what Veeam is trying to do with incorporating the run of the Post-Backup Script into the results of the job, however it would have been nice to know in advance that this change was coming with v8, or at least give us the ability to not monitor the script as part of the job processing.

[cstemaly]

I'm going to have to also complain about this. I use the post-job action sections to kick off things that take a very long time. Also, I USED to use the success or failure of a number of jobs to kick off other tasks. Now since the job doesn't report until the script is finished, this isn't going to work anymore.

While the ability to report on the success or failure of a post-job action within the job report is admirable, I think most scripts/tasks either already have their only reporting built in, or have the capacity to.

How am I supposed to chain together my 5 SureBackup jobs that used to run via script after the last backup job? The only way to chain multiple SureBackup Jobs is via script, so now my last backup job is going to take 12 hours+ before I get alerted to failure. Actually, the last job always ends in warning, so that won't work either. Crap.

Editing because I found another shortcoming: I have 2 monthly backup jobs. When they're done, a script used to run that uses robocopy to synchronize the folder containing the backup job to my DR. That copy job takes almost a whole day. I can't use a BackupCopy job because it doesn't do any sync period longer than daily.

I guess my conclusion is this: There were some things I wasn't able to do with Veeam, but it was all okay because I was able to use the post-script activity to customize it's actions. From the examples above, these things are:
1. Ability to chain Surebackup jobs so that when the last backup job finishes, the my SureBackup jobs run one after the other.
2. The ability to copy Veeam backup files to my DR outside of the sync schedules afforded me by the BackupCopy feature.
3. The ability to do multiple actions based on the outcome of a backup windows' jobs

[foggy]

I can't use a BackupCopy job because it doesn't do any sync period longer than daily.

It actually does, you can set any number of days while configuring the copy interval.

[jelloir]

Will Veeam wait for an after job script to complete before allowing the Veeam job that called the script to finish?

Reason for the question: It occurred to me a nice way to trigger Server 2012 R2 De-duplication operations would be as a post (after) job script. But it wouldn't any good if Veeam waited for the script to complete.

I suppose a way to resolve it may be if the post script creates a separate thread so Veeam thinks it's complete?

Thanks

James

[foggy]

James, starting v8, Veeam B&R does wait for the post-job script to complete. Please review this thread for some hints on how to work around this.

[pkelly_sts]

What is the best way to get a (powershell) post-job script configured in a Backup Copy job which is triggering a "replication from backup copy job" to return/complete success if it successfully /started/ the replica job?

Within the backup copy job I'm not interested in whether the replica completed sucessfully or not, I just want to know that it was started correctly. The replica job obviously creates its own reports so happy with monitoring there. I'm not interested in the post-job script sitting & waiting for the manually-started job to complete.

I'm only having to do this because I can't chain the replica job to start after the "completion" of a backup copy job.

[foggy]

The script will return success once it is itself executed correctly, it doesn't wait for replica job to complete unless you explicitly create script that waits for the replica job completion status.

[obould]

Hello,

on one of my mysql servers I use a pre-freeze script to backup the databases.
After 15 minutes the job fails with a timeout error. Can anybody tell me if and where on a linux (centos) server I can change the timeout setting? I already tried to add a tools.conf file with timeout=3600 param but to no avail.

Best regards

[foggy]

Please use the registry values mentioned in the beginning of this thread.

[ghaxians]

Default timeouts of 900 seconds (15 minutes) is customizable for both pre- and post-job scripts via the following registry values:

Code: Select all

PreJobScriptTimeoutSec (DWORD)
PostJobScriptTimeoutSec (DWORD)

Unlike with previous versions, in v8 we are executing these scripts as a part of the job, instead of just launching them in a separate shell window as before. This allows us to reflect script execution results (depending on the exit code) in the job status, so that the job notification email tells you everything you need to know. Without this in place, our customers had to monitor additional activities spawned by post-job script separately.

I hope you will find this useful and yes, this is one feature that I forgot to include in the What's New document.

Hi Gostev, are these registry values still supported/relevant for VM backup with B&R V10 (and V11 ?)
Regards,
G

[Vitaliy S.]

Hi Gilles, as far as I see from the internal KBs there were changes since these keys were introduced in v8. Regardless of that, can you please give us a bit more info on what you're trying to achieve with these keys? Do you see any issues with post-job scripts in v10/11?