Scheduled jobs not starting

[nickmancini]

Greetings.

We manage about 20 separate Veeam deployments for our clients.

We've noticed that since the installation of 4.1, many jobs are not starting when they are scheduled to.

When we open Veeam, the jobs state that they are scheduled to run, but when opening the job history (HTML report or Realtime stats), the last job run was when it was done manually. When we open session history, there is no activity since the last manual job.

Veeam services appear to be running fine.

Veeam is installed on either Server 2003, WinXP or Win7. (all physical, not virtual)

Machines may be logged in or out, but always on.

any ideas?

[Gostev]

Hello Nick, there were no changes in 4.1 which could cause this, so most likely this is not related to upgrade to 4.1 (assuming you have upgraded from 4.0). So this issue would require full logs research. Please open a support case and send the sessions history screenshot, as well as job logs from Help | Support Information to support@veeam.com

[gmacdonald]

Greetings Nick,

I have the same problem with Veeam Backup & Replication 4.0. It is running on a physical XP box. The XP box is running 24/7 with occassional reboots to clean up windows. Either logged in or out. I have a backup job set to run every day at 6PM.

Veeam shows success in the client GUI yet the most recent vbk/vrb files and the HTML report show the last backup as 4 days ago?

This has happened a few times. When I catch it I run a manual backup and then Veeam behaves for a few weeks until it does it again.

Any news from support on your issue?

Thanks,
Graham

I will be upgrading to 4.1 but since it is stated that there have been no changes from 4.0 to 4.1 to cause this problem I just thought I would add my 2 cents.

[nickmancini]

I basically double checked the jobs, rebooted the machines, and watched everything like a hawk. Jobs started up again as scheduled.

Not quite sure what the issue was, but everything seems OK right now.

[Gostev]

Some account/domain security changes may require logoff/logon or reboot to fully apply, so may be this is whay happened.

[nickmancini]

It's happening again......

Job was scheduled and has been working fine for a few weeks, and, all of a sudden, it stops running as scheduled.

Veeam is installed on a freshly created Windows Server 2003 (x32) box with no other software installed - It's a machine dedicated to running Veeam.

O/S is fully patched and updated.

Veeam services appear to be running.

All other automatic services running OK also.

What can we do to get to the bottom of this?

(I realize that a reboot will likley solve the issue, but, we have many clients running Veeam. We need the scheduled jobs to run as scheduled, eveywhere, all the time....)

Thanks.

[Gostev]

Nick, please work with our support to get to the bottom of this. They would need to investigate debug logs to see what prevented the job from starting in the particular occasion.

[Arnold]

Good morning all

It would seem that our new shiney Veeam installation has started to play up a little, so I thought I would post here first to see if anyone has any suggestions.

For the second time now we have come in only to find that all our scheduled jobs have not run. When you look in the console as far as it is concerned they did run, but when you browse to the directory where the VBK and VRB files are located there are no new VRB files for the previous night.

Now we do seem to have a slightly funny issue with this server at present in so much as it seems to want to re-boot itself automatically everynow and then for no apparent reason. The reason I am mentioning this is that the last 2 failures of Veeam backups have been after a unscheduled re-boot.

I have taken a look in the VeeamBackupService.log.

The time stamp is just after re-boot as the system is coming back up. If I look back through the log to the last time it happened I get the same. This almost looks like Veeam is starting up before SQL has a chance to and then causing us issues with the scheduled jobs.

I have started the Veeam console, set a new scheduled job and this works, and now I am sure that tonight everything will run smoothly.

Yes I know that one of the first things to do is try and source the sporadic re-boot issue, but I am wondering if there is anything that I can put in place that may resolve this scheduled job problem?

Thanks in advanced .

A.

[Arnold]

Just as a small addition, I have added MSSQLSERVER as a dependecy within the Veeam Service, so this should hopefully stop it starting up before SQL, but I am not sure if this will resolve the issue, as this may not be what is actually causing our problem.

[Vitaliy S.]

Hello Arnold,

That's right, I would try catching the reason for your continuous machine reboots first, as those two issues might be connected to each other. However SQL server connection problems, usually state about either slow responding remote SQL server or high loaded one. In those both cases any application would experience such problems. You don't see updated status for your scheduled jobs, as all job configuration, schedulers etc. are stored in the SQL server, that suffers from connectivity issues.

I would recommend to look closely at the SQL server, do you have any other application using it? Is it a remote or local installation?

Thank you!

[Arnold]

Vitaliy, thank you for the quick reply . I guess it would have helped if I gave a little background on the system

This is a Windows 2008 Standard x64 server, running Veeam 4.1.1 and SQL 2008 Standard on the same box. The system has plenty of RAM and CPU power and I have never seen it stretched.

The server is used for Veeam and Veeam only, nothing else. Things have been working fine without any issues, but the last 2 unscheduled re-boots have resulted in none of the scheduled jobs running.

As I mentioned I have placed a dependency within the Veeam service for SQL Server. Just in case something funny is happening upon startup. Although, SQL is running all the time and it will have an ungraceful shutdown when the system re-boots.

The reason for my thinking behind Veeam trying to start work to soon was this section in the above log:

System.Data.SqlClient.SqlException: Database 5 cannot be autostarted during server shutdown or startup.

But I could be barking up the wrong tree

I have logged a call for the server to be looked at. But I just thought it would be worth getting peoples views here as well

[Vitaliy S.]

Arnold,

Thanks for giving the background, I might also suggest reviewing Windows Event log for more information on the unexpected reboots. As far as I got it right, you have a proper view about your scheduled jobs if your server runs smoothly. So let's keep an eye on your SQL server, because I'm pretty sure that the scheduled problem is caused by those unexpected reboots.

[Arnold]

Just a quick update....It turns out that there was a know bug with the model of server we were using for our Veeam box . If these models had a serial number below a certain level then they require a flash update to prevent.....yes you guessed it, sporadic re-boots for no reason.

I have now applied this patch and it is just a case of waiting to see if it has solved the problem .

As I mentioned, I had placed MSSQLSERVER in the Veeam service dependencies list. This was because I thought that after a sporadic re-boot Veeam was trying to use the database before SQL had started, I am not sure if SQL takes longer upon startup after an unscheduled re-boot. This then placed a lot of errors in the VeeamBackupService.log. The server did re-boot last night, but all scheduled jobs ran as normal, and upon checking the VeeamBackupService.log none of the earlier errors were logged

But hopefully after the new flash update the server will behave itself or it's off to the scrap yard with it

[Vitaliy S.]

Nice catch! I always try applying all the latest updates before using any hardware, but this is the first time I see such behavior. I'd call it "Undocumented server feature"

[Arnold]

I think this was one of those "You have to speak to the vendor" patches as it only affects a select few servers in the range. The sort that states if you are having these issues please call and we will provide you with a patch

The server was fully patched and upto date prior to go live, and all hardware was checked with diags over a 48hr period. Everything seemed fine....other than the fact the server thought it would be funny to just re-boot whenever it felt like it

Still hopefully all is now sorted and we can let this server do it's job without having to baby sit it