Healthchecks still starting after disabling jobs

[FrancWest]

Hi,

Case # 07382328

We had a 3 day problem with the network connection to our remote repository. So I disabled all copy jobs that use that repository to prevent them from running. I was surprised that even when the jobs were disabled, the healthchecks configured within those jobs were still starting. Of course they all failed since the connection to the repo wasn’t there. Support told me this is by design to ensure backupfile integrity. To me this sounds strange, I did disable the jobs for a reason. According to support, the only way to prevent this is editing all affected copy jobs and disable healthchecks in the settings.

Is this really how it should be, or is this an oversight in the way health checks are scheduled in the latest versions?

[jason.berry]

Hi,

If we look at this from a long-term retention perspective having the health checks running irrespective of backup copy job or primary job status would make sense. Veeam doesn't know that you have disabled those jobs because those workloads you were protecting are being retired and now that data in the repo is something that will be kept for a long period or it is because you having networking issues like in the case you referenced. Healthchecks ensures that when you do go to restore the data you can and there hasn't been any silent corruption in the storage layer. Because of that important function, they are enabled by default and require you to explicitly set it to off to make sure that it is not overlooked.

[FrancWest]

Hi,

Ok, but is it best practice for jobs that you don’t use anymore to keep them in Veeam in a disabled state? If I delete the job, it’s backup files move to the ‘orphaned’ node. Background retention still processes those orphaned backup files (https://helpcenter.veeam.com/docs/backu ... ml?ver=120) , so why don’t health checks do the same?

This way you can temporarily disable the job and stop health checks also, instead of going through the jobs one by one.

[jason.berry]

Yeah good point, it must be because the logic is that the health check is part of the job scheduling so when the job is gone then the health check scheduling is gone as well. I will chat with a few folks internally to see if there are recommendations for disabling/deleting jobs when they are no longer needed.

The only thing I could find for stopping the health checks would be using PowerShell to iterate through the jobs to disable them.

[FrancWest]

GFS retention is also part of the job, but that still works after deleting the job.

[FrancWest]

Another use case for turning off health checks when disabling a job:

today a drive in our repository raid array failed. The raid array started rebuilding. During a rebuild you don’t want intensive I/O like health checks to occur because that will slow down the raid rebuild significantly. I also disabled all copy jobs going to this repo in order for the rebuilt to complete asap. However, the health checks still run and I have to disable them. Because there isn’t, as far a I know, a way to see the schedules for all healthchecks in a simple view, I have to go through each job manually to see when the health check runs and disable it accordingly.

[mcz]

why not create a seperate section under jobs "health checks". If you enable it in the job, it will create/enable it in the health-check and then you could delete your backup job and it would still be listed under "health check"-jobs. You had full controll over it and could even change it long after the backupjob was gone...

[MartinO]

In instances like this, wouldn't it also be a good idea to place the repository into maintenance mode, since that's exactly what you're carrying out? That'll stop jobs of all kinds running that target that repo.

[FrancWest]

You can only put Scale-Out backup repositories in maintenance mode, not regular repositories.

[SnakeSK]

Im baffled that the RND team fails to acknowledge that the HC separation was a huge step back.

[mcz]

Im baffled that the RND team fails to acknowledge that the HC separation was a huge step back.

I see it the other way round: If it was clearly seperated (by a visible single job), then it would be very clear and understandable.