v12 Health Check Email Notifications for Failures Not Sending

[cgsm]

Since v12 and the changes to how Health Checks are performed (scheduled separately, not part of first backup job for the day), I do not get email notifications when the health check for a backup job fails. I have the backup job set to use the global notification settings which is set to send emails on warnings or failures.

With v11, I would get an email if the health check failed. Now, I have to manually go check the History > System > Health Check entries.

This seems to be a regression...I want to know when the health checks fails. Thoughts?

[Gostev]

This would be a regression indeed.

[cgsm]

Is there any way to run the Health Check manually, so I can cancel it mid-running and "show" someone the bug I am seeing (no email sent)?

[PetrM]

Hello,

There is no option to run this task manually but you can adjust the schedule.

Thanks!

[cgsm]

Petr,
I will try scheduling for a different time and trying to manually make the job fail for testing.

I have also opened case # 05985729 trying to explain the issue the best I can.

[Gostev]

I heard QA has already confirmed the issue, so no worries about reproducing.

[cgsm]

@Gostev,

Great to hear it wasn't "just me" configuring or doing something wrong! Is this a confirmed regression at this point? What should I say to my support contact for my case?

[Gostev]

I guess you can just close the case, there's no point as this is not even a bug. Rather a missing feature: we missed this requirement in the task of splitting health check into a separate process. No one have thought about the loss of the job-based email notification and the need to create a new one, a fair mistake by PM that QA did not catch.

[mkretzer]

@Gostev do you think this is fixable in the first update? Or is it too late for that?

[Gostev]

Normally it would not be even considered for the cumulative patch anyway, since it's not a bug. Remember our patches don't even change a build number, and they are about making very granular fixes to the exiting code. An example of bug would be: the email notification functionality is there but notifications are not being sent when it's a full moon (figuratively speaking). Whereas here we're talking about non-existing functionality that will first need to be implemented.

[cgsm]

For the time being, I implemented the following workaround: I disabled the Backup Job's Health Check and created a new SureBackup job that is linked to the backup and has "validate entire disk..." enabled. This SureBackup job has email notifications enabled.

[JPMS]

I disabled the Backup Job's Health Check and created a new SureBackup job that is linked to the backup and has "validate entire disk..." enabled.

The problem with that solution is that the check takes about five times longer doing it from Surebackup (even though it is the same data check being performed)

You can read more about it here Multi-threaded backup file validation for SureBackup

[cgsm]

@JPMS Yes, the SureBackup validation of backup files does take dramatically longer (I started the thread on the slow SureBackup file validation; cause is because validation runs sequentially instead of in parallel like on a Health Check). In mind though, getting a notification upon validation failure is more important than the time validation takes.

[JPMS]

I started the thread on the slow SureBackup file validation

LOL. Talk about teaching your grandmother how to suck eggs!

I agree that getting the notification is important but sometimes it is not possible to accommodate the additional time required by using the Surebackup method. For the moment I think we will check the console history weekly. Our infrastructure seems pretty stable and we have never suffered from data corruption in the 4-5 years we have used B&R.

What we have suffered from is our health checks not running at all for the last 10 weeks since we installed v12 and were completely oblivious to the issue (v12 health check defaults to running at 22:00 when our repo isn't available)!

It's only one of the issues I have with the changes in health checks in v12 - Confused by v12 storage-level corruption guard

[ENBS]

Hi,

since i hadn't found anything on that i make a new Thread. After upgrading to VBR12 the Health-Checks are run independent from the Jobs (which is fine) and i see them in History > System > Health check but how can i activate an email-Notification for this? I get the Joblog and the Configuration Backup - Log but not this "Healthjob".

regards ENBS

[Mildur]

Hello ENBS

Mail Notifications will be added again in one of our next patches. Please see the comments from my colleagues in this topic.

Best,
Fabian

[ENBS]

@Mildur: Thanks, and sorry for the "repost" i hadn't found this thread before.

regards, ENBS

[JPMS]

Mail Notifications will be added again in one of our next patches. Please see the comments from my colleagues in this topic.

Are any other changes currently planned for Health Check?

I'm particularly thinking about the ability to chain it as a job and Powershell support.

[Gostev]

@Egor Yakovlev did email notification for health check jobs made it into 12.1?

[Egor Yakovlev]

Nope, slipped off onwards.

[JPMS]

You're kidding right?

You broke/deleted/forgot (choose your own adjective) this functionality when you moved Health Checks out of Backup Jobs and seven months later you haven't even started to fix it?!

It is completely ridiculous that we have to login to a console and check this manually for every Health Check.

[Gostev]

I'm not too happy about this situation either...

[Gostev]

...but after checking with the corresponding team, the requirement is still in their pipeline so there's still a chance this makes it into 12.1
Plus I found that the other connected requirement (Windows Events Log event for same) has already been implemented.

[vitami]

Veeam 12.1 has been released but no mention of this issue/limitaion/regression: https://www.veeam.com/veeam_backup_12_1 ... tes_rn.pdf

Should I contact support to place my concern? We only found out by pure luck, at least they should put it into the release notes (even if it isn't classified as an issue or bug by veeam). Many failed health jobs could have gone undetected because of this.

[Mildur]

Hi Vitami

Health Check Notifications are part of the v12.1 release. Please see our "What's new - v12.1" document on page 8:
https://www.veeam.com/veeam_backup_12_1 ... new_wn.pdf

Health check email notification — storage-level corruption guard functionality will now send email
notifications with the session results. This functionality was lost in V12 due to the separation of health
checks into a separate activity, instead of being an integration part of backup jobs and thus its email
notifications

Best,
Fabian

[FrancWest]

It appears that successful health check notifications aren’t sent immediately just like other jobs. Instead they are summarized in a daily email sent at 10 p.m. Why this difference and how about failure emails, are they send out immediately or also only reported in the notification at 10 p.m.?

[Gostev]

I guess the team was acting on my general guidance to not flood customers with instant emails for every single thing that happens. Because we have so many features by now with their own reports that customers tell us they stop reading them altogether!

But perhaps this can be made an option for those who want to be notified instantly.

Me personally, I think I would schedule health check for some weekend hours and would prefer to receive a summary email before I wake up on Monday morning, which is when I'm going to action it.

[FrancWest]

I understand, but how about health check failures? Are they send out immediately or do they also only appear in the summary email? If the latter then it should be an option to send out immediately, because that’s something you want to know asap so that you can take appropiate actions.

[cgsm]

I understand the reasoning of both sides of this: a summary email and an immediate email. To me, an immediate email seems like a good/better option to me, because if a health check fails I am absolutely going to investigate it right away.

On a related note, it doesn't look like you can turn off emails for "successful" health check. I really do not need an email if a health check is successful. That is just flooding my inbox with unnecessary info. It doesn't seem like the health checks takes account of the job's related "notify on success/warning/error" settings.

[Gostev]

@Egor Yakovlev that would be a bug.