Comprehensive data protection for all workloads
cgsm
Enthusiast
Posts: 95
Liked: 21 times
Joined: Oct 05, 2021 3:55 pm
Contact:

v12 Health Check Email Notifications for Failures Not Sending

Post by cgsm »

Since v12 and the changes to how Health Checks are performed (scheduled separately, not part of first backup job for the day), I do not get email notifications when the health check for a backup job fails. I have the backup job set to use the global notification settings which is set to send emails on warnings or failures.

With v11, I would get an email if the health check failed. Now, I have to manually go check the History > System > Health Check entries.

This seems to be a regression...I want to know when the health checks fails. Thoughts?
Gostev
Chief Product Officer
Posts: 31741
Liked: 6914 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by Gostev »

This would be a regression indeed.
cgsm
Enthusiast
Posts: 95
Liked: 21 times
Joined: Oct 05, 2021 3:55 pm
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by cgsm »

Is there any way to run the Health Check manually, so I can cancel it mid-running and "show" someone the bug I am seeing (no email sent)?
PetrM
Veeam Software
Posts: 3369
Liked: 552 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by PetrM »

Hello,

There is no option to run this task manually but you can adjust the schedule.

Thanks!
cgsm
Enthusiast
Posts: 95
Liked: 21 times
Joined: Oct 05, 2021 3:55 pm
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by cgsm » 1 person likes this post

Petr,
I will try scheduling for a different time and trying to manually make the job fail for testing.

I have also opened case # 05985729 trying to explain the issue the best I can.
Gostev
Chief Product Officer
Posts: 31741
Liked: 6914 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by Gostev »

I heard QA has already confirmed the issue, so no worries about reproducing.
cgsm
Enthusiast
Posts: 95
Liked: 21 times
Joined: Oct 05, 2021 3:55 pm
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by cgsm »

@Gostev,

Great to hear it wasn't "just me" configuring or doing something wrong! Is this a confirmed regression at this point? What should I say to my support contact for my case?
Gostev
Chief Product Officer
Posts: 31741
Liked: 6914 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by Gostev »

I guess you can just close the case, there's no point as this is not even a bug. Rather a missing feature: we missed this requirement in the task of splitting health check into a separate process. No one have thought about the loss of the job-based email notification and the need to create a new one, a fair mistake by PM that QA did not catch.
mkretzer
Veeam Legend
Posts: 1172
Liked: 395 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by mkretzer »

@Gostev do you think this is fixable in the first update? Or is it too late for that?
Gostev
Chief Product Officer
Posts: 31741
Liked: 6914 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by Gostev »

Normally it would not be even considered for the cumulative patch anyway, since it's not a bug. Remember our patches don't even change a build number, and they are about making very granular fixes to the exiting code. An example of bug would be: the email notification functionality is there but notifications are not being sent when it's a full moon (figuratively speaking). Whereas here we're talking about non-existing functionality that will first need to be implemented.
cgsm
Enthusiast
Posts: 95
Liked: 21 times
Joined: Oct 05, 2021 3:55 pm
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by cgsm » 1 person likes this post

For the time being, I implemented the following workaround: I disabled the Backup Job's Health Check and created a new SureBackup job that is linked to the backup and has "validate entire disk..." enabled. This SureBackup job has email notifications enabled.
JPMS
Expert
Posts: 120
Liked: 36 times
Joined: Nov 02, 2019 6:19 pm
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by JPMS » 1 person likes this post

I disabled the Backup Job's Health Check and created a new SureBackup job that is linked to the backup and has "validate entire disk..." enabled.
The problem with that solution is that the check takes about five times longer doing it from Surebackup (even though it is the same data check being performed)

You can read more about it here Multi-threaded backup file validation for SureBackup
cgsm
Enthusiast
Posts: 95
Liked: 21 times
Joined: Oct 05, 2021 3:55 pm
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by cgsm » 1 person likes this post

@JPMS Yes, the SureBackup validation of backup files does take dramatically longer (I started the thread on the slow SureBackup file validation; cause is because validation runs sequentially instead of in parallel like on a Health Check). In mind though, getting a notification upon validation failure is more important than the time validation takes.
JPMS
Expert
Posts: 120
Liked: 36 times
Joined: Nov 02, 2019 6:19 pm
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by JPMS »

I started the thread on the slow SureBackup file validation
LOL. Talk about teaching your grandmother how to suck eggs!

I agree that getting the notification is important but sometimes it is not possible to accommodate the additional time required by using the Surebackup method. For the moment I think we will check the console history weekly. Our infrastructure seems pretty stable and we have never suffered from data corruption in the 4-5 years we have used B&R.

What we have suffered from is our health checks not running at all for the last 10 weeks since we installed v12 and were completely oblivious to the issue (v12 health check defaults to running at 22:00 when our repo isn't available)!

It's only one of the issues I have with the changes in health checks in v12 - Confused by v12 storage-level corruption guard
ENBS
Novice
Posts: 9
Liked: 1 time
Joined: Jan 16, 2023 10:07 am
Full Name: ENBS
Contact:

[MERGED] VBR 12 Health check E-Mails

Post by ENBS »

Hi,

since i hadn't found anything on that i make a new Thread. After upgrading to VBR12 the Health-Checks are run independent from the Jobs (which is fine) and i see them in History > System > Health check but how can i activate an email-Notification for this? I get the Joblog and the Configuration Backup - Log but not this "Healthjob".

regards ENBS
Mildur
Product Manager
Posts: 9146
Liked: 2407 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by Mildur » 1 person likes this post

Hello ENBS

Mail Notifications will be added again in one of our next patches. Please see the comments from my colleagues in this topic.

Best,
Fabian
Product Management Analyst @ Veeam Software
ENBS
Novice
Posts: 9
Liked: 1 time
Joined: Jan 16, 2023 10:07 am
Full Name: ENBS
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by ENBS »

@Mildur: Thanks, and sorry for the "repost" i hadn't found this thread before.

regards, ENBS
JPMS
Expert
Posts: 120
Liked: 36 times
Joined: Nov 02, 2019 6:19 pm
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by JPMS » 1 person likes this post

Mildur wrote: Jul 10, 2023 1:57 pm Mail Notifications will be added again in one of our next patches. Please see the comments from my colleagues in this topic.
Are any other changes currently planned for Health Check?

I'm particularly thinking about the ability to chain it as a job and Powershell support.
Gostev
Chief Product Officer
Posts: 31741
Liked: 6914 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by Gostev »

@Egor Yakovlev did email notification for health check jobs made it into 12.1?
Egor Yakovlev
Product Manager
Posts: 2557
Liked: 692 times
Joined: Jun 14, 2013 9:30 am
Full Name: Egor Yakovlev
Location: Prague, Czech Republic
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by Egor Yakovlev »

Nope, slipped off onwards.
JPMS
Expert
Posts: 120
Liked: 36 times
Joined: Nov 02, 2019 6:19 pm
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by JPMS »

You're kidding right?

You broke/deleted/forgot (choose your own adjective) this functionality when you moved Health Checks out of Backup Jobs and seven months later you haven't even started to fix it?!

It is completely ridiculous that we have to login to a console and check this manually for every Health Check.
Gostev
Chief Product Officer
Posts: 31741
Liked: 6914 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by Gostev » 1 person likes this post

I'm not too happy about this situation either...
Gostev
Chief Product Officer
Posts: 31741
Liked: 6914 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by Gostev »

...but after checking with the corresponding team, the requirement is still in their pipeline so there's still a chance this makes it into 12.1
Plus I found that the other connected requirement (Windows Events Log event for same) has already been implemented.
vitami
Lurker
Posts: 2
Liked: never
Joined: Oct 28, 2021 2:30 pm
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by vitami »

Veeam 12.1 has been released but no mention of this issue/limitaion/regression: https://www.veeam.com/veeam_backup_12_1 ... tes_rn.pdf

Should I contact support to place my concern? We only found out by pure luck, at least they should put it into the release notes (even if it isn't classified as an issue or bug by veeam). Many failed health jobs could have gone undetected because of this.
Mildur
Product Manager
Posts: 9146
Liked: 2407 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by Mildur » 2 people like this post

Hi Vitami

Health Check Notifications are part of the v12.1 release. Please see our "What's new - v12.1" document on page 8:
https://www.veeam.com/veeam_backup_12_1 ... new_wn.pdf
Health check email notification — storage-level corruption guard functionality will now send email
notifications with the session results. This functionality was lost in V12 due to the separation of health
checks into a separate activity, instead of being an integration part of backup jobs and thus its email
notifications
Best,
Fabian
Product Management Analyst @ Veeam Software
FrancWest
Veteran
Posts: 499
Liked: 98 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by FrancWest »

It appears that successful health check notifications aren’t sent immediately just like other jobs. Instead they are summarized in a daily email sent at 10 p.m. Why this difference and how about failure emails, are they send out immediately or also only reported in the notification at 10 p.m.?
Gostev
Chief Product Officer
Posts: 31741
Liked: 6914 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by Gostev »

I guess the team was acting on my general guidance to not flood customers with instant emails for every single thing that happens. Because we have so many features by now with their own reports that customers tell us they stop reading them altogether!

But perhaps this can be made an option for those who want to be notified instantly.

Me personally, I think I would schedule health check for some weekend hours and would prefer to receive a summary email before I wake up on Monday morning, which is when I'm going to action it.
FrancWest
Veteran
Posts: 499
Liked: 98 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by FrancWest »

I understand, but how about health check failures? Are they send out immediately or do they also only appear in the summary email? If the latter then it should be an option to send out immediately, because that’s something you want to know asap so that you can take appropiate actions.
cgsm
Enthusiast
Posts: 95
Liked: 21 times
Joined: Oct 05, 2021 3:55 pm
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by cgsm » 1 person likes this post

I understand the reasoning of both sides of this: a summary email and an immediate email. To me, an immediate email seems like a good/better option to me, because if a health check fails I am absolutely going to investigate it right away.

On a related note, it doesn't look like you can turn off emails for "successful" health check. I really do not need an email if a health check is successful. That is just flooding my inbox with unnecessary info. It doesn't seem like the health checks takes account of the job's related "notify on success/warning/error" settings.
Gostev
Chief Product Officer
Posts: 31741
Liked: 6914 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v12 Health Check Email Notifications for Failures Not Sending

Post by Gostev »

@Egor Yakovlev that would be a bug.
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 168 guests