Notification Email Problems v7.0.0.771

[mc1903]

For the past 6 months my Veeam backup jobs have been very smoothly and I have been getting email notifications for every job. In the early hours of yesterday morning this all stopped!

My vSphere vCenter server rebooted during the first few minutes of a Veeam backup session - Veeam had requested a snapshot and vCenter had completed the task, but then the communications stopped and Veeam timed out the jobs.

Afer vCenter was back on-line, one of the subsequent jobs continued to fail (it is a 8Tb server and runs every 2 hours) until I manually deleted the failed job's snapshot and then I was able to run it again. Other smaller servers continued to backup OK even with their failed job snapshots still being there.

Whilst I was deleting the snapshots in vCenter I disabled the effected jobs (highlighted all of them at once and selected 'disable). I re-enabled them one-by-one, as I cleared the snapshots and I started each job to test - they all ran OK.

However, after doing none of the completed jobs sent email notifications to me - Including those that were not effected by the vCenter issue (and that had not been disabled). I did not reconfigure the jobs in any way - just disabled and re-enabled them.

I can send the test notifications OK , but the scheduled jobs do not send anything when they complete.

My SMTP relay host IP is a cluster, so I am happy it was available when the jobs completed.

I am unsure about upgrading to Patch 3a or 4 as I feel it may introduce more issues that may fix for me.

Any thoughts?

Martin

[foggy]

Martin, what are notification settings both for individual jobs and specified in general Veeam B&R options?

[mc1903]

Hi,

In the general options - I have the SMTP server IP address and the 'To:' email address set, along with the default 'Subject: [%JobResult%] %JobName% (%VmCount% VMs) %Issues%'. I do not have a 'From:' email address set (only ever had an address here when I tested the SMTP worked). Notify on Success, Failure and Warning check boxes are all checked.

In each of the individual jobs - I have a single email address in the 'Send notifications to'. This has not changed since it was first set-up over 6 months ago and has been working as expected for each job up until the issues I had yesterday.

If you need any more information or if I should open a support case please let me know.

Thanks
Martin

[Gostev]

Hi Martin, I do recall some of the recent patches did fix some issue with SMTP notifications. Before you install the latest patch, you can always open a support case and make sure you are experiencing the issue that was fixed. Which is actually very unlikely in this case, because you said it was working before, so this seems to be an environmental issue with 99.99% chance. Thanks!

[mc1903]

Hi Gostev,

This is getting weird..!

The email notifications have started working again. The last time I edited the job notification settings was about 11:00 today (13/06) and they started to appear after the 19:15 backup completed..!

Not complaining - just a little confused..!

Code: Select all

Job Name	Start Date	Start Time	Stop Time	Duration	Result	Email Received
2 Hourly Backup of x	13/06/2014	21:15	21:21	00:05:58	Success	Yes
2 Hourly Backup of x	13/06/2014	19:15	19:21	00:05:57	Success	Yes
2 Hourly Backup of x	13/06/2014	17:15	17:21	00:06:34	Success	No 
2 Hourly Backup of x	13/06/2014	15:15	15:21	00:06:05	Success	No 
2 Hourly Backup of x	13/06/2014	13:15	13:20	00:05:50	Success	No 
2 Hourly Backup of x	13/06/2014	11:15	11:21	00:06:12	Success	No 
2 Hourly Backup of x	13/06/2014	09:15	09:21	00:06:05	Success	No 
2 Hourly Backup of x	13/06/2014	07:15	07:18	00:03:28	Success	No 
2 Hourly Backup of x	13/06/2014	05:15	05:19	00:04:05	Success	No 
2 Hourly Backup of x	13/06/2014	03:15	03:19	00:04:08	Success	No 
2 Hourly Backup of x	13/06/2014	01:15	01:20	00:05:20	Success	No 
2 Hourly Backup of x	12/06/2014	23:15	23:20	00:05:19	Success	No 
2 Hourly Backup of x	12/06/2014	21:15	21:22	00:07:40	Success	No 
2 Hourly Backup of x	12/06/2014	19:15	19:22	00:06:53	Success	No 
2 Hourly Backup of x	12/06/2014	17:15	17:20	00:05:39	Success	No 
2 Hourly Backup of x	12/06/2014	15:53	16:07	00:13:39	Success	No ***
2 Hourly Backup of x (Retry 3)	12/06/2014	13:52	13:55	00:02:25	Failed	Yes
2 Hourly Backup of x (Retry 2)	12/06/2014	13:40	13:42	00:02:14	Failed	N/A
2 Hourly Backup of x (Retry 1)	12/06/2014	13:27	13:30	00:02:20	Failed	N/A
2 Hourly Backup of x	12/06/2014	13:15	13:17	00:02:19	Failed	N/A
2 Hourly Backup of x (Retry 3)	12/06/2014	11:52	11:55	00:02:11	Failed	Yes
2 Hourly Backup of x (Retry 2)	12/06/2014	11:40	11:42	00:02:27	Failed	N/A
2 Hourly Backup of x (Retry 1)	12/06/2014	11:27	11:30	00:02:13	Failed	N/A
2 Hourly Backup of x	12/06/2014	11:15	11:17	00:02:27	Failed	N/A
2 Hourly Backup of x (Retry 3)	12/06/2014	09:52	09:54	00:02:13	Failed	Yes
2 Hourly Backup of x (Retry 2)	12/06/2014	09:40	09:42	00:02:12	Failed	N/A
2 Hourly Backup of x (Retry 1)	12/06/2014	09:27	09:29	00:02:11	Failed	N/A
2 Hourly Backup of x	12/06/2014	09:15	09:17	00:02:21	Failed	N/A
2 Hourly Backup of x (Retry 3)	12/06/2014	07:52	07:54	00:02:13	Failed	Yes
2 Hourly Backup of x (Retry 2)	12/06/2014	07:40	07:42	00:02:13	Failed	N/A
2 Hourly Backup of x (Retry 1)	12/06/2014	07:27	07:29	00:02:13	Failed	N/A
2 Hourly Backup of x	12/06/2014	07:15	07:17	00:02:27	Failed	N/A
2 Hourly Backup of x (Retry 3)	12/06/2014	05:52	05:54	00:02:15	Failed	Yes
2 Hourly Backup of x (Retry 2)	12/06/2014	05:40	05:42	00:02:11	Failed	N/A
2 Hourly Backup of x (Retry 1)	12/06/2014	05:27	05:29	00:02:13	Failed	N/A
2 Hourly Backup of x	12/06/2014	05:15	05:17	00:02:25	Failed	N/A
2 Hourly Backup of x (Retry 3)	12/06/2014	04:04	04:07	00:02:13	Failed	Yes
2 Hourly Backup of x (Retry 2)	12/06/2014	03:52	03:54	00:02:15	Failed	N/A
2 Hourly Backup of x (Retry 1)	12/06/2014	03:40	03:42	00:02:13	Failed	N/A
2 Hourly Backup of x	12/06/2014	03:15	03:30	00:14:45	Failed	N/A
2 Hourly Backup of x	12/06/2014	01:15	01:20	00:05:08	Success	Yes

*** Prior to this manual backup running I disabled the job and deleted the unwanted snapshot via vCenter.

Martin

[veremin]

Strange, indeed. I do recommend opening a ticket with our support team in case the issue shows up again. Thanks.

[mc1903]

Hi v.Eremin,

I will open a support request on Monday morning.

Overnight the 2 Hourly Web Servers job stopped emailing notifications again, while it's partner 2 Hourly Database Servers job (that starts at exactly the same time) continued to email it's notifications as expected.

I have written a Powershell report to send notifications from outside of Veeam B&R, so I have some visibility of the job status. Now looking to get the hourly job status info into our network monitoring software (PRTG by Paessler) as well.

Have a good weekend and thanks for your help.

Martin

[foggy]

It would be highly appreciated if you could post the case ID number here, so we could track down the resolution.

[mc1903]

Morning,

This problem is now getting worst. Yesterday I got 16 notification emails for 50 jobs.

I have finally logged a case - 00641734

Thanks
Martin