No notification even if job fails

[FrancWest]

Hi,

case # 04931222. After upgrading to V11 a copy job fails to start. It fails with 'Job has failed unexpectedly'. However, no notification mail is being send out. According to the engineer, the job fails at the database phase, reaching it's 900 seconds timeout. After opening the job properties and without changing anything and just go through the wizard, the job started working again. Looks like a V11 upgrade glitch.

My main concern though is, that apparently the notification procedure is dependent on which stage a job fails. If it fails to early in the job, it's not able to send out a failure notification for some reason. Now we missed the failed job for several days and only noticed it when going through the job history.

[PetrM]

Hi Franc,

I agree, the behavior does not seem to be optimal. I've asked our support team to analyze the issue a bit deeper.

Thanks!

[spiritie]

I've seen many jobs in Veeam that stops working with the behaviour you are mentioning, but not in relation to upgrading (both on v10 + v11)

Not even rebooting the entire VBR server would solve it. The symptom would be the timeout which could be seen in the console under last 24 hours,
and looking in the console the "last run" date would be blank even when the job was not running.

In the beginning I simply cloned the job, deleted the original one, and then renamed the cloned one to the original ones name again, and mapped the backup and it would work again.
After some times I found and easier fix. Rename the affected job and click all the way through the wizard, and then rename it back again to the original name and then start the job again.

We are an MSP and we have seen this across many of our customers as well as our internal VBR server, so this is not limited to a single VBR server (I've seen it on at least 7 different VBR installations now)

The first time it happened Veeam support did not really acknowledge it and asked to open a case if it ever happened again (wasn't bothered with using my time on it again with support)

I'd really like to Veeam to change their alarting logic on this, since no alarms are being sent to to SMTP, it can only be seen in the console, this is making us really paranoid since we cannot trust the Veeam e-mails we are getting (we only receive warnings and errors)

[FrancWest]

Exactly my point. We can't trust the notification system. Last run date was indeed blank. We have quite a few job and it's not feasible to launch the console every day to see the status of the jobs. Why else would you have a notification system for...

[mkretzer]

Case 04992408
Hello,

today we found that one of our copy jobs did not run for several days (with not any more information in the console):

29.08.2021 23:41:50 :: Job XXXX\yyyyyyy cannot be started. Timeout: 900,2135556 sec
29.08.2021 23:41:57 :: Job has failed unexpectedly

The worst thing: Veeam did not inform us via mail about the failing copy job! It failed again and again... I searched the forum but did not find similar issues.

Has someone seen something similar?

Markus

[Natalia Lupacheva]

Hello,

Moved your post to the existing thread with a similar problem.
Please keep working with Support team to get the solution.

Thanks!

[mkretzer]

Thank you, this indeed looks similar and also seems to be fixed by going through the job settings!

Since the issue seems to be well known will this be fixed in 11A?

[PetrM]

Hi Markus,

We're planning to fix the issue reported within the support request 04931222 in 11a except the cases in which the issue already appeared before upgrade. Anyway, I'd recommend to ask our engineer to check that you've faced exactly the same problem as the one described above. At first sight, it looks to be identical but it's better to confirm this assumption by debug logs analysis.

Thanks!

[spiritie]

@PetrM, can you confirm if this behaviour has been changed/fixed now in 11a?

[PetrM]

Hi Gert,

Yes, it's fixed in 11a.

Thanks!

[FrancWest]

Hi,

may we know how? For this specific issue only? Is the upgrade procedure fixed, so that the job doesn’t fail to start anymore? Or in general a more robust notification system that can cope with such situations?

[PetrM]

Hi Franc,

Yes, I'm talking about the specific issue that you reported in the first post. The issue is fixed at the level of logic of backup copy itself, it's not in the upgrade procedure nor in notification system.

Thanks!

[spiritie]

Hi @PetrM

I can confirm that this behaviour is not fixed in v11a, or we are seeing a different bug, but I'd expect the logic you changed would have fixed it so that it would have sent an SMTP notification anyhow.

I'm almost sure that this is the same bug where the jobs won't start. (Timeout 900 seconds on the job)

No SMTP was sent. The fail can be viewed in the console under "Last 24 Hours"

22-10-2021 14:49:09 :: Job Copy-Backup 037\Backup 037 cannot be started. Timeout: 900,2603688 sec
22-10-2021 14:49:09 :: Job has failed unexpectedly

Running VBR: 11.0.1.1261

I've detailed exactly in this post which behaviour we are seeing (see my last 2 posts)
veeam-backup-replication-f2/backup-copy ... 75525.html

[wishr]

Hi Gert,

Thanks for the heads up!

Do you have a support case where we can get all the details?

[spiritie]

Hi @wishr

I've created a support case (#05092467) to help you guys out, I'll upload the logs for this job later.

Unfortunately I cannot keep this job is a broken state for you guys, so I hope the logs will be enough.

EDIT:
Something more I've noticed is, that the console reports "Last Result: Success". even though the job is failed.

Wonder if this will impact the SMTP logic, as we've chosen only the receive warning + error on SMTP

[johnwatson]

Case 04992408 (Not My Case number, but with some possible solutions from this thread and my own troubleshooting) i.e. could not post fix without Support ID...LOL
Hey Veeam forum,

I had the same issue. My backup Copy job would time out with the following errors:
• Job … cannot be started. Timeout: 900.0453694 sec
• Job has failed unexpectedly
• Job finished with error at …
• No Alerts from VBR via SMTP

Steps that I took to fix the issue:

Attempt 1:
1. Upgraded from Veeam VBR 11 (11.0.0.837 P20210507) to Veeam 11a (Build 11.0.1.1261)
2. Restarted the server
3. Kicked off “Sync Now” on Backup Job
4. Backup failed with same error

Attempt 2: (Fixed the Issue) Some steps could have skipped but here they are
1. From Backup Infrastructure
a. Under Managed Servers Right clicked on Server that was having issues selected Properties and Click Next, Next, … Finish through the monkey prompts

b. Under Backup Repositories Right clicked on Repository that was having issues selected Properties and Click Next, Next, … Finish through the monkey prompts

2. From Home Right clicked on the Failed backup copy job selected Edit
a. Clicked Next

b. Noticed that under Objects the failed Backup Copy Job had 0 (Zero) for Size. So I removed it and Re-Added it. The size updated from 0 to its actual size.

c. Clicked Next

d. Clicked “Map Backup” and drilled down to Backup Location

e. Clicked Next …. Finish through rest of the monkey prompts

3. Kicked off “Sync Now” on Backup Copy Job

[suntech]

I hate to be a "me too" person, but me too.

I didn't have this issue but I updated from B&R Ver 11 to 11A yesterday (dec 21) and came in today to find the one backup copy job for offloading to be failing with a timeout and not sending a message. We don't have alerts for successful jobs enabled so, if the job thought it was successful, it didn't send anything.

I'm going to try this first because it looks pretty quick - "just rename the job and the rename it back again, this resolves this issue" from spiritie in the thread post422628.html#p422628 (Case # 04910549)

Then I'm going to try johnwatson's suggestion and will let you know if one or neither action works

[wishr]

Hi Everyone,

If you are experiencing any issues which look like the ones described in this thread, please open a support case so we could collect additional information and apply changes where and when necessary.

Thanks

[suntech]

My Case is #05196362

I did as John Watson suggested and it seems to have worked. The first few jobs that followed had errors but jobs that started afterwards over the course of a few days are okay. I'm waiting another 24 hours to confirm everything is good.

1. From Backup Infrastructure
a. Under Managed Servers Right clicked on Server that was having issues selected Properties and Click Next, Next, … Finish through
the prompts
b. Under Backup Repositories Right clicked on Repository that was having issues selected Properties and Click Next, Next, … Finish
through the prompts

2. From Home Right clicked on the Failed backup copy job selected Edit
a. Clicked Next
b. Noticed that under Objects the failed Backup Copy Job had 0 (Zero) for Size. So I removed it and Re-Added it. The size updated
from 0 to its actual size.
c. Clicked Next
d. Clicked “Map Backup” and drilled down to Backup Location
e. Clicked Next …. Finish through rest of the prompts

3. Kicked off “Sync Now” on Backup Copy Job"