Veeam Synthetic full failure

[exarchbcn]

Greetings,

Considering the crucial role a synthetic full backup plays in a backup chain, is there a mechanism to identify if such a job fails?

I haven’t come across any reports or alerts in Veeam One that could assist in identifying this type of failure. If not manually noticed, it could potentially lead to an endless incremental chain.

Best Regards!

[PetrM]

Hello,

The job will be failed and you will see a corresponding error in the email report and job session statistics.

Thanks!

[Kosk]

Hi PetrM

exarchbcn teammate here.
We have set all jobs to do Synthetic Full on Saturday.

The issue is that in these days some jobs fails. Monitoring tool opened a ticket about this Failure (which did not differ from other Incremental Failure)
Veeam keeps running and on Sundays, it completes the Incremental Backup correctly. So monitoring team takes this Failure ticket and check's the last Status for the job.
As the last status for this job was Success, the ticket is closed without taking care of Synthetic full not completed.

Now we are trying to avoid this, and we were looking for any Veeam report able to detect this failures on Synthetic Full (that means chain backup is Forever Incremental until Synthethic Full completes)
We faced this issue more times in our infrastructure. We noticed this by vCenter alert about datastore full, and it was painful to work with this as we have to move/delete the whole chain backup...

P.D there is no option to force Synthetic Full from Console (only possible changing job settings) and we have more than 700 jobs running on our infrastructure

Thanks

[tyler.jurgens]

Very difficult to troubleshoot why your synthetic fulls are failing. That said, sounds like your support team needs some additional training.

Sometimes merge operations can fail. I've dug into them in the past and it can be very tricky to determine why, but double-check your Antivirus (even Windows Defender) settings if you are using a Windows repo. Also, check to see if your repository is being overwhelmed and simply can't handle the IO load at that time.

You can always change the times when synthetic full backups are created - that may help.

I've generally tackled merge failures with a nuclear option. If the job fails to merge multiple times an active full backup can often fix any of those challenges. It will start a new chain and your backups will often work fine to merge the next time.

If you can't do that, just keep working with support.

[PetrM]

Hello,

@Kosk
From my point of view, you should explicitly say to the team that takes care of tickets that they cannot close them unless the root cause of the failure is found. To find this very root cause, you need to contact our support team.

Also, I recommend configuring email notifications so that you get info about failures directly to your mailbox and can work on it.

Thanks!

[AlexHeylin]

We also faced this recently - our backups run seven days a week, but our backup check system & process were excessively focused on our working week.
Given how lightweight a synthetic full is, we're experimenting with moving them to between 6pm Sunday - 2am Friday so our backup checks will catch any errors until we can rework our backup check system / process to reliably catch failures that occur outside that timeslot (at weekends).

[tyler-no-work]

+1 to this 'feature' request. To balance the load on our repo, synthetic fulls run on different days. Even just an additional notation in the notification email or Veeam ONE reports would help prioritize the resolution on weekends and holidays.

[exarchbcn]

I agree that all failures must be checked, but I also believe that the synthetic full has an important role to mantain the full backup chain that probably deserves special attention when failing.

[RubinCompServ]

Hello,

@Kosk
From my point of view, you should explicitly say to the team that takes care of tickets that they cannot close them unless the root cause of the failure is found. To find this very root cause, you need to contact our support team.

In our case, it's an automated system ("Backup Radar") that opens a ticket when a backup fails, and automatically closes the ticket if it gets a good run on it. Otherwise, we'd have literally hundreds of tickets that need to be reviewed because of transient failures that self-healed but tickets didn't close, and >99% of those tickets would be wastes of time.

Also, I recommend configuring email notifications so that you get info about failures directly to your mailbox and can work on it.

Same issue as above.

[PetrM]

Hi David,

Ok, understood. At the moment, I don't have any ideas on how to handle it on our side except for some unofficial and unreliable workarounds like automated parsing of debug logs. Probably, it would be possible to make some kind of customization at the level of your system that opens and closes tickets. The feature request is noted, we'll think about implementation in one of our future versions as long as we have enough similar requests.

Thanks!

[RubinCompServ]

@PetrM,

It's possible that even an extra variable in the email subject would be sufficient. If there was a variable for "Run Type" (or something similar) that would differentiate between, for example, "Full", "Incremental", & "Synthetic Full (all of which must exist somewhere because they show up in the job history) we could build rules based upon those. Thanks!

[PetrM]

Hi David,

Got it! I truly appreciate your feedback, very good idea of implementation!

Thanks!