Storage Maintenance Days

[Matthew Dickens]

Hello All,

This is hopefully a simple question with a simple answer but I haven't yet found either on these forums.

I am reviewing the settings on our backup jobs (Forward Incremental, 5.29TB source, 3.15TB compressed, non deduped storage), specifically the performing of Storage-level corruption guard and Full backup file maintenance. One aspect of these options I have not been able to work out is when the check is performed to determine whether these tasks are run as they can significantly extend the length of a job and have a significant impact on whether or not subsequent jobs (replicas) have sufficient time to complete. Is the check performed at the start of job instance or when a job instance reaches the point in its task sequence that these tasks would be performed?

Example scenario:

Scheduled start time - 19:00
'Normal' runtime - 4 - 6 hours
Storage-level corruption guard - Saturdays
Full backup file maintenance - Saturdays

If the above job ran on a Friday evening and took 6 hours to backup the VMs would the Storage-level corruption guard and Full backup file maintenance run as part of this instance (assessed when the task is reached) or during the instance which starts on the Saturday evening (assessed when the job instance is started).

I am trying to schedule these appropriately to ensure that they run as early as possible during the weekend.

[foggy]

Hi Matthew and welcome,

Whether health check is performed at the beginning or the end of a job depends on the job type - backup copy or regular backup correspondingly. It is performed during the job cycle that has started on the day the health check is scheduled at. As for compact/defragmentation then it is performed at the end of the job but works for forever forward incremental or reverse incremental backup chains only, so is not applicable in your case (you're talking about regular backup with forward incremental mode, right?).

[oleg.feoktistov]

Hi Matthew,

Adding to foggy's comment, I would also like to highlight that health check is not performed during an active full backup job session.
So, in your scenario health check will take place at the end of Saturday's run started at 19:00 given that it is not an active full.

Thanks,
Oleg

[Matthew Dickens]

Hi both, thank you for the prompt response.

In my scenario I was referring to a normal backup job configured for forever forward incremental (no active or synthetic fulls or transforms), though it is interesting to know that there is a difference in behaviour between those and a backup copy job.

I always thought this was the case but have on occasions seen job instances from the Friday perform these tasks which has always cast doubt in my mind.

Regards,

Matt

[foggy]

Any chance those were actually started on Saturday (due to job chaining, for example)?

[Matthew Dickens]

No, this job is the first in the chain so it is the one with a scheduled start time

[foggy]

Sounds unexpected. If you see this again, feel free to contact support for a closer investigation.

[Matthew Dickens]

The only possible explanation I can consider is it occurring on a retry if that is starting after midnight. Now I know for certain the designed behaviour I can make an assessment if I see something unexpected in the future.

Thanks,

Matt

[JPMS]

Adding to foggy's comment, I would also like to highlight that health check is not performed during an active full backup job session.

Why? My understanding is that Health Check verifies the metadata against the backup therefore checking for corruption of the backup. Corruption can occur for an active full so why no Health Check?

[foggy]

Active Full reads all the data anew from the source, there's not much sense in verifying data that has just landed - metadata has just been calculated for it.

[JPMS]

I strongly disagree. If there is a problem with your repo (bit rot is one example) then your Active Full could be worthless - that's the point of Health Check! It is in effect how you verify your backup,

Your suggested logic also makes no sense to me. As an example, you run an Active Full and a day later you run a forward incremental and, say, 3% of your data has changed. If you then run a Health Check, 97% of what it is checking is in the Active Full. So why is it worth checking 97% and not 100%?

I'm actually shocked that Health Check won't run on an Active Full. How do I verify what has been written is good?

[ITP-Stan]

In my understanding directly after writing blocks of backup data, a calculation is made to verify the data written is the same as the production data.
So promptly re-calculating at the end of the job is almost pointless.

[JPMS]

Thanks for the response ITP-Stan.

It would be good to hear from Veeam exactly what verification is done of an Active Full. Unfortunately I am in the middle of trying to identify an intermittent issue with our backups with the aid of tech support so I don't want to do any testing myself ATM. From memory, a Health Check takes a significant time, whatever happens at the end of an Active Full doesn't so my concern would be how thorough the check is.

In the past we would run an Active Full as part of the 'weekend cycle' when other resources aren't being used and we have a bigger time window. As part of this cycle we would also run a Health Check and a Surebackup job. From what foggy is saying, the Health Check would not actually happen. My concern would be that an unverified Active Full would leave us with a useless backup. Even worse, as we only scheduled a Health Check on the days we did an Active Full, a Health Check would never happen and if we had an ongoing issue with our repo, all our backups could potentially be useless.

Veeam seem to know what they are doing so I suspect I am missing some information but as I'm paranoid about backup I would like to understand how I can be sure that an Active Full has been properly verified.

[wishr]

Hi,

SureBackup and Backup Validator are the ways to go to not get into trouble when your target storage fails.

Btw, @JPMS, could you please let us know your support case ID?

Thanks

[JPMS]

I'm not familiar with Backup Validator but looking at your link this is a command line utility that is primarily designed for manual use. Yes I suppose you could run it as a batch file linked to a job but nobody has explained why you can't just use Health Check on an Active Full which would effectively do the same job. It's a simple tick box on the backup job configuration, not a command line utility that I suspect a lot of people will be unaware of. I also wonder how many people tick the Health Check box unaware that it doesn't work with an Active Full (yes it is stated in the documentation but how many people read the documentation for every part of the software they use when they think they understand what an option means).

Surebackup isn't available in Community or Standard editions. So that isn't an option for some people. Being able to verify a backup's integrity is an essential part of any backup software and should be as simple as ticking a box to say you want it done but if you have these editions it isn't possible as part of a standard backup job.

My cases aren't related to this topic but are detailed here:post349109.html

[wishr]

Heath Check does not make any sense for a full backup. It's a time and resource-consuming operation that involves the validation of both the data blocks on the target and corresponding metadata itself. Both data blocks on the target and backup metadata are built during the full backup run. So there is nothing to compare the initial data set with yet. This means that more time passes more valuable health check becomes. The value growth exponentially keeping in mind how much time and resources it takes. If the inability of data transmission protocols to handle data transfer failures and report them (meaning you get a respective job failure) and the inability of storage to write and hold the blocks scares you so much, it's a subject to make sure that you use proper hardware and additionally build appropriate processes involving use of SureBackup or Backup Validator - it's up to you what to use.

Hope it helps!

[foggy]

Even worse, as we only scheduled a Health Check on the days we did an Active Full, a Health Check would never happen and if we had an ongoing issue with our repo, all our backups could potentially be useless.

This is not the case. If the health check is scheduled for the day the job itself does not run, it will be performed after the next scheduled job run completes.

[JPMS]

Sorry for the delay responding but everybody seems to see the Christmas holiday as a deadline for jobs that they have been procrastinating about for months!

Heath Check does not make any sense for a full backup. It's a time and resource-consuming operation that involves the validation of both the data blocks on the target and corresponding metadata itself. Both data blocks on the target and backup metadata are built during the full backup run. So there is nothing to compare the initial data set with yet.

You are looking at this from a Veeam point of view, not a backup point for view. You are only considering what Veeam does; I'm interested in what has actually been recorded in the backup. For me it does make sense to Health Check a full backup because it is verifying what has actually been written, not just what Veeam has requested be written.

If the inability of data transmission protocols to handle data transfer failures and report them (meaning you get a respective job failure) and the inability of storage to write and hold the blocks scares you so much, it's a subject to make sure that you use proper hardware and additionally build appropriate processes involving use of SureBackup or Backup Validator - it's up to you what to use.

I'm not "scared", I'm experienced. 30 years supporting IT infrastructure has taught me that even the best hardware/software goes wrong or has bugs. Also, as I've previously pointed out, Surebackup isn't an option if you are using Standard or Community editions. To use Backup Validator on a automated basis would require writing a script, finding a way to securely store the required credentials, test for a result code and then find a solution for emailing the result. This is 2019, I don't expect to have to go to this sort of effort to properly validate a backup when I have backup software that should do that for me!