Backup files health checks and defragmentations

[perjonsson1960]

Folks,

For the first time I activated the "Backup files health check" in one of our backup jobs. The job is backing up six SQL servers in VMware with a retention cycle of 14 days. The data size is approx. 5 TB, and the full backup size approx. 2.7 TB. I noticed that the health check took 2 hours and 51 minutes in total for all six servers. How much is that affecting the overall performance of the backup servers, when other jobs are running at the same time? We have around 210 servers in VMware, and 15 physical servers, and backup jobs are pretty much running nonstop from 17:00 in the afternoon up until 07:00 in the morning. And then we are running backup copy jobs during the day.

When the health checks are performed, do they check all the restore points in the chain?

And what about the full backup file defragmentation feature? How much does that affect the performance of the backup servers?

I was thinking about running the health checks and also the defragmentations once a month, but I really don't know when to run them all... I realize that I must spread them out on separate days throughout the month, and maybe not run the health check and the defrag on the same day in a certain job. Is there a best practice for environments like ours, with loads of servers and loads of data? Can I just activate these features for all the backup jobs on different days throughout the month, and then forget about it, or do I have to give it even more thought than I already have?

Sincerely,
Per Jonsson
Sweden

[ejenner]

We had issues running health checks because we found the healthcheck wouldn't succeed over a certain size of backup. Somewhere between 1.5 and 6TB. As yours still work at 2.7TB your limit for a successful healthcheck may be higher due to differences in hardware configuration. Or we may find our limit would be similar and the limit is somewhere between 2.7 and 6. We don't have any backups of a size between 1.5 and 6 so don't know what would and wouldn't work.

As a workaround to the healthcheck issue our plan is to use a second Backup and Replication server to perform copy jobs and then we'll healthcheck those instead. That way, healthchecks can run forever without impacting on the daily backup.

[perjonsson1960]

The 2.7 TB is the total for all the six servers in that job. The biggest backup file is approx. 1 TB, so I guess your 1.5 to 6 TB range may still apply, if it is the file sizes that is the problem, and not the total backup size for the job.

We have full backup files that are around 2.5 and 2.9 TB in size, but I haven't dared to turn on the health checks for those...
When it comes to the backup copy jobs, the health checks are activated in all of them. That seems to be the default setting for backup copy jobs.
As you say, maybe it is enough to do health checks on the backup copies, and skip it on the backups?
What about the full backup file defragmentation? When is that necessary? We are backing up all our virtual servers using reverse incremental, and with the physical servers we are creating synthetic fulls every Saturday, and the retention cycle is 14 days for all backups.

PJ

[ejenner]

Probably best to follow Microsoft's best practice for defragmentation, probably taking into account whether or not you use ReFS formatted disk.

[perjonsson1960]

Microsoft? I am talking about the defrag of full backup files in Veeam.
And we are using ReFS on all backup repositories.

[ejenner]

If Veeam defrags the file or MS does it... I'm assuming the same effect? Veeam does it by creating a new file. MS does it by making the original file consistent. Either way... it's going to shift a load of data around on the disk and that may be a negative if your filesystem is ReFS. Hence, I'd probably have a look at what MS says. Not that I know the answer. If you were using NTFS I'd say it makes sense to run it before creating synthetic fulls (if you are doing that?) or performing a health check. But as you're not using NTFS I'd do some further reading to try and find out whether or not ReFS likes other programs to perform maintenance on files. I'd say probably not.

[foggy]

Hi Per, health check verifies the latest restore points state only and is performed by the data mover installed on the repository server (or the gateway, in case of CIFS or dedupe type repository), so other servers are not affected. Spreading them out throughout the month is a good idea though. The default monthly schedule for both health checks and compact operations is recommended. Btw, if you're using ReFS, compact operations should be fast and spaceless, due to Fast Clone functionality.

[perjonsson1960]

We have two physical MS Exchange servers. One backup job backs up the first server, and when that job is finished, the backup job for the second server starts. If the first job runs a backup files health check, doesn't the second backup job start until the health check is finished? Or does the second job start when the backup portion of the first job is finished? Otherwise the second job will probably be delayed for, I'm guessing, at least two hours. And quite frankly, we are running out of hours here... 24 hours a day is not enough...

[foggy]

Do you mean you chain the jobs to be performed one after another? In this case the second job will start after health check for the first one completes.

[perjonsson1960]

Correct, the jobs are chained. So, since the full backup file is about 2.8 TB in the first job, and not counting the incrementals that may exist, then I guess the health check will take between two and three hours, because we have another job that backs up six SQL servers, and that health check took almost three hours, and the six full backup files are about 2.4 TB together. I don't know if one single file will take longer or shorter time than six full backups of roughly the same total backup size.

[foggy]

It doesn't depend on the amount of VMs in restore point, but more on the number of blocks to verify, so I'd assume time will be comparable.

[perjonsson1960]

I wonder if it is necessary to do health checks when synthetic fulls are created once a week, and we also do backup copies continuously? I just noticed that the health check on one of the Exchange servers took six hours... The full backup file in this case is approx. 2.8 TB in size.

[foggy]

Yes, in this case periodic health check is recommended.

[Robvil]

I do health check every month on a 12TB backup set without issues on a REFS volume. It takes a little over 24 hours.

[perjonsson1960]

It took between six and seven hours each for our Exchange servers. I don't know how many files that are checked in the backup chain, but the full backup file is approx. 3 TB.