Comprehensive data protection for all workloads
Post Reply
perjonsson1960
Veteran
Posts: 443
Liked: 44 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

Backup files health checks and defragmentations

Post by perjonsson1960 »

Folks,

For the first time I activated the "Backup files health check" in one of our backup jobs. The job is backing up six SQL servers in VMware with a retention cycle of 14 days. The data size is approx. 5 TB, and the full backup size approx. 2.7 TB. I noticed that the health check took 2 hours and 51 minutes in total for all six servers. How much is that affecting the overall performance of the backup servers, when other jobs are running at the same time? We have around 210 servers in VMware, and 15 physical servers, and backup jobs are pretty much running nonstop from 17:00 in the afternoon up until 07:00 in the morning. And then we are running backup copy jobs during the day.

When the health checks are performed, do they check all the restore points in the chain?

And what about the full backup file defragmentation feature? How much does that affect the performance of the backup servers?

I was thinking about running the health checks and also the defragmentations once a month, but I really don't know when to run them all... I realize that I must spread them out on separate days throughout the month, and maybe not run the health check and the defrag on the same day in a certain job. Is there a best practice for environments like ours, with loads of servers and loads of data? Can I just activate these features for all the backup jobs on different days throughout the month, and then forget about it, or do I have to give it even more thought than I already have?

Sincerely,
Per Jonsson
Sweden
ejenner
Veteran
Posts: 636
Liked: 100 times
Joined: Mar 23, 2018 4:43 pm
Full Name: EJ
Location: London
Contact:

Re: Backup files health checks and defragmentations

Post by ejenner »

We had issues running health checks because we found the healthcheck wouldn't succeed over a certain size of backup. Somewhere between 1.5 and 6TB. As yours still work at 2.7TB your limit for a successful healthcheck may be higher due to differences in hardware configuration. Or we may find our limit would be similar and the limit is somewhere between 2.7 and 6. We don't have any backups of a size between 1.5 and 6 so don't know what would and wouldn't work.

As a workaround to the healthcheck issue our plan is to use a second Backup and Replication server to perform copy jobs and then we'll healthcheck those instead. That way, healthchecks can run forever without impacting on the daily backup.
perjonsson1960
Veteran
Posts: 443
Liked: 44 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

Re: Backup files health checks and defragmentations

Post by perjonsson1960 »

The 2.7 TB is the total for all the six servers in that job. The biggest backup file is approx. 1 TB, so I guess your 1.5 to 6 TB range may still apply, if it is the file sizes that is the problem, and not the total backup size for the job.

We have full backup files that are around 2.5 and 2.9 TB in size, but I haven't dared to turn on the health checks for those... ;-)
When it comes to the backup copy jobs, the health checks are activated in all of them. That seems to be the default setting for backup copy jobs.
As you say, maybe it is enough to do health checks on the backup copies, and skip it on the backups?
What about the full backup file defragmentation? When is that necessary? We are backing up all our virtual servers using reverse incremental, and with the physical servers we are creating synthetic fulls every Saturday, and the retention cycle is 14 days for all backups.

PJ
ejenner
Veteran
Posts: 636
Liked: 100 times
Joined: Mar 23, 2018 4:43 pm
Full Name: EJ
Location: London
Contact:

Re: Backup files health checks and defragmentations

Post by ejenner »

Probably best to follow Microsoft's best practice for defragmentation, probably taking into account whether or not you use ReFS formatted disk.
perjonsson1960
Veteran
Posts: 443
Liked: 44 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

Re: Backup files health checks and defragmentations

Post by perjonsson1960 »

Microsoft? I am talking about the defrag of full backup files in Veeam.
And we are using ReFS on all backup repositories.
ejenner
Veteran
Posts: 636
Liked: 100 times
Joined: Mar 23, 2018 4:43 pm
Full Name: EJ
Location: London
Contact:

Re: Backup files health checks and defragmentations

Post by ejenner »

If Veeam defrags the file or MS does it... I'm assuming the same effect? Veeam does it by creating a new file. MS does it by making the original file consistent. Either way... it's going to shift a load of data around on the disk and that may be a negative if your filesystem is ReFS. Hence, I'd probably have a look at what MS says. Not that I know the answer. If you were using NTFS I'd say it makes sense to run it before creating synthetic fulls (if you are doing that?) or performing a health check. But as you're not using NTFS I'd do some further reading to try and find out whether or not ReFS likes other programs to perform maintenance on files. I'd say probably not.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup files health checks and defragmentations

Post by foggy » 1 person likes this post

Hi Per, health check verifies the latest restore points state only and is performed by the data mover installed on the repository server (or the gateway, in case of CIFS or dedupe type repository), so other servers are not affected. Spreading them out throughout the month is a good idea though. The default monthly schedule for both health checks and compact operations is recommended. Btw, if you're using ReFS, compact operations should be fast and spaceless, due to Fast Clone functionality.
perjonsson1960
Veteran
Posts: 443
Liked: 44 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

Re: Backup files health checks and defragmentations

Post by perjonsson1960 »

We have two physical MS Exchange servers. One backup job backs up the first server, and when that job is finished, the backup job for the second server starts. If the first job runs a backup files health check, doesn't the second backup job start until the health check is finished? Or does the second job start when the backup portion of the first job is finished? Otherwise the second job will probably be delayed for, I'm guessing, at least two hours. And quite frankly, we are running out of hours here... 24 hours a day is not enough... ;-)
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup files health checks and defragmentations

Post by foggy »

Do you mean you chain the jobs to be performed one after another? In this case the second job will start after health check for the first one completes.
perjonsson1960
Veteran
Posts: 443
Liked: 44 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

Re: Backup files health checks and defragmentations

Post by perjonsson1960 »

Correct, the jobs are chained. So, since the full backup file is about 2.8 TB in the first job, and not counting the incrementals that may exist, then I guess the health check will take between two and three hours, because we have another job that backs up six SQL servers, and that health check took almost three hours, and the six full backup files are about 2.4 TB together. I don't know if one single file will take longer or shorter time than six full backups of roughly the same total backup size.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup files health checks and defragmentations

Post by foggy »

It doesn't depend on the amount of VMs in restore point, but more on the number of blocks to verify, so I'd assume time will be comparable.
perjonsson1960
Veteran
Posts: 443
Liked: 44 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

Re: Backup files health checks and defragmentations

Post by perjonsson1960 »

I wonder if it is necessary to do health checks when synthetic fulls are created once a week, and we also do backup copies continuously? I just noticed that the health check on one of the Exchange servers took six hours... The full backup file in this case is approx. 2.8 TB in size.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup files health checks and defragmentations

Post by foggy »

Yes, in this case periodic health check is recommended.
Robvil
Expert
Posts: 172
Liked: 20 times
Joined: Oct 03, 2016 12:41 pm
Full Name: Robert
Contact:

Re: Backup files health checks and defragmentations

Post by Robvil »

I do health check every month on a 12TB backup set without issues on a REFS volume. It takes a little over 24 hours.
perjonsson1960
Veteran
Posts: 443
Liked: 44 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

Re: Backup files health checks and defragmentations

Post by perjonsson1960 »

It took between six and seven hours each for our Exchange servers. I don't know how many files that are checked in the backup chain, but the full backup file is approx. 3 TB.
Post Reply

Who is online

Users browsing this forum: No registered users and 258 guests