Very slow health check (3+ days)

[thomasrensvold]

Hello!

I'm running B & R 9.5 U1 (9.5.0.823) on a VM with windows server 2012 R2. I have 1 backup job with about 160 VMs and 35,6TB. I run incremental with 1 active full monthly. Health check is set to run every other month. I also use "Per-VM backup files"

I have no problems with regular backup time, it normally finishes in about 4 hours on incremental and I guess about 35 hours on active full. My backup repository is a freenas box with cifs share. Both is connected with 10gbit.

The cpu usage on veeam server is very low (I run only 1 proxy server), and same with memory. The network speeds I see on the veeam server is about 500mbit read while the health check is running.

Cpu usage on backup repository is 0-1%, there is no problems handling what veeam server reads..

Any ideas on how to speed things up? How necessary is it to run the health check? Maybe I don't really need to do it since I run periodically active fulls anyway?

My healthcheck has now run for 42 hours and is 57% done

[thomasrensvold]

81 hours now - 99% complete

[foggy]

Hi Thomas, health check performance is mostly defined by the backup size and by what the storage can provide in terms of the read speed. In your case, I suppose, both factors play the role, since the backup is pretty large and the storage is not top grade in terms of IOPS. Health check protects against storage bit rot and allows to detect in-flight backup file data corruptions. With periodic active fulls the need for health check is not that high, since even if existing backup file is corrupt, active full will create a brand new one, but it still allows to be informed about existing backups health.

[jencalma]

HI Good Day

Just want to ask if Veeam Validator (https://www.veeam.com/kb2086) is same as Veeam Backup Health Check?

TIA

[DGrinev]

Hi Jennilyn,

No, it's not the same as Health Check, it uses a different algorithm to check backups.
You can review additional information about Validator in the User Guide.
Can you clarify a bit why are you asking? Thanks!

[PTide]

My 2 cents: this explanation might be helpful as well.

Thanks

[jencalma]

Hi Jennilyn,

No, it's not the same as Health Check, it uses a different algorithm to check backups.
You can review additional information about Validator in the User Guide.
Can you clarify a bit why are you asking? Thanks!

We have a backup copy job and i noticed that backup health check is running quite slow - 14% for 3days

This might be a network connection issue in which the target site is on a different location. I'm thinking if i could just schedule to run a Veeam Validator on the target site and then disable the health check on the copy job.

This is the reason I asked

[PTide]

Health check speed depends on different factors, such as storage type, location, backup size, backup chain length, the way how restore point's data blocks are distributed within the chain. What is your setup?

Yes, validator will also work, however it won't automatically correct errors if case any found.

Thanks

[csydas]

@Jennilyn,

Do you have an actual managed server over on the remote side as a gateway? When going through our growing pains with Veeam we saw this happening cause the gateway for the remote device was being chosen on the source side so some data validation must have to go back to the source site. We spun up a proxy on the remote site and set that as the Gateway and it sped up the checks considerably.

[jencalma]

@Jennilyn,

Do you have an actual managed server over on the remote side as a gateway? When going through our growing pains with Veeam we saw this happening cause the gateway for the remote device was being chosen on the source side so some data validation must have to go back to the source site. We spun up a proxy on the remote site and set that as the Gateway and it sped up the checks considerably.

@csydas
Not sure how to check this one. Our remote server is a standalone. Created a copy job via unc and this is added in backup infrastructure
Local site can access the remote shared folder via an external IP. No site to site vpn configured between both servers.

Health check speed depends on different factors, such as storage type, location, backup size, backup chain length, the way how restore point's data blocks are distributed within the chain. What is your setup?

Yes, validator will also work, however it won't automatically correct errors if case any found.

Thanks

I am not aware that health check can fix corruption/errors. Here's the setup

Local Site
- Backup are scheduled to run every 4hrs (incremental)
- Synthetic full backup every Saturday
- Retention is 14
- Local storage is Synology NAS
- Total VM size: 2TB

Copy job
- interval is every 3 days
- Currently health check is disabled


Thank You so much for all the help. I really appreciate it.

[jayscarff]

Hi Thomas, health check performance is mostly defined by the backup size and by what the storage can provide in terms of the read speed. In your case, I suppose, both factors play the role, since the backup is pretty large and the storage is not top grade in terms of IOPS. Health check protects against storage bit rot and allows to detect in-flight backup file data corruptions. With periodic active fulls the need for health check is not that high, since even if existing backup file is corrupt, active full will create a brand new one, but it still allows to be informed about existing backups health.

Hi Foggy,
I just enabled health checks, 32TB job, 24% after 45 hours. I've read the above but i guess my problem is that i loose RPO with this enabled so essentially cant run health checks often. I could enable this on the copy job for this job but not sure if solves the problem (if in fact there is a problem).
Forward incremental with 14 restore points
Copy job - 7 RP's plus 2 weeklys

Thanks

[foggy]

Do you have per-VM chains setting enabled on the repository? With this setting, health check is running in parallel for multiple VM chains.

[jayscarff]

Thanks, i do not. The Repo is fairly big - 400TB ReFS window 2016 LUN. Would you recommend adding that tick box? As i mentioned the chain is only 14 RP's and then copy job takes 7 then weeklys. Should i maybe enable this on the copy job repository and enable the health checks on the copy job?

Thanks

[foggy]

You can try. Performance will depend on the storage capabilities, anyway.

[rikH]

Right now I see by a customer they are running checkhealth on a backup of a VM of 11TB. The backup has the minimum of 2 restore points, so it is merging every day. The check health takes a long time. Right now it is 70% complete after 130 hours. Not very handy as it blocks the backup. I looked back to last month and then it took 110hours(4,5 days) also very long> i was wondering if we were to run a defrag and compact using the full backup file maintenance option if that may improve the time taken for the health check? Or if that is likely to also take a long time and not make any difference?
Would a new full(active full) make any difference or is this not really dependant on the backup image and more the HW performance on how long it takes?

If any body could advise here I would be most appreciative.

[Mgamerz]

What is the underlying storage volume for the backup? ReFS? NTFS?

[rikH]

This is ReFS with indexing in file properties allowed.

[foggy]

Hi Rick, some hints can be found in the thread above. F.ex., do they use per-VM chains? What kind of storage is involved?

[rikH]

Hi Foggy, thanks for your input.
After running a synthetic full the health check now takes around 19hours which on a monthly basis is ok.
So for the moment this is resolved.