Host-based backup of VMware vSphere VMs.
Post Reply
thomasrensvold
Novice
Posts: 3
Liked: never
Joined: Dec 14, 2011 4:49 pm
Full Name: Thomas Rensvold
Contact:

Very slow health check (3+ days)

Post by thomasrensvold »

Hello!

I'm running B & R 9.5 U1 (9.5.0.823) on a VM with windows server 2012 R2. I have 1 backup job with about 160 VMs and 35,6TB. I run incremental with 1 active full monthly. Health check is set to run every other month. I also use "Per-VM backup files"

I have no problems with regular backup time, it normally finishes in about 4 hours on incremental and I guess about 35 hours on active full. My backup repository is a freenas box with cifs share. Both is connected with 10gbit.

The cpu usage on veeam server is very low (I run only 1 proxy server), and same with memory. The network speeds I see on the veeam server is about 500mbit read while the health check is running.

Cpu usage on backup repository is 0-1%, there is no problems handling what veeam server reads..

Any ideas on how to speed things up? How necessary is it to run the health check? Maybe I don't really need to do it since I run periodically active fulls anyway?

My healthcheck has now run for 42 hours and is 57% done
thomasrensvold
Novice
Posts: 3
Liked: never
Joined: Dec 14, 2011 4:49 pm
Full Name: Thomas Rensvold
Contact:

Re: Very slow health check (3+ days)

Post by thomasrensvold »

81 hours now - 99% complete
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Very slow health check (3+ days)

Post by foggy »

Hi Thomas, health check performance is mostly defined by the backup size and by what the storage can provide in terms of the read speed. In your case, I suppose, both factors play the role, since the backup is pretty large and the storage is not top grade in terms of IOPS. Health check protects against storage bit rot and allows to detect in-flight backup file data corruptions. With periodic active fulls the need for health check is not that high, since even if existing backup file is corrupt, active full will create a brand new one, but it still allows to be informed about existing backups health.
jencalma
Novice
Posts: 6
Liked: never
Joined: Jul 19, 2017 6:22 am
Full Name: Jennilyn Calma
Contact:

[MERGED] Veeam Validator

Post by jencalma »

HI Good Day

Just want to ask if Veeam Validator (https://www.veeam.com/kb2086) is same as Veeam Backup Health Check?

TIA
DGrinev
Veteran
Posts: 1943
Liked: 247 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

[MERGED] Re: Veeam Validator

Post by DGrinev »

Hi Jennilyn,

No, it's not the same as Health Check, it uses a different algorithm to check backups.
You can review additional information about Validator in the User Guide.
Can you clarify a bit why are you asking? Thanks!
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

[MERGED] Re: Veeam Validator

Post by PTide » 2 people like this post

My 2 cents: this explanation might be helpful as well.

Thanks
jencalma
Novice
Posts: 6
Liked: never
Joined: Jul 19, 2017 6:22 am
Full Name: Jennilyn Calma
Contact:

[MERGED] Re: Veeam Validator

Post by jencalma »

DGrinev wrote:Hi Jennilyn,

No, it's not the same as Health Check, it uses a different algorithm to check backups.
You can review additional information about Validator in the User Guide.
Can you clarify a bit why are you asking? Thanks!
We have a backup copy job and i noticed that backup health check is running quite slow - 14% for 3days

This might be a network connection issue in which the target site is on a different location. I'm thinking if i could just schedule to run a Veeam Validator on the target site and then disable the health check on the copy job.

This is the reason I asked
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Very slow health check (3+ days)

Post by PTide »

Health check speed depends on different factors, such as storage type, location, backup size, backup chain length, the way how restore point's data blocks are distributed within the chain. What is your setup?

Yes, validator will also work, however it won't automatically correct errors if case any found.

Thanks
csydas
Expert
Posts: 193
Liked: 47 times
Joined: Jan 16, 2018 5:14 pm
Full Name: Harvey Carel
Contact:

Re: Very slow health check (3+ days)

Post by csydas »

@Jennilyn,

Do you have an actual managed server over on the remote side as a gateway? When going through our growing pains with Veeam we saw this happening cause the gateway for the remote device was being chosen on the source side so some data validation must have to go back to the source site. We spun up a proxy on the remote site and set that as the Gateway and it sped up the checks considerably.
jencalma
Novice
Posts: 6
Liked: never
Joined: Jul 19, 2017 6:22 am
Full Name: Jennilyn Calma
Contact:

Re: Very slow health check (3+ days)

Post by jencalma »

csydas wrote:@Jennilyn,

Do you have an actual managed server over on the remote side as a gateway? When going through our growing pains with Veeam we saw this happening cause the gateway for the remote device was being chosen on the source side so some data validation must have to go back to the source site. We spun up a proxy on the remote site and set that as the Gateway and it sped up the checks considerably.
@csydas
Not sure how to check this one. Our remote server is a standalone. Created a copy job via unc and this is added in backup infrastructure
Local site can access the remote shared folder via an external IP. No site to site vpn configured between both servers.
PTide wrote:Health check speed depends on different factors, such as storage type, location, backup size, backup chain length, the way how restore point's data blocks are distributed within the chain. What is your setup?

Yes, validator will also work, however it won't automatically correct errors if case any found.

Thanks
I am not aware that health check can fix corruption/errors. Here's the setup

Local Site
- Backup are scheduled to run every 4hrs (incremental)
- Synthetic full backup every Saturday
- Retention is 14
- Local storage is Synology NAS
- Total VM size: 2TB

Copy job
- interval is every 3 days
- Currently health check is disabled


Thank You so much for all the help. I really appreciate it. :)
jayscarff
Service Provider
Posts: 114
Liked: 12 times
Joined: Nov 15, 2016 6:56 pm
Location: Cayman Islands
Contact:

Re: Very slow health check (3+ days)

Post by jayscarff »

foggy wrote:Hi Thomas, health check performance is mostly defined by the backup size and by what the storage can provide in terms of the read speed. In your case, I suppose, both factors play the role, since the backup is pretty large and the storage is not top grade in terms of IOPS. Health check protects against storage bit rot and allows to detect in-flight backup file data corruptions. With periodic active fulls the need for health check is not that high, since even if existing backup file is corrupt, active full will create a brand new one, but it still allows to be informed about existing backups health.
Hi Foggy,
I just enabled health checks, 32TB job, 24% after 45 hours. I've read the above but i guess my problem is that i loose RPO with this enabled so essentially cant run health checks often. I could enable this on the copy job for this job but not sure if solves the problem (if in fact there is a problem).
Forward incremental with 14 restore points
Copy job - 7 RP's plus 2 weeklys

Thanks
Jason
VMCE
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Very slow health check (3+ days)

Post by foggy »

Do you have per-VM chains setting enabled on the repository? With this setting, health check is running in parallel for multiple VM chains.
jayscarff
Service Provider
Posts: 114
Liked: 12 times
Joined: Nov 15, 2016 6:56 pm
Location: Cayman Islands
Contact:

Re: Very slow health check (3+ days)

Post by jayscarff »

Thanks, i do not. The Repo is fairly big - 400TB ReFS window 2016 LUN. Would you recommend adding that tick box? As i mentioned the chain is only 14 RP's and then copy job takes 7 then weeklys. Should i maybe enable this on the copy job repository and enable the health checks on the copy job?

Thanks
Jason
VMCE
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Very slow health check (3+ days)

Post by foggy »

You can try. Performance will depend on the storage capabilities, anyway.
rikH
Service Provider
Posts: 19
Liked: 1 time
Joined: May 22, 2015 1:44 pm
Full Name: RickH
Contact:

[MERGED] Checkhealth takes a long time

Post by rikH »

Right now I see by a customer they are running checkhealth on a backup of a VM of 11TB. The backup has the minimum of 2 restore points, so it is merging every day. The check health takes a long time. Right now it is 70% complete after 130 hours. Not very handy as it blocks the backup. I looked back to last month and then it took 110hours(4,5 days) also very long> i was wondering if we were to run a defrag and compact using the full backup file maintenance option if that may improve the time taken for the health check? Or if that is likely to also take a long time and not make any difference?
Would a new full(active full) make any difference or is this not really dependant on the backup image and more the HW performance on how long it takes?

If any body could advise here I would be most appreciative.
Regards Richard
Mgamerz
Expert
Posts: 159
Liked: 28 times
Joined: Sep 29, 2017 8:07 pm
Contact:

Re: Checkhealth takes a long time

Post by Mgamerz »

What is the underlying storage volume for the backup? ReFS? NTFS?
rikH
Service Provider
Posts: 19
Liked: 1 time
Joined: May 22, 2015 1:44 pm
Full Name: RickH
Contact:

Re: Checkhealth takes a long time

Post by rikH »

This is ReFS with indexing in file properties allowed.
Regards Richard
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Very slow health check (3+ days)

Post by foggy »

Hi Rick, some hints can be found in the thread above. F.ex., do they use per-VM chains? What kind of storage is involved?
rikH
Service Provider
Posts: 19
Liked: 1 time
Joined: May 22, 2015 1:44 pm
Full Name: RickH
Contact:

Re: Very slow health check (3+ days)

Post by rikH » 1 person likes this post

Hi Foggy, thanks for your input.
After running a synthetic full the health check now takes around 19hours which on a monthly basis is ok.
So for the moment this is resolved.
Regards Richard
Post Reply

Who is online

Users browsing this forum: No registered users and 91 guests