Discussions specific to the VMware vSphere hypervisor
Post Reply
thomasrensvold
Novice
Posts: 3
Liked: never
Joined: Dec 14, 2011 4:49 pm
Full Name: Thomas Rensvold
Contact:

Very slow health check (3+ days)

Post by thomasrensvold » May 09, 2017 2:46 pm

Hello!

I'm running B & R 9.5 U1 (9.5.0.823) on a VM with windows server 2012 R2. I have 1 backup job with about 160 VMs and 35,6TB. I run incremental with 1 active full monthly. Health check is set to run every other month. I also use "Per-VM backup files"

I have no problems with regular backup time, it normally finishes in about 4 hours on incremental and I guess about 35 hours on active full. My backup repository is a freenas box with cifs share. Both is connected with 10gbit.

The cpu usage on veeam server is very low (I run only 1 proxy server), and same with memory. The network speeds I see on the veeam server is about 500mbit read while the health check is running.

Cpu usage on backup repository is 0-1%, there is no problems handling what veeam server reads..

Any ideas on how to speed things up? How necessary is it to run the health check? Maybe I don't really need to do it since I run periodically active fulls anyway?

My healthcheck has now run for 42 hours and is 57% done

thomasrensvold
Novice
Posts: 3
Liked: never
Joined: Dec 14, 2011 4:49 pm
Full Name: Thomas Rensvold
Contact:

Re: Very slow health check (3+ days)

Post by thomasrensvold » May 11, 2017 6:11 am

81 hours now - 99% complete

foggy
Veeam Software
Posts: 17087
Liked: 1396 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Very slow health check (3+ days)

Post by foggy » May 11, 2017 11:29 am

Hi Thomas, health check performance is mostly defined by the backup size and by what the storage can provide in terms of the read speed. In your case, I suppose, both factors play the role, since the backup is pretty large and the storage is not top grade in terms of IOPS. Health check protects against storage bit rot and allows to detect in-flight backup file data corruptions. With periodic active fulls the need for health check is not that high, since even if existing backup file is corrupt, active full will create a brand new one, but it still allows to be informed about existing backups health.

jencalma
Novice
Posts: 6
Liked: never
Joined: Jul 19, 2017 6:22 am
Full Name: Jennilyn Calma
Contact:

[MERGED] Veeam Validator

Post by jencalma » Mar 15, 2018 12:04 am

HI Good Day

Just want to ask if Veeam Validator (https://www.veeam.com/kb2086) is same as Veeam Backup Health Check?

TIA

DGrinev
Veeam Software
Posts: 1292
Liked: 139 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

[MERGED] Re: Veeam Validator

Post by DGrinev » Mar 15, 2018 11:57 am

Hi Jennilyn,

No, it's not the same as Health Check, it uses a different algorithm to check backups.
You can review additional information about Validator in the User Guide.
Can you clarify a bit why are you asking? Thanks!

PTide
Veeam Software
Posts: 4536
Liked: 371 times
Joined: May 19, 2015 1:46 pm
Contact:

[MERGED] Re: Veeam Validator

Post by PTide » Mar 15, 2018 12:18 pm 2 people like this post

My 2 cents: this explanation might be helpful as well.

Thanks

jencalma
Novice
Posts: 6
Liked: never
Joined: Jul 19, 2017 6:22 am
Full Name: Jennilyn Calma
Contact:

[MERGED] Re: Veeam Validator

Post by jencalma » Mar 15, 2018 11:10 pm

DGrinev wrote:Hi Jennilyn,

No, it's not the same as Health Check, it uses a different algorithm to check backups.
You can review additional information about Validator in the User Guide.
Can you clarify a bit why are you asking? Thanks!
We have a backup copy job and i noticed that backup health check is running quite slow - 14% for 3days

This might be a network connection issue in which the target site is on a different location. I'm thinking if i could just schedule to run a Veeam Validator on the target site and then disable the health check on the copy job.

This is the reason I asked

PTide
Veeam Software
Posts: 4536
Liked: 371 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Very slow health check (3+ days)

Post by PTide » Mar 16, 2018 9:49 am

Health check speed depends on different factors, such as storage type, location, backup size, backup chain length, the way how restore point's data blocks are distributed within the chain. What is your setup?

Yes, validator will also work, however it won't automatically correct errors if case any found.

Thanks

csydas
Expert
Posts: 139
Liked: 29 times
Joined: Jan 16, 2018 5:14 pm
Full Name: Harvey Carel
Contact:

Re: Very slow health check (3+ days)

Post by csydas » Mar 17, 2018 7:51 am

@Jennilyn,

Do you have an actual managed server over on the remote side as a gateway? When going through our growing pains with Veeam we saw this happening cause the gateway for the remote device was being chosen on the source side so some data validation must have to go back to the source site. We spun up a proxy on the remote site and set that as the Gateway and it sped up the checks considerably.

jencalma
Novice
Posts: 6
Liked: never
Joined: Jul 19, 2017 6:22 am
Full Name: Jennilyn Calma
Contact:

Re: Very slow health check (3+ days)

Post by jencalma » Mar 19, 2018 12:15 am

csydas wrote:@Jennilyn,

Do you have an actual managed server over on the remote side as a gateway? When going through our growing pains with Veeam we saw this happening cause the gateway for the remote device was being chosen on the source side so some data validation must have to go back to the source site. We spun up a proxy on the remote site and set that as the Gateway and it sped up the checks considerably.
@csydas
Not sure how to check this one. Our remote server is a standalone. Created a copy job via unc and this is added in backup infrastructure
Local site can access the remote shared folder via an external IP. No site to site vpn configured between both servers.
PTide wrote:Health check speed depends on different factors, such as storage type, location, backup size, backup chain length, the way how restore point's data blocks are distributed within the chain. What is your setup?

Yes, validator will also work, however it won't automatically correct errors if case any found.

Thanks
I am not aware that health check can fix corruption/errors. Here's the setup

Local Site
- Backup are scheduled to run every 4hrs (incremental)
- Synthetic full backup every Saturday
- Retention is 14
- Local storage is Synology NAS
- Total VM size: 2TB

Copy job
- interval is every 3 days
- Currently health check is disabled


Thank You so much for all the help. I really appreciate it. :)

jayscarff
Service Provider
Posts: 74
Liked: 3 times
Joined: Nov 15, 2016 6:56 pm
Location: Cayman Islands
Contact:

Re: Very slow health check (3+ days)

Post by jayscarff » May 30, 2018 5:25 pm

foggy wrote:Hi Thomas, health check performance is mostly defined by the backup size and by what the storage can provide in terms of the read speed. In your case, I suppose, both factors play the role, since the backup is pretty large and the storage is not top grade in terms of IOPS. Health check protects against storage bit rot and allows to detect in-flight backup file data corruptions. With periodic active fulls the need for health check is not that high, since even if existing backup file is corrupt, active full will create a brand new one, but it still allows to be informed about existing backups health.
Hi Foggy,
I just enabled health checks, 32TB job, 24% after 45 hours. I've read the above but i guess my problem is that i loose RPO with this enabled so essentially cant run health checks often. I could enable this on the copy job for this job but not sure if solves the problem (if in fact there is a problem).
Forward incremental with 14 restore points
Copy job - 7 RP's plus 2 weeklys

Thanks
Jason
VMCE v9

foggy
Veeam Software
Posts: 17087
Liked: 1396 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Very slow health check (3+ days)

Post by foggy » May 30, 2018 5:30 pm

Do you have per-VM chains setting enabled on the repository? With this setting, health check is running in parallel for multiple VM chains.

jayscarff
Service Provider
Posts: 74
Liked: 3 times
Joined: Nov 15, 2016 6:56 pm
Location: Cayman Islands
Contact:

Re: Very slow health check (3+ days)

Post by jayscarff » May 30, 2018 5:39 pm

Thanks, i do not. The Repo is fairly big - 400TB ReFS window 2016 LUN. Would you recommend adding that tick box? As i mentioned the chain is only 14 RP's and then copy job takes 7 then weeklys. Should i maybe enable this on the copy job repository and enable the health checks on the copy job?

Thanks
Jason
VMCE v9

foggy
Veeam Software
Posts: 17087
Liked: 1396 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Very slow health check (3+ days)

Post by foggy » May 30, 2018 5:55 pm

You can try. Performance will depend on the storage capabilities, anyway.

Post Reply

Who is online

Users browsing this forum: raudi, vmJoe and 22 guests