Very slow health check (3+ days)

Discussions specific to VMware vSphere hypervisor

Very slow health check (3+ days)

Veeam Logoby thomasrensvold » Tue May 09, 2017 2:46 pm

Hello!

I'm running B & R 9.5 U1 (9.5.0.823) on a VM with windows server 2012 R2. I have 1 backup job with about 160 VMs and 35,6TB. I run incremental with 1 active full monthly. Health check is set to run every other month. I also use "Per-VM backup files"

I have no problems with regular backup time, it normally finishes in about 4 hours on incremental and I guess about 35 hours on active full. My backup repository is a freenas box with cifs share. Both is connected with 10gbit.

The cpu usage on veeam server is very low (I run only 1 proxy server), and same with memory. The network speeds I see on the veeam server is about 500mbit read while the health check is running.

Cpu usage on backup repository is 0-1%, there is no problems handling what veeam server reads..

Any ideas on how to speed things up? How necessary is it to run the health check? Maybe I don't really need to do it since I run periodically active fulls anyway?

My healthcheck has now run for 42 hours and is 57% done
thomasrensvold
Novice
 
Posts: 3
Liked: never
Joined: Wed Dec 14, 2011 4:49 pm
Full Name: Thomas Rensvold

Re: Very slow health check (3+ days)

Veeam Logoby thomasrensvold » Thu May 11, 2017 6:11 am

81 hours now - 99% complete
thomasrensvold
Novice
 
Posts: 3
Liked: never
Joined: Wed Dec 14, 2011 4:49 pm
Full Name: Thomas Rensvold

Re: Very slow health check (3+ days)

Veeam Logoby foggy » Thu May 11, 2017 11:29 am

Hi Thomas, health check performance is mostly defined by the backup size and by what the storage can provide in terms of the read speed. In your case, I suppose, both factors play the role, since the backup is pretty large and the storage is not top grade in terms of IOPS. Health check protects against storage bit rot and allows to detect in-flight backup file data corruptions. With periodic active fulls the need for health check is not that high, since even if existing backup file is corrupt, active full will create a brand new one, but it still allows to be informed about existing backups health.
foggy
Veeam Software
 
Posts: 16024
Liked: 1228 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

[MERGED] Veeam Validator

Veeam Logoby jencalma » Thu Mar 15, 2018 12:04 am

HI Good Day

Just want to ask if Veeam Validator (https://www.veeam.com/kb2086) is same as Veeam Backup Health Check?

TIA
jencalma
Novice
 
Posts: 6
Liked: never
Joined: Wed Jul 19, 2017 6:22 am
Full Name: Jennilyn Calma

[MERGED] Re: Veeam Validator

Veeam Logoby DGrinev » Thu Mar 15, 2018 11:57 am

Hi Jennilyn,

No, it's not the same as Health Check, it uses a different algorithm to check backups.
You can review additional information about Validator in the User Guide.
Can you clarify a bit why are you asking? Thanks!
DGrinev
Veeam Software
 
Posts: 910
Liked: 97 times
Joined: Thu Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev

[MERGED] Re: Veeam Validator

Veeam Logoby PTide » Thu Mar 15, 2018 12:18 pm 2 people like this post

My 2 cents: this explanation might be helpful as well.

Thanks
PTide
Veeam Software
 
Posts: 3711
Liked: 308 times
Joined: Tue May 19, 2015 1:46 pm

[MERGED] Re: Veeam Validator

Veeam Logoby jencalma » Thu Mar 15, 2018 11:10 pm

DGrinev wrote:Hi Jennilyn,

No, it's not the same as Health Check, it uses a different algorithm to check backups.
You can review additional information about Validator in the User Guide.
Can you clarify a bit why are you asking? Thanks!


We have a backup copy job and i noticed that backup health check is running quite slow - 14% for 3days

This might be a network connection issue in which the target site is on a different location. I'm thinking if i could just schedule to run a Veeam Validator on the target site and then disable the health check on the copy job.

This is the reason I asked
jencalma
Novice
 
Posts: 6
Liked: never
Joined: Wed Jul 19, 2017 6:22 am
Full Name: Jennilyn Calma

Re: Very slow health check (3+ days)

Veeam Logoby PTide » Fri Mar 16, 2018 9:49 am

Health check speed depends on different factors, such as storage type, location, backup size, backup chain length, the way how restore point's data blocks are distributed within the chain. What is your setup?

Yes, validator will also work, however it won't automatically correct errors if case any found.

Thanks
PTide
Veeam Software
 
Posts: 3711
Liked: 308 times
Joined: Tue May 19, 2015 1:46 pm

Re: Very slow health check (3+ days)

Veeam Logoby csydas » Sat Mar 17, 2018 7:51 am

@Jennilyn,

Do you have an actual managed server over on the remote side as a gateway? When going through our growing pains with Veeam we saw this happening cause the gateway for the remote device was being chosen on the source side so some data validation must have to go back to the source site. We spun up a proxy on the remote site and set that as the Gateway and it sped up the checks considerably.
csydas
Enthusiast
 
Posts: 69
Liked: 11 times
Joined: Tue Jan 16, 2018 5:14 pm
Full Name: Harvey Carel

Re: Very slow health check (3+ days)

Veeam Logoby jencalma » Mon Mar 19, 2018 12:15 am

csydas wrote:@Jennilyn,

Do you have an actual managed server over on the remote side as a gateway? When going through our growing pains with Veeam we saw this happening cause the gateway for the remote device was being chosen on the source side so some data validation must have to go back to the source site. We spun up a proxy on the remote site and set that as the Gateway and it sped up the checks considerably.

@csydas
Not sure how to check this one. Our remote server is a standalone. Created a copy job via unc and this is added in backup infrastructure
Local site can access the remote shared folder via an external IP. No site to site vpn configured between both servers.

PTide wrote:Health check speed depends on different factors, such as storage type, location, backup size, backup chain length, the way how restore point's data blocks are distributed within the chain. What is your setup?

Yes, validator will also work, however it won't automatically correct errors if case any found.

Thanks


I am not aware that health check can fix corruption/errors. Here's the setup

Local Site
- Backup are scheduled to run every 4hrs (incremental)
- Synthetic full backup every Saturday
- Retention is 14
- Local storage is Synology NAS
- Total VM size: 2TB

Copy job
- interval is every 3 days
- Currently health check is disabled


Thank You so much for all the help. I really appreciate it. :)
jencalma
Novice
 
Posts: 6
Liked: never
Joined: Wed Jul 19, 2017 6:22 am
Full Name: Jennilyn Calma


Return to VMware vSphere



Who is online

Users browsing this forum: Google [Bot] and 24 guests