B&R 9 : Health check of VM Backup Files needs so long time

Discussions specific to VMware vSphere hypervisor

B&R 9 : Health check of VM Backup Files needs so long time

Veeam Logoby WinstonWolf » Thu Jan 28, 2016 8:22 am

Hello ,

i activated the Option "Perform Backup Files Health Check" , but it needs 10 Hours on some Backup Jobs .
Why this Option needs so long time ?

Michael
WinstonWolf
Expert
 
Posts: 184
Liked: 4 times
Joined: Thu Jan 06, 2011 8:33 am

Re: B&R 9 : Health check of VM Backup Files needs so long ti

Veeam Logoby PTide » Thu Jan 28, 2016 10:23 am

Hi,

During health check VBR calculates CRC values for backup metadata and hash values for VM disks data blocks in the backup file and compares them with the CRC and hash values that are already stored in the backup file. The process can take a lot of time for large restore points. What's the average size of "some Backup Jobs" restore point?

Thank you.
PTide
Veeam Software
 
Posts: 3308
Liked: 274 times
Joined: Tue May 19, 2015 1:46 pm

Re: B&R 9 : Health check of VM Backup Files needs so long ti

Veeam Logoby WinstonWolf » Thu Jan 28, 2016 12:10 pm

We have an retention Time of 30 Days .
The Problem is that we have an Tape Job after the VM Backup Job running and this Job was waiting long time for finishing the check Backup Files Job .

I think it should be possible may be as an Feature to create own Jobs for the Health Check option .
WinstonWolf
Expert
 
Posts: 184
Liked: 4 times
Joined: Thu Jan 06, 2011 8:33 am

Re: B&R 9 : Health check of VM Backup Files needs so long ti

Veeam Logoby PTide » Thu Jan 28, 2016 12:50 pm

So, how big is your average restore point?
PTide
Veeam Software
 
Posts: 3308
Liked: 274 times
Joined: Tue May 19, 2015 1:46 pm

Re: B&R 9 : Health check of VM Backup Files needs so long ti

Veeam Logoby WinstonWolf » Thu Jan 28, 2016 2:03 pm

What you mean ? The vib Files or the vbk Files .
The vbk File is 6 TB big the everage vib files are 100 GB big .
WinstonWolf
Expert
 
Posts: 184
Liked: 4 times
Joined: Thu Jan 06, 2011 8:33 am

Re: B&R 9 : Health check of VM Backup Files needs so long ti

Veeam Logoby PTide » Thu Jan 28, 2016 3:22 pm

So you have 6Tb vbk + 29 .vibs (average size) 100GB. In order to make sure that your last restore point is vaild and can be used for recovery veeam needs to check all .vibs since your last full. In your case, if no intermediate fulls are present it is 9Tb of data to process. 10 hours looks good when talking about CRC check of 9Tb of data. To reduce the amount of time needed for health check consider configuring periodic fulls.

UPDATE at 3:46 PM (EST) 29/01.2016:

I checked with devs and it turned out that we actually check only blocks that compose the last restore point. Since those blocks can be spread across the whole backup chain that can be a lot of data to check, depending on many factors. Anyway 6Tb of data (even if compressed) that's still a lot work to do.

Thank you.
PTide
Veeam Software
 
Posts: 3308
Liked: 274 times
Joined: Tue May 19, 2015 1:46 pm

Re: B&R 9 : Health check of VM Backup Files needs so long ti

Veeam Logoby Gostev » Thu Jan 28, 2016 9:56 pm

Pasha, this is not correct as only the latest VM state is checked for consistency. As such, we do not read the entire content of each and every backup file in the chain. As such, periodic fulls will make no difference to the health check performance (as this will not change the amount of data that we will need to read from the backup storage). Please, double-check with the devs. Thanks!
Gostev
Veeam Software
 
Posts: 21648
Liked: 2433 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: B&R 9 : Health check of VM Backup Files needs so long ti

Veeam Logoby WinstonWolf » Fri Jan 29, 2016 11:59 am

Ok , the i can forget the Health Check Option . It is too bad . Because the Time runs on the Backup Job Time .
And after this Backup Job there comes an Tape Job .

What i say before - I think it will be a good feature to have the possibility to create an own "Health Check Job "

I have the Feeling that on all new features from V9 something is wrong and other necessary features are Missing :cry:
WinstonWolf
Expert
 
Posts: 184
Liked: 4 times
Joined: Thu Jan 06, 2011 8:33 am

Re: B&R 9 : Health check of VM Backup Files needs so long ti

Veeam Logoby lennis40 » Fri Feb 12, 2016 12:57 pm

We've had a few lengthy health checks as well, which has brought up some questions on what might be best practice. Health checks by default are disabled on backup jobs, and enabled on copy jobs. There must be a reason why the option is available on both, even though they're using the same backup files for the health check process. Other than spreading out the times when health checks run, I assume Veeam recommends to run on copy job, as that's the default setting? Without parallel processing to cloud repository on copy jobs, would it make more sense to run a health check on the backup job where parallel processing is available?

When we run into a health check taking several hours, it's holding up the other copy jobs from running any tasks. Even running them on backup files may delay the time the copy job starts transferring during the interval. I'm just curious to get the different opinions on what might be the best way to go about alleviating some of the wait time on health checks. Is it also safe to say that if SureBackup is being used, with the backup file verification option enabled, that health checks can be disabled on both backup jobs and copy jobs? Thanks for any input!
lennis40
Service Provider
 
Posts: 122
Liked: 3 times
Joined: Tue Nov 11, 2014 11:03 pm
Full Name: Michael

Re: B&R 9 : Health check of VM Backup Files needs so long ti

Veeam Logoby PTide » Fri Feb 12, 2016 3:47 pm

There must be a reason why the option is available on both, even though they're using the same backup files for the health check process. Other than spreading out the times when health checks run, I assume Veeam recommends to run on copy job, as that's the default setting?
Health-check option has been added recently and is set to "disabled" state so it does not change the behaviour of backup jobs that had been already configured.

ven though they're using the same backup files for the health check process
That's not correct. Backup files produces by backup job and backup files produced by backup copy job are different sets of files. Backup copy health check protects your backup copy chain from storage corruption whereas backup health check protects you main backup chain from storage corruption.

would it make more sense to run a health check on the backup job where parallel processing is available?
No, it would not. However, there are some improvements on health-check planned in future releases.

Is it also safe to say that if SureBackup is being used, with the backup file verification option enabled, that health checks can be disabled on both backup jobs and copy jobs?
Yes if you run SureBackup on both backup and backup copy jobs. Did you mean backup validator when you said "backup file verification option"?

Thank you.
PTide
Veeam Software
 
Posts: 3308
Liked: 274 times
Joined: Tue May 19, 2015 1:46 pm

Re: B&R 9 : Health check of VM Backup Files needs so long ti

Veeam Logoby lennis40 » Fri Feb 12, 2016 4:09 pm

The backup file integrity check option, in the settings menu of the SureBackup wizard, is the validation I was talking about.

So if the copy job is checking the copy job chain, that health check is actually happening on the target repository, or in this case the cloud repository?

Thanks for the information. We certainly look forward to improvements in future releases.
lennis40
Service Provider
 
Posts: 122
Liked: 3 times
Joined: Tue Nov 11, 2014 11:03 pm
Full Name: Michael

[MERGED] : backup copy health check very slow

Veeam Logoby bkc » Mon Feb 15, 2016 4:57 pm

Hi,

Our backup copy job started a health check at the end of January and today I finally killed it off with only 61% complete. During the past 2 weeks the backup copy job didn't copy any jobs. (hardware details below)

I've disabled the health check feature in the job and restarted it.

After restarting I see that it is processing 4 out of 31 VMs simultaneously.

Would disabling parallel data processing reduce disk fragmentation on the target backup repository?

Would enabling use per-vm backup files possibly reduce the health check time?

infrastructure notes..

we have a 3 node vsphere 6 cluster (w/ CBT disabled) and SAN. Veeam backup software runs on a VM and stores data to an external physical linux server w/ RAID 1 via 1GB ethernet connection. The main backup repository is currently using 1.2TB of space

The backup copy job target repository is a ReadyNas 6 box w/ 6 hard drives in Raid 5, also reachable via 1GB ethernet. Yeah, the Readynas isn't great but it's not terrible. Those 4 vms are currently showing "x% completed at 3MB/sec"

The main backup repository machine is currently running 6 VeeamAgents, 0% load and 0.1 I/O wait state.

suggestions?
bkc
Influencer
 
Posts: 20
Liked: 3 times
Joined: Tue Dec 21, 2010 10:31 pm
Full Name: brad clements

Re: B&R 9 : Health check of VM Backup Files needs so long ti

Veeam Logoby PTide » Mon Feb 15, 2016 5:48 pm

Hi,

Would disabling parallel data processing reduce disk fragmentation on the target backup repository?
No, it would not. If you want to reduce fragmentation please use compact full backup file feature.

Would enabling use per-vm backup files possibly reduce the health check time?
No, as the amount of data to be checked stays the same.

Please provide your bottleneck statistics.

Thank you.
PTide
Veeam Software
 
Posts: 3308
Liked: 274 times
Joined: Tue May 19, 2015 1:46 pm

Re: B&R 9 : Health check of VM Backup Files needs so long ti

Veeam Logoby bkc » Mon Feb 15, 2016 9:56 pm

Can you explain how fragmentation would not be reduced by having only one VM processed at a time? With 4 VMs all writing to the target at the same time their data blocks will be interleaved with each other as the target allocates free space to write to. Simultaneous writes is a classic cause of file fragmentation.

If I enable compact full, and all 4 vms start compacting their fulls at the same time, I'll have the same problem ... simultaneous writes to the same data store = fragmented files.

regarding bottleneck stats, the backup copy job just finished the first VM, 30 vms to go.

stats are very bad: 2/15/2016 4:31:13 PM :: Busy: Source 0% > Proxy 0% > Network 0% > Target 99%

The backup copy job doesn't report much for stats.. 7.4GB read at 533Kb/sec
Would be nice to have more stats regarding the target repository .. avg write speed vs avg read speed.

I guess this ReadyNas is a dog.
bkc
Influencer
 
Posts: 20
Liked: 3 times
Joined: Tue Dec 21, 2010 10:31 pm
Full Name: brad clements

Re: B&R 9 : Health check of VM Backup Files needs so long ti

Veeam Logoby bkc » Mon Feb 15, 2016 10:21 pm 2 people like this post

wait.. 2 ethernet interfaces on the Readynas box.

eth0 - rsync speed is about 600kb/sec (this is the interface veeam is using)

eth1 - rsync speed 19 MB/sec

I see lots of framing errors on eth0.. well that's hopeful, something I can actually troubleshoot

--

after reconfiguring veeam to use the good interface, I'm now seeing processing rate around 35MB/sec. yeah!
bkc
Influencer
 
Posts: 20
Liked: 3 times
Joined: Tue Dec 21, 2010 10:31 pm
Full Name: brad clements

Next

Return to VMware vSphere



Who is online

Users browsing this forum: Google [Bot] and 1 guest