Health Check on Copy Backup Job stalled the job

VMware specific discussions

Health Check on Copy Backup Job stalled the job

Veeam Logoby cfvonner » Thu Jul 03, 2014 9:56 pm

I recently created a copy backup job that copies VMs from three different nightly backup jobs stored on a Nimble SAN into a single repository on a Drobo B1200i. I have archival restore points enabled, and currently have about 4.2 TB of full and incremental backups stored. I have the copy job configured to keep 90 restore points, and to keep some restore points for archival purposes (montly, quarterly, yearly). I have not yet reached 90 restore points.

I have the "Health check" feature turned on and set to do a check on the last Sunday of the month. Since this past Sunday was the last Sunday of June, the copy backup job started a new interval on schedule, and then hung at "Starting backup file verification". The Duration counter keeps running, and I can see from the Windows Resource Monitor that the VeeamAgent has open file handles on pretty much all of the .vib files in the repository. This has been running for four days now, and no new backups have been copied by this job. Is this normal behavior?

-Carl V.
cfvonner
Enthusiast
 
Posts: 32
Liked: 3 times
Joined: Fri Nov 16, 2012 9:58 pm
Full Name: Carl Von Stetten

Re: Health Check on Copy Backup Job stalled the job

Veeam Logoby Gostev » Thu Jul 03, 2014 10:31 pm

No, it should complete as fast as it takes to read all your backup files.
Gostev
Veeam Software
 
Posts: 21611
Liked: 2409 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Health Check on Copy Backup Job stalled the job

Veeam Logoby cfvonner » Thu Jul 03, 2014 10:58 pm

Of course, as soon as I post, it finishes and starts running a copy job... :oops:

Unfortunately, it only seems to be copying last night's backup, so I'll have a gap of a few days. Any way to make Veeam go back and copy the nighly backups from the previous three nights?

-Carl V.
cfvonner
Enthusiast
 
Posts: 32
Liked: 3 times
Joined: Fri Nov 16, 2012 9:58 pm
Full Name: Carl Von Stetten

Re: Health Check on Copy Backup Job stalled the job

Veeam Logoby foggy » Fri Jul 04, 2014 7:10 am

No, backup copy job copies the latest available state only.
foggy
Veeam Software
 
Posts: 15289
Liked: 1133 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

[MERGED] Stopping long running Health Check

Veeam Logoby readie » Wed Apr 29, 2015 9:23 am

I have a Backup Copy Health Check which has now been running for over 3 days and is only 70% done! I should like to stop it, as I have some server changes to make (which might actually make it faster in the long run), but unlike other jobs there is no option to 'stop session'.
Can I just stop the veeam services somewhere, or do I just have to be patient and wait another day or two?
Bob
Bob Eadie
Computer Manager at Bedford School, UK (since 1999).
Veeam user since 2009.
readie
Expert
 
Posts: 156
Liked: 30 times
Joined: Sun Dec 05, 2010 9:29 am
Full Name: Bob Eadie

Re: Health Check on Copy Backup Job stalled the job

Veeam Logoby Shestakov » Wed Apr 29, 2015 10:46 am

Hello Bob,
Health check is performed by target side. In general it should not take 3 days. Could you check if CPU or Memory use 100% if their capacity? And what is the size of backups of the job?
What version of VBR are you at?
readie wrote:Can I just stop the veeam services somewhere, or do I just have to be patient and wait another day or two?

Even if you stop the process, health check will be started again from the beginning. You can disable it permanently though.
Thanks!
Shestakov
Veeam Software
 
Posts: 5140
Liked: 430 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Health Check on Copy Backup Job stalled the job

Veeam Logoby readie » Thu Apr 30, 2015 8:13 am

Thanks. Version 8.0.0.917.
I have left it and it has finished the Health Check (I've never noticed this before? is it a new feature?) and has now started its backup copy - 23% through after about 3 hours (though I am not sure when it finished the health check and started the backup copy.
I'm fairly sure it is Windows iSCSI to fairly slow SANs (Netstor and Synology) which is limiting it - the total size of the job is about 7TB!! Do these timings seem OK, or way slower than you would expect?
Can you confirm whether the health check is comparing source with target - so reading from both?
(I think now I've spotted this, I shall have to think which months it is sensible to run - probably December, July - when the School is on holiday. (Feature request - it would be nice to select a date rather than day of the week - e.g. starting on 23rd December would be ideal. Last day of December (whatever weekday I choose) may well run on into 3rd or 4th Jan when school might be restarting . . . . but I could manage this manually each year by changing the day of the week!)
At least I am now happier that I know what's going on.

Follow up question. If the backup copy job is not due to run on the Sunday of the health check, will it just do the health check and NOT continue to the backup cope job?
Bob Eadie
Computer Manager at Bedford School, UK (since 1999).
Veeam user since 2009.
readie
Expert
 
Posts: 156
Liked: 30 times
Joined: Sun Dec 05, 2010 9:29 am
Full Name: Bob Eadie

Re: Health Check on Copy Backup Job stalled the job

Veeam Logoby Shestakov » Thu Apr 30, 2015 3:43 pm

readie wrote:Thanks. Version 8.0.0.917.

There is a newer version available. Worth updating.
readie wrote:is it a new feature?

No, it`s not. Can`t remember when it appeared, but it was in v7 for sure.
readie wrote:Do these timings seem OK, or way slower than you would expect?

Waiting the health check for 3 days doesn`t seem ok for me, but it depends on your target proxy capabilities a lot. Please, answer the question I asked in the previous post? (Could you check if CPU or Memory use 100% if their capacity?)
readie wrote:Can you confirm whether the health check is comparing source with target - so reading from both?

Health check procedure is explained here. Please get familiar and ask additional questions if you have any.
readie wrote:(Feature request - it would be nice to select a date rather than day of the week - e.g. starting on 23rd December would be ideal. Last day of December (whatever weekday I choose) may well run on into 3rd or 4th Jan when school might be restarting . . . . but I could manage this manually each year by changing the day of the week!)

Normally it should not take several days. And if you want be sure your backups are recoverable with no issues, I`d suggest to use our new feature Surebackup.
Thanks!
Shestakov
Veeam Software
 
Posts: 5140
Liked: 430 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Health Check on Copy Backup Job stalled the job

Veeam Logoby ferrus » Tue Sep 27, 2016 2:22 pm

Sorry to dig up an old thread - but do these time estimations still count for dedup devices?

We're about to head into the third night without a backup copy of a file server, to an EMC Data Domain - because the Health Check is still running from the weekend.
The Health Check is scheduled for once a month and loses us at least one full night of backup copy's each time.
ferrus
Veeam ProPartner
 
Posts: 147
Liked: 20 times
Joined: Thu Dec 03, 2015 3:41 pm
Location: UK

Re: Health Check on Copy Backup Job stalled the job

Veeam Logoby foggy » Tue Sep 27, 2016 4:34 pm

There're no general estimation, everything depends on the particular storage. But 3 days looks long indeed.
foggy
Veeam Software
 
Posts: 15289
Liked: 1133 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Health Check on Copy Backup Job stalled the job

Veeam Logoby ferrus » Fri Sep 30, 2016 9:20 am

I thought part of it may have been congestion because of (almost) all the Health Checks starting concurrently.
But it seems there was one that was delayed - and on it's own, it has the same performance.

Parent job - Two file server VMs - 3.5TB
Copy job, from Veeam DAS to EMC DD2500 - both fibre connected

This morning it stands at 79% complete, after 57 hours.

Is there any checking I do to find the cause/bottleneck?
ferrus
Veeam ProPartner
 
Posts: 147
Liked: 20 times
Joined: Thu Dec 03, 2015 3:41 pm
Location: UK

Re: Health Check on Copy Backup Job stalled the job

Veeam Logoby foggy » Fri Sep 30, 2016 3:02 pm

Health check needs to rehydrate a lot of data to calculate checksums for each of the stored block, so slow performance is expected on dedupe storage.
foggy
Veeam Software
 
Posts: 15289
Liked: 1133 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

[MERGED] Backup Copy Job Health Checks

Veeam Logoby efranklin » Thu Jan 26, 2017 8:45 pm

I managed multiple B&R servers and I've found that the backup job monthly health checks are taking multiple days to finish on large backup jobs. Majority of the destinations are to a Cloud Connect server which I also manage. I'm trying to find ways to improve the time to finish the health check. Any suggestions would be appreciated.

One question I haven't been able to find an answer to is when performing a health check on a backup copy job, is it all processed on the local B&R server or is it on the remote destination(i.e. Cloud Connect server)? If it's run on the local B&R server, how much does the available bandwidth relate to the speed of the health check? Would WAN acceleration help speed up the health check?

Thanks,
Eric
efranklin
Service Provider
 
Posts: 4
Liked: never
Joined: Thu Jan 26, 2017 8:36 pm
Full Name: Eric Franklin

Re: Health Check on Copy Backup Job stalled the job

Veeam Logoby PTide » Fri Jan 27, 2017 9:31 am

Hi,

What kind of repository is used on the Cloud Connect side? During health check checksums for data blocks in the backup files stored on the target repo are calculated, so the most part of the job is performed at the destination (be it repository or gateway in case of CIFS repo). Neither increased bandwidth nor WAN accelerator will give a significant boost as there is not so much data to transfer between sites.

Thank you
PTide
Veeam Software
 
Posts: 3245
Liked: 272 times
Joined: Tue May 19, 2015 1:46 pm

Re: Health Check on Copy Backup Job stalled the job

Veeam Logoby efranklin » Fri Jan 27, 2017 2:36 pm

The repository on the Cloud Connect side is a Synology NAS hooked up SMB.

If I'm understanding this correctly, I would need to improve the speed on the Cloud Connect side to improve the health check times? Should I focus more on storage I/O versus CPU & memory?
efranklin
Service Provider
 
Posts: 4
Liked: never
Joined: Thu Jan 26, 2017 8:36 pm
Full Name: Eric Franklin

Next

Return to VMware vSphere



Who is online

Users browsing this forum: Bing [Bot] and 1 guest