Health Check on a Data Domain Recommendations

Availability for the Always-On Enterprise

Health Check on a Data Domain Recommendations

Veeam Logoby ferrus » Mon Nov 28, 2016 5:08 pm

We have an issue with our monthly Backup Copy Job Health Checks.
The performance is so slow, it's preventing days of backups across a number of Backup Copy Jobs:

Health Check durations:

Code: Select all
Exchange Backup Copy Job:           11TB, 13 days (now disabled)
File Server Backup Copy Job 1:       5TB, 5 days (estimated)
File Server Backup Copy Jobs 2&3:    4TB, 3.5-4 days (estimated)


The Veeam servers are Cisco C240 M4s, with 2x Intel Xeon E5-2660v2, and 96GB RAM, fibre connected with MPIO
The device the Copy Jobs are stored on, is an EMC Data Domain 2500, fibre connected and fully loaded with DD boost

I've read warnings in a few places that Health Checks to dedup devices will be slower - but we can't tolerate performance which loses us so many backup copy cycles.
I've mentioned it in another thread - but the date of the Health Check appeared again this weekend, and I don't think we can leave it unchanged for another month.

Can anyone offer recommendations, to either:

a) optimize the Health Check performance

or perhaps more realistically

b) find a strategy, which doesn't require monthly Health Checks on the DD2500
ferrus
Veeam ProPartner
 
Posts: 127
Liked: 20 times
Joined: Thu Dec 03, 2015 3:41 pm
Location: UK

Re: Health Check on a Data Domain Recommendations

Veeam Logoby ferrus » Thu Dec 01, 2016 11:06 am

Sorry for bumping this.
Trying a last time before putting a support call in with Veeam. This is the type of question where real world community advice, can sometimes be more beneficial.
ferrus
Veeam ProPartner
 
Posts: 127
Liked: 20 times
Joined: Thu Dec 03, 2015 3:41 pm
Location: UK

[MERGED] Frequent Backup Copy Job Health Checks, since upgra

Veeam Logoby ferrus » Fri Jan 06, 2017 10:41 am

Is there a reason or cause, for frequent Backup Copy Job Health Checks?

Since around the time of the v9.5 upgrade, we've had Health Checks at least once per fortnight, perhaps once per week.
The Health Check settings are all set to the First Sunday of the month, but they've started again this morning.
ferrus
Veeam ProPartner
 
Posts: 127
Liked: 20 times
Joined: Thu Dec 03, 2015 3:41 pm
Location: UK

Re: Frequent Backup Copy Job Health Checks, since upgrade

Veeam Logoby Vitaliy S. » Fri Jan 06, 2017 12:06 pm

Hi,

Please take a look at this UG page for more info when the health check is performed > Health Check for Copied Backups

Let me know if that help!
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Frequent Backup Copy Job Health Checks, since upgrade

Veeam Logoby ferrus » Fri Jan 06, 2017 12:18 pm

According to that, Health Checks should only occur after two conditions - in the set schedule, or on the next interval after a previous failed Health Check.

Neither condition is true this morning. It's not the first Sunday of the month, and the last few Copy Jobs were normal Incrementals - without a Health Check.
I'm not sure why this is happening so often - but it has a knock on effect on the performance of the DataDomain, and the Repository scans,
ferrus
Veeam ProPartner
 
Posts: 127
Liked: 20 times
Joined: Thu Dec 03, 2015 3:41 pm
Location: UK

Re: Health Check on a Data Domain Recommendations

Veeam Logoby Vitaliy S. » Fri Jan 06, 2017 1:56 pm

Hmm... what about other times when the health check operation was executed? Do you see any pattern in your case?
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Health Check on a Data Domain Recommendations

Veeam Logoby ferrus » Fri Jan 06, 2017 2:44 pm

Thanks for merging the threads Vitaliy. We're really having a bad time with the Data Domains recently.
It seems with every Veeam release and update, our Tier 1 DAS backups get faster and better, but our Tier 2 Copy Jobs to the DataDomains get slower and buggier.

I think I found a solution to the original issue at the start of the thread, from another post -> https://forums.veeam.com/veeam-backup-replication-f2/health-check-for-gfs-archives-t39707.html#p222020
Since the Copy Jobs are replicated to a second DD - via DD replication, we could run manual, scheduled Backup Validator jobs against the second copy. This would leave all resources on the First DD to the Veeam jobs, without interruption.

Now the biggest problem with that, is the outstanding post v9 bug, that's killed out DD replication:
https://forums.veeam.com/veeam-backup-replication-f2/datadomain-replication-performance-issue-t36545.html#p201882
https://forums.veeam.com/veeam-backup-replication-f2/datadomain-replication-performance-issue-t36545-30.html#p219600

I've tried the advice from Veeam and EMC, to create additional MTree's for parallel replication streams, and we now have our SQL and File Server VMs replicating within the daily window - to their own MTrees.
But the backlog of the Exchange and other VMs is still over 1000TB pre-compressed.

So we're stuck in a bit of a catch-22. We're hoping to Health Check our Copy jobs on the offsite DD, but DD replication is still crippled and the backlog isn't clearing. And now Health Checks are running frequently on the first DD, which is bringing down the performance on those.
ferrus
Veeam ProPartner
 
Posts: 127
Liked: 20 times
Joined: Thu Dec 03, 2015 3:41 pm
Location: UK

Re: Health Check on a Data Domain Recommendations

Veeam Logoby ferrus » Fri Jan 06, 2017 2:56 pm

As well as that, the above mentioned Exchange Copy Job seems to be doing something like an Active Full for no apparent reason.
Not a GFS merge - which we have scheduled, it would appear to be reading the full source backup job.

For the last few days, the Backup Copy Job size matches the source Incremental Backup Job size very closely (around 300GB).
But yesterday for some reason, instead of reading the source Backup Jobs 300GB, the Copy Job is processing the full 11TB.
ferrus
Veeam ProPartner
 
Posts: 127
Liked: 20 times
Joined: Thu Dec 03, 2015 3:41 pm
Location: UK

Re: Health Check on a Data Domain Recommendations

Veeam Logoby ferrus » Fri Jan 06, 2017 3:21 pm

Vitaliy S. wrote:Hmm... what about other times when the health check operation was executed? Do you see any pattern in your case?


I disabled the Backup Copy Jobs temporarily yesterday, to get a clean Repository Rescan.
Could that start a new Health Check window, or affect the Exchange Copy Job?
ferrus
Veeam ProPartner
 
Posts: 127
Liked: 20 times
Joined: Thu Dec 03, 2015 3:41 pm
Location: UK

Re: Health Check on a Data Domain Recommendations

Veeam Logoby Vitaliy S. » Mon Jan 09, 2017 5:04 pm

ferrus wrote:As well as that, the above mentioned Exchange Copy Job seems to be doing something like an Active Full for no apparent reason.

Maybe frequent health checks were triggered by the same situation as this active full for a backup copy job? What do logs show for this situation?
ferrus wrote:I disabled the Backup Copy Jobs temporarily yesterday, to get a clean Repository Rescan.
Could that start a new Health Check window, or affect the Exchange Copy Job?

Health checks are per job, so repository rescan shouldn't cause a new health check operation.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Health Check on a Data Domain Recommendations

Veeam Logoby ferrus » Thu Jan 12, 2017 7:44 am

Health Checks again today - so that rules out the rescan.
What location are the relevant logs at?
ferrus
Veeam ProPartner
 
Posts: 127
Liked: 20 times
Joined: Thu Dec 03, 2015 3:41 pm
Location: UK

Re: Health Check on a Data Domain Recommendations

Veeam Logoby v.Eremin » Thu Jan 12, 2017 2:35 pm

Either check "C:\ProgramData\Veeam\Backup\Job Name" directory or export corresponding logs to whatever location you want to (menu -> help -> support information -> export logs for this job). Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Google Feedfetcher, Yahoo [Bot] and 13 guests