Comprehensive data protection for all workloads
Post Reply
ferrus
Veeam ProPartner
Posts: 299
Liked: 43 times
Joined: Dec 03, 2015 3:41 pm
Location: UK
Contact:

Health Check on a Data Domain Recommendations

Post by ferrus »

We have an issue with our monthly Backup Copy Job Health Checks.
The performance is so slow, it's preventing days of backups across a number of Backup Copy Jobs:

Health Check durations:

Code: Select all

Exchange Backup Copy Job:           11TB, 13 days (now disabled)
File Server Backup Copy Job 1:       5TB, 5 days (estimated)
File Server Backup Copy Jobs 2&3:    4TB, 3.5-4 days (estimated)
The Veeam servers are Cisco C240 M4s, with 2x Intel Xeon E5-2660v2, and 96GB RAM, fibre connected with MPIO
The device the Copy Jobs are stored on, is an EMC Data Domain 2500, fibre connected and fully loaded with DD boost

I've read warnings in a few places that Health Checks to dedup devices will be slower - but we can't tolerate performance which loses us so many backup copy cycles.
I've mentioned it in another thread - but the date of the Health Check appeared again this weekend, and I don't think we can leave it unchanged for another month.

Can anyone offer recommendations, to either:

a) optimize the Health Check performance

or perhaps more realistically

b) find a strategy, which doesn't require monthly Health Checks on the DD2500
ferrus
Veeam ProPartner
Posts: 299
Liked: 43 times
Joined: Dec 03, 2015 3:41 pm
Location: UK
Contact:

Re: Health Check on a Data Domain Recommendations

Post by ferrus »

Sorry for bumping this.
Trying a last time before putting a support call in with Veeam. This is the type of question where real world community advice, can sometimes be more beneficial.
ferrus
Veeam ProPartner
Posts: 299
Liked: 43 times
Joined: Dec 03, 2015 3:41 pm
Location: UK
Contact:

[MERGED] Frequent Backup Copy Job Health Checks, since upgra

Post by ferrus »

Is there a reason or cause, for frequent Backup Copy Job Health Checks?

Since around the time of the v9.5 upgrade, we've had Health Checks at least once per fortnight, perhaps once per week.
The Health Check settings are all set to the First Sunday of the month, but they've started again this morning.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Frequent Backup Copy Job Health Checks, since upgrade

Post by Vitaliy S. »

Hi,

Please take a look at this UG page for more info when the health check is performed > Health Check for Copied Backups

Let me know if that help!
ferrus
Veeam ProPartner
Posts: 299
Liked: 43 times
Joined: Dec 03, 2015 3:41 pm
Location: UK
Contact:

Re: Frequent Backup Copy Job Health Checks, since upgrade

Post by ferrus »

According to that, Health Checks should only occur after two conditions - in the set schedule, or on the next interval after a previous failed Health Check.

Neither condition is true this morning. It's not the first Sunday of the month, and the last few Copy Jobs were normal Incrementals - without a Health Check.
I'm not sure why this is happening so often - but it has a knock on effect on the performance of the DataDomain, and the Repository scans,
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Health Check on a Data Domain Recommendations

Post by Vitaliy S. »

Hmm... what about other times when the health check operation was executed? Do you see any pattern in your case?
ferrus
Veeam ProPartner
Posts: 299
Liked: 43 times
Joined: Dec 03, 2015 3:41 pm
Location: UK
Contact:

Re: Health Check on a Data Domain Recommendations

Post by ferrus »

Thanks for merging the threads Vitaliy. We're really having a bad time with the Data Domains recently.
It seems with every Veeam release and update, our Tier 1 DAS backups get faster and better, but our Tier 2 Copy Jobs to the DataDomains get slower and buggier.

I think I found a solution to the original issue at the start of the thread, from another post -> veeam-backup-replication-f2/health-chec ... ml#p222020
Since the Copy Jobs are replicated to a second DD - via DD replication, we could run manual, scheduled Backup Validator jobs against the second copy. This would leave all resources on the First DD to the Veeam jobs, without interruption.

Now the biggest problem with that, is the outstanding post v9 bug, that's killed out DD replication:
veeam-backup-replication-f2/datadomain- ... ml#p201882
veeam-backup-replication-f2/datadomain- ... ml#p219600

I've tried the advice from Veeam and EMC, to create additional MTree's for parallel replication streams, and we now have our SQL and File Server VMs replicating within the daily window - to their own MTrees.
But the backlog of the Exchange and other VMs is still over 1000TB pre-compressed.

So we're stuck in a bit of a catch-22. We're hoping to Health Check our Copy jobs on the offsite DD, but DD replication is still crippled and the backlog isn't clearing. And now Health Checks are running frequently on the first DD, which is bringing down the performance on those.
ferrus
Veeam ProPartner
Posts: 299
Liked: 43 times
Joined: Dec 03, 2015 3:41 pm
Location: UK
Contact:

Re: Health Check on a Data Domain Recommendations

Post by ferrus »

As well as that, the above mentioned Exchange Copy Job seems to be doing something like an Active Full for no apparent reason.
Not a GFS merge - which we have scheduled, it would appear to be reading the full source backup job.

For the last few days, the Backup Copy Job size matches the source Incremental Backup Job size very closely (around 300GB).
But yesterday for some reason, instead of reading the source Backup Jobs 300GB, the Copy Job is processing the full 11TB.
ferrus
Veeam ProPartner
Posts: 299
Liked: 43 times
Joined: Dec 03, 2015 3:41 pm
Location: UK
Contact:

Re: Health Check on a Data Domain Recommendations

Post by ferrus »

Vitaliy S. wrote:Hmm... what about other times when the health check operation was executed? Do you see any pattern in your case?
I disabled the Backup Copy Jobs temporarily yesterday, to get a clean Repository Rescan.
Could that start a new Health Check window, or affect the Exchange Copy Job?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Health Check on a Data Domain Recommendations

Post by Vitaliy S. »

ferrus wrote:As well as that, the above mentioned Exchange Copy Job seems to be doing something like an Active Full for no apparent reason.
Maybe frequent health checks were triggered by the same situation as this active full for a backup copy job? What do logs show for this situation?
ferrus wrote:I disabled the Backup Copy Jobs temporarily yesterday, to get a clean Repository Rescan.
Could that start a new Health Check window, or affect the Exchange Copy Job?
Health checks are per job, so repository rescan shouldn't cause a new health check operation.
ferrus
Veeam ProPartner
Posts: 299
Liked: 43 times
Joined: Dec 03, 2015 3:41 pm
Location: UK
Contact:

Re: Health Check on a Data Domain Recommendations

Post by ferrus »

Health Checks again today - so that rules out the rescan.
What location are the relevant logs at?
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Health Check on a Data Domain Recommendations

Post by veremin »

Either check "C:\ProgramData\Veeam\Backup\Job Name" directory or export corresponding logs to whatever location you want to (menu -> help -> support information -> export logs for this job). Thanks.
Post Reply

Who is online

Users browsing this forum: Google [Bot], NWmybns and 202 guests