Health Check on a Data Domain Recommendations

[ferrus]

We have an issue with our monthly Backup Copy Job Health Checks.
The performance is so slow, it's preventing days of backups across a number of Backup Copy Jobs:

Health Check durations:

Code: Select all

Exchange Backup Copy Job:           11TB, 13 days (now disabled)
File Server Backup Copy Job 1:       5TB, 5 days (estimated)
File Server Backup Copy Jobs 2&3:    4TB, 3.5-4 days (estimated)

The Veeam servers are Cisco C240 M4s, with 2x Intel Xeon E5-2660v2, and 96GB RAM, fibre connected with MPIO
The device the Copy Jobs are stored on, is an EMC Data Domain 2500, fibre connected and fully loaded with DD boost

I've read warnings in a few places that Health Checks to dedup devices will be slower - but we can't tolerate performance which loses us so many backup copy cycles.
I've mentioned it in another thread - but the date of the Health Check appeared again this weekend, and I don't think we can leave it unchanged for another month.

Can anyone offer recommendations, to either:

a) optimize the Health Check performance

or perhaps more realistically

b) find a strategy, which doesn't require monthly Health Checks on the DD2500

[ferrus]

Sorry for bumping this.
Trying a last time before putting a support call in with Veeam. This is the type of question where real world community advice, can sometimes be more beneficial.

[ferrus]

Is there a reason or cause, for frequent Backup Copy Job Health Checks?

Since around the time of the v9.5 upgrade, we've had Health Checks at least once per fortnight, perhaps once per week.
The Health Check settings are all set to the First Sunday of the month, but they've started again this morning.

[Vitaliy S.]

Hi,

Please take a look at this UG page for more info when the health check is performed > Health Check for Copied Backups

Let me know if that help!

[ferrus]

According to that, Health Checks should only occur after two conditions - in the set schedule, or on the next interval after a previous failed Health Check.

Neither condition is true this morning. It's not the first Sunday of the month, and the last few Copy Jobs were normal Incrementals - without a Health Check.
I'm not sure why this is happening so often - but it has a knock on effect on the performance of the DataDomain, and the Repository scans,

[Vitaliy S.]

Hmm... what about other times when the health check operation was executed? Do you see any pattern in your case?

[ferrus]

Thanks for merging the threads Vitaliy. We're really having a bad time with the Data Domains recently.
It seems with every Veeam release and update, our Tier 1 DAS backups get faster and better, but our Tier 2 Copy Jobs to the DataDomains get slower and buggier.

I think I found a solution to the original issue at the start of the thread, from another post -> veeam-backup-replication-f2/health-chec ... ml#p222020
Since the Copy Jobs are replicated to a second DD - via DD replication, we could run manual, scheduled Backup Validator jobs against the second copy. This would leave all resources on the First DD to the Veeam jobs, without interruption.

Now the biggest problem with that, is the outstanding post v9 bug, that's killed out DD replication:
veeam-backup-replication-f2/datadomain- ... ml#p201882
veeam-backup-replication-f2/datadomain- ... ml#p219600

I've tried the advice from Veeam and EMC, to create additional MTree's for parallel replication streams, and we now have our SQL and File Server VMs replicating within the daily window - to their own MTrees.
But the backlog of the Exchange and other VMs is still over 1000TB pre-compressed.

So we're stuck in a bit of a catch-22. We're hoping to Health Check our Copy jobs on the offsite DD, but DD replication is still crippled and the backlog isn't clearing. And now Health Checks are running frequently on the first DD, which is bringing down the performance on those.

[ferrus]

As well as that, the above mentioned Exchange Copy Job seems to be doing something like an Active Full for no apparent reason.
Not a GFS merge - which we have scheduled, it would appear to be reading the full source backup job.

For the last few days, the Backup Copy Job size matches the source Incremental Backup Job size very closely (around 300GB).
But yesterday for some reason, instead of reading the source Backup Jobs 300GB, the Copy Job is processing the full 11TB.

[ferrus]

Hmm... what about other times when the health check operation was executed? Do you see any pattern in your case?

I disabled the Backup Copy Jobs temporarily yesterday, to get a clean Repository Rescan.
Could that start a new Health Check window, or affect the Exchange Copy Job?

[Vitaliy S.]

As well as that, the above mentioned Exchange Copy Job seems to be doing something like an Active Full for no apparent reason.

Maybe frequent health checks were triggered by the same situation as this active full for a backup copy job? What do logs show for this situation?

I disabled the Backup Copy Jobs temporarily yesterday, to get a clean Repository Rescan.
Could that start a new Health Check window, or affect the Exchange Copy Job?

Health checks are per job, so repository rescan shouldn't cause a new health check operation.

[ferrus]

Health Checks again today - so that rules out the rescan.
What location are the relevant logs at?

[veremin]

Either check "C:\ProgramData\Veeam\Backup\Job Name" directory or export corresponding logs to whatever location you want to (menu -> help -> support information -> export logs for this job). Thanks.