DataDomain replication performance issue

[ferrus]

I think the DD2500 was ordered fully loaded.
Not sure what the max spec is, but the inventory shows 57x 3TB drives - 54 in use.

[ferrus]

Ferrus,

It was discussed with DEV and QC today, but unfortunately it seems there is indeed something with the base file relationships which causes the Data Domain VSR capability to break. So yes, this means that the DD replication takes longer at this point in time. There is a discussion going on right now between Veeam and EMC to see where a potential solution can be defined, but I'm afraid it might take some time

Sorry, but at least I hope it helps a bit in your struggle that you have today and that you know we are aware of it

M/

Do you know if a fix for this is pending, in U1 - that Gostev announced in the Digest e-mail?
v9.5 has been a good release for us so far, but if a resolution isn't on the horizon for this - we'll have to design a strategy round it.

[Gostev]

We've determined what can be done to address the issue, but this requires a major re-architecture of our interaction with Data Domain - so not something we would be able to include in the update, unfortunately...

[jpreou]

So when on the roadmap? ('cos I am currently looking at a solution like this ... much smaller data sizes, but do not want to be affected by this issue and customer won't own the remote DD either so will have limited capability to drive change)

[rereduck]

Hello,

We have a DD2500 with Veeam since 3 years now and we just buy a DD6300 to replicate our DD2500 on our recovery site.

We use one MTREE with 400TB pre-comp and the replication job says that there is 3.9PB of data to replicate and it announce 3 months to replicate it .... (with 2 10GB link). Why 3.9 PB ??

I think our problem is the same that all people have in this post unfortunately

What solution did you find to figure this issue ? Using ultiples MTREE instead of big one would be helpfull ?

@Gostev : When Veeam will provide a fix for this issue ? For V10 ?

Thnaks

[mk2311]

Our DD replication between two DD990's was eventually up to 850tb!

As mentioned in a previous post, Dell/EMC and Veeam are looking into a resolution but it won't be here quickly

Instead, we have reverted to using backup copy jobs.

To reduce the workload on our Veeam B&R server, we have created two VM's which we use as gateway servers. The backup jobs use one gateway, the backup copy jobs use the other. When we had the gateway on the Veeam servers, it was frequently running out of memory.

The backup copy jobs appear to be running fine, except that some of the very largest jobs never seem to complete. We are having ongoing discussions with Veeam Tech Support

I guess the very frustrating thing with this is that we chose Veeam (at v8) as our backup tool largely due to it fully integrating with our Data Domain. Obviously no longer the case.....

We started with replication issues at v9 and at v9.5 it became so bad that we had to move away from mtree replication to BCJ's

[ferrus]

Hello,

We have a DD2500 with Veeam since 3 years now and we just buy a DD6300 to replicate our DD2500 on our recovery site.
We use one MTREE with 400TB pre-comp and the replication job says that there is 3.9PB of data to replicate and it announce 3 months to replicate it .... (with 2 10GB link). Why 3.9 PB ??
I think our problem is the same that all people have in this post unfortunately

What solution did you find to figure this issue ? Using multiple MTREE instead of big one would be helpful ?

@Gostev : When Veeam will provide a fix for this issue ? For V10 ?

Thanks

@rereduck

After starting this thread, last year - I finally managed to get our DD replication backlog cleared last month, which had been there since the Veeam 9 upgrade.
At it's maximum, our backlog was over 1PB; however, this is the uncompressed figure, and we found it doesn't reduce in linear steps - but sometimes drops dramatically.

Following the advice on this thread, I created another four Mtree's, and assigned Backup Copy Jobs to those. It had the effect of starting the replication again from the beginning for those jobs, but used 4x the bandwidth as the replication jobs worked in parallel.
It's certainly a workaround rather than a fix, as each job still works through it's backlog slowly, but we usually see a point every week where all Mtree replication jobs are complete. Something unthinkable since Veeam v8.

The good feature of Mtree's that makes this possible, it the fact that you don't have to pre-define or reserve a capacity for them. They all share the same free capacity.

[Gostev]

I guess the very frustrating thing with this is that we chose Veeam (at v8) as our backup tool largely due to it fully integrating with our Data Domain. Obviously no longer the case.....

Just wanted to correct you a little bit on this statement: Veeam has never supported native Data Domain replication, and moreover always explicitly recommended against using all types of storage-based replication for copying backups - this is based on multiple support cases where this practice has caused our customers irreversible data loss.

That said, we're looking at ways of leveraging storage-based replication engine as the underlying transport for Backup Copy jobs.

[ferrus]

Don't think I can agree with this.

Our design, including DD integration and replication was sent for validation by Veeam, with no 'explicit recommendation' not to use it. DD replication may not be supported by Veeam (nor should it be, as it's purely an EMC operation), but it was working in v8 - then broken by Veeam. This has caused us a huge amount of disruption, and negatively affected our DR capabilities for a long time now.
Veeam must surely understand the impact on DR this has caused its customers.

I'll look into using a secondary Backup Copy Job as an alternative - but I can't believe this would be viable. I thought the recommendation for BCJ to DD was to use the primary backup jobs as the source (not use merges) - and use full backups instead. This would create huge amounts of WAN traffic every week for the secondary BCJ.

[Gostev]

I am sorry to hear the people you worked at Veeam did not raise a red flag on this solution, I will be sure to discuss this at the upcoming SE trainings.

[ferrus]

On balance, every other aspect of Veeam has far exceeded our expectations. It really has improved our Backup and DR strategies immeasurably. It's just aspects of the DD integration that has caused us issues.
I look forward to any improvements in the BCJ replication you mentioned.
In the meantime, for others experiencing the same issue, adding MTree's does provide a workaround which should work for many deployments (the more Mtree's the better the performance, bandwidth permitting).

Now, if Veeam could just do something to improve heath check performance to BCJ on DD's - eg allow health checks that last >1 week to run in parallel with running jobs, rather than cancel them - I'd have all the DD issues fixed!

[Gostev]

You're actually spot on. Increasing health check performance is one critical aspect we're looking at in conjunction with potentially leveraging storage-based replication, as using the latter makes it impossible to validate the copied data on the fly as Backup Copy job currently does. And without such validation, you can never be sure if the data you're copying is any good.

[chris.mcdonald]

Victor, were you provided a defect / escalation number from EMC by chance? If so, would you care to share?

Thanks!

[bpayne]

I'd like to jump start this thread again. I also am in the same boat. I have two DD4500's. Also, extremely frustrated.

Here is my thread as I've tried using Veeam Backup Copy Jobs with MANY pains, and have recently started DD to DD storage replication and also have similar issues.
https://forums.veeam.com/veeam-backup-r ... 50116.html

With DD to DD replication, I have huge amounts of replication lag! It just cannot keep up, and I have a dedicated 10Gb fiber path for replication only.

This thread was started in 2016 and here we are halfway through 2018 and the issue has been identified (according to this thread) but I have not seen a true solution, just workarounds. Unfortunately, I am in the same boat as many customers that we have made the investment into Data Domain and now have to live with this issue. Part of that problem is Veeam's approach in letting any storage vendor state they work the best with Veeam (different problem, different topic, sorry)!

Yes, I have an active ticket with EMC and its the same problem mentioned in this thread - the Veeam data has changed since v9. We the customer have less than adequate DR backup replication.

So what is Veeam and EMC doing to fix this? Are there any updates to share?

Thank you

[foggy]

Hi Brandon, we did research in this regard and planned some particular improvements for one of the future updates.

[weem]

Hi,

any news regarding this issue? We have the same problem.

Thanks

[foggy]

We did certain optimizations since then. Could you please describe the observed behavior and your setup in more detail? In many cases, suboptimal infrastructure setup is the issue so it is hard to give universal advice. Do you have a case open with our technical support?

[foggy]

I was able to check your support case. In Veeam B&R v9.5u4 we've added a registry value that allows for delaying of incremental backup deletion for scenarios involving Data Domain MTree replication preventing it from querying those from source instead of synthesizing on target. Judging on the symptoms, this might help in your case - to enable this functionality, set the DDBoostDeleteIncrementStorageAfterHours (DWORD) value under HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication registry key on the backup server to the desired number of hours.

[weem]

We did certain optimizations since then. Could you please describe the observed behavior and your setup in more detail? In many cases, suboptimal infrastructure setup is the issue so it is hard to give universal advice. Do you have a case open with our technical support?

Hi,

thanks for answering!

We did some Backups with veeam to a DataDomain. Both Systems are in the same network without firewalls between each other, connected with 2 x 10GBe each. The entire data was about 35 TB. After that we disabled the veeam jobs and created a Mtree Replication to another DataDomain. The DataDomain replication should transfer 35 TB of data because both systems are using deduplication and are transferring the data deduplicated. From the first point of view the transferred data was about 800 TB and took forever. No Veeam job or interaction took place on the source side during the replication.

[Gostev]

I recommend you open a case with DELL EMC then, and have them troubleshoot how 35TB of the source data turned into the 800TB transfer.