DataDomain replication performance issue

Availability for the Always-On Enterprise

Re: DataDomain replication performance issue

Veeam Logoby ferrus » Mon Nov 21, 2016 3:27 pm

I think the DD2500 was ordered fully loaded.
Not sure what the max spec is, but the inventory shows 57x 3TB drives - 54 in use.
ferrus
Veeam ProPartner
 
Posts: 134
Liked: 20 times
Joined: Thu Dec 03, 2015 3:41 pm
Location: UK

Re: DataDomain replication performance issue

Veeam Logoby ferrus » Mon Dec 05, 2016 4:52 pm

Mike Resseler wrote:Ferrus,

It was discussed with DEV and QC today, but unfortunately it seems there is indeed something with the base file relationships which causes the Data Domain VSR capability to break. So yes, this means that the DD replication takes longer at this point in time. There is a discussion going on right now between Veeam and EMC to see where a potential solution can be defined, but I'm afraid it might take some time

Sorry, but at least I hope it helps a bit in your struggle that you have today and that you know we are aware of it

M/


Do you know if a fix for this is pending, in U1 - that Gostev announced in the Digest e-mail?
v9.5 has been a good release for us so far, but if a resolution isn't on the horizon for this - we'll have to design a strategy round it.
ferrus
Veeam ProPartner
 
Posts: 134
Liked: 20 times
Joined: Thu Dec 03, 2015 3:41 pm
Location: UK

Re: DataDomain replication performance issue

Veeam Logoby Gostev » Mon Dec 05, 2016 10:50 pm

We've determined what can be done to address the issue, but this requires a major re-architecture of our interaction with Data Domain - so not something we would be able to include in the update, unfortunately...
Gostev
Veeam Software
 
Posts: 21442
Liked: 2361 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: DataDomain replication performance issue

Veeam Logoby jpreou » Tue Jan 17, 2017 10:33 pm

So when on the roadmap? ('cos I am currently looking at a solution like this ... much smaller data sizes, but do not want to be affected by this issue and customer won't own the remote DD either so will have limited capability to drive change)
jpreou
Service Provider
 
Posts: 2
Liked: never
Joined: Tue Jan 17, 2017 10:08 pm

Re: DataDomain replication performance issue

Veeam Logoby rereduck » Wed Mar 01, 2017 1:45 pm

Hello,

We have a DD2500 with Veeam since 3 years now and we just buy a DD6300 to replicate our DD2500 on our recovery site.

We use one MTREE with 400TB pre-comp and the replication job says that there is 3.9PB of data to replicate and it announce 3 months to replicate it .... (with 2 10GB link). Why 3.9 PB ??

I think our problem is the same that all people have in this post unfortunately :(

What solution did you find to figure this issue ? Using ultiples MTREE instead of big one would be helpfull ?

@Gostev : When Veeam will provide a fix for this issue ? For V10 ?

Thnaks
rereduck
Influencer
 
Posts: 24
Liked: 1 time
Joined: Mon Dec 09, 2013 9:11 am

Re: DataDomain replication performance issue

Veeam Logoby mk2311 » Mon Mar 06, 2017 8:29 am

Our DD replication between two DD990's was eventually up to 850tb!

As mentioned in a previous post, Dell/EMC and Veeam are looking into a resolution but it won't be here quickly

Instead, we have reverted to using backup copy jobs.

To reduce the workload on our Veeam B&R server, we have created two VM's which we use as gateway servers. The backup jobs use one gateway, the backup copy jobs use the other. When we had the gateway on the Veeam servers, it was frequently running out of memory.

The backup copy jobs appear to be running fine, except that some of the very largest jobs never seem to complete. We are having ongoing discussions with Veeam Tech Support

I guess the very frustrating thing with this is that we chose Veeam (at v8) as our backup tool largely due to it fully integrating with our Data Domain. Obviously no longer the case.....

We started with replication issues at v9 and at v9.5 it became so bad that we had to move away from mtree replication to BCJ's
mk2311
Novice
 
Posts: 3
Liked: never
Joined: Tue Apr 28, 2015 2:12 pm
Full Name: Jeff White

Re: DataDomain replication performance issue

Veeam Logoby ferrus » Mon Mar 06, 2017 12:43 pm

rereduck wrote:Hello,

We have a DD2500 with Veeam since 3 years now and we just buy a DD6300 to replicate our DD2500 on our recovery site.
We use one MTREE with 400TB pre-comp and the replication job says that there is 3.9PB of data to replicate and it announce 3 months to replicate it .... (with 2 10GB link). Why 3.9 PB ??
I think our problem is the same that all people have in this post unfortunately :(

What solution did you find to figure this issue ? Using multiple MTREE instead of big one would be helpful ?

@Gostev : When Veeam will provide a fix for this issue ? For V10 ?

Thanks


@rereduck

After starting this thread, last year - I finally managed to get our DD replication backlog cleared last month, which had been there since the Veeam 9 upgrade.
At it's maximum, our backlog was over 1PB; however, this is the uncompressed figure, and we found it doesn't reduce in linear steps - but sometimes drops dramatically.

Following the advice on this thread, I created another four Mtree's, and assigned Backup Copy Jobs to those. It had the effect of starting the replication again from the beginning for those jobs, but used 4x the bandwidth as the replication jobs worked in parallel.
It's certainly a workaround rather than a fix, as each job still works through it's backlog slowly, but we usually see a point every week where all Mtree replication jobs are complete. Something unthinkable since Veeam v8.

The good feature of Mtree's that makes this possible, it the fact that you don't have to pre-define or reserve a capacity for them. They all share the same free capacity.
ferrus
Veeam ProPartner
 
Posts: 134
Liked: 20 times
Joined: Thu Dec 03, 2015 3:41 pm
Location: UK

Re: DataDomain replication performance issue

Veeam Logoby Gostev » Mon Mar 06, 2017 2:01 pm

mk2311 wrote:I guess the very frustrating thing with this is that we chose Veeam (at v8) as our backup tool largely due to it fully integrating with our Data Domain. Obviously no longer the case.....

Just wanted to correct you a little bit on this statement: Veeam has never supported native Data Domain replication, and moreover always explicitly recommended against using all types of storage-based replication for copying backups - this is based on multiple support cases where this practice has caused our customers irreversible data loss.

That said, we're looking at ways of leveraging storage-based replication engine as the underlying transport for Backup Copy jobs.
Gostev
Veeam Software
 
Posts: 21442
Liked: 2361 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: DataDomain replication performance issue

Veeam Logoby ferrus » Mon Mar 06, 2017 2:53 pm

Don't think I can agree with this.

Our design, including DD integration and replication was sent for validation by Veeam, with no 'explicit recommendation' not to use it. DD replication may not be supported by Veeam (nor should it be, as it's purely an EMC operation), but it was working in v8 - then broken by Veeam. This has caused us a huge amount of disruption, and negatively affected our DR capabilities for a long time now.
Veeam must surely understand the impact on DR this has caused its customers.

I'll look into using a secondary Backup Copy Job as an alternative - but I can't believe this would be viable. I thought the recommendation for BCJ to DD was to use the primary backup jobs as the source (not use merges) - and use full backups instead. This would create huge amounts of WAN traffic every week for the secondary BCJ.
ferrus
Veeam ProPartner
 
Posts: 134
Liked: 20 times
Joined: Thu Dec 03, 2015 3:41 pm
Location: UK

Re: DataDomain replication performance issue

Veeam Logoby Gostev » Mon Mar 06, 2017 7:01 pm

I am sorry to hear the people you worked at Veeam did not raise a red flag on this solution, I will be sure to discuss this at the upcoming SE trainings.
Gostev
Veeam Software
 
Posts: 21442
Liked: 2361 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: DataDomain replication performance issue

Veeam Logoby ferrus » Mon Mar 06, 2017 7:21 pm

On balance, every other aspect of Veeam has far exceeded our expectations. It really has improved our Backup and DR strategies immeasurably. It's just aspects of the DD integration that has caused us issues.
I look forward to any improvements in the BCJ replication you mentioned.
In the meantime, for others experiencing the same issue, adding MTree's does provide a workaround which should work for many deployments (the more Mtree's the better the performance, bandwidth permitting).

Now, if Veeam could just do something to improve heath check performance to BCJ on DD's - eg allow health checks that last >1 week to run in parallel with running jobs, rather than cancel them - I'd have all the DD issues fixed! :D
ferrus
Veeam ProPartner
 
Posts: 134
Liked: 20 times
Joined: Thu Dec 03, 2015 3:41 pm
Location: UK

Re: DataDomain replication performance issue

Veeam Logoby Gostev » Mon Mar 06, 2017 8:35 pm

You're actually spot on. Increasing health check performance is one critical aspect we're looking at in conjunction with potentially leveraging storage-based replication, as using the latter makes it impossible to validate the copied data on the fly as Backup Copy job currently does. And without such validation, you can never be sure if the data you're copying is any good.
Gostev
Veeam Software
 
Posts: 21442
Liked: 2361 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: DataDomain replication performance issue

Veeam Logoby chris.mcdonald » Mon Jul 24, 2017 1:55 pm

Victor, were you provided a defect / escalation number from EMC by chance? If so, would you care to share?

Thanks!
chris.mcdonald
Veeam Software
 
Posts: 2
Liked: never
Joined: Sat Jul 08, 2017 3:25 am
Full Name: Chris McDonald

Previous

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Exabot [Bot], Yahoo [Bot] and 23 guests