Comprehensive data protection for all workloads
Post Reply
rgarrison
Novice
Posts: 7
Liked: never
Joined: Jan 08, 2015 1:45 pm
Full Name: Ryan Garrison
Contact:

DDBoost Offsite Backup Copy

Post by rgarrison »

We've been struggling getting Veeam configured with DDBoost. Through a combination of conversations with support and these forums, we finally have local DDBoost backup jobs running fast, but are still having a deal-breaking problem with offsite "replication" (backup copy). Our offsite backup copy job for DDBoost is painfully slow, and is essentially useless for our requirements. We have a 100mb pipe between the local datacenter and the offsite datacenter. Data Domains are DD2500s, boost enabled, running OS 5.4.2.1-423209. The processing rate we're getting on the backup copy job is 10MB/s. The two data domains perform their built in replication for cifs much faster than this.

Is anyone getting rates faster than this? I've talked with support several times and the last call ended along the lines of "maybe we shouldn't use ddboost". I'm not ready to give up yet.

Job details:
1 VM, 6.5TB
In-line dedupe: disabled
Compression: Dedupe-friendly
Storage Optimization: Local target (16TB+) <-- "recommended" Veeam setting

I'm happy to post other details as needed.

Thanks for the feedback.
jadams159
Enthusiast
Posts: 80
Liked: 4 times
Joined: Apr 16, 2012 11:44 am
Full Name: Justin Adams
Location: United States
Contact:

Re: DDBoost Offsite Backup Copy

Post by jadams159 »

If you have 2 Data Domains, why are you leveraging Backup copy to get your data to the remote site? Data Domain replication will be much more efficient and it will free up resources in your Veeam infrastructure.

If it were me, I would write initial backups to one MTREE on your local DD. Then you can use a backup copy job (or even faster: script it in the DD leveraging fast copy)to copy to another MTREE on the same DD(I'm assuming your do this to meet a retention SLA). And then replicate that 2nd MTREE to the other site.
rgarrison
Novice
Posts: 7
Liked: never
Joined: Jan 08, 2015 1:45 pm
Full Name: Ryan Garrison
Contact:

Re: DDBoost Offsite Backup Copy

Post by rgarrison »

With DDBoost, we are forced to use Backup Copy instead of the standard Data Domain replication. For some reason, it is billed as a "feature" of DDBoost such that you control replication (backup copy) through the Veeam interface. EMC has also told us that replication for DDBoost must be handled within Veeam, and is not possible in the "traditional" Data Domain way.

Veeam support directed me to create a backup copy job which points directly to the offsite Data Domain via DDBoost. I'm still working with them to figure out if there is anything we can do about the slowness. I was hoping someone out there has tackled this problem and has some numbers they could share.
jadams159
Enthusiast
Posts: 80
Liked: 4 times
Joined: Apr 16, 2012 11:44 am
Full Name: Justin Adams
Location: United States
Contact:

Re: DDBoost Offsite Backup Copy

Post by jadams159 »

I hadn't heard that limitation. Can someone from Veeam verify that?

In an effort to help solve your problem, I would still try using the fastcopy command in the data domain to move the Veeam backup files you want to replicate to another MTREE and then replicate that MTREE. It may not be officially supported, but it would be a cool workaround if it worked. And it would be very fast.
SGalbincea
Enthusiast
Posts: 55
Liked: 6 times
Joined: May 25, 2012 2:09 pm
Full Name: Steve Galbincea
Location: Houston, TX
Contact:

Re: DDBoost Offsite Backup Copy

Post by SGalbincea »

I can confirm that this is the case (must use backup copy job.) You cannot, through the DD interface, replicate a DDBoost enabled MTREE from one DD to another (at least I have not found a way). We are coming from EMC NetWorker (migration to Veeam in progress) and with that program the replication (called DDBoost Managed Replication, not supported by Veeam (yet)) is configured via the backup software.

Currently we have a 1Gb link from HQ to DR and I have seen total backup copy throughput reach about 100MB/sec. One thing that baffles me though: why is the Veeam VM doing the work for the backup copy job and not one of my physical proxy servers? Seems like that would be better, but I am sure there is a reason...?
Senior Solutions Engineer, SLED - VMware
jadams159
Enthusiast
Posts: 80
Liked: 4 times
Joined: Apr 16, 2012 11:44 am
Full Name: Justin Adams
Location: United States
Contact:

Re: DDBoost Offsite Backup Copy

Post by jadams159 »

I would imagine that the Veeam server is doing the work because it is acting as the DD Boost library Gateway on the Veeam side. Normally Backup Copy jobs are processed be repository servers(notice you cannot select a proxy in the backup copy job), and the repo servers perform the transform process. So the transport service providing the transform process is likely the Veeam server.

Veeam folks, am I correct on that?
rgarrison
Novice
Posts: 7
Liked: never
Joined: Jan 08, 2015 1:45 pm
Full Name: Ryan Garrison
Contact:

Re: DDBoost Offsite Backup Copy

Post by rgarrison »

Thanks for the feedback, folks.

Long story short, EMC support told me that Data Domain OS version 5.5.x supports native DDBoost replication, so we wouldn't have to use Veeam or a backup copy job.

I was also told to disable all compression and deduplication by Veeam for jobs going to Data Domain (EMC considers this Best Practice). This has helped our normal DDBoost backup times noticeably.

We're upgrading the OS on our Data Domains and will look to get this going next week. I'll post back once we've had a chance to mess with it.
meilicke
Influencer
Posts: 22
Liked: 4 times
Joined: Sep 02, 2014 2:51 pm
Full Name: Scott Meilicke
Contact:

Re: DDBoost Offsite Backup Copy

Post by meilicke »

I have a very similar environment, a pair of DD2500s separated by a 100Mbps link. I know the DD replication would be faster, but won't you lose visibility of your offsite backups within Veeam? Or perhaps you can use a powershell after each backup, allowing for some DD replication time, to then rescan the remote repository?

Also, how do you handle keeping the replication in sync with the backup? DD snaps and a known schedule?

I am using the Veeam WAN Accelerators with acceptable results, but nothing like native DD.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: DDBoost Offsite Backup Copy

Post by Vitaliy S. »

jadams159 wrote:I would imagine that the Veeam server is doing the work because it is acting as the DD Boost library Gateway on the Veeam side. Normally Backup Copy jobs are processed be repository servers(notice you cannot select a proxy in the backup copy job), and the repo servers perform the transform process. So the transport service providing the transform process is likely the Veeam server.

Veeam folks, am I correct on that?
Yes, that's correct, but you can choose where all processing should be done via selecting the gateway server when configuring DDBoost repository. By default, this will be your backup server.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: DDBoost Offsite Backup Copy

Post by Gostev »

rgarrison wrote:We're upgrading the OS on our Data Domains and will look to get this going next week.
I was never a fan of storage-based replication for data protection purposes, as with that your data remains in the same fault domain. Native replication will replicate corrupted data just as well as it replicates "good" data.

Always remember the 3-2-1 rule:
3 copies of data
2 different media
1 copy offsite

And two identical storage devices that stay in a complete sync can never be considered as different medias ;)

Yes, what you are planning will save you against a complete site loss due to a fire, flood, etc... but honestly, how often do we experience those events comparing to data corruption events?
KevinK
Enthusiast
Posts: 28
Liked: 10 times
Joined: Apr 24, 2013 9:18 am
Full Name: Kevin Kissack
Contact:

Re: DDBoost Offsite Backup Copy

Post by KevinK »

I've just manged to setup DD Replication for a DDBoost MTree - OS 5.4.2.1-423209
KevinK
Enthusiast
Posts: 28
Liked: 10 times
Joined: Apr 24, 2013 9:18 am
Full Name: Kevin Kissack
Contact:

Re: DDBoost Offsite Backup Copy

Post by KevinK »

DD Replication has complete successfully, I've created a CIFS share on the remote DD/destination Mtree and can read the Veeam files without issue.
tjgrie
Novice
Posts: 9
Liked: 5 times
Joined: Mar 24, 2011 2:37 pm
Contact:

Re: DDBoost Offsite Backup Copy

Post by tjgrie »

We're on 5.4.0.7 of Data Domain's OS and we replicate our DDBoost enabled Veeam MTREE to our offsite DD. Sounds like it's a limitation in an older version of the DD OS.
vStorage
Influencer
Posts: 11
Liked: 3 times
Joined: Nov 06, 2014 9:32 pm
Contact:

Re: DDBoost Offsite Backup Copy

Post by vStorage »

Also successfully replicating DD Boost enabled MTREEs between DD2500 units running 5.5 firmware.

A key point is that ddboost user credentials must be the same across both units including group ID
Barrie
https://twitter.com/vStorage
SGalbincea
Enthusiast
Posts: 55
Liked: 6 times
Joined: May 25, 2012 2:09 pm
Full Name: Steve Galbincea
Location: Houston, TX
Contact:

Re: DDBoost Offsite Backup Copy

Post by SGalbincea »

Quick question then: if your backups become corrupted on the primary side and you have the DD managing the replication, won't the backups that are replicated to the secondary site immediately become corrupt as well? How are you all designing for this possibility?
Senior Solutions Engineer, SLED - VMware
KevinK
Enthusiast
Posts: 28
Liked: 10 times
Joined: Apr 24, 2013 9:18 am
Full Name: Kevin Kissack
Contact:

Re: DDBoost Offsite Backup Copy

Post by KevinK »

Run a DD Fastcopy command to replicate the data to another folder on the destination DD outside of the replication pair.
rgarrison
Novice
Posts: 7
Liked: never
Joined: Jan 08, 2015 1:45 pm
Full Name: Ryan Garrison
Contact:

Re: DDBoost Offsite Backup Copy

Post by rgarrison »

We were able to get replication going via mtree. The key thing to do on the destination is to attach your ddboost user to the mtree, which designates it as a ddboost storage unit. EMC very specifically said do NOT enable cifs share on the ddboost repository, as you can screw up the data in the storage unit that way.

I added a DDBoost repository pointing to the replication destination, did a rescan, and saw the imported backups. Performing a file level restore from the replicated copy worked fine. We're going to have to slowly roll stuff into this replicating boost repository so the replication can keep up, but it's looking like we're in business.
SGalbincea
Enthusiast
Posts: 55
Liked: 6 times
Joined: May 25, 2012 2:09 pm
Full Name: Steve Galbincea
Location: Houston, TX
Contact:

Re: DDBoost Offsite Backup Copy

Post by SGalbincea »

I think I have solved the corruption issue, at least for us. The DD has the ability to snapshot the MTREE that is being replicated, and in turn replicate that snapshot to the other DD. I have confirmed that I can mess up the .vbk on the primary copy and the replica, yet still access the backup successfully by mounting the snapshot on the DR side. As of last night I have disabled our copy jobs and am now using DD replication to copy our backups to DR.
Senior Solutions Engineer, SLED - VMware
Dave1337
Novice
Posts: 8
Liked: 1 time
Joined: Jan 16, 2015 4:07 pm
Contact:

Re: DDBoost Offsite Backup Copy

Post by Dave1337 »

Hello everyone,

I am in a similar boat to many in this thread. I have replicating Data Domain DD2200's, using Mtree replication. Everything works as of now, but the setup is not ideal.

DDBoost is only enabled on the primary DD, with the secondary DD being accessible in Veeam via CIFS.

I would like to have them both be DDBoost enabled but haven't figured out how to make this work... Is this even currently possible?

The production and DR site each have at least 1 dedicated Veeam VM.

DDOS 5.5.0.8
Veeam 8.0.0.917

Thanks in advance,
-Dave
Dave1337
Novice
Posts: 8
Liked: 1 time
Joined: Jan 16, 2015 4:07 pm
Contact:

Re: DDBoost Offsite Backup Copy

Post by Dave1337 »

Re-reading everything again, my question is:

How specifically do I attach a DDBoost user to a destination Mtree replication folder that currently has CIFS enabled. Step by step would be very much appreciated.

Also, does this break current replication or cause some type of re-sync to occur?

Kind Regards,
-Dave
rgarrison
Novice
Posts: 7
Liked: never
Joined: Jan 08, 2015 1:45 pm
Full Name: Ryan Garrison
Contact:

Re: DDBoost Offsite Backup Copy

Post by rgarrison »

FYI I was told explicity by EMC to NOT enable cifs for DDBoost storage units, as data corruption is a significant risk.

Here is the command to attach a ddboost user to a replicated mtree (storage unit):

Code: Select all

ddboost storage-unit modify <SU name> user <ddboost user name>
If you run the command "ddboost storage show", you should now see the replicated storage unit with RO/RD for status. This will enable the ability to make a DDBoost repository on the DR side. Rescanning the repository will bring the replicated backups into Veeam under the "Imported" section.
Dave1337
Novice
Posts: 8
Liked: 1 time
Joined: Jan 16, 2015 4:07 pm
Contact:

Re: DDBoost Offsite Backup Copy

Post by Dave1337 » 1 person likes this post

Thanks a lot!!! This is exactly what I needed.

I removed the repo/jobs from the remote Veeam server, and removed the CIFS share from the remote DD. Then I ran that and got it to work! Mounted it properly via the Veeam console and can confirm its working with no replication hiccups or otherwise.

Thank you for your help and I appreciate the prompt reply. Hopefully this helps someone else as well.

I still have a throughput issue on the remote site but I think it is related to the iSCSI connectivity to storage.
haslund
VeeaMVP
Posts: 839
Liked: 149 times
Joined: Feb 16, 2012 7:35 am
Full Name: Rasmus Haslund
Location: Denmark
Contact:

Re: DDBoost Offsite Backup Copy

Post by haslund »

vStorage wrote:Also successfully replicating DD Boost enabled MTREEs between DD2500 units running 5.5 firmware.

A key point is that ddboost user credentials must be the same across both units including group ID
Can you explain this in more details please?
What happens if the user credentials are not the same across both units?
Do you have a source for the information?

Thank you in advance.
Rasmus Haslund | Twitter: @haslund | Blog: https://rasmushaslund.com
haslund
VeeaMVP
Posts: 839
Liked: 149 times
Joined: Feb 16, 2012 7:35 am
Full Name: Rasmus Haslund
Location: Denmark
Contact:

Re: DDBoost Offsite Backup Copy

Post by haslund »

Is anyone seeing any issues with the performance/speed of merging oldest VIB file into VBK file at the end of the Backup Copy Job?
Rasmus Haslund | Twitter: @haslund | Blog: https://rasmushaslund.com
vStorage
Influencer
Posts: 11
Liked: 3 times
Joined: Nov 06, 2014 9:32 pm
Contact:

Re: DDBoost Offsite Backup Copy

Post by vStorage »

rhaslund wrote: Can you explain this in more details please?
What happens if the user credentials are not the same across both units?
Do you have a source for the information?

Thank you in advance.
As per the Data Domain command guide - page 93-95 I think. If you create a DDBoost Storage Unit on one Data Domain and then replicate that MTREE to another DD - the MTREE is replicated fine, but the DDboost Storage unit config is not carried over

This is expected behavior according to EMC.

The solution is to make sure your boost users are the created as the same group type from the start (either user or admin on both DataDomains) and then when you have replicated the MTREE to the second DataDomain, from the CLI of that unit - setting DD Boost ownership to the target Mtree, e,g, ddboost storage-unit modify <MyReplicaMTREE> user <MyBootUser>
Barrie
https://twitter.com/vStorage
NightBird
Expert
Posts: 242
Liked: 57 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: DDBoost Offsite Backup Copy

Post by NightBird »

rhaslund wrote:Is anyone seeing any issues with the performance/speed of merging oldest VIB file into VBK file at the end of the Backup Copy Job?
With GFS enabled ? If yes, Veeam made a private fix for one of my customer because each time at the end of the merge of the vib file into the vbk file, Veeam made a sort of compact of the vbk file (recompacting the .vbk file for limiting grow of it by ddboost design).

Yet the GFS merde do not do the "compact" anymore in the vbk file during the week (just the saturday on sunday during the weekly full).

open a support case
haslund
VeeaMVP
Posts: 839
Liked: 149 times
Joined: Feb 16, 2012 7:35 am
Full Name: Rasmus Haslund
Location: Denmark
Contact:

Re: DDBoost Offsite Backup Copy

Post by haslund »

NightBird wrote: With GFS enabled ? If yes, Veeam made a private fix for one of my customer because each time at the end of the merge of the vib file into the vbk file, Veeam made a sort of compact of the vbk file (recompacting the .vbk file for limiting grow of it by ddboost design).

Yet the GFS merde do not do the "compact" anymore in the vbk file during the week (just the saturday on sunday during the weekly full).

open a support case
I already have a support case open and feel like it is going nowhere at the moment.
Could you do me a major favor and reference the case # where the private fix was issued? Then I can point to it when I call Veeam Support again.
Also, was this private hotfix for v8 before patch 1 or after patch 1?

Thank you
Rasmus Haslund | Twitter: @haslund | Blog: https://rasmushaslund.com
NightBird
Expert
Posts: 242
Liked: 57 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: DDBoost Offsite Backup Copy

Post by NightBird »

After patch 1.

I will check with my customer for the case #
NightBird
Expert
Posts: 242
Liked: 57 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: DDBoost Offsite Backup Copy

Post by NightBird »

Here is the case number
Case # 00689492
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot], jorgedlcruz, mbrzezinski and 190 guests