DDBoost Offsite Backup Copy

[rgarrison]

We've been struggling getting Veeam configured with DDBoost. Through a combination of conversations with support and these forums, we finally have local DDBoost backup jobs running fast, but are still having a deal-breaking problem with offsite "replication" (backup copy). Our offsite backup copy job for DDBoost is painfully slow, and is essentially useless for our requirements. We have a 100mb pipe between the local datacenter and the offsite datacenter. Data Domains are DD2500s, boost enabled, running OS 5.4.2.1-423209. The processing rate we're getting on the backup copy job is 10MB/s. The two data domains perform their built in replication for cifs much faster than this.

Is anyone getting rates faster than this? I've talked with support several times and the last call ended along the lines of "maybe we shouldn't use ddboost". I'm not ready to give up yet.

Job details:
1 VM, 6.5TB
In-line dedupe: disabled
Compression: Dedupe-friendly
Storage Optimization: Local target (16TB+) <-- "recommended" Veeam setting

I'm happy to post other details as needed.

Thanks for the feedback.

[jadams159]

If you have 2 Data Domains, why are you leveraging Backup copy to get your data to the remote site? Data Domain replication will be much more efficient and it will free up resources in your Veeam infrastructure.

If it were me, I would write initial backups to one MTREE on your local DD. Then you can use a backup copy job (or even faster: script it in the DD leveraging fast copy)to copy to another MTREE on the same DD(I'm assuming your do this to meet a retention SLA). And then replicate that 2nd MTREE to the other site.

[rgarrison]

With DDBoost, we are forced to use Backup Copy instead of the standard Data Domain replication. For some reason, it is billed as a "feature" of DDBoost such that you control replication (backup copy) through the Veeam interface. EMC has also told us that replication for DDBoost must be handled within Veeam, and is not possible in the "traditional" Data Domain way.

Veeam support directed me to create a backup copy job which points directly to the offsite Data Domain via DDBoost. I'm still working with them to figure out if there is anything we can do about the slowness. I was hoping someone out there has tackled this problem and has some numbers they could share.

[jadams159]

I hadn't heard that limitation. Can someone from Veeam verify that?

In an effort to help solve your problem, I would still try using the fastcopy command in the data domain to move the Veeam backup files you want to replicate to another MTREE and then replicate that MTREE. It may not be officially supported, but it would be a cool workaround if it worked. And it would be very fast.

[SGalbincea]

I can confirm that this is the case (must use backup copy job.) You cannot, through the DD interface, replicate a DDBoost enabled MTREE from one DD to another (at least I have not found a way). We are coming from EMC NetWorker (migration to Veeam in progress) and with that program the replication (called DDBoost Managed Replication, not supported by Veeam (yet)) is configured via the backup software.

Currently we have a 1Gb link from HQ to DR and I have seen total backup copy throughput reach about 100MB/sec. One thing that baffles me though: why is the Veeam VM doing the work for the backup copy job and not one of my physical proxy servers? Seems like that would be better, but I am sure there is a reason...?

[jadams159]

I would imagine that the Veeam server is doing the work because it is acting as the DD Boost library Gateway on the Veeam side. Normally Backup Copy jobs are processed be repository servers(notice you cannot select a proxy in the backup copy job), and the repo servers perform the transform process. So the transport service providing the transform process is likely the Veeam server.

Veeam folks, am I correct on that?

[rgarrison]

Thanks for the feedback, folks.

Long story short, EMC support told me that Data Domain OS version 5.5.x supports native DDBoost replication, so we wouldn't have to use Veeam or a backup copy job.

I was also told to disable all compression and deduplication by Veeam for jobs going to Data Domain (EMC considers this Best Practice). This has helped our normal DDBoost backup times noticeably.

We're upgrading the OS on our Data Domains and will look to get this going next week. I'll post back once we've had a chance to mess with it.

[meilicke]

I have a very similar environment, a pair of DD2500s separated by a 100Mbps link. I know the DD replication would be faster, but won't you lose visibility of your offsite backups within Veeam? Or perhaps you can use a powershell after each backup, allowing for some DD replication time, to then rescan the remote repository?

Also, how do you handle keeping the replication in sync with the backup? DD snaps and a known schedule?

I am using the Veeam WAN Accelerators with acceptable results, but nothing like native DD.

[Vitaliy S.]

I would imagine that the Veeam server is doing the work because it is acting as the DD Boost library Gateway on the Veeam side. Normally Backup Copy jobs are processed be repository servers(notice you cannot select a proxy in the backup copy job), and the repo servers perform the transform process. So the transport service providing the transform process is likely the Veeam server.

Veeam folks, am I correct on that?

Yes, that's correct, but you can choose where all processing should be done via selecting the gateway server when configuring DDBoost repository. By default, this will be your backup server.

[Gostev]

We're upgrading the OS on our Data Domains and will look to get this going next week.

I was never a fan of storage-based replication for data protection purposes, as with that your data remains in the same fault domain. Native replication will replicate corrupted data just as well as it replicates "good" data.

Always remember the 3-2-1 rule:
3 copies of data
2 different media
1 copy offsite

And two identical storage devices that stay in a complete sync can never be considered as different medias

Yes, what you are planning will save you against a complete site loss due to a fire, flood, etc... but honestly, how often do we experience those events comparing to data corruption events?

[KevinK]

I've just manged to setup DD Replication for a DDBoost MTree - OS 5.4.2.1-423209

[KevinK]

DD Replication has complete successfully, I've created a CIFS share on the remote DD/destination Mtree and can read the Veeam files without issue.

[tjgrie]

We're on 5.4.0.7 of Data Domain's OS and we replicate our DDBoost enabled Veeam MTREE to our offsite DD. Sounds like it's a limitation in an older version of the DD OS.

[vStorage]

Also successfully replicating DD Boost enabled MTREEs between DD2500 units running 5.5 firmware.

A key point is that ddboost user credentials must be the same across both units including group ID

[SGalbincea]

Quick question then: if your backups become corrupted on the primary side and you have the DD managing the replication, won't the backups that are replicated to the secondary site immediately become corrupt as well? How are you all designing for this possibility?

[KevinK]

Run a DD Fastcopy command to replicate the data to another folder on the destination DD outside of the replication pair.

[rgarrison]

We were able to get replication going via mtree. The key thing to do on the destination is to attach your ddboost user to the mtree, which designates it as a ddboost storage unit. EMC very specifically said do NOT enable cifs share on the ddboost repository, as you can screw up the data in the storage unit that way.

I added a DDBoost repository pointing to the replication destination, did a rescan, and saw the imported backups. Performing a file level restore from the replicated copy worked fine. We're going to have to slowly roll stuff into this replicating boost repository so the replication can keep up, but it's looking like we're in business.

[SGalbincea]

I think I have solved the corruption issue, at least for us. The DD has the ability to snapshot the MTREE that is being replicated, and in turn replicate that snapshot to the other DD. I have confirmed that I can mess up the .vbk on the primary copy and the replica, yet still access the backup successfully by mounting the snapshot on the DR side. As of last night I have disabled our copy jobs and am now using DD replication to copy our backups to DR.

[Dave1337]

Hello everyone,

I am in a similar boat to many in this thread. I have replicating Data Domain DD2200's, using Mtree replication. Everything works as of now, but the setup is not ideal.

DDBoost is only enabled on the primary DD, with the secondary DD being accessible in Veeam via CIFS.

I would like to have them both be DDBoost enabled but haven't figured out how to make this work... Is this even currently possible?

The production and DR site each have at least 1 dedicated Veeam VM.

DDOS 5.5.0.8
Veeam 8.0.0.917

Thanks in advance,
-Dave

[Dave1337]

Re-reading everything again, my question is:

How specifically do I attach a DDBoost user to a destination Mtree replication folder that currently has CIFS enabled. Step by step would be very much appreciated.

Also, does this break current replication or cause some type of re-sync to occur?

Kind Regards,
-Dave

[rgarrison]

FYI I was told explicity by EMC to NOT enable cifs for DDBoost storage units, as data corruption is a significant risk.

Here is the command to attach a ddboost user to a replicated mtree (storage unit):

Code: Select all

ddboost storage-unit modify <SU name> user <ddboost user name>

If you run the command "ddboost storage show", you should now see the replicated storage unit with RO/RD for status. This will enable the ability to make a DDBoost repository on the DR side. Rescanning the repository will bring the replicated backups into Veeam under the "Imported" section.

[Dave1337]

Thanks a lot!!! This is exactly what I needed.

I removed the repo/jobs from the remote Veeam server, and removed the CIFS share from the remote DD. Then I ran that and got it to work! Mounted it properly via the Veeam console and can confirm its working with no replication hiccups or otherwise.

Thank you for your help and I appreciate the prompt reply. Hopefully this helps someone else as well.

I still have a throughput issue on the remote site but I think it is related to the iSCSI connectivity to storage.

[haslund]

Also successfully replicating DD Boost enabled MTREEs between DD2500 units running 5.5 firmware.

A key point is that ddboost user credentials must be the same across both units including group ID

Can you explain this in more details please?
What happens if the user credentials are not the same across both units?
Do you have a source for the information?

Thank you in advance.

[haslund]

Is anyone seeing any issues with the performance/speed of merging oldest VIB file into VBK file at the end of the Backup Copy Job?

[vStorage]

Can you explain this in more details please?
What happens if the user credentials are not the same across both units?
Do you have a source for the information?

Thank you in advance.

As per the Data Domain command guide - page 93-95 I think. If you create a DDBoost Storage Unit on one Data Domain and then replicate that MTREE to another DD - the MTREE is replicated fine, but the DDboost Storage unit config is not carried over

This is expected behavior according to EMC.

The solution is to make sure your boost users are the created as the same group type from the start (either user or admin on both DataDomains) and then when you have replicated the MTREE to the second DataDomain, from the CLI of that unit - setting DD Boost ownership to the target Mtree, e,g, ddboost storage-unit modify <MyReplicaMTREE> user <MyBootUser>

[NightBird]

Is anyone seeing any issues with the performance/speed of merging oldest VIB file into VBK file at the end of the Backup Copy Job?

With GFS enabled ? If yes, Veeam made a private fix for one of my customer because each time at the end of the merge of the vib file into the vbk file, Veeam made a sort of compact of the vbk file (recompacting the .vbk file for limiting grow of it by ddboost design).

Yet the GFS merde do not do the "compact" anymore in the vbk file during the week (just the saturday on sunday during the weekly full).

open a support case

[haslund]

With GFS enabled ? If yes, Veeam made a private fix for one of my customer because each time at the end of the merge of the vib file into the vbk file, Veeam made a sort of compact of the vbk file (recompacting the .vbk file for limiting grow of it by ddboost design).

Yet the GFS merde do not do the "compact" anymore in the vbk file during the week (just the saturday on sunday during the weekly full).

open a support case

I already have a support case open and feel like it is going nowhere at the moment.
Could you do me a major favor and reference the case # where the private fix was issued? Then I can point to it when I call Veeam Support again.
Also, was this private hotfix for v8 before patch 1 or after patch 1?

Thank you

[NightBird]

After patch 1.

I will check with my customer for the case #

[NightBird]

Here is the case number
Case # 00689492