EMC Data Domain

[myrdin]

hold on, @Sgalbincea

are you saying that replicating with EMC Replicator is a lot faster than Veeam right? That is one of the things i was wondering about. Do you actually use replication with Veeam as well at the same time ? Did you try to fire up virtual machines from direclty the DD using Veeam failover? Does it work? or do you have to restore the VM first to a faster storage?

cheers

[dellock6]

I feel you are talking about two different things.
Veeam replication uses hypervisor datastores as target, so it's unlikely you will use DD share as an NFS share inside vSphere to run VMs from. I think we are more talking about backup copy in a secondary location, that can be achieved both with Veeam Backup Copy jobs (that however is a synthetic creation of a new backup file in the secondary location) or EMC Replicator (that if I uderstand correctly, is a replication of the content between two DD appliances).
Both have pros and cons, first of all you need to understand your business case.

Luca.

[SGalbincea]

In my case, I was weighing the pros and cons of having the DD do the replication via their own mechanisms vs. having Veeam do it via a copy job. The DD internal method is MUCH faster as it basically happens in near real time to the initial backup itself (we have 1Gb WAN links to DR where the other DD lives). The challenge though, is that if you mount up the DR replica share and look, the jobs are named identically (expected) since you are essentially looking at identical data. I believe I have solved this though, as I am going to spin up another full Veeam server at DR for the purposes of recovery from that replicated data as well as a copy job off to our tape library there. I wish there was some way to better delineate the replicas if I had to mount them both to the same Veeam server instance, but it is not the end of the world because in that scenario more than likely I will only bee seeing one of them anyway.

BTW, can't wait for v8 to come out....

[SGalbincea]

hold on, @Sgalbincea

are you saying that replicating with EMC Replicator is a lot faster than Veeam right? That is one of the things i was wondering about. Do you actually use replication with Veeam as well at the same time ? Did you try to fire up virtual machines from direclty the DD using Veeam failover? Does it work? or do you have to restore the VM first to a faster storage?

cheers

DD replicator is a LOT faster, but to make Veeam aware of the replica you have to rescan the repository or wait 24 hours. Tried both ways several times. I also have tried instant recovery from the DD - technically it "works", but you will wait two days for the VM to boot up. Rehydrating to a temp location will be a much faster way to get it back up and running. I am considering a LUN on the SAN as the initial target (until our SAN is supported directly in Veeam), with a copy job to the DD and then subsequently replicating to DR via the DD replication. Once at DR, I will run a Powershell script via task scheduler to scan the repo and also schedule full copies to the tape library once a month.

[Gostev]

To be honest, I am not really fond of storage replication for offsite protection when it comes to deduplication storage, because the data remains in the same fault domain. Should the data corruption appear in the primary system, replication process will carry it over to the secondary system just as nicely as it replicates "good" data. But unlike regular storage, where you can store multiple full backups in GFS schedule for redundancy, this corruption may mean immediate loss of your entire backup set (which can spawn years). And I've seen this happen in more than one occasion in the past years...

So, it's important to realize that replication between deduplicating storage, while indeed very efficient and will protect you from entire site loss, does not address MUCH more common events of storage-level corruptions, which in turn are extremely dangerous for deduplicating storage due to a nature of one (no data redundancy).

[SGalbincea]

Thanks Gostev, that is a very important point, and one that I really need to take into consideration as I design this new system. So, given what I have written earlier, it would seem best to do the following:

Data mover is a hardware Veeam proxy server - 64GB RAM, lots of CPU, 20Gb connectivity to LAN and SAN networks respectively each. Total data size for backup is usually around 120TB or so. DataDomains have 20Gb LAN connectivity. HQ to DR is a 1Gb link that will most likely be upgraded to 2Gb as part of this project.

1. Initial Veeam backup to LUN on fast SAN at HQ, retention limited to a couple days to ease size restrictions (to be replaced with array integrated snapshots when available in Veeam)
2. Copy job from HQ SAN to HQ DataDomain
3. Copy job from HQ DataDomain to DR DataDomain
4. Copy job once per month from DR DataDomain to LTO6 tape library

My biggest backup challenge is the size of the data we are protecting. The majority of this data is seismic (40+TB of it) and does not change over time, but it is just huge and time consuming. Thanks for the input, definitely looking forward to any additional thoughts or comments.

[Gostev]

Hi, Steve

To be honest, I think your design is close to perfect, now that you've explained the entire architecture.

That tape out in bullet 4 is really the critical piece of the puzzle that protects you from so many common data loss scenarios, most of which people don't normally even think about (like some angry admin deleting all of their disk backups). It makes me very happy to see users doing tape out even being heavily invested in DataDomain, the guys who love to pitch that tape is dead

The only two things I would look at changing is:

1. Do tape out directly from primary storage, instead of DR DataDomain. Right now you are basically doing tape out from 3rd hop. There is just too many processing steps data goes through before it finally reaches tape, each increasing the chance of data corruption (lots of complex logic in play), and these chances stack up quickly. Remember that every software has bugs, and complex software has complex bugs which are hard for QC to find

The rule of thumb is that you never want too many moving pieces in the picture, unless it absolutely cannot be avoided. Try to follow the KISS principle (Keep It Simple Stupid), as your data makes its way to tape (which will always be your last resort). For example, some people don't even trust incremental backups, and tape out Active Full backups to ensure what they put on tape is fresh raw data right from productions storage (which is an overkill, in my opinion - but you get the idea).

2. If possible at all, do tape out at least weekly to reduce potential data loss to just a few days - instead of potentially weeks.

But overall - very happy to see well thought out data protection design, and Veeam implemented according to the reference architecture

Hope this helps!

[SGalbincea]

It absolutely does. Unfortunately I will most likely have to fight a battle to get the library installed at HQ instead (management and space decision), but I will definitely give it a go. Worst case, it lives at DR and I do like you said and pull an active full onto it somehow. Given the volume of data we have, and the other levels of redundancy, the client feels that once a month is sufficient for now (but I will work on getting it done more frequently, you can be certain). Thanks again for the feedback, it is very much appreciated.

[veremin]

In case of tape library still sitting in DR site, you might want to perform test restorations from time to time just to make sure that the backup data is restorable and can be used, if something goes wrong with production site. Thanks.

[chrisdearden]

Given that we can write to the DD in a Lanfree way - can we use boost to tape out over fibre?

[Gostev]

Chris, not sure I follow the idea or how this is relevant. My point is that neither HQ nor (especially) DR DD box is ideal source of tape out jobs. Thanks!

[jrugg]

We are looking at both Data Domain 2500 and an ExaGrid EX5000.
Since ExaGrid has a non- deduped landing zone I would think that Sure Backup would work ok as long as there is enough space.
I have seen one post here about Sure Backup and Data Domain and would find it helpful if someone could expound on if they work together, if not what is the solution as well as performance issues.

TIA!

[veremin]

Since ExaGrid has a non- deduped landing zone I would think that Sure Backup would work ok as long as there is enough space.

The Surebackup will work fine for the recent backups, as they will be retrieved from landing zone without the rehydration penalty. For the retired backups that are no longer located on landing zone the performance will be still slowed.

Thanks.

[dellock6]

They work, problem is vPower needs to power on VMs from backup files, and those are compressed and deduped, so do not expect great performances at all.
A non-deduped area works intead like a plain storage, so performances on read operations are not impacted by deduplication and rehydration.

[luckyinfil]

Until version 8 is released, I can say that the Data Domain is NOT compatible with Veeam v7 as primary storage, no matter what their documentation tells you. I've had tickets with EMC about this and they've assured me that they are fully compatible with Veeam (their documentation had version 6), and were very reluctant to admit what was completely obvious.
Yes, Veeam can backup to a DD. However, you have to disable everything on Veeam to make it work. To me, that's like saying Veeam is compatible with a USB thumb drive. Sure, but you're not going to like the results.

I'm very excited that Veeam v8 will support DD Boost, and it will make it so much more bearable. However, it will still only make it partially compatible, and here is why:

Data Domain is designed to be a replacement of tape backups. It's a streaming backup appliance that does in-line deduplication, which means that all data that comes in gets automatically deduped. You can't turn that feature off, it's the whole point of the product.

Here's one thing the marketing brochures and sales guys never talk about: "rehydration". All data going out must be "rehydrated" (more on that later). With Veeam v7, writing backups to the DD is actually pretty fast. There are some extra settings, like you need to disable compression and use the "dedupe friendly" deduplication option. You also need to enable aligning storage blocks for the Data Domain repository. But it's fast because the DD is designed to take in streaming data. Pushing data back out is another matter.

You'll think it's running pretty great during test backups and the first few days of your backup schedule, until you reach the number of restore points you set up in Veeam. Then the problem arises when you try to do anything that requires reading the data, like doing a synthetic full backup. Synthetic fulls require a huge amount of read IO as well as write IO. That's where the "rehydration" comes in. The Data Domain must rehydrate data in real-time as it's being read, which is very CPU intensive and very, very slow. For example, I have a job that's about 1.5 terabytes of data. The daily incremental backups take a couple hours or less. The synthetic full took days to finish.

What else is heavy on read IO? SureBackup jobs, Instant Recovery, file browsing, Virtual Lab, and even Backup Copy jobs. Forget about those features if you're using a Data Domain, unless you don't mind waiting an hour to boot up a single VM.

Veeam v8 is going to address the issue with synthetic fulls and backup copy jobs by using some magic and Data Domain's "DD Boost" technology (*requires an additional license from EMC). But it's not going to have any effect on the performance of SureBackup jobs or Instant Recovery. The Data Domain must rehydrate data in order for it to be read, and nothing short of redesigning the Data Domain's hardware is going to change that. Other in-line dedupe appliances include staging partitions that negate this problem, but DD does not.

Let me be clear, the Data Domain works as intended, and I like the product. But it is simply not designed for backup solutions such as Veeam. My beef is with EMC selling the Data Domain as a primary storage device, and not properly documenting or disclosing the facts about the incompatibilities with some of Veeam's main selling features. You will need a primary storage for staging your backups, and for secondary storage or long-term archiving, the Data Domain works well.

And I hope that Veeam will make it very clear about the limitations when they are pushing v8 with DD Boost support.

I've been having the same issues with dedupe appliances for the last few months. While trying to use any dedupe appliance (I've mainly tested with HP's StoreOnce) as a primary storage, I found the read performance from file browsing/file level recovery to be abysmal. What's interesting to me is that there are no issues with sequential reads (restoring a full VM), but the random reads from File Level restores just kills the performance of the restore job completely. While you can argue that I shouldn't be using the dedupe appliance as primary storage, it still doesn't change the fact that FLR will have extremely slow performance if I'm restoring anything that is not in the non-dedupe landing pad making the FLR feature useless.

[luckyinfil]

The Surebackup will work fine for the recent backups, as they will be retrieved from landing zone without the rehydration penalty. For the retired backups that are no longer located on landing zone the performance will be still slowed.

Thanks.

In the case of file level restores on backups that are no longer in the landing zone, there should be a disclaimer that recovery may not always be possible due to timeout values. In order to even browse the files within a 3TB deduped VBK, I had to extend the timeouts to 1.5 hours.

[Vitaliy S.]

In order to even browse the files within a 3TB deduped VBK, I had to extend the timeouts to 1.5 hours.

Hmm...was it a Windows (built-in) FLR or you were using "Other OS" FLR wizard for this?

[Gostev]

Yep, that's definitely not normal, because all it takes before you can start browsing the guest files is FLR engine to retrieve a few virtual disk blocks from backup file (those containing MFT), and I don't have any logical explanation how this operation can take 1.5 hours with any dedupe storage. We are certainly not seeing this with our own test unit, which is by the way the lowest end ExaGrid unit in history (we are getting new one soon though).

However, let's please respect that this discussion thread is about EMC DataDomain. There are plenty of existing topics about ExaGrid

[luckyinfil]

Was the FLR tested using a large VM? I found that I had no issues doing FLR on dedupe appliances with smaller VMs (couple hundred GBs).

Hmm...was it a Windows (built-in) FLR or you were using "Other OS" FLR wizard for this?

Windows FLRs

[SGalbincea]

Was the FLR tested using a large VM? I found that I had no issues doing FLR on dedupe appliances with smaller VMs (couple hundred GBs).

Just tested this on a 280GB VM straight off of a DD640 - took about 15-17 minutes to get the file tree populated. Granted, this may or may not have been on an unreleased version...

[tsightler]

I have always recommended leveraging file indexing when using a dedupe appliance to store backups. Then you can use EM to restore files and browsing the filesystem is quick and painless. It may still take some time to do the actual restore though, but even that is usually significantly faster since we don't have to load so much data to build the entire file tree, only go get the specific items selected.

[Gostev]

Perhaps it's not the size of the VM, but rather how fragmented MFT is. In case of very fragmented file server with millions of files, MFT can become really big (remember that small files are stored entirely inside MFT), and of course very fragmented too. In which case, reading the entire MFT may potentially mean hundreds GB worth of somewhat random I/O.

[Matts N]

Hello Steve,
I have worked a little with the DD's and would actually make one change to your plan here.

Instead of
3. Copy job from HQ DataDomain to DR DataDomain

I would
3. Copy job from HQ SAN to DR DataDomain

Anything to avoid reading large amounts of data from the DD box. Don't think you will have any issue with the HQ SAN not being able to deliver and if I guess correctly, the data will still (more or less) use the same LAN paths to the DR site.

Just another 2 cents...

// Matts

[jj281]

Would someone mind clarifying this, for the upcoming VBR v8 and DataDomain Boost integration, will this be provided for all Editions (Standard/Enterprise/Enterprise Plus)?

I understand DD Boost licensing will be needed

[foggy]

The feature list for all Veeam B&R editions will be available closer to the release date. We are unable to comment on it at this time, as the current plans may change. DDBoost license will be required.

[Gostev]

Current plan is that DataDomain Boost integration will require Enterprise Edition license from Veeam perspective, but as Foggy mentioned, don't hold us to it until the official v8 editions document is published.

[dgapinski]

Hello Steve,
I have worked a little with the DD's and would actually make one change to your plan here.

Instead of
3. Copy job from HQ DataDomain to DR DataDomain

I would
3. Copy job from HQ SAN to DR DataDomain

Anything to avoid reading large amounts of data from the DD box. Don't think you will have any issue with the HQ SAN not being able to deliver and if I guess correctly, the data will still (more or less) use the same LAN paths to the DR site.

Just another 2 cents...

// Matts

I have a couple questions along a similar line about this, that I hope Gostev and/or Foggy could comment on:

If DDBoost would save all unnecessary traffic from leaving the VM, what are the ramifications of having Veeam write directly to the HQ SAN and - like Matts is suggesting but without the copy job? Not saying that we wouldn't want HQ storage, we would. I'm questioning the need for a copy job.

Also, given it's purpose, would there be any problem with making EVERY DR backup an active full, or does Veeam's DD boost function only work with incrementals? If it works with fulls, it might suggest that storage space and communication are minimized to the lowest possible need, leaving to consider how much it would take for a VM to process what chunks to send.

Thanks for discussing this topic btw - perfect reference for use as we consider a 2200.

[Gostev]

Hi,

I don't understand the first part of your post or architecture you have in mind.
But yes, DDBoost will work with both active full backups and incrementals.

Thanks!

[dgapinski]

That's helpful. I was piggybacking off of Steve's idea, but I wrote it wrongly - rather than saying "having Veeam write directly to the HQ SAN", it should have said "having Veeam write directly to the DR DataDomain", but then I wasn't considering the extra processing time saved by the copy job. Never mind that, but thanks for answering my second question - THAT makes the periodic active full not such a traffic hog, or space hog on the DR side too.

[EzE]

So it seems pretty well established that lower end Data Domain devices are not a good primary Veeam target, mainly because rehydration of data is slow and affects restores, surebackup, and instant recovery. If that is the case, I am curious how V8 integration with DDBoost affects a 3-2-1 best practice design. To boost restore times, I have seen several people much smarter than I suggest writing backups to a different target before copying to the DD. This was my take on it:

1.) VEEAM backup jobs should be written to fast disk, with enough space for maybe a few days.
2.) VEEAM backup copy jobs from the fast disk to larger/slower disk - potentially Data Domain.
3.) VEEAM backup copy jobs from the fast disk to offsite target. In my case this will likely be a VEEAM server in Azure with attached disk and WAN accelerator.

With this design concept, I can assume that DDBoost only affects the backup copy job speed? Would DDBoost give me any performance benefit for the production backups? Also, how would this affect the job settings at each level. Would step 1 backups be set to "dedupe friendly compression" knowing that the data will eventually be copied to the Dedupe appliance? Or would I leave Veeam dedupe and compression settings to their recommended, knowing that the copy job has it's own compression options, and DDBost will be affecting the transfer?

On the same topic, I am curious if anyone has any deduplication numbers running Data Domain & VEEAM. Newer forums and documentation seem to point to keeping VEEAM duplication enabled to reduce network traffic, and I'm wondering what type of additional dedupe people are seeing with DD & VEEAM together. A server 2012R2 volume with deduplication is an interesting alternative. http://www.veeam.com/blog/how-to-get-un ... ation.html

Thanks!


PS Notes:
- I am new to VEEAM and currently just running a test environment.
- Full environment backup is 9TB
- 1Gbps infrastructure - hoping to avoid adding 10G