Dell EMC Data Domain + SureBackup?

[namiko78]

Hello

We are considering getting a data domain for our primary backup. We currently use Surebackup to check all backups once a week, which take about 8 hrs to process, using a 12 disk raid array. Is Surebackup supported on data domain, and what will my speeds be like? We're looking at a single shelf data domain, and they claim it's got a built in SSD now which will of course speed up certain operations.

Thank you

[Rick.Vanover]

Hi Namiko78. The official guidance is here:

https://www.veeam.com/kb1956

However, i can provide perspective having dealt with this for years. The I/O requirements for SureBackup by its nature has random reads, which is not an ideal experience for a deduplication solution. I'm sure the SSD will improve performance - but I can not imagine it to be on-par with general purpose disk.

One option is to use the Scale-Out Backup Repository with a deduplication appliance and a general purpose disk. Basically, you would put your full backups on the appliance and increments on the general purpose storage, what is called a performance policy in the UI. This may help.

As a general rule, our highest performant environments using all of the vPower capabilities (Instant VM Recovery, Data Labs, SureBackup, etc.) have the first backup on general purpose disk.

[cb1ocker]

I recently purchased a pair of DD6800s that have the SSD tier and use them as primary backup storage. Backups work great. DD replication between sites works well too. I was originally looking forward to running SureBackup jobs off of them also, but that hasn't worked out so well. Most VMs will boot fine, but with some VMs the VMware Tools service times out when starting, which causes the SureBackup job to fail. If I back these same VMs up to any other storage and run the SureBackup job from that storage, with the same settings, the VMware Tools service starts in time and the job succeeds. At this point we're looking at re-architecting our backup infrastructure to send initial backups that require SureBackup validation to other storage first with very low retention, then copying the backups to the DD6800 for longer term storage, which from what I've seen here is actually best practice for dedupe appliances anyway, but obviously requires a lot more storage allocated to backups as well as administrative overhead.

[Rick.Vanover]

@cb1ocker - thank you for that feedback.

Your last sentence validates in the field what I have seen in the lab and in conversation with many other customers about.

One option is to also only have some VMs tagged critical - those that need SureBackup - and have them go to raw storage repository first.

[ChrisSnell]

What Rick describes can also be achieved using an ExaGrid architecture - but all in one unit (or many similar units) instead of different types of disks. Our landing zone keeps the most recent week of backup data on straight disk, which means that all the Veeam vPower features work as intended. Then historical data is kept in our deduped storage, giving the best of both worlds. The solution supports SOBR for easy expansion.

See: https://www.veeam.com/exagrid-storage-solutions.html

[namiko78]

Great suggestion. I'll look into Exagrid. I'd looked into it years ago but not recently. One of the big reasons we're looking at an appliance is to increase security. The data domain has some features to prevent data from being deleted should the veeam console ever be accessed by someone malicious.

Thanks to everyone else for your comments also.

[Gostev]

Storage appliances with "read-only-ness" implemented in firmware is certainly one of a few valid ways to achieve air gap for your backups. I like to highlight this option among others in my presentations on backup best practices.

Just be sure to treat any deduplicating storage as a secondary backup repository for your backup copies and long-term retention. Don't land backup there directly, or your RTOs will suffer, just as per your observations. Besides, you still need another backup storage to achieve 3-2-1 rule regardless - so just land your backups to a fast but small primary repository, as our reference architecture suggests. Then, copy them to a secondary repository, for which the performance will no longer matter that much.

[jtupeck]

We have been doing this in our environment for about 5 years, now. A secondary storage location for short term, 3-5 day backup storage (originally an HP/LeftHand SAN, then EMC 5500 array, now a Cisco S3260 proxy/repo) with backup copy jobs to the DataDomain infrastructure.

The DataDomains, even with the SSD tier, are just too slow to do anything meaningful that has a time sensitive component. Their dedupe is absolutely insane, though. 17x (ish) in our enterprise wide environment. 4.8PB logical data footprint in about 275TB of consumed DD space.

[tjgrie]

SureBackup is the bane of Data Domains. We're on a DD6300. If you want your SureBackup job to have a chance at working, you need to bump the 'Maximum allowed boot time' for the VM up to around 35-45 minutes (2,100 seconds min.). This should give it enough time to be responsive.

I'm sort of surprised they even bothered to put SSD's in a Data Domain. In our testing of DD Virtual Edition, having the storage exist on an AFA did nothing for its performance. DD seems to be bound mostly by CPU and memory.

[jtupeck]

What Rick describes can also be achieved using an ExaGrid architecture - but all in one unit (or many similar units) instead of different types of disks. Our landing zone keeps the most recent week of backup data on straight disk, which means that all the Veeam vPower features work as intended. Then historical data is kept in our deduped storage, giving the best of both worlds. The solution supports SOBR for easy expansion.

See: https://www.veeam.com/exagrid-storage-solutions.html

In my experience, Exagrid is really great for environments that do not have lengthy retention requirements on backups, have a LOT of redundant data, have a small data set, or an insanely huge WAN connection to wherever their DR data center is for off-site copying of data. When we compared Exagrid costs against what our per TB/GB costs were for logical data retention on Data Domain, the DD architecture won handily because it is still best in class for dedupe, supports fiber channel connectivity so our backups never hit the general network and would take up less of a rack footprint. 15-20x dedupe across our environment is something that we have not seen any other system come even close to achieving.

I desperately want to get away from a two tiered architecture for backup and backup copy job targeting in order to simplify the environment, but the relatively low cost of a UCS S3260 with a pile of disk and a well provisioned DD6300 architecture make it nearly impossible until our retention policies are reduced (hopefully coming soon). The added benefit is that because the DD duplication is so good, our nightly backup footprint of ~50TB compresses to ~300-400GB that automatically gets off-sited across our 800 meg P2P link to a DD2500 on the other end. By 7am when I arrive, all of the data is generally off campus. Coming from a tape based backup environment where we used to send someone, every day, on an hour long 'tape run' to an off site storage facility that was still in town, mind you...the way everything functions now is worlds better, so even with a two tiered architecture I can't really complain. Veeam just makes the world better because there are so many options to fit any environment.

Just be sure to treat any deduplicating storage as a secondary backup repository for your backup copies and long-term retention. Don't land backup there directly, or your RTOs will suffer, just as per your observations. Besides, you still need another backup storage to achieve 3-2-1 rule regardless - so just land your backups to a fast but small primary repository, as our reference architecture suggests. Then, copy them to a secondary repository, for which the performance will no longer matter that much.

1000%, this.

[Rick.Vanover]

Another twist regarding the two-tiered approach - and the fact that you mentioned long retention....

We will soon have object storage support. This will make the scale-out backup repository effectively unlimited. You could have a happy situation where the following happens:

-A backup job has many restore points
-A (relatively) small amount of high-performance general purpose disk exists on-premises
-Retention is set to use the object storage implementation we are planning
-SureBackup done on the first storage/most recent backups completed

In this situation - you may be in a really good arrangement.

[bpayne]

I know others have shared their Data Domain experiences, but why not add another with a different DD model. I have a set of DD4500's (no SSD Tier) and as suggested above, deduplication/compression is amazing, backups to the DD's work great and DD to DD replication (method not supported by Veeam) works great as well. But any kind of restores are painfully slow. SureBackup does not work. Which was enough for us to look elsewhere. I am very glad someone above mentioned their experiences about having a DD SSD tier as I was told adding the SSD tier would solve all my problems - I was skeptical. I'll never forget, five years ago, a very respected Veeam SE flat out said we should not use Data Domain as our primary backup storage platform, when we were beginning our Veeam implementation! I can't end without a "but" - this is not to say DD is not a good fit b/c as already mentioned, it can do alot of great things if its a fit for you.

I agree with Veeam's recommendation about faster primary backup storage and then for example DD for secondary backups and this is proven to work better from my experience and others, but do know that can be an expensive setup and add's some complexity, especially if the amount of VMs/data is quite large and you have lengthy retentions. In my case, ExaGrid's architecture seemed to fit my company best. I do love all the options Veeam gives us, but it can be hard on the customer when every backup storage vendor says they're the best. You really have to do your homework! Hope this was helpful and am very happy with everyone working together and sharing their experiences!

[Rick.Vanover]

Cheers, Brandon - thanks for sharing your perspective. This again validates our recommendation.

To the Data Domain's credit- I remember at one point in the lab, no matter what I could put on it, it would never go down in available space. It's storage efficiency is incredible.

[Gostev]

You should have tried that with X-ray images, or even just Microsoft Office documents