Deduplicating Backup Storage Suggestions

[mongie]

Hello,

We're currently in need of some extra storage for our Veeam backups, and I'm thinking along the lines of a de-dupe appliance. We're currently storing ~ 60TB of backups on 2x Dell MD1200 direct attached shelves with 3TB disks in RAID 50. I find that the MD's are (I assume this is the problem) too slow to do surebackup restores on my high IO servers (specifically Exchange 2003).

We' ve identified that we're going to need to store at least 120TB of backups on the new appliance. Ideally, we'll be
So far I've spoken to Dell - we normally use their gear - about the DR4000 appliance. The model in question will have 18tb of disk and is supposed to store 270TB. We're also looking at a datadomain (or potentially anything from IBM, HP or NetApp). I've only got pricing on the Dell unit so far, and it is probably within my budget (around 35K AUD).

Does anyone have any advice on what to buy or what not to buy? I've seen threads on here criticising datadomain for slow ingest rates... and I think I may have seen the same mentioned about HP. From my brief conversation with the vendor, the DD model we'd be looking at is the DD640. Is there any support for DD Boost within Veeam? HP (Catalyst) software? Is that on the roadmap? I know that Exagrid are support Veeam, but they dont appear to be expandable, and thats one of the things I have been looking at as a positive. My vendors have also not really heard of them. I know they have postprocess dedupe which is different to most of the other vendors... What about FalconStor, Quantum?

Compared to plain DAS storage, are there any major differences with using Veeam with a dedupe appliance as the storage? Is there anything (apart from space and ingest rate) that I should be considering? Is it possible to run surebackup restores from the appliances?

Any advice you can provide would be appreciated.

[Gostev]

Hi,

Massive Exchange 2003 servers is just naturally bad workload for vPower (since Exchange 2003, Microsoft has reduced Exchange I/O requirements a few times both in Exchange 2007, and especially in Exchange 2010 - one of the major reasons why we upgraded, btw).

Your current storage is certainly much faster than any inline deduplicating storage mentioned above (pretty much any raw storage is much faster than deduplicating storage). Typically, you are not able to run SureBackup from any inline deduplicating storage. So one thing you should not hope is that this new storage will improve vPower performance. ExaGrid is different though, as it does post-process dedupe and features raw disk landing zone.

I believe ExaGrid's main differentiator is expandability (thus "grid" in their name), but you'd better check with them directly.

At this time, we do not plan specific integrations with any backup storage - we like to remain storage-agnostic, and happy with the current performance of writing to those boxes anyway. While integration will not help with vPower performance anyway, because it does not accelerate random I/O.

Regarding best settings for writing to dedupe devices, they depend on device - most vendors provide integration guides, and their recommendations is usually different from ours because of different goals anyway, there are plenty of existing discussions regarding this here on these forums, so please just use search to find those.

Thanks!

[deduplicat3d]

I would recommend not getting a dedupe device. I use dedupe for the sole reason that I am required to send backups to tape which works best with incremental (weekly fulls). If I were you I would get some really fast backup storage and just do reverse incrementals.

[Gostev]

Indeed, deduplicating devices are best used for the long term data archival purposes (as a tape substitute).

[mongie]

So you're suggesting that I would continue backing up to raw disk, and then archive older backups somehow on the dedupe appliance?

Is there anything the roadmap around making it easier to transfer backups around without breaking the database? At the moment that is looming as a complication to using multiple types of disk... I'm using forward incremental with active fulls at the moment and its working fine... I dont really want to use reverse inc if I can get away with it because the extra IO makes the backups SLOOWWW.

As far as buying really fast storage - well I'd love to, but try getting the $100k or whatever it would cost for 190TB of disk signed off

Looks like I'll just do a POC on a datadomain or something and see how I go.

[Gostev]

Is there anything the roadmap around making it easier to transfer backups around without breaking the database?

Please clarify a little bit what you mean here.

[mongie]

At the moment, you can move a whole backup job to another location... but it would be good if somehow I could move parts of a job...

Archive locations would be ideal... e.g. keep x restore points on location 1 and then y restore points on location 2.

[kortasma]

Another idea is to consider an appliance that enables deduplication at the primary location of the VMDKs. A NetApp FAS unit can be deployed with low cost storage. The deduplication space savings occurs at the source, so you consume less capacity when storing the VMDKs. And, you can also enable disk based snapshot to protect against logical failures. You can then look at a less expensive secondary disk solution to simply write full backups to.

My proposed solution works in reverse too, you can configure at NetApp FAS to receive backups via NFS, CIFS or SAN protocols and do post-process dedupe and/or inline compression. However, the NetApp FAS device is better suited to serving primary data for VMs versus being relegated to a secondary disk solution. Good luck!

[mongie]

I'm going to be running a POC with a data domain appliance. Their "key" to throughput is the number of streams being written to disk at once (up to 90?) so, is there any way to know or adjust the number of streams being written to disk by veeam?

Its a shame that you're not looking at implementing DD Boost, it looks like it provides a good benefit - BackupExec and vRanger both support it.

[Gostev]

1 stream per job.

DD Boost will certainly provide a good benefit for any solution that does not feature inline source-side dedupe and compression, like the above-mentioned products (and unlike Veeam).

[Saintly]

Hi,
I am currently making selections to replace my backup systems including hardware and software.
In this thread I was choosing between AppAssure and Veeam and have decided on Veeam.

Now i am considering the backup target to save the backups too.
Dell is pushing their DR4000 Disk Backup Appliance. It has features like deduplication, compression and replication built in.
It has 18Tb of space within it but the Dell tech rep claims that with the deduplication and compression that it's actualy equivilant to about 270Tb of storage. I personaly don't understand how it can deduplicate backup files as each file would be unique wouldn't it?
I would still be asking Veeam to deduplicate and compress so the amount of data being pushed accross the network is smaller so im not sure if the DR4000 would find more that it could reduce.
They also say that a benifit is that having the device handle the replication to another site (a second DR4000) frees up the backup software and reduces our backup window.

Does anyone have experience with this device or does anyone think i should just get large disk arrays with heaps of disk space and rely on Veeam to handle the deduplication, compression and replication?
Any other options that i should consider?

A bit about my current environment:
I have 3, 2 socket servers running VMware Essentials Plus ESXi (i.e. 6 sockets)
These connect to a SAN (Dell MD3200) to house the VMs
Current backups is with BackupExec to a tape autoloader. Very slow and clunky. Not really looking after the Linux servers.
Running 8 Windows 2008r2 VMs and about a dozen Linux VMs.

Thanks
Ian

[dellock6]

Hi, I've never used the Dell unit, but I have a fair experience with deduplication appliances, mostly DataDomain and ExaGrid.
First, do not trust their numbers: usually are there as "best case". If you fill them with txt files deduplication is huge and can probably reach those numbers, but what if you save zip files or jpeg images, already compressed? Your best choice is to ask them for real numbers about Veeam backup files.
Also, remember Veeam does not do file backups, it saves whole VMs, so deduplication must occur at byte level. So it does not matter if there are unique files or duplicated ones, dedup happens at the block level.

About replication, they are correct: since Veeam can save only to a single destination for any given job, you can run Veeam only once and save to the first appliance, and then let the appliances replicate.

Luca.

[Vitaliy S.]

Ian, I would also recommend taking a look at Windows Server 2012 as a storage device for your backup files, with deduplication feature enabled you should get pretty decent global deduplication results.

For further reading, please check out our blog post: http://www.veeam.com/blog/how-to-get-un ... ation.html

[Gostev]

Windows Server 2012 dedupe might not suit well depending on amount of data that needs to be backed up, as it has pretty limited post-process dedupe performance. So, I would not compare it directly to hardware storage appliances.

[jpeake]

Saintly

I just went down this path with the DR4000 also. The claims sound nice, but in the end Dell wasn't able to prove it with real-world data. I asked Dell for a demo unit, they don't offer one. They did send me a spreadsheet that the Dell storage engineers use to size systems. The macro's are locked so i can't see the code (was hoping to find the formulas). But entering in our estimated workload data - it came back with an expected dedupe ratio of 5:1. A far cry from the 15:1 they claim in the marketing materials.

So I ended up ordering a Dell MD3600i, using 3TB disks and 10Gbe. It's not been delivered yet, but will let you know how it goes when I get it. I was looking at the 5.4TB/70TB DR4000. The MD3600i was about $4000 cheaper than the DR. Will use it combined with Veeam's dedupe, hosted on a Win 2012 box using Windows dedupe. Veeam seems to be pushing this combo (not the storage hardware, but the combo of Veeam and Win2012). Check out the "whiteboard Friday" from last week. They detail the setup pretty well.

The dedupe ratios are pretty dang nice with this combo, and seems to have less headaches than you MIGHT have with dedupe appliances. And it's faster and cheaper.

[chimera]

Dell is pushing their DR4000 Disk Backup Appliance. It has features like deduplication, compression and replication built in.
It has 18Tb of space within it but the Dell tech rep claims that with the deduplication and compression that it's actualy equivilant to about 270Tb of storage. I personaly don't understand how it can deduplicate backup files as each file would be unique wouldn't it?

Not done at file level. It only keeps unique blocks of data with an index/pointers to each 'unique' block thus saving on storage space.

I would still be asking Veeam to deduplicate and compress so the amount of data being pushed accross the network is smaller so im not sure if the DR4000 would find more that it could reduce.

On the contrary. Dedup appliances are purpose built for just that, dedup'ing and compressing data. Even with Veeam dedup/compression on, I would guarantee you'll get further deduplication/compression on a dedup device. The biggest benefit I personally see with the DR4000 (or in fact, any dedup device that supports it - I've only had experience with DD) is the replication feature by automatically replicating dedup'd/compress blocks of data across the LAN/WAN to a similar device (preferably) in an offsite location. To me thats a big selling point, especially when I've got 30TB of data (if running a full) to backup. Fire in the server room? Flood? Earthquake? Meteorite? Having offsite backups largely "automated" (via the replication) equals peace of mind.

Of course, if you're running EqualLogic currently and aren't in a hurry, then wait til Dell port the Ocarina dedup code into the EqualLogic firmware, add a member (tray) of SATA disk, utilise that as a backup target, then buy another member house it offsite and utilise EqualLogic SAN replication to transfer your backups offsite.

[emachabert]

Hi,

We are using two HP D2D 4112 (9TB) replicating between two sites over a 40mbit/s optic fiber line (150 km), those boxes are only used as Veeam target:
Here is our global dedup ratio:
Disk Space Used 3.5 TB of 8.9 TB (39.6%)
Deduplication Ratio (7.0 : 1)
User Data Stored 24.7 TB

Here are the stats for each CIFS share:
VEEAM-LINUX 29 restore points (real data: 2.7 TB, size on disk: 288.7 GB, ratio 9.5:1)
VEEAM-WINDOWS 37 restore points (real data: 7.1 TB , size on disk: 1.0 TB, ratio 7.0:1)
VEEAM-SHAREPOINT 43 restore points (real data: 12.2 TB , size on disk: 1.2 TB, ratio 10.2:1 )
VEEAM-EXCHANGE 18 restore points (real data: 2.7 TB , size on disk: 753.9 GB , ratio 3.6:1 )

Write speed is between 60 and 80 MB/s.

The only problem we are facing at the moment (we have an ongoing support ticket with both Veeam and HP) is read speed.
We can't really use vPower NFS based jobs as random reads are too bad (5 hours to restore an item from our sharepoint app).

[dellock6]

I suspect HP appliance is doing inline dedup, right? Read speed is always a problem with inline, since it needs to rehydrate data before exporting it. And vPower + Instant VM Recovery is almost the worst foe of this configuration
BTW, the debate about inline vs post-processing deduplication is almost never ending, like discussions about religion...

Luca.

[Rumple]

Saintly

I just went down this path with the DR4000 also. The claims sound nice, but in the end Dell wasn't able to prove it with real-world data. I asked Dell for a demo unit, they don't offer one. They did send me a spreadsheet that the Dell storage engineers use to size systems. The macro's are locked so i can't see the code (was hoping to find the formulas). But entering in our estimated workload data - it came back with an expected dedupe ratio of 5:1. A far cry from the 15:1 they claim in the marketing materials.

So I ended up ordering a Dell MD3600i, using 3TB disks and 10Gbe. It's not been delivered yet, but will let you know how it goes when I get it. I was looking at the 5.4TB/70TB DR4000. The MD3600i was about $4000 cheaper than the DR. Will use it combined with Veeam's dedupe, hosted on a Win 2012 box using Windows dedupe. Veeam seems to be pushing this combo (not the storage hardware, but the combo of Veeam and Win2012). Check out the "whiteboard Friday" from last week. They detail the setup pretty well.

The dedupe ratios are pretty dang nice with this combo, and seems to have less headaches than you MIGHT have with dedupe appliances. And it's faster and cheaper.

Did you get this up and running yet? Curious about results and amount of data you can retain.

[cramry]

"We are using two HP D2D 4112 (9TB) replicating between two sites over a 40mbit/s optic fiber line (150 km), those boxes are only used as Veeam target:
Here is our global dedup ratio:
Disk Space Used 3.5 TB of 8.9 TB (39.6%)
Deduplication Ratio (7.0 : 1)
User Data Stored 24.7 TB

Here are the stats for each CIFS share:
VEEAM-LINUX 29 restore points (real data: 2.7 TB, size on disk: 288.7 GB, ratio 9.5:1)
VEEAM-WINDOWS 37 restore points (real data: 7.1 TB , size on disk: 1.0 TB, ratio 7.0:1)
VEEAM-SHAREPOINT 43 restore points (real data: 12.2 TB , size on disk: 1.2 TB, ratio 10.2:1 )
VEEAM-EXCHANGE 18 restore points (real data: 2.7 TB , size on disk: 753.9 GB , ratio 3.6:1 )"

Could you elaborate, if possible, on your backup to the HP D2D? I only ask due to I'm currently using 2x D2D4106 (8TB) for offsite backups and the dedup I'm getting is absolutely terrible! I've tried multiple scenarios with no improvement. Current Scenario is that I have about 2+ TBs disk space used, with 6TBs user data stored, and total storage is showing at 6TBs. I contacted HP who stated that the difference was due to the complex calcualations and the "metadata".

[emachabert]

Here is the link to the white paper written on that setup :
http://www.veeam.com/wp-veeam-hp-machab ... goals.html