Comprehensive data protection for all workloads
Post Reply
markwyeth
Novice
Posts: 8
Liked: never
Joined: Oct 01, 2010 1:38 am
Full Name: Mark Wyeth
Contact:

Recommended settings when using a dedupe appliance

Post by markwyeth »

We are testing a the FalconStor dedupe appliance and interested to know what is recommended when using these type of devices. Specifically:

1. Should inline data deduplication be enabled or disabled
2. Assuming we turn off compression
3. Do we set storage optimization to: Local, LAN or WAN

Cheers Mark
tfloor
Veteran
Posts: 270
Liked: 15 times
Joined: Jan 03, 2012 2:02 pm
Full Name: Tristan Floor
Contact:

Re: Recommended settings when using a dedupe appliance

Post by tfloor »

We always have used:

1. Dedup enabled
2. Compression definitly off!
3. Storage to LAN.

You can play with option 3, because it depends also on the VM's you're backing up.
If you for example backup an oracle vm with much disk read/writes then you can set it to WAN if you think you're incrementals are too big.
But if it's a CIFS/SMB dedup appliance i recommend LAN

Cheers,

Tristan.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Recommended settings when using a dedupe appliance

Post by dellock6 »

Uhm, not sure about the inline deduplication.
To date , the only appliance where both Veeam and the dedup vendor stated it can be left enabled is ExaGrid. Tristan, which dedup appliance are you using?
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Recommended settings when using a dedupe appliance

Post by Gostev »

So far, I have not seen a dedupe appliance where overall resulting data reduction ratio would suffer from keeping Veeam deduplication on. Remember that Veeam dedupe is done with large block size. At the same time, keeping Veeam dedupe on may speed up processing significantly.

Consider using Low compression as well, besides None. Low compression algorithm does not mess up advanced dedupe, but may speed up processing significantly.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Recommended settings when using a dedupe appliance

Post by dellock6 »

After several tests with ExaGrid, I found out the best combination is inline deduplication on and compression off.
But dedup appliances are really different one another, so everyone of them need best practices and vendor advice to be used at its best.
Do not know the official position by FalconStor.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Recommended settings when using a dedupe appliance

Post by tsightler » 1 person likes this post

Inline deduplication on and compression off is pretty much the answer for every deduplication solution, however, some vendors focus heavily on wanting to show "maximum" dedupe ratio, so they suggest turning off Veeam dedupe. If your criteria for using a dedupe appliance is that you want the largest "dedupe ratio" possible, then of course you want to turn off Veeam dedupe. However, in most cases, the better goal is to focus on storing and transferring the least amount of data, as that will give the best performance, and in some cases, even be willing to give up a little dedupe/storage for performance.

To illustrate why some vendors like to recommend disabling Veeam deduplication consider this example. Let's say you are backing up a disk that has 40 identical 256K blocks, that's 10MB of identical data. Now lets assume that one of those blocks can be compressed and deduped 4:1 for a 75% savings. If you leave Veeam dedupe on Veeam will write only 1 of those 40 blocks, since the others are identical, that's a 40:1 savings in network traffic to the dedupe appliance, but the dedupe appliance will only report 4:1 dedupe. If we disable Veeam dedupe, Veeam will copy all 40 of the blocks to the dedupe appliance, thus copying 10MB instead of 256K, however, the dedupe appliance will deduplicate the 40 blocks, and report a dedupe ratio of 160:1. The amount of data on disk remained the same, but the amount transferred across the wire is 40x more. Once again, I recognize the contrived aspect of this example, but I'm only illustrating the point.

Using Veeam low compression can be a nice performance improvement, saving 10-20% in the total amount of data that must transfer across the wire, while increasing the size on disk typically by less the 5% do to slightly poorer dedupe. This 10-20% savings across the wire can really add up to some improved performance if you have many TB of full backups to run. Dedupe appliances can only ingest a fixed amount of TB's an hour and thus this performance improvement can translate into shorter backups, and faster restores due to less data having to be transferred across the wire during restores as well, but at the penalty of some additional loss of storage dedupe.

As with everything, it's all about striking the right balance for your environment.
tfloor
Veteran
Posts: 270
Liked: 15 times
Joined: Jan 03, 2012 2:02 pm
Full Name: Tristan Floor
Contact:

Re: Recommended settings when using a dedupe appliance

Post by tfloor »

dellock6 wrote:Uhm, not sure about the inline deduplication.
To date , the only appliance where both Veeam and the dedup vendor stated it can be left enabled is ExaGrid. Tristan, which dedup appliance are you using?
I WAS using a datadomain. DD610. Too slow. So I've placed it in /dev/null. :)
stormlight
Enthusiast
Posts: 48
Liked: 3 times
Joined: Apr 28, 2011 5:34 pm
Full Name: JG
Contact:

Re: Recommended settings when using a dedupe appliance

Post by stormlight »

I can't remember where I read this, or maybe watched it on a Veeam webinar:

Go to your Dedup appliance in Veeam under the backup repositories. Go to the advance tab where location is

Enable Align backup file data blocks
Enable Decompress backup data blocks before storing.

These two have helped me with my dedup box. DataDomain.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Recommended settings when using a dedupe appliance

Post by Gostev » 1 person likes this post

Definitely avoid enabling backup file data blocks align for deduplication storage that uses variable block size dedupe (such as DataDomain). What saved you here is that this checkbox does not currently work until you add one missing registry key ;)
stormlight
Enthusiast
Posts: 48
Liked: 3 times
Joined: Apr 28, 2011 5:34 pm
Full Name: JG
Contact:

Re: Recommended settings when using a dedupe appliance

Post by stormlight »

What you are saying is in conflict with this webinar or maybe the webinar was to generic and did not define the importance of storage that uses variable block size deDup. (I would hope that the presenter would outline that next time)

In addition the webinar makes no mention of the registry key.

http://go.veeam.com/webinar-12202011-ri ... am-ty.html

Is the decompress data blocks setting ok to check? Does this help with restores at all ?

Edit: I see the webinar was based on exagrid.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Recommended settings when using a dedupe appliance

Post by Gostev »

Yes, not using compression is always a good idea with any deduplicating storage.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Recommended settings when using a dedupe appliance

Post by dellock6 »

As an addition for those using ExaGrid, you can ignore block align since ExaGrid uses variable size, compression is needed only if you have the ExaGrid off-site, so you're best compressing the backup in the local proxy and then de-compress at the target site before saving data to ExaGrid.
Uhm, I'm better write a blog post about these settings so I will not need to copy/paste them in so many threads :)

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
bengoa@tulane.edu
Novice
Posts: 7
Liked: never
Joined: Sep 21, 2011 7:55 pm
Full Name: Alex Bengoa

Veean and Deduplication Appliances

Post by bengoa@tulane.edu »

[merged]

Is any one out there using a Deduplication Appliances with their VEEAM backups?

What appliance are you using? What kind of compression ratio are you seeing?

Are you backing up raw/ or are you using VEEAM deduping option as well?

We are looking at the DELL DR4000 backup appliance. Any insight would be appreciated.

Thanks,
Alex
pcmmitham
Influencer
Posts: 18
Liked: never
Joined: Jan 17, 2012 6:18 pm
Full Name: Pierre Mitham
Contact:

Re: Veean and Deduplication Appliances

Post by pcmmitham »

we're using steelheads for deduplication, nothing is better than them for that. but interacting with veeam....we're still working on it. its seemless for everyother app we use, but I think it might be whats causing veeam to fail miserably at replication. once we can replicate successfully I'll update this thread!!
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veean and Deduplication Appliances

Post by tsightler »

@pcmmitham - Are you referring to WAN deduplication? I think the OP was referring to deduplication storage appliances. If you are attempting to use Veeam replication with Steelheads than you'll want to disable multiple TCP connections as well as Veeam compression. Should work great. I have quite a few clients using the combination successfully.
ibis69
Lurker
Posts: 2
Liked: never
Joined: Oct 27, 2011 3:03 pm
Contact:

Re: Veean and Deduplication Appliances

Post by ibis69 »

Hi, i'm using Datadomain DD670 appliance atm.

On veeam, storage options are configured like this :

- Compression Level : None
- LAN target
- Deduplication ON

We have 20 jobs, half of them are reverse incremental and others incremental with synthetics fulls
Retention set to 3 months
Compression ratio : 18X ( 163 To of data for 9 To used on datadomain)
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Recommended settings when using a dedupe appliance

Post by Gostev »

Nice! Can to share some more details please? We could use some real-world feedback since this is a common question.
- Are you getting any decent backup performance?
- How did you hook up DD to Veeam, via CIFS-based repository?
- Did you try running some VMs off those backups (any vPower feature)?
- Anything else worth sharing?
NightBird
Expert
Posts: 242
Liked: 57 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: Recommended settings when using a dedupe appliance

Post by NightBird »

Gostev wrote:Definitely avoid enabling backup file data blocks align for deduplication storage that uses variable block size dedupe (such as DataDomain). What saved you here is that this checkbox does not currently work until you add one missing registry key ;)
So with a DataDomain appliance, should we use "Align backup file data blocks" as says "Veeam Backup & Replication Integration Guide, Version 6" or not ?

What are the Veeam Best Practices with DataDomain ?
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Recommended settings when using a dedupe appliance

Post by Gostev »

Dedupe-friendly compression, all other settings at default.
Post Reply

Who is online

Users browsing this forum: Bing [Bot], MarkBoothmaa, Semrush [Bot] and 145 guests