Comprehensive data protection for all workloads
Post Reply
mongie
Expert
Posts: 152
Liked: 24 times
Joined: May 16, 2011 4:00 am
Full Name: Alex Macaronis
Location: Brisbane, Australia
Contact:

DR4100 as a target

Post by mongie »

Hello,

Just wondering if anyone is using a Dell DR4100 as their backup target. We're currently talking with Dell about some issues, and they don't seem to know of anyone using the appliance with Veeam, so I was hoping to find someone here so we could compare experiences.

Thanks,
Alex
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: DR4100 as a target

Post by dellock6 »

I Alex,
I have no experience with the 4100, I tested the 4000 a while ago (http://www.virtualtothecore.com/en/dell ... uick-test/) but I don't know how much improvement has come with the 4100, so if that test is already valuable or outdated.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: DR4100 as a target

Post by Vitaliy S. »

Alex, can you please tell us what issues you're experiencing with this storage? Are they performance related? What is your configuration?
mongie
Expert
Posts: 152
Liked: 24 times
Joined: May 16, 2011 4:00 am
Full Name: Alex Macaronis
Location: Brisbane, Australia
Contact:

Re: DR4100 as a target

Post by mongie »

Vitaliy S. wrote:Alex, can you please tell us what issues you're experiencing with this storage? Are they performance related? What is your configuration?
Hi Vitaliy,

My colleague has a ticket open (with Veeam) about this, and it has been an ongoing issue. We have two problems with the appliance. Firstly, that we're not achieving the dedupe ratio that is advertised (We get 4:1 at best, they advertise 10-15:1). We have 72TB RAW, and we have around 300TB logical data stored. The appliance is now showing 15TB free raw disk.

Secondly, performance of the appliance is poor. We have a fully populated unit (36x3TB). We've been told by Dell support that in the 12 or so months since it was put in place, the "cleaner" has never completed a cycle. We've now had it running with no other I/O for 9 days, and while it has reclaimed ~ 15TB of raw space, its not realistic to expect this to happen on a regular basis.

Sequential write seems to peak at ~ 100-150MB/s, so running more than one or two active full backups doesn't work very well. Running the cleaner 24x7 also kills write speeds. Running Tape backups with the dedupe appliance as the source also shows significantly slower performance than raw disk (30-50MB/s). I should mention - my preliminary testing shows this is the same when reading or writing directly to the appliance with Windows.

Our local Dell sales team are looking at the issue with us, but they're very keen to point at Veeam as the culprit. They also seem to think that nobody (apart from us) is using the DR4100 with Veeam, despite it being supported by Dell. The DR4100 integration guide suggests we use Veeam dedupe, with compression disabled, and apparently this is why we're only getting 4:1 dedupe.

I'd be interested to know if DD users are getting better dedupe ratios, especially when they have Veeam doing inline dedupe.

Vitaliy - Is there any way to scan files on disk for their Dedupe/Compression ratio as seen by Veeam? Apparently, despite Local Target Dedupe being enabled, we're still only seenig 1.0x or perhaps 1.1x dedupe displayed in Veeam.

If we could get some assistance in determining whether Veeam is in fact the culprit (at least for the dedupe ratio) that may assist.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: DR4100 as a target

Post by dellock6 »

Alex,
just a note on the dedup results: you need to sum both Veeam and the appliance dedup. Disabling Veeam dedupe just to see higher ratios in the appliance is good for the appliance results, but the overall result is what counts. If Veeam can give you 2:1 and Dr4100 4:1, the total is 8:1. If you get 8:1 from the appliance when Veeam dedupe is disabled, the final result is going to be the same.
Also, be aware 10,15:1 is probably the highest expected ratio, that can probably be reached in almost perfect conditions. I've seen in many environments 4 and 5 be more realistic values.
Finally, 1.0 is probably coming out during incremental runs, and this is expected since we already extract only new blocks, that are supposed to non exists in the previous backup files. Full backups are usually better to evaluate dedupe ratios.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: DR4100 as a target

Post by Vitaliy S. »

mongie wrote:Firstly, that we're not achieving the dedupe ratio that is advertised (We get 4:1 at best, they advertise 10-15:1).
I actually believe you can achieve this advertised dedupe ratio (10-15:1), but this would be for only for dedupe-friendly data :) I would not expect these numbers from production VMs running SQL and Exchange servers.
mongie wrote:Vitaliy - Is there any way to scan files on disk for their Dedupe/Compression ratio as seen by Veeam? Apparently, despite Local Target Dedupe being enabled, we're still only seenig 1.0x or perhaps 1.1x dedupe displayed in Veeam.

If we could get some assistance in determining whether Veeam is in fact the culprit (at least for the dedupe ratio) that may assist.
If you want to test the dedupe of this device, you can completely disable this option in Veeam and then check the dedupe stats. On the other hand, If I were you I wouldn't chase for the dedupe numbers advertised by Dell. I would try to configure your backup jobs in the way that would allow you to have the smallest backup files possible, probably even with compression option enabled. Just my 2 cents.
mongie
Expert
Posts: 152
Liked: 24 times
Joined: May 16, 2011 4:00 am
Full Name: Alex Macaronis
Location: Brisbane, Australia
Contact:

Re: DR4100 as a target

Post by mongie »

Hello again...

My colleague was told by Veeam support that we should not use backup copy jobs on our dedupe appliance, because they use a transform process.

I can understand how this would affect performance, but is there going to be any issue with dedupe ratio or file integrity?

We're currently running backup copies with the same appliance as source and destination of the copy, which I realise is going to impact performance. I expect that we should be able to use a dedupe appliance as the target for a backup copy job with no issues?
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: DR4100 as a target

Post by dellock6 »

Hi Alex.
Yes, backup copy jobs created the same file types of backup jobs. So both dedupe and file integrity will work as in primary backup jobs.
The problem of having a dedupe appliance at source is the fact Veeam has to read its content, so asking to the appliance to rehydrate data, thus the performance impact. But this is the "only" problem in such a scenario. If you only use a dedupe appliance at target you will be fine. By the way, is the kind of design we usually suggest, a plain storage as primary target and a dedupe appliance as secondary target.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
mongie
Expert
Posts: 152
Liked: 24 times
Joined: May 16, 2011 4:00 am
Full Name: Alex Macaronis
Location: Brisbane, Australia
Contact:

Re: DR4100 as a target

Post by mongie »

Thanks Luca,

One more question... V8 DD Boost support... Our Dell sales guy suggested DD Boost is a trade name for OST by EMC. Is this true? By this measure... Should we expect our Dell appliance to integrate with the boost functionality care of its current OST support?

Thanks!
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: DR4100 as a target

Post by veremin »

My colleague was told by Veeam support that we should not use backup copy jobs on our dedupe appliance, because they use a transform process.
In fact, we recommend using dedupe appliance as the target for backup copy job. Even though the transformation activity might take some time, it happens at target location and doesn't affect production environment by any means. So, you shouldn't be that concerned about it.
We're currently running backup copies with the same appliance as source and destination of the copy, which I realize is going to impact performance.
It's not advised to point both backup and backup copy jobs to one location, since in case of disaster you will inevitably loose primary backups, as well as, secondary copies.

If applicable, use some local disks on backup server as a target for backup job (short retention), and copy data afterwards to the said dedupe appliance via backup copy job (longer retention).

Thanks.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: DR4100 as a target

Post by dellock6 »

Honestly this is the first time I hear this, and I hardly believe it's true. DD Boost is a specific EMC protocol used in DataDomain, I'm not aware of any other hardware vendor support it. In fact, there is a DDBoost plugin even for Symantec products; if what was said was true, it would not be needed.

Maybe, he meant to say both protocols has the same advantages and goals, just like also HP has Catalyst protocol for example. But to my knowledge they are not compatible each other...
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
mongie
Expert
Posts: 152
Liked: 24 times
Joined: May 16, 2011 4:00 am
Full Name: Alex Macaronis
Location: Brisbane, Australia
Contact:

Re: DR4100 as a target

Post by mongie »

v.Eremin wrote:It's not advised to point both backup and backup copy jobs to one location, since in case of disaster you will inevitably loose primary backups, as well as, secondary copies.

If applicable, use some local disks on backup server as a target for backup job (short retention), and copy data afterwards to the said dedupe appliance via backup copy job (longer retention).
We also use tape at the moment to store for disaster. Long term plan would be to backup to 10/15K disk, and backup copy to DR4100[s] (we have one in the office and one at DR site). Unfortunately, our full backup is ~ 26TB, so even a small amount of "fast disk" means 50-60TB minimum. We're looking to implement this at the moment, but also need Dell to sort our our dedupe storage first.


You guys have been very helpful. Thanks.
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: DR4100 as a target

Post by tsightler »

OST is definitely not equal to DDBoost. DDBoost is an API from Data Domain that can be used by any vendor interested in writing to a Data Domain to accelerate the communication with the Data Domain appliance by offloading much of the work to the host doing the writing and provides other advanced features such as synthetic full processing directly on the DD, etc.

On the other hand, OST is an API from Symantec that provides a "standard" way for vendors to integrate with Symantec products. Basically, it allows storage vendors to write a OST compliant plugin which exposes the capabilities of their storage appliance to the Symantec product. In other words, there is a DDBoost plugin for OST which is how Symantec products can leverage the DDBoost API without having specific support for DDBoost directly in the Symantec product.

Probably the funniest thing about OST from my perspective is that it stands for "OpenStorage API" which makes a lot of people think that somehow it's some type of industry standard, but the only products that use this API are Symantec products and to get access to it you have to be a member of the Symantec Technology Enabled Program (STEP). So I guess it's fair to say that it's an "API open to Symantec partners for integrating with Symantec products" but that's about it.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: DR4100 as a target

Post by dellock6 »

The beauty of standards is that everyone has his own :D
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
mongie
Expert
Posts: 152
Liked: 24 times
Joined: May 16, 2011 4:00 am
Full Name: Alex Macaronis
Location: Brisbane, Australia
Contact:

Re: DR4100 as a target

Post by mongie »

Just to be clear... I didn't think they were the same... One of the Dell storage sales guys who is handling our case threw that one at me when I mentioned DDboost in V8. Oh we'll... I almost believed him for a minute.

Our dedupe box finally finished its cleaner cycle today after 10 days :)

We now have a ratio of somewhere around 5:1

I had a look at the Veeam dedupe ratios on our full backups, and they range from 1.3x to 3x. Obviously this is handy, and I can see how this would reduce the ability for the appliance to dedupe this data, but when you consider that each of the jobs (there are about 15 backup jobs) has about 3 fulls, and between 5-40 incremental with 1x dedupe, but we also then have about 8 archive jobs with between 5-10 fulls and 10-30 incrementals, surely the appliance still has a lot of data to dedupe.

Anyway, a dell guy is coming in tomorrow to have a look at our config.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: DR4100 as a target

Post by dellock6 »

Oh, I didn't thought you agreed on that statement, don't worry ;)
Keep us updated on the results.
Just as a second try, you could try to increase the block size on Veeam dedupe engine, I hope you are already using local target (1 MB block) but you can also try out the 8MB block, called "local target 16+ TB". This could give bigger blocks to the Dell dedupe engine so result in faster dedupe. Obviously, the expected dedupe ratio will be worse.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
marcel.mertens
Influencer
Posts: 11
Liked: 2 times
Joined: Sep 27, 2012 7:46 am
Full Name: Marcel Mertens

Re: DR4100 as a target

Post by marcel.mertens » 1 person likes this post

We have currently one customer (county council in germany) running DR4100 as a B&R repository.
But the customer is backup up to a local B2D repository (Dell T620 with local 4TB disks) and then copy the restore points to the DR4100 (Backup Copy Feature).
This DR4100 is for long time archiv and also replicated to a second DR4100 on a offsite location.

Information i can provide from my memories (implementation end of 2013):

DR4100 18TB. 10GbE connected to Backup Server (T620).
Backup Copy speed for a full backup ~250-300MB/s
Space savings around 81% @ 75% disk usage (most Windows 2003/2008 VMs, Exchange, File, Citrix Server...)


There is also a Set Up Guide avaiable:
"Setting Up the Dell™ DR Series System on Veeam"
ftp://ftp.dell.com/Manuals/all-products ... _en-us.pdf
samh22a
Enthusiast
Posts: 35
Liked: 4 times
Joined: Nov 10, 2013 2:03 pm
Contact:

Re: DR4100 as a target

Post by samh22a »

We are actually using 2 Dr4100 at the moment as primary target

Dedupe ratio is 5.1X on one of them and 4.1x on the other one.
fijiboy
Novice
Posts: 7
Liked: 1 time
Joined: Jan 17, 2012 12:05 am
Full Name: fijiboy
Contact:

Re: DR4100 as a target

Post by fijiboy »

Hello,

We are using a DR4100 (3 X trays total = ~70TB) since January now.

We get a total saving of "90.78%"; that would equate to ~10:1 dedupe ratio. It took about 2 months of daily backups to reach that ratio.

I disabled Veeam dedupe and also do synthetic fulls. I perform Active Fulls at the end of each month.

I hope this helps.

The Dell dedupe box is a simple target appliance for backups- that 'just works' (just like Veeam)

Cheers,
Rishan
lhenshaw@advizex.com
Lurker
Posts: 2
Liked: never
Joined: Apr 27, 2012 7:26 pm
Full Name: L H
Contact:

Re: DR4100 as a target

Post by lhenshaw@advizex.com »

I know this is an older post but I have a customer who is also using DR4100.

In doing research before an install I came across the following from Veeam KB: http://www.veeam.com/kb1745
This has general guidelines around De-dupe devices as follows:

It is advised to configure the job as following when backing up to a dedupe appliance:
◾Use “Incremental”
◾Uncheck “Enable synthetic fulls”
◾Check “Perform active full backups periodically”
◾Uncheck “Enable inline data deduplication”
◾Change the Compression Level to “DeDupe Friendly”
◾Change Optimize for to “Local target (16TB+ backup files)”


Here is also another link on EMC DD with and w/o DDBoost
http://www.veeam.com/kb1956
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: DR4100 as a target

Post by veremin »

Actually, if you enable "decompress data block" in repository settings, you can set compression level to optimal just to reduce traffic sent between datamovers. Thanks.
btmaus
Expert
Posts: 138
Liked: 10 times
Joined: Jul 17, 2015 9:02 am
Full Name: Glenn L
Contact:

Re: DR4100 as a target

Post by btmaus »

marcel.mertens wrote:DR4100 18TB. 10GbE connected to Backup Server (T620).
Backup Copy speed for a full backup ~250-300MB/s
Space savings around 81% @ 75% disk usage (most Windows 2003/2008 VMs, Exchange, File, Citrix Server...)
I'm using a Dell DR6000 and getting similar figures to you so far.

I used this document for setting up my appliance: http://downloads.dell.com/Manuals/Commo ... _en-us.pdf
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: DR4100 as a target

Post by foggy »

Also, there's generally not much sense in disabling inline deduplication, since it has almost zero impact on storage dedupe, while provides some advantages in terms of incremental backup performance.
rreed
Veteran
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: DR4100 as a target

Post by rreed » 1 person likes this post

We've been using a couple of them for a couple months now w/o issue. Same real-world results as btmaus and Marcel; about 250-389-ish MBps @ 4-5x dedupe/85%-ish reduction. About the same dedupe/compression as our old EMC Data Domains, BTW... 10GB connected on the Dells, 1GB on the old EMC which run about 35-89-ish MBps speed. We're using ALB on the pairs but Dell did say switching to LACP would give a slight improvement in network throughput. Please don't ever believe the sales person or brochures on capabilities. Set your expectations real-world I'm afraid. If you did full backups of powered off VM's every single night, sure you might see dedupe and compression hit the stratosphere and I understand management loves to see that. But weekend fulls w/ weekday incrementals, like most folks might typically do, won't see those inflated advertised numbers. By their nature, incremental backups are changed data so there's nothing to dedupe - since it's not duplicate! And yes, do NOT do synthetics since the device has to rehydrate, uncompress, etc. work w/ that data live - which is NOT what they're meant to do. Just straight forward old fashioned weekend fulls/weekday incrementals. Also, this new fad of dedupe boxes (yes, I said that) are meant for blazing ingest, which they do quite well, but if you need data back out of them, go find something to do for a while and set your users' expectations accordingly. A LONG while. Follow the device's manufacture best practices for Veeam settings; I think we're decompressing blocks before storing on both DR's and EMC DD's; enabling inline data dedupe w/ no compression for the DR's, disabling inline data dedupe w/ no compression for the DD's, 16+ backup files (large blocks) for both. Basically the only difference being enable inline dedupe for the Dells and disable inline dedupe for EMCs. I think this was last-known best practices recommended but I need to see what the latest is since these devices are "new" to Veeam and the manufacturers are leaving it up to us to figure out what works best. When I last talked to Dell they said as long as you're decompressing blocks before sending on the Repository, we're on our own to figure out what works best for us on the individual Job settings.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
btmaus
Expert
Posts: 138
Liked: 10 times
Joined: Jul 17, 2015 9:02 am
Full Name: Glenn L
Contact:

Re: DR4100 as a target

Post by btmaus »

For those of you using Dell DR4100 / DR6000 appliances, have any of youu tried using these new features in Veeam v9 yet?

  • Per-VM backup file chains.
    Active full backups with backup copy jobs.
Would be nice to know whether you have seen any improvements with these features?

For the Veeam employee's, do any of you know if the above are recommended with the Dell DR appliances?
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: DR4100 as a target

Post by Gostev »

Yes, these are the recommended settings for Dell DR appliances.
Post Reply

Who is online

Users browsing this forum: ThomasIkoula and 168 guests