DD 670 vs HP StoreOnce advice

[jgremillion]

I am a little press for time in making a decsion about which dedupe appliance to purchase for our infrastructure. I have narrowed our selection down to two companies: HP and DD. My plan was to get demo units in our environment and test but, we have some stimulis money we need to spend in a few weeks and so we don't have time to get teh demo units. I wanted to clarify a few points that HP and DD said about their units.

If there is anyone on reading who can help clarify this, I would appreciate any anad all replies.

Anton, I know we kicked this can this can down the road already, but I really need some help filtering out all of this marketing crap.

1. DD says that sending data to them that is already deduped from Veeam is usless as they can't dedupe any further. .... From all that I read here, this sounds like like it is a false and ridiculus statement. ...
HP says that teh deduped and compressed data that Veeam sends to it is pretty clean and they can dedupe it farther. -- Can anyone offer some advice here. CAN HP and/or DD dedupe Veeam images farther? I really think that the HP statement is teh correct one.

2. DD says that HP does not dedupe the first 24MB of a file or any files smaller then 24 MBs. -- Can anyone confirm or deny this based on their experience with HP?

I am really leaning towards teh HP device but want to make sure all claims are valid before I jump.

Thanks.

[Gostev]

Hi Joe,

We did not do any testing with DD recently, so I cannot say much about it.

HP recommends to keep our dedupe on, but disable compression and I fully agree with this recommendation. HP storage is also certified for vPower functionality, however one DD user reported here on this forum before that vPower functionality is unbearably slow when backup files reside on DD device. Again, we have not tested but that it what was reported by actual user.

Concerning 24MB, I do not believe this is the case, but best for you would be to just ask technical HP resource directly (I am sure they will not lie). I can also ask myself using my channels if you'd like, just let me know.

I've attached recent whitepaper on combined HP storage and Veeam solution for VMware, this included HP StoreOnce, hope this helps.

Update: Sorry, this looks to be more of a high-level paper, not the one I had in mind. Technical brief on how to best deploy and configure both solutions together (including compression and dedupe settings) is still coming, and should be made available here > http://go.veeam.com/hp_storeonce.html

Thanks!

[jgremillion]

Thanks for info Anton. I appreciate it, as always.

I was hoping someone could provide some real world experience, with regard to HP. Our real goal here is to keep the Veeam backup speed up. I to am wary of teh IR speeds on a dedupe appliance, but my boss is willing to risk so we can us an in-line dedupe storage appliance.

-Joe

[Gostev]

Joe,

What I am saying is that there are *NO* issues with IR speed when backups are located on HP StoreOnce appliance (unlike it was reported to be the case with DD).

We did actual real world testing of v5 in HP lab just a few months ago, and found performance to be great for both backups and restore (including vPowered restores). The IR performance loss we were observing was noticeable, but pretty insignificant (just a few seconds longer boot time for IR VM). As for backup performance, there were no noticeable performance loss at all comparing to backing up to a local hard drive, although backup server was a bit underpowered so we could not really go super fast anyway (about 50MB/s for full backup).

Only possible concern is that HP has quite a few different StoreOnce models, and they may have different hardware (more/less powerful). I simply do not know, so just a heads up. We tested on 4106FC G2 presented as a CIFS share on 1Gb LAN (which is definitely not ideal, given that it has FC interface).

Thanks.

[jgremillion]

Anto,

Thanks again.

As always, you are a gentleman and a scholar.

BTW -- How's the Google position working out?

[Gostev]

I am on probation period, so for now responsible for handling search queries on this forum only.
Don't know how long the probation period will last, Google is so hard to please

[tsightler]

We did actual real world testing of v5 in HP lab just a few months ago.

I'm sorry, but I couldn't pass this up, it was just too funny. I suspect the OP was looking for some feedback from an actual person using the system in production, not the Veeam bizzarro world where "Lab" = "Real World". I'll repeat one of my all time favorite quotes, "In theory, theory and practice are the same. In practice, they are not".

Of course sometimes the lab results are all you've got, and you just hope that your lab testing is close enough to the "real world" that the end result is good. Perhaps that the case in this situation. Unfortunately life is full of examples where "real world lab testing" had it's lunch eaten by the actual world.

Good Luck!!

[Gostev]

Yes, it was quite decent environment (much better than our own production). Although testing backup storage capabilities around vPower does not really require fancy production-mirror-lab. Just backup single VM, and do instant recovery. And VM either boots up under 1 min and you're good, or it will take 5 min to draw that text mode progress bar that is shown in the beginning of Windows boot process, you know what I mean?

[tsightler]

Sure, because we all know in the real world we'll all only ever need to recover one VM at a time, and that VM will be silent and barely used at all.

[Gostev]

No, but this does not really make the test I suggested flawed. Such test is something anyone can run in under 5 mins, yet even single IR VM boot up speed shows a lot on solution capabilities right away, because OS boot is a about heavy random I/O, and this is type of workload that most dedupe appliances seems to struggle with.

Another important thing to note, is that once VM is booted, things go pretty quiet with most applications. For example, for most VMs in our production, you can hardly see any I/O activity in Monitor at all. But you can clearly guess when some VM starts rebooting. Of course, this is not the case with high load databases, but I would not want to IR those even with backups on raw storage anyway. These things are better off replicated, for sure. Which is why we sell Backup and Replication, not Backup and Instant VM Recovery

[tsightler]

I didn't say your results were not valid. I'm only point out that results in the "lab" are generally not very reflective of "real world" results because labs have a tendency to focus on a specific goal, such as compatibility between two products, and generally cover a very limited time period. A "real world" deployment has to work for years, hopefully at the same performance level, and many cases serves far more than a single purpose. For example, what if the StoreOnce appliance was used as a VTL for a traditional backup and a backend for Veeam? In the "real world" this might be likely since it's possible the customer has some physical servers. Also, did you test IR while backups were running, while the backed dedupe was running, etc? You may have been very thorough with this, but since the results are not published it's hard for us as users to know.

Even without though, it's customers who usually find out where a products weak points are. What happens in six months when fragmentation levels start to get high thus increasing the IOP load on the repetitively slow and limited drives. The difference in deployment between a product like that from day one, and 12 months down the road can be quite significant. Labs generally don't show those kinds of things, real deployments do.

I suspect the OP was hoping to find at least one person who had actually deployed a StoreOnce with Veeam for a few months and would tell their experience because there is nothing in the industry more valued than an actual customer sharing their true experience with a product. Unfortunately it looks like all they'll have is the lab results for the HP, which may be OK since real world customers have reported nothing but poor performance from most of the DD products. Someone's got to be the first.

[Gostev]

Agree. I promise to never use "real-world" on these forums again when talking about lab results, even if the lab is more stuffed than our production.

Bottomline is, this HP offering is fairly new, and these G2 (2nd generation) devices we tested with are even newer, so I do not hope we hear from existing customers. On the other hand, I am quite happy with the lab provided by HP and the results we saw, although I agree this is not a substitute for longer term real-world testing. Based on all the information I have, at this time I am more confident in HP solution as it comes to Veeam B&R integration and vPower performance. To be fair to DD, I did hear EMC claiming very recently 7x performance increase with their latest generation DD devices, but I am not sure what performance specifically they are referring to, and I have no testing results to confirm that current performance levels are acceptable (not even from lab).

[Gostev]

Updated my first reply above, because apparently I have attached wrong document, not the one I was thinking about (which was a technical paper I have reviewed a couple of weeks ago).

[tsightler]

Agree. I promise to never use "real-world" on these forums again when talking about lab results, even if the lab is more stuffed than our production.

OK, but if you do, I'll call you on it. I didn't mean to start such a big discussion anyway, it was just funny because I'm currently working with multiple vendors trying to design a new storage architecture and their always wanting me to see their "real-world" labs which are only "real-world" in the sense that Jupiter is a world, it's just not quite the same as the "world" I live on.

But if all you have is lab results, well, I'll trust your results over the vendors, that's for sure.

[Gostev]

I am looking at the HP dedupe appliance now and can say that I am very disappointed in it. ... The price was good and the recovery was fast, howerver, the injest rate and overall depude % is very low. The Hp device we are considering HP D2D4324 only takes in 15 threads and after dumping the same data to it in different fashion I am only getting 3.1:1 dedupe. However, if you wanted to use it as a VTL, it is very fast.

I am also looking at a Data Domain DD670. 90 streams and I am getting a dedupe rate of 26.1:1 with the same data. The IR may be a little slower but I am thinking we can live with that.

Joe, I missed your post with results because you posted it in different topic. Replying here not to derail that other one (which is about ExaGrid). From the above, it looks like you were testing 1st generation HP D2D device. Our testing I am referring above was on 2nd generation device (G2 in the product name). This may explain why you are seeing worse results than expected.

Also, can you share any more details on DD670 performance you are seeing. I am very interested since we have not had a chance to test this ourselves.

[jgremillion]

[merged into existing discussion]

As of last Friday, I now have Veeam backing up my VMware environment to a Data Domain DD670 and replicating to a second DD670n at our DR site. The implementation was very easy and, as of the weekend, I am getting pretty good performance with the units.

The compression ratio on the DD670, after a four runs is at 7.0x and growing. Also, my backup speeds have increased since I stopped dumping the backup images to a local target.

Veeam seems to be working very well with the DD670. (Not that I expected any different)

On a side note, the EMC/DD domain arch. has a VCP so he was using his free Veeam product to setup a lab and test with the DD670. He did not know how to use the product, so naturally I gave him a quick lesson. He seemed pretty impressed with the product.

[dodgint]

Interesting results you are getting with the DD. As of last week a Veeam rep advised against a DD due to the In-Line Dedup causing performance issues and advised to look at Post-Dedup solutions. Are you De-Duping on both sides?

[jgremillion]

Yes. I am deduping on both sides. In Veeam, I am using the local target option and am turning off compression. The DD670 has an ingest rate of 2 TB/Hr and can handle something like 20 multiple streams at once. Since everything is deduped inline, there is less of a need for a ton of disk spindles to handle the IO from Veeam.

Maybe the Veeam rep was advising against inline dedupe for the vPower IR? I know that vPower IR is a little bit slower on the DD but we found we can live with it, especially since we don't use it a whole awful lot.

[MisterL]

i need the same quick lesson jgremillion, i've just been asked to get two DD670 devices up and going to backup our vm environment and i don't know where to start. its not as simple as i thought - adding a system to the console and configuring what you want it to backup, help. and thanks. i attemped to add a cifs share and wanted to point veeam jobs to that share but i ran into an authentication issue with the same account i'm logged on to the DD device with. help.