Dedup device for Veeam backups

[zak2011]

I have been talking lately with EMC to find the best way to store Veeam backups and the options we have are either upgrade our current DD610 dedup device to more capacity or getting a new DD640. Upgrading the DD610 wouldn't give us any long term benefits so the DD640 seems to be more promising as it is expandable to 42TB.
However as EMC has advised that in order to get the maximum benefit of the DD dedup device, we need to disable the dedup and compression from Veeam . However i don't see any point in spending a lot of money on a dedup device to store Veeam backups as I am already getting good dedup and compression from Veeam. How much more can i expect from a dedup device !
So why do i actually need a dedup device then..isn't it better to go for some cheap non dedup storage and use the dedup and compression of Veeam to store the backups?

[sieb]

Depends. I don't know how the EMC is calculating it's dedupe: per job or storage as a whole. On my Exagrid, I let Veeam dedupe each job as it sends it to the Exagrid, which keeps backups small. Then the Exagrid dedupes the entire Veeam datastore post-backup. I don't use compression however as was recommended by Veeam when using Exagrids. I think you really only need compression if your dumping to a non-dedupe device. So far I am getting a 3:1 dedupe ratio on 5TB of backups (all Windows servers). It also depends on what your backup window is, where the overhead is causing a bottleneck, and what your restore window is since you will have to wait for re-hydration.

[tsightler]

My general rule of thumb for recommending clients look into a dedupe appliance when using Veeam is based on how long they want to retain backups and how often they want to run full backups. Once the retention gets beyond 2-3 full backups, and/or 30-60 days, then dedupe appliance start making a lot of sense. Anything less than those values and it's very difficult to justify them from a cost perspective unless you have other considerations (limited rack space, other backups to store on dedupe like SQL dumps, etc).

With the advent of dedupe built into the OS/filesystems, it seems likely that even these cases may start to diminish. Solutions like SDFS, Windows 2012 dedupe, and ZFS storage systems, which all offer both inline and post-process deduplication. In initial testing Windows 2012 dedupe seems to work quite well at reducing storage of Veeam backups and I've also been playing with SDFS and it's built in, dedupe based replication to replicate backups to other targets (including cloud storage). These technologies are maturing rapidly. I can hardly believe I'm saying this but I'm particularly impressed with Windows 2012 dedupe, it appears that this will truly bring dedupe to the masses. If Microsoft continues to mature this technology to allow for transparently storing/replicating/distributing deduped chunks across multiple servers/cloud storage it could offer a huge improvement in unstructured file storage.

[zak2011]

That is a really good news.
Since my organization isnt really keen on tapes in the near future..i need to make a choice between buying a cheap storage or buying a dedup device.
wouldnt it be a better choice to use a cheap non dedupe storage for Veeam backups and upgrade the OS running Veeam server to Windows Server 2012..rather than spending so much on a dedup device?

[zak2011]

I am a bit surprised..by the answer that i got from EMC when i asked them the justification of storing the backups on a Data Domain as they had initially recommended disabling compression and dedup of Veeam. I don't agree completely to what they say especially the fact that Veeam does not provide deduplication ...Read on
"Veeam does not provide deduplication but changes only for backup. We are talking about thing like 4-5 times dedup, may be 10 if you are lucky. With Data Domain, we are talking 25 may be 50 time reduction.
So if you buy normal disk for let say 10 TB, you will store 40-50TB.
With Data domain. You will need 8TB and will store up to 400TB of backup data.
Performance is a real pain when writing to disk. When writing to Data Domain unit, you write to memory , the CPU do the dedup then you eventually write to the disk if the data cannot be find from before. So you get 10 time better performance with a dedup unit than a disk system…"

[Gostev]

Veeam does not provide deduplication but changes only for backup.

Well, guess they know our product better than we do

We are talking about thing like 4-5 times dedup [with Veeam], may be 10 if you are lucky. With Data Domain, we are talking 25 may be 50 time reduction.

Marketing vs. reality > Deduplication contest – Veeam or DataDomain

And I'd really like to see at least one customer with 50x dedupe ratio on standard workloads. That is, real 50x reduction, not marketing numbers which assume that every backup taken is full backup, and there is no such thing as incremental backups.

Where deduplicating storage really shines is global dedupe (between multiple Veeam backups). However, when talking a single job, it is really hard to challenge our built-in dedupe and compression. Having access to all kinds of deduplicating storage, we've done plenty of testing around data reduction technologies on real-world data past years, and know exactly what to expect... and I am just not buying 50x any longer not even close!

[sieb]

If I were you, I'd make sure to talk to an engineer and not a sales guy. They should have a best-practices for each backup application, not just ones that make their system look the best. I can see disabling compression since that will affect dedupe calculations on the storage side, but no dedupe within Veeam sounds like sales talk. Why wouldn't you? The only thing I can think of is they like to quote their RAW numbers for dedupe instead of splitting it with a third party application which would give them lower numbers. In the end, your ratio should be as good or better with both.

Like tsightler said, the longer you keep it, the better the ratio. I only keep Veeam backups for a couple weeks since by then, too much has changed for the VM to be worth keeping, but my SQL and Exchange backups get crazy dedupe ratios since I keep those longer (also on the Exagrid). In your case, how much data do you need to hold onto and and for how long is what you need to be asking?

[dellock6]

I also tested ExaGrid in the past, and 50x is a pure sales number. For sure you can reach those numbers by completely disabling all Veeam optimizations, but at this point would be as copy/pasting virtual disks inside DataDomain.
Another fun stuff I read in their preso is this:

Performance is a real pain when writing to disk. When writing to Data Domain unit, you write to memory , the CPU do the dedup then you eventually write to the disk if the data cannot be find from before. So you get 10 time better performance with a dedup unit than a disk system…"

Come on, inline dedup IS the limiter when saving to the appliance. don't get me wrong, DD is a really cool product, but if you take the same storage without deduplication, it runs faster. Otherwise why ExaGrid or Windows 2012 preferred the post-processing dedup instead of inline?

I agree with others, talk with an engineer. And read this article from Justin Paul about "real" DD performances: http://jpaul.me/?p=3149

Luca.

[andersonts]

I would also take a look at this video recently posted related to Windows 2012 Dedupe Storage...very exciting for customers who already own licensing and may want to use some commodity hardware for back end storage.

[tsightler]

Just a warning, while the settings in this video are fine for demo purposes, I would never suggest using reverse incremental backups with dedupe in a production setup as the impact on performance is likely to be significant.

[Gostev]

Since the author already said he is redoing testing using the recommended settings, and re-creating the video, I have removed link to the video from post above for now, just to avoid spreading bad practices. I will ask him to post the new video on the forums when he is done, hopefully this will get him some "likes" for his hard work too

[zak2011]

Exactly as Tom said..

I just heard one of the whiteboard Friday sessions now .and Doug Hazelman was presenting the benefits and the dedup of Windows Server 2012.
Seems like dedup appliances wont live for long. He explained how Windows Server 2012 gives excellent global dedup for storing Veeam backups..
I think that simply answers my question..i actually dont need a dedup appliance anymore to store Veeam backups. I could install Veeam server on a Windows Server 2012, add the backup repository volume of a cheap storage and enable deduplication on that volume using Windows Server 2012, enable forward incrementals. So i get to use cheap non dedupe storage at production and DR site to store Veeam backups and have a copy of backups offsite too.
Why buy a dedup appliance in the first place then?

[tsightler]

How are you getting the backups offsite?

[JoshRountree]

How are you getting the backups offsite?

DFS replication for the backups, Veeam Replication for VMs.

[tsightler]

Have you tested this? I've been told that DFS replication has some serious limitations with very large files, and I believe it will consume a significant amount more bandwidth than a hardware dedupe appliance such as DataDomain since the DD will only replicate the changes after dedupe, but I don't think that DFS takes any dedupe into account.

[danielec]

So why do i actually need a dedup device then..isn't it better to go for some cheap non dedup storage and use the dedup and compression of Veeam to store the backups?

A storage can dedup beter because:

• A Storage can use a small blocksize, so better deduplication ratio.
  If I remember correctly Veeam use: 1 MByte block size for Local Target, 512KB for LAN Target and 256KB for WAN target. A storage can use a blocksize of 4KByte or even smaller.
• A Storage can dedup all the data, so if you have multiple veeam jobs, the storage "share" the duplication...

Other advance are: Less Cpu utilization on Veeam, because compression must be turned off.
Normaly, but it depends on the storage that you use, the deduplication options on veeam can be left on (LAN Target 512KB), this will also speed up writing backups on storage, because there are less bytes to transfer.

Storage Deduplication can also be useful for Storage Replication, because it reduces the volume of data that must be syncronized between storages.
See for examples: HP D2D (http://h20195.www2.hp.com/v2/GetPDF.asp ... 685ENW.pdf)

IMHO I will not buy a new storage, with deduplication, only for veeam, because veeam can do an excellent job alone.
I will buy a storage with deduplication if I can do replication (at storage level) or if I will not use the storage only for Veeam.

Bye

[zak2011]

Hmm..well in my case i was thinking of running an additional Backup Job to the offsite backup repository..as both sites are connected by fiber.
Would it possible to use rsync and get both backups synchronized...instead of going in for two DDs at both locations or using DFS replication?

[foggy]

Would it possible to use rsync and get both backups synchronized...instead of going in for two DDs at both locations or using DFS replication?

Yes, there are a number of existing topics on how to replicate backup files offsite using rsync, please search to review them.

[zak2011]

@danielec- thanks for your comment. However from what i understand and seen so far from the experts and other resources..Windows Server 2012 global dedupe and SMB 3.0 is here to stay and provide stiff competiton to hard ware dedupe appliances and to SANs itself.

[haslund]

Has anyone tested Veeam backups to Windows Server 2012 storage with dedup enabled?

[Gostev]

Yes, and it is awesome.
See 3rd post of this topic.

[tsightler]

While I've performed basic testing of Windows 2012 dedupe, and I'm sure others have as well, want remains to be seen is how well this technology scales, and how reliable it is over the long haul. In other words, while it seems to work very well with backups in the 100's of GB to perhaps a couple of TBs range, what happens when a customers creates a 50TB repository with dedupe enabled? How much memory will be needed, what dedupe will we see on that, how long will the backend processing and cleanup stages take and what impact will they have on performance when running? I think that it will take time for stories to unfold regarding how well Windows 2012 dedupe works at scale, from a performance perspective and otherwise, but initial testing is quite promising.

[Gostev]

Considering dedupe is post-process vs. inline, I think there will be no huge memory requirements or scalability issues. May be only initially and due to lack of optimizations, but that will surely be addressed. At least in theory, this approach is most scalable (at the cost of landing zone requirement).

[dellock6]

Some good information on how well post-dedupe works can be grabbed from ExaGrid, they use post-processing and their servers are not super-fast or loaded with tons of ram. But nonetheless, with low performance server post-processing can be really time consuming and going on too much time, so some sort of design around this should be done. ExaGrid has preconfigured appliances, sized for the amount of storage they offer, with a software-only solution as in Windows 2012, there are risks people under-sizing their storage and then complaining about deduplication taking too long.
I hope to do some deep tests once I'll be be at office next week, our backup is about 9 TB full and we keep 7 days, so I think it could be a good test for Windows 2012. And I also want to test OpenDedupe side by side.

Luca.

[tsightler]

Actually, the idea that post-process dedupe is the most scalable is open to debate. There's more disadvantage to post-process than just the landing zone, there's also a LOT more I/O since, with post process dedupe, I/O must be written uncompressed to the storage, then eventually all of that uncompressed data must be read again, and processed, and perhaps moved to a dedupe pool. With inline dedupe, because the data is deduped prior to writing to disks it significantly reduces the amount of I/O that actually hits the physical disks, this is why inline dedupe solutions are exceptionally good at processing multiple streams. For example, with ZFS inline dedupe running on a powerful multi-core system, some customers actually see better disk throughput than when using raw disks because of the lower disk I/O.

Garbage collection is another issue which even impacts hardware appliances like Exagrid and Data Domain. The garbage collection process is both CPU and I/O intensive and if it runs during backup operations it will significantly impact performance.

All I'm saying is there is a lot to test, I don't really anticipate major issues, but with any new technology that is storing all of your backup data, caution is always warranted.

[TheJourney]

I think DFS rep for 2012 is dedup aware, but I am not sure about the file size limits. Anyone else?

[RGlintmeijer]

According to this blog post:

http://blogs.technet.com/b/filecab/arch ... -2012.aspx

- Windows 2012 DFSR will replicate deduped files as normal files. (But it would be a really cool feature if it does only replicate the unique dedupe chunks, it would be THE solution to replicate Veeam data offsite).
- 64GB files are still the supported maximum

[yps]

How large files does Veeam safe as backup files? Does Veeam splits vmdk-files so they are smaller than 64Gb on backup device?

[Vitaliy S.]

Hi Magnus,

How large files does Veeam safe as backup files?

There are no such limits, however generally we see that our customers do not have any issues with the backup files up to 5TB.

Does Veeam splits vmdk-files so they are smaller than 64Gb on backup device?

No, Veeam does not split any VM files in the backup file.

Thanks!

[mongie]

Exactly as Tom said..

I just heard one of the whiteboard Friday sessions now .and Doug Hazelman was presenting the benefits and the dedup of Windows Server 2012.
Seems like dedup appliances wont live for long. He explained how Windows Server 2012 gives excellent global dedup for storing Veeam backups..
I think that simply answers my question..i actually dont need a dedup appliance anymore to store Veeam backups. I could install Veeam server on a Windows Server 2012, add the backup repository volume of a cheap storage and enable deduplication on that volume using Windows Server 2012, enable forward incrementals. So i get to use cheap non dedupe storage at production and DR site to store Veeam backups and have a copy of backups offsite too.
Why buy a dedup appliance in the first place then?

I dont know about you guys, but I'm using Win2012 Dedup on my 2 backup storage servers... It works reasonably well, but it is EXTREMELY slow. This is because its not designed for this sort of use case.

I'm talking... a 30TB repository and a 60TB respository (dell dumb disks) and full backup sizes up to 8TB.

I hoped that it would be good enough to use as a permanent solution, but I dont think this is the case... it does achieve good savings, but the job times are ridiculous (1-2 weeks for a full Dedupe job) and I don't know if I trust it for this amount of data.