One month with Win 2012 dedupe and Veeam

[jpeake]

I've been on Veeam and Win 2012 for just over a month now. Backing up to a Dell MD3600i in RAID 10 (16 TB usable). I have enabled Windows dedupe for files older than 7 days. Veeam compression is off, Veeam dedupe set to LAN.

The results are excellent. 12.88 TB worth of files in the optimize policy (after Veeam dedupe) has been shrunk down to 2.5 TB in Windows. In total, my repo is holding 16.1TB of data using 3.23 TB of disk space.

Everything seems to be running great. i can run VM's directly from the deduped images with no problems. So far so good. I should be able to store 4-6 months of images on disk.

[Gostev]

Hi, this is excellent! Thanks so much for some real-world feedback and numbers!

[chrisdearden]

With a forward or reverse incremental ?

[tsightler]

Yes, thanks for sharing this. I've been testing Win 2012 dedupe myself in a lab environment but I'm wondering if you might be able to share any more real world data. For example, how big are you full Veeam backups (i.e. VBK files) when uncompressed? What about your daily VIB files?

[jpeake]

Yes, thanks for sharing this. I've been testing Win 2012 dedupe myself in a lab environment but I'm wondering if you might be able to share any more real world data. For example, how big are you full Veeam backups (i.e. VBK files) when uncompressed? What about your daily VIB files?

I have my VM's split amongst 5 different jobs.

The individual VBK size per job is as follows (Veeam's dedupe ratio noted):
App Servers VBK: 972 GB (51%)
Mail Servers VBK: 528 GB (25%)
Domain Controllers VBK: 55 GB (37%)
Linux Servers VBK: 52 GB (23%)
File Servers VBK: 1.5 TB (63%)

So each full after Veeam dedupe is about 3 TB in total stored to disk. Then Windows dedupe kicks in after a week and takes that 3 TB down to 600 GB

Here is what an average week's worth of incrementals amount to:
App Servers VIBs: 115 GB
Mail Servers VIBs: 260 GB
Domain Controllers VIBs: 6 GB
Linux Servers VIBs: 16 GB
File Servers VIBs: 40 GB

With a forward or reverse incremental ?

Forward incremental

[tsightler]

Thanks a lot for this. What you're seeing in the real world lines up pretty closely to my projected results. Part of the reason that I was curious was because I'm trying to outline when this makes sense to use Windows dedupe as opposed to just Veeam compression with reverse incremental. It's hard to know for sure, but I'd bet you'd actually use less space with Veeam reverse incremental at the 30 day mark, and probably very similar at the 60 day mark.

This guess is based on the fact that, if you enabled Veeam compression, which you can't use now because of dedupe, your 3TB VBK backups would likely be only 1.5TB (I almost always see at least 50% compression, usually more, but I use this as a conservative estimate). If I understand your numbers above, you're seeing about ~72GB of uncompressed incremental data per day, so that would be ~36GB with compression enabled. Assuming these estimates are anywhere near accurate, using Veeam reverse incremental to keep a month of restore points would be 36GB * 30 = ~1.1TB month. That means the estimated space using reverse incremental for a month, would be about 2.6TB (1.5TB for full + 1.1TB for rollbacks), without using dedupe at all. As the number of restore points increase to 60+, forward incremental with Windows dedupe will eventually catch up, and then slowly pull ahead in space efficiency (this would line up with my testing of other dedupe appliances).

Note that I'm not saying this isn't cool, improving the storage efficiency for forward incremental using Windows 2012 dedupe is awesome, especially for the use cases where it makes sense (retention longer than 60 days, staging to tape or offsite, faster incremental performance, etc.), but just pointing out the fact that reverse incremental (a feature of Veeam since V1) offers similar storage efficiency for retention periods of 30-60 days and environments of this size. I'd love if you would continue to share your results as your retention builds as I'm continuing to collect both lab and real world results.

[brentmccrea]

I've been dabbling in server 2012 dedupe a bit now. Have yet to put any large amounts of data on it (we'd need to upgrade our production backup environment to 2012).

I'm curious what memory (ram) requirements there are for server 2012 dedupe. How much memory are any of you running and how much data are you deduping...?

[jpeake]

It seems to handle memory usage fairly well. The background dedupe process runs in low priority with a hard cap of 25% of the system memory. The scheduled jobs have a default cap of 50% of memory. I have just a single server running Veeam and Win 2012/dedupe. It has 44GB of RAM and I have no complaints. I don't think there are any additional memory requirements other than Win 2012 specs and Veeam requirements. I wouldn't want less than 16 GB

[brentmccrea]

Thanks. Our production box has 12GB of ram and has about 28TB of veeam data stored on it. Definitley will need a ram upgrade then.

[ryang]

Hi, I just wanted to share my the results of my new dedupe implementation for those of you considering it. It was extremely easy to setup. Even migrating Veeam to the new install of server 2012 was easy. But in summary after Veeam's built in dedupe, Veeams compression and windows dedupe I have a total 4.64tb of data backed up and it is only consuming 583.38gb of disk space!

More details can be found on my blog post here: Server 2012 Dedupe Results

[jnalldr]

So, I'm assuming 2012 Dedup is only feasible if forward incrementals are used? I have 7 TB of full backups which are modified nightly with reverse incrementals. 2012 Dedup only processes at a maximum of 100GB per hour so I'd never catch up and would be in a never ending dedup cycle.

[jpeake]

That's my take on it. I don't think it would work well at all with reverse inc. But I have not tried it. There would be far less to dedupe with only one full on disk, so you probably wouldn't gain much anyways.

[Delo123]

Seems like 2012 Dedupe not really fits Veeam. Since Deduping Files larger then 1Tb is not really possible nobody with large VM's can use this feature.
We have 2 "big" FileServer, each 2.5TB in Size. Windows will never finish the Dedupe Job. (even on high priority, 50% Mem etc...)
The only "Solution" would be if Veeam could split up Backupfiles in smaller Chunks, but i have read this has been proposed many times before but has not been realized.
Sugestions to split up Veeam Backups in less Vm's, though not a solution anyway only works if one has many small Vm's ofcourse, but is no option for big ones...
Too bad

[jpeake]

I would argue that it works great. For my environment anyways. I have 10 weeks of backups on disk, 32.3 TB of Veeam files deduped down to 3.37 TB. I do split my servers up into separate jobs (I have 5 jobs, largest full is 1.5 TB).

I am doing forward incremental for all jobs.

My take on it is that I get super fast backups by using forward incremental, and excellent storage savings using Windows dedupe.

[tsightler]

Having two big fileserver that are 2.5TB shouldn't really be a problem. If you allow the dedupe process to run 24x7 it will dedupe both of them in only 48 hours. Remember you can always choose to run full backups monthly, which means that you can split the "ingest" across multiple within the month. I actually think Windows 2012 dedupe works exceptionally well as long as you keep the limits in mind, even for fairly large customers, although I suppose some of this is based on the definition of "large". I work with customers that backup 100+TB of fulls each week, so yes, a single volume repository running dedupe will not work for them, but at the sizes you are referring to, I actually think that's right in the "sweet spot" for Windows dedupe.

As long as you are not ingesting more data per week than can be processed by dedupe process in a week then it works quite well. The Microsoft recommendation is the plan for about 100GB/hr, so that 2.4TB a day or 16.8TB/wk if you allow the dedupe process to run 24x7.