Windows Server 2016 w/ Dedupe or Dedupe Appliance?

[cbrasga]

We are looking to create a long term backup repository. I'm interested in the community's thoughts and recommendations on whether to use a Windows Server 2016 packed with disks and the deduplication feature enabled, or a dedicated deduplication appliance?

[PTide]

Hi,

That really depends on your RPO and RTO, as well as on the amount and size of VMs that you are going to backup. To give you an example of one Windows 2016 limitation:

<...>This has all been redesigned for Windows Server 2016 <...>, with the results being that you can go ahead and dedup files up to 1TB without worrying about them not being good candidates.

For some infrastructures 1 TB is not a limit of size for a backup.

Thanks

[sanjaykrk]

We are looking to create a long term backup repository. I'm interested in the community's thoughts and recommendations on whether to use a Windows Server 2016 packed with disks and the deduplication feature enabled, or a dedicated deduplication appliance?

As pointed out in above comment it depends on your requirement. Windows dedupe might serve you well.
On the other hand, dedupe appliances may offer some benefit (may or may not be relevant to you). For e.g. as I understand Windows dedupe will be effective at volume level with current recommended size around 64TB.
In case you have data > 64TB, you will be creating multiple volumes but they will be deduped independently. Dedupe appliances may be able to provide you global dedupe i.e. dedupe across multiple volumes. so that even across volumes only unique data is stored.

[planetcoop]

here are my 2016 dedupe stats for the year as i have updated from tp4 to 5 and now final:

PS C:\WINDOWS\system32> Get-DedupVolume | Format-List
Volume : D:
VolumeId : \\?\Volume{2fc167ff-9d98-45cb-b5fe-13b772ce848c}\
Enabled : True
UsageType : Backup
DataAccessEnabled : True
Capacity : 3.5 TB
FreeSpace : 2.68 TB
UsedSpace : 836.79 GB
UnoptimizedSize : 9.02 TB
SavedSpace : 8.2 TB
SavingsRate : 90 %
MinimumFileAgeDays : 0
MinimumFileSize : 32768
NoCompress : False
ExcludeFolder :
ExcludeFileType :
ExcludeFileTypeDefault : {bin, vsv, slp, xml...}
NoCompressionFileType : {asf, mov, wma, wmv...}
ChunkRedundancyThreshold : 100
Verify : False
OptimizeInUseFiles : True
OptimizePartialFiles : False
InputOutputScale : 0

[Mike Resseler]

Hi Chris,

That is a ridiculous large amount of savings! Nice. What file types are you storing on that volume? Is it a volume containing Veeam backups only or also other files? (I see you use the type Backup but you might use the volume for something else also...)

Thanks

Mike

[Mike Resseler]

And to give some feedback on the original question:

I agree with Pavel that sometimes Windows Dedupe is not always the perfect solution for every company. There is indeed that limitation of 1 TB (although that is not a hard limit and I have seen files over 2 TB to be deduped also...). The other side is that Windows Dedupe is post-process which means it needs to land on the target FULL size first before it can get deduped. That means you need to have that actual storage available to start with. Also, when rehydrating a backup (at the time of restore), you need the storage available also. Last but not least, it requires resources to do this work (not only the dedupe but also the scrubbing and clean-up of old data on a weekly basis) so you might need a beefier server for that purpose.

A dedup appliance is specifically built and has the necessary resources but comes with a price. It might have less of the above limitations, but it does come with a higher price.

So imho, the best you can do is actually look at your environment. See if per-VM backups will get you over that 1TB files or not, make a cost-calculation of a beefier server + storage and compare it with the price of a dedup aplliance. Then see if the additional cost of that appliance is acceptable to not have those limitations (if they actually apply to you, because for many environments Windows Dedupe is more than enough)

My 2 cents
Mike

[nmdange]

Using ReFS on Windows Server 2016 will be a better option since Veeam 9.5 will use the new ReFS APIs so that when a synthetic full is run, both the original file and the new file point to the same physical blocks. Essentially, duplicates won't be created in the first place so dedupe won't be needed.

[Mike Resseler]

That is true when you use incremental backups and use a synthetic full. We expect to see the same sort of savings, but do remember this is not deduplication so there still will be a difference and depending on your specific setup and the data you are protecting your mileage will vary!

[planetcoop]

my backup data for the awesome space savings. i have a combination of jobs but the underlying result is:
-Exchange 2016 - every 4 hours with a full at midnight and a retention of 1 week (42 copies)
-Daily All - servers (including Cisco VOIP) daily incremental with a full on the weekend and a 365 day retention and 365 copies
-Weekly All - servers (including Cisco VOIP) weekly incremental with a full on the first Wednesday of the month and a 52 week retention and a total of 52 copies

Exchange seems to dedupe well and you can imagine the total number of copies are almost 500 copies. The reset of the servers also do well with a low change rate. The stand out is that once of the windows 2016 servers doe shave a 26TB storage drive that is excluded. I use a separate process to store and forward its data (low change rate archive) to another server mostly kept offline. I have looked at ReFs and 2016, but i cannot speak on behalf of Veeam 9.5 as i am not in the beta testing, however it has not proven to be as beneficial for me as dedupe has been. I can also share my dedupe settings. Basically, the type is "backup" which is new in 2016. I also delete the canned scheduling enabled with default dedupe. I have devised a better schedule for my workload and change rate. Daily dedupe with 95% memory and no throttling at 5AM. By 10PM I scavenge and then at 11PM I collect garbage.

I cant share as many details but another production environment I have two vaults of 60TB each and the same process is scheduled on a weekly basis just based on change rate and size. Additionally this larger environment revolves around a 30 day retention with only a few server archiving exceptions with 999 copies max.

[Delo123]

A lot of savings can be made even with incrementals. Our largest full Backup is 8TB. When setting things up with scale out repositories and clever placement of files windows dedupe still rules.
Just make sure you test, test test! and follow the 3,2,1 Rule. In our Case O,Q,S and U have Full Backups only where P,R,T and V are for incrementals only. This is on physically 270TB where 169 are still free. 440TB are "saved" by dedupe until now.

PS C:\Windows\system32> get-dedupstatus

FreeSpace SavedSpace OptimizedFiles InPolicyFiles Volume
--------- ---------- -------------- ------------- ------
48.23 TB 81.37 TB 372 372 O:
43.42 TB 99.09 TB 8589 8589 R:
44.35 TB 26.48 TB 10 11 S:
50.49 TB 67.57 TB 407 407 Q:
49.77 TB 47.49 TB 4471 4471 P:
36.93 TB 65.38 TB 197 198 T:
55.72 TB 40.05 TB 3761 3761 V:
57.92 TB 13.42 TB 204 204 U:

[cousinsp]

The other thing to factor in is offsite replication. We are currently using Windows 2012R2 with Dedupe, but due to only having 30Mb bandwidth to our other sites we are unable to replicate much offsite, as we only have Enterprise Edition which doesn't do anything clever with offsite replication - data is just copied.
We looked at Enterprise Plus Edition, but it's extremely expensive for us to upgrade.
So we are currently looking at the HP Store Once devices, as we're successfully using these for Backup Exec (but haven't got one at our main site!). I believe they will handle offsite replication better and only send different data blocks.

Pete Cousins

[Delo123]

Yes, you would certainly need some kind of wan accelerator here, windows file copy will be full blown.
Not sure if Veeam only offers this in enterprise, i'll take your word for it. Yes, HP Storeonce will probably do this also so it's a matter of comparing costs for them against windows dedupe + oem jbod + veeam licenses.
If you are a HP shop Storeonce devices probably make sense i guess...

[veremin]

WAN Acceleration is indeed an ENT+ feature. However, have you tried to seed backup copy job first in order to avoid initial time-consuming cycle over WAN? Starting from that moment, backup copy job will be forever incremental, meaning only changes will be copied on pre-defined basis. Thanks.

[Delo123]

I shared our results with 2012R2 on several threads (not runnins 2016 in production yet btw) but again I can also report huge savings with windows dedupe. Biggest vbk's are between 4 and 8TB.

PS C:\Windows\system32> get-dedupstatus

FreeSpace SavedSpace OptimizedFiles InPolicyFiles Volume
--------- ---------- -------------- ------------- ------
47.75 TB 85.22 TB 387 393 O:
42.8 TB 100.06 TB 7965 8145 R:
48.25 TB 30.39 TB 11 11 S:
53.1 TB 69.92 TB 441 442 Q:
49.62 TB 50.58 TB 4740 4740 P:
33.19 TB 65.67 TB 199 199 T:
55.68 TB 39.64 TB 3756 3756 V:

Nearly 440TB saved on 279Tb physical jbod and still 140TB's free

[nmdange]

Here's an interesting concept, storing backups on ReFS inside a VM with the host volume formatted as NTFS with dedupe enabled. This is for DPM 2016 (which is also utilizing ReFS in the same way Veeam will in 9.5) but it could be done the same way with Veeam: https://charbelnemnom.com/2016/10/how-t ... p-storage/

[Mike Resseler]

That is an interesting concept. Done with Veeam that would mean we first create a volume with NTFS and dedup enabled on it. After that, we create a VM, and attach a disk to it (our repository) and the disks VHDX will be stored on that NTFS volume. Of course, inside that disk, we format as ReFS. So we get the benefit of having the ReFS advantages in our repository, and underneath that we gain from the NTFS dedupe advantages (at least that means that our repository disks are max 4 TB and probably we will use SOBR then to connect multiple repo's)

While it really seems interesting, I do have concerns...

* In this setup, will be not loose some performance that we want to gain by using ReFS?
* Dedup is #1 feature request in ReFS, MSFT might release this quicker and then we can redesign the entire setup again...
* ReFS has disk savings thanks to block clone API, I wonder how much additional savings there will be by placing dedup underneath it
* ReFS with integrity streams on (as we will do in v9.5) is a good solution for running backups on the backup storage, aren't we going to loose that advantage...

Last but not least, the dedup schedule needs to run outside of the backup schedule... Which might become very difficult to achieve

So again, interesting concept, but I would need to do some serious testing on this first because I have more questions for it then I have answers

Mike

[nmdange]

In the design, the VM holding the backups uses simple storage spaces to aggregate 1TB VHDX files into a single disk, so in that case SOBR wouldn't be needed. But I agree the benefits may not be enough over just using a physical server with ReFS.

[PTide]

UPDATE:

Users are having troubles with Windows Server 2016 deduplication and large files (> 2TB). Please take a look at this topic.

[mkaec]

The holy grail will be when Windows supports dedup on ReFS using block cloning. Maybe we'll see that in the 2019 release.

[BradJensen]

For those of you using Windows Deduplication in Server 2012 or 2016, we have a commercial product that does replication for the deduplicated volume.

This is real replication and does not reflate the deduplicated files.

It runs on the server that host the deduplicated volume, and another copy on the remote server with a deduplicated volume. You can run it on azure or Google Cloud also. You can replicate multiple volumes and to multiple locations.

Basically it allows you to 'roll your own' Data Domain-type appliance at a much lower cost. It can run inside a VM too.

See the 'Replacador' product on LaserVault.com

[mkaec]

Brad,

This sounds like a product that could fill a big need. Windows dedup is great, but it needs replication to really compare to the repository appliances. When I looked on the web site, I couldn't find pricing. Please relay the feedback that transparent pricing would be appreciated. Products that require contacting sales to get pricing information get negative points in our product evaluations.

[bkothari@hach.com]

Any update based on new features in Windows 2019 and Refs ?

[mkaec]

ReFS in Server 2019 gained the capability of deduplication, but it is the same style deduplication that is used with NTFS. I really would like to see it implemented using block cloning. That type of implementation would lose the compression feature, but there would no need for garbage collection.

[foggy]

Please see this thread for more details.