Windows Server 2016 w/ Dedupe or Dedupe Appliance?

Availability for the Always-On Enterprise

Windows Server 2016 w/ Dedupe or Dedupe Appliance?

Veeam Logoby cbrasga » Thu Oct 20, 2016 4:38 pm

We are looking to create a long term backup repository. I'm interested in the community's thoughts and recommendations on whether to use a Windows Server 2016 packed with disks and the deduplication feature enabled, or a dedicated deduplication appliance?
cbrasga
Influencer
 
Posts: 11
Liked: never
Joined: Sat Apr 27, 2013 2:09 am
Full Name: Cazi Brasga

Re: Windows Server 2016 w/ Dedupe or Dedupe Appliance?

Veeam Logoby PTide » Thu Oct 20, 2016 5:06 pm

Hi,

That really depends on your RPO and RTO, as well as on the amount and size of VMs that you are going to backup. To give you an example of one Windows 2016 limitation:
<...>This has all been redesigned for Windows Server 2016 <...>, with the results being that you can go ahead and dedup files up to 1TB without worrying about them not being good candidates.

For some infrastructures 1 TB is not a limit of size for a backup.

Thanks
PTide
Veeam Software
 
Posts: 3291
Liked: 274 times
Joined: Tue May 19, 2015 1:46 pm

Re: Windows Server 2016 w/ Dedupe or Dedupe Appliance?

Veeam Logoby sanjaykrk » Thu Oct 20, 2016 11:33 pm

cbrasga wrote:We are looking to create a long term backup repository. I'm interested in the community's thoughts and recommendations on whether to use a Windows Server 2016 packed with disks and the deduplication feature enabled, or a dedicated deduplication appliance?


As pointed out in above comment it depends on your requirement. Windows dedupe might serve you well.
On the other hand, dedupe appliances may offer some benefit (may or may not be relevant to you). For e.g. as I understand Windows dedupe will be effective at volume level with current recommended size around 64TB.
In case you have data > 64TB, you will be creating multiple volumes but they will be deduped independently. Dedupe appliances may be able to provide you global dedupe i.e. dedupe across multiple volumes. so that even across volumes only unique data is stored.
sanjaykrk
Influencer
 
Posts: 11
Liked: 3 times
Joined: Sat Apr 09, 2016 12:12 am
Full Name: Sanjay kumar

Re: Windows Server 2016 w/ Dedupe or Dedupe Appliance?

Veeam Logoby planetcoop » Fri Oct 21, 2016 1:53 am 1 person likes this post

here are my 2016 dedupe stats for the year as i have updated from tp4 to 5 and now final:

PS C:\WINDOWS\system32> Get-DedupVolume | Format-List
Volume : D:
VolumeId : \\?\Volume{2fc167ff-9d98-45cb-b5fe-13b772ce848c}\
Enabled : True
UsageType : Backup
DataAccessEnabled : True
Capacity : 3.5 TB
FreeSpace : 2.68 TB
UsedSpace : 836.79 GB
UnoptimizedSize : 9.02 TB
SavedSpace : 8.2 TB
SavingsRate : 90 %
MinimumFileAgeDays : 0
MinimumFileSize : 32768
NoCompress : False
ExcludeFolder :
ExcludeFileType :
ExcludeFileTypeDefault : {bin, vsv, slp, xml...}
NoCompressionFileType : {asf, mov, wma, wmv...}
ChunkRedundancyThreshold : 100
Verify : False
OptimizeInUseFiles : True
OptimizePartialFiles : False
InputOutputScale : 0
planetcoop
Novice
 
Posts: 6
Liked: 1 time
Joined: Wed Jul 01, 2015 4:12 pm
Full Name: Chris Cooper

Re: Windows Server 2016 w/ Dedupe or Dedupe Appliance?

Veeam Logoby Mike Resseler » Fri Oct 21, 2016 5:24 am

Hi Chris,

That is a ridiculous large amount of savings! Nice. What file types are you storing on that volume? Is it a volume containing Veeam backups only or also other files? (I see you use the type Backup but you might use the volume for something else also...)

Thanks

Mike
Mike Resseler
Veeam Software
 
Posts: 3490
Liked: 390 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Windows Server 2016 w/ Dedupe or Dedupe Appliance?

Veeam Logoby Mike Resseler » Fri Oct 21, 2016 5:35 am 1 person likes this post

And to give some feedback on the original question:

I agree with Pavel that sometimes Windows Dedupe is not always the perfect solution for every company. There is indeed that limitation of 1 TB (although that is not a hard limit and I have seen files over 2 TB to be deduped also...). The other side is that Windows Dedupe is post-process which means it needs to land on the target FULL size first before it can get deduped. That means you need to have that actual storage available to start with. Also, when rehydrating a backup (at the time of restore), you need the storage available also. Last but not least, it requires resources to do this work (not only the dedupe but also the scrubbing and clean-up of old data on a weekly basis) so you might need a beefier server for that purpose.

A dedup appliance is specifically built and has the necessary resources but comes with a price. It might have less of the above limitations, but it does come with a higher price.

So imho, the best you can do is actually look at your environment. See if per-VM backups will get you over that 1TB files or not, make a cost-calculation of a beefier server + storage and compare it with the price of a dedup aplliance. Then see if the additional cost of that appliance is acceptable to not have those limitations (if they actually apply to you, because for many environments Windows Dedupe is more than enough)

My 2 cents
Mike
Mike Resseler
Veeam Software
 
Posts: 3490
Liked: 390 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Windows Server 2016 w/ Dedupe or Dedupe Appliance?

Veeam Logoby nmdange » Fri Oct 21, 2016 2:15 pm

Using ReFS on Windows Server 2016 will be a better option since Veeam 9.5 will use the new ReFS APIs so that when a synthetic full is run, both the original file and the new file point to the same physical blocks. Essentially, duplicates won't be created in the first place so dedupe won't be needed.
nmdange
Expert
 
Posts: 235
Liked: 60 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: Windows Server 2016 w/ Dedupe or Dedupe Appliance?

Veeam Logoby Mike Resseler » Fri Oct 21, 2016 2:24 pm 1 person likes this post

That is true when you use incremental backups and use a synthetic full. We expect to see the same sort of savings, but do remember this is not deduplication so there still will be a difference and depending on your specific setup and the data you are protecting your mileage will vary!
Mike Resseler
Veeam Software
 
Posts: 3490
Liked: 390 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Windows Server 2016 w/ Dedupe or Dedupe Appliance?

Veeam Logoby planetcoop » Fri Oct 21, 2016 4:09 pm

my backup data for the awesome space savings. i have a combination of jobs but the underlying result is:
-Exchange 2016 - every 4 hours with a full at midnight and a retention of 1 week (42 copies)
-Daily All - servers (including Cisco VOIP) daily incremental with a full on the weekend and a 365 day retention and 365 copies
-Weekly All - servers (including Cisco VOIP) weekly incremental with a full on the first Wednesday of the month and a 52 week retention and a total of 52 copies

Exchange seems to dedupe well and you can imagine the total number of copies are almost 500 copies. The reset of the servers also do well with a low change rate. The stand out is that once of the windows 2016 servers doe shave a 26TB storage drive that is excluded. I use a separate process to store and forward its data (low change rate archive) to another server mostly kept offline. I have looked at ReFs and 2016, but i cannot speak on behalf of Veeam 9.5 as i am not in the beta testing, however it has not proven to be as beneficial for me as dedupe has been. I can also share my dedupe settings. Basically, the type is "backup" which is new in 2016. I also delete the canned scheduling enabled with default dedupe. I have devised a better schedule for my workload and change rate. Daily dedupe with 95% memory and no throttling at 5AM. By 10PM I scavenge and then at 11PM I collect garbage.

I cant share as many details but another production environment I have two vaults of 60TB each and the same process is scheduled on a weekly basis just based on change rate and size. Additionally this larger environment revolves around a 30 day retention with only a few server archiving exceptions with 999 copies max.
planetcoop
Novice
 
Posts: 6
Liked: 1 time
Joined: Wed Jul 01, 2015 4:12 pm
Full Name: Chris Cooper

Re: Windows Server 2016 w/ Dedupe or Dedupe Appliance?

Veeam Logoby Delo123 » Mon Oct 24, 2016 7:55 am 1 person likes this post

A lot of savings can be made even with incrementals. Our largest full Backup is 8TB. When setting things up with scale out repositories and clever placement of files windows dedupe still rules.
Just make sure you test, test test! and follow the 3,2,1 Rule. In our Case O,Q,S and U have Full Backups only where P,R,T and V are for incrementals only. This is on physically 270TB where 169 are still free. 440TB are "saved" by dedupe until now.

PS C:\Windows\system32> get-dedupstatus

FreeSpace SavedSpace OptimizedFiles InPolicyFiles Volume
--------- ---------- -------------- ------------- ------
48.23 TB 81.37 TB 372 372 O:
43.42 TB 99.09 TB 8589 8589 R:
44.35 TB 26.48 TB 10 11 S:
50.49 TB 67.57 TB 407 407 Q:
49.77 TB 47.49 TB 4471 4471 P:
36.93 TB 65.38 TB 197 198 T:
55.72 TB 40.05 TB 3761 3761 V:
57.92 TB 13.42 TB 204 204 U:
Delo123
Expert
 
Posts: 355
Liked: 103 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Windows Server 2016 w/ Dedupe or Dedupe Appliance?

Veeam Logoby cousinsp » Mon Oct 24, 2016 1:30 pm

The other thing to factor in is offsite replication. We are currently using Windows 2012R2 with Dedupe, but due to only having 30Mb bandwidth to our other sites we are unable to replicate much offsite, as we only have Enterprise Edition which doesn't do anything clever with offsite replication - data is just copied.
We looked at Enterprise Plus Edition, but it's extremely expensive for us to upgrade.
So we are currently looking at the HP Store Once devices, as we're successfully using these for Backup Exec (but haven't got one at our main site!). I believe they will handle offsite replication better and only send different data blocks.

Pete Cousins
cousinsp
Novice
 
Posts: 9
Liked: 1 time
Joined: Fri Jan 15, 2016 11:20 am
Full Name: Pete Cousins

Re: Windows Server 2016 w/ Dedupe or Dedupe Appliance?

Veeam Logoby Delo123 » Mon Oct 24, 2016 1:57 pm

Yes, you would certainly need some kind of wan accelerator here, windows file copy will be full blown.
Not sure if Veeam only offers this in enterprise, i'll take your word for it. Yes, HP Storeonce will probably do this also so it's a matter of comparing costs for them against windows dedupe + oem jbod + veeam licenses.
If you are a HP shop Storeonce devices probably make sense i guess...
Delo123
Expert
 
Posts: 355
Liked: 103 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Windows Server 2016 w/ Dedupe or Dedupe Appliance?

Veeam Logoby v.Eremin » Mon Oct 24, 2016 2:08 pm 1 person likes this post

WAN Acceleration is indeed an ENT+ feature. However, have you tried to seed backup copy job first in order to avoid initial time-consuming cycle over WAN? Starting from that moment, backup copy job will be forever incremental, meaning only changes will be copied on pre-defined basis. Thanks.
v.Eremin
Veeam Software
 
Posts: 13845
Liked: 1036 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Windows Server 2016 w/ Dedupe or Dedupe Appliance?

Veeam Logoby Delo123 » Wed Oct 26, 2016 1:21 pm 2 people like this post

I shared our results with 2012R2 on several threads (not runnins 2016 in production yet btw) but again I can also report huge savings with windows dedupe. Biggest vbk's are between 4 and 8TB.

PS C:\Windows\system32> get-dedupstatus

FreeSpace SavedSpace OptimizedFiles InPolicyFiles Volume
--------- ---------- -------------- ------------- ------
47.75 TB 85.22 TB 387 393 O:
42.8 TB 100.06 TB 7965 8145 R:
48.25 TB 30.39 TB 11 11 S:
53.1 TB 69.92 TB 441 442 Q:
49.62 TB 50.58 TB 4740 4740 P:
33.19 TB 65.67 TB 199 199 T:
55.68 TB 39.64 TB 3756 3756 V:

Nearly 440TB saved on 279Tb physical jbod and still 140TB's free
Delo123
Expert
 
Posts: 355
Liked: 103 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Windows Server 2016 w/ Dedupe or Dedupe Appliance?

Veeam Logoby nmdange » Wed Oct 26, 2016 10:13 pm

Here's an interesting concept, storing backups on ReFS inside a VM with the host volume formatted as NTFS with dedupe enabled. This is for DPM 2016 (which is also utilizing ReFS in the same way Veeam will in 9.5) but it could be done the same way with Veeam: https://charbelnemnom.com/2016/10/how-t ... p-storage/
nmdange
Expert
 
Posts: 235
Liked: 60 times
Joined: Thu Aug 20, 2015 9:30 pm

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot] and 1 guest