My server Windows Server 2012 deduplication results

[johnlong]

First I'll just get right to the good stuff. It's pretty awesome.

Volume : R:
VolumeId : \\?\Volume{2abc0d3c-e0e0-4285-a50c-dc718d40641f}\
Capacity : 6.36 TB
FreeSpace : 1.65 TB
UsedSpace : 4.71 TB
UnoptimizedSize : 8.38 TB
SavedSpace : 3.67 TB
SavingsRate : 43 %
OptimizedFilesCount : 8229
OptimizedFilesSize : 6.75 TB
OptimizedFilesSavingsRate : 54 %
InPolicyFilesCount : 8229
InPolicyFilesSize : 6.75 TB
LastOptimizationTime : 3/4/2013 9:45:17 AM
LastOptimizationResult : 0x00000000
LastOptimizationResultMessage : The operation completed successfully.
LastGarbageCollectionTime : 3/3/2013 2:57:41 PM
LastGarbageCollectionResult : 0x00000000
LastGarbageCollectionResultMessage : The operation completed successfully.
LastScrubbingTime : 3/3/2013 2:58:56 PM
LastScrubbingResult : 0x00000000
LastScrubbingResultMessage : The operation completed successfully.

Now, onto the boring stuff. If you are thinking about using dedup, powershell is going to be the best way to work with deduplication. I used the GUI to enable it and then it's all powershell from there. This is the webpage I refer back to the most. http://technet.microsoft.com/en-us/libr ... 48450.aspx

In my setup I had a server 2008 R2 server with an external 6.3TB array consisting of 8 - 1TB drives in raid 5. I had another server that I setup as a server 2012 box and I simply moved the array controller to the new server and plugged in the drive array. I reassigned the drive letter to R, installed the Data Deduplication role and enabled dedup for that volume.

If you simply enable dedup and let it run it will run in its default schedule. You can look at the schedule in the task scheduler under Microsoft/windows/deduplication/backgroundoptimization. I won't go into the command switches that are used because I don't fully understand them myself yet.

I manually ran a dedup job and it appears to have run at full speed until the job finished. In all for me it ran for about 3 days or so. It consumed 10% cpu on 1 core, and about 14gb ram, on my server anyway. It was hitting the disks pretty hard however. Luckily the only thing this server really does is veeam backups so that wasn't an issue for me.

The dedup results I posted above are on backup files from my previous veeam jobs which were setup like this.
Reversed incremental, Perform active full backups periodically (last **** of the month, different for each job), Enable inline data dedup, compression optimal, local target.

Here is where things get a little unclear to me on what the best settings/options are.

I changed my jobs to run like this.
Incremental, Perform active full backups periodically (last **** of the month, different for each job), Uncheck enable dedup, compression dedup-friendly, optimize for local storage.

Here is one thing I didn't do, on the repository there is an advanced option for "Align backup file data blocks" and "Decompress backup data blocks before storing". I didn't check these options. Should I check one or both? I really cannot answer that question. If I did check decompress would I really see a big gain?

Hope this helps someone and if anyone has any suggestions please feel free to share!

[dellock6]

Pretty nice numbers!
About the repository configurations, is it the same machine as also proxy role? In that case you do not need to enable those configurations, they are used to reduce at the minimum data travelling via network from proxy to repository, and let nonetheless reporitory expand again data before writing them to the storage, this is pretty useful to save bandwidth while shipping data to a dedupe appliance with a linux/windows "head" in front of it. If it's the same server, there is really no need to compress and decompress data inside the same VM.
I do not know if win2012 works with variable size blocks, or it can benefit from block alignment. On some dedupe appliances like DataDomain it helps.

Luca.

[Delo123]

Was doing some testing since we were unable to Dedupe Big backup Sets with 2012...
Thought i start again comparing big vs small Jobs, Dedupe vs Compression etc...
Let me know what you think!

Backing up 13 Citrix Xenapp Servers (8x2008R2, 5x2003)
First Backup, no other Data on the Disks.
All located on the same Backup Repository. Dual AMD 6100, 24 Logical Cpu's, 64Gb RAM. CIFS Target 1x1GB.

E: 1 Job, Forward Incremental, Dedupe enabled, Compression Optimal, Optimize for local Target.
F: 3 Jobs splitted in 5 Vm's each, Forward Incremental, Dedupe enabled, Compression Optimal, Optimize for Local Target.
G: 1 Job, Forward Incremental, Dedupe disabled, Compression disabled, Optimize for local Target.
H: 1 Job, Forward Incremental, Dedupe disabled, Compression enabled, Optimize for local Target.

Full Backups
File Sizes on Target:
E: (1 Backup File) 135GB -> Windows Dedupe -> 129GB
F: (3 Backup Files) 422GB -> Windows Dedupe -> 66GB
G: (1 Backup File) 550GB -> Windows Dedupe -> 132GB
H: (1 Backup File) 209GB -> windows Dedupe -> 137GB

Ps. i am not sure why G: doesn't show Dedupe Savings, maybe the Stats need some Time...!?

FreeSpace SavedSpace OptimizedFiles InPolicyFiles Volume
--------- ---------- -------------- ------------- ------
6.37 TB 2.83 GB 2 2 E: 1:45:21
4.72 TB 356.55 GB 3 3 F: 31:15,29:00,29:00 -> 1:29:15
4.69 TB 0 B 0 1 G: 1:59:43
4.63 TB 65.5 GB 2 2 H: 1:56:36

PS C:\Windows\system32> get-dedupstatus e: |fl
Volume : E:
UsedSpace : 129.17 GB
UnoptimizedSize : 132 GB
SavedSpace : 2.83 GB
SavingsRate : 2 %
OptimizedFilesCount : 2
OptimizedFilesSize : 129.13 GB
OptimizedFilesSavingsRate : 2 %

PS C:\Windows\system32> get-dedupstatus f: |fl
Volume : F:
UsedSpace : 66.31 GB
UnoptimizedSize : 422.85 GB
SavedSpace : 356.55 GB
SavingsRate : 84 %
OptimizedFilesCount : 3
OptimizedFilesSize : 419.56 GB
OptimizedFilesSavingsRate : 84 %

PS C:\Windows\system32> get-dedupstatus g: |fl
Volume : G:
UsedSpace : 132.48 GB
UnoptimizedSize : 132.48 GB
SavedSpace : 0 B
SavingsRate : 0 %
OptimizedFilesCount : 0
OptimizedFilesSize : 0 B
OptimizedFilesSavingsRate : 0 %
InPolicyFilesCount : 1
InPolicyFilesSize : 94.28 GB

PS C:\Windows\system32> get-dedupstatus h: |fl
Volume : H:
UsedSpace : 136.81 GB
UnoptimizedSize : 202.31 GB
SavedSpace : 65.5 GB
SavingsRate : 32 %
OptimizedFilesCount : 2
OptimizedFilesSize : 199.37 GB

Concluding it seems like more Jobs with less VM's are fastest and give best Dedupe results.
Also since Backup Files are smaller. Windows Dedupe has a bigger chance of getting Dedupe done within the Jobs Timeframe,
can skip optimized files and does not have to start over again on the next run.

Will do a second run today with a second synthetic Full to see how it works out.
But i still think if Veeam would be able to split backup Files in Chunks this would be great for 2012 Dedupe.
Previous Tests with a 3TB Fileserver Backup showed it was not possible to dedupe the File within 24 Hours...

[johnlong]

I've got relatively small vm's. 300-500 GB.

Here are my latest stats.
Volume : R:
VolumeId : \\?\Volume{2abc0d3c-e0e0-4285-a50c-dc718d40641f}\
Capacity : 6.36 TB
FreeSpace : 2.48 TB
UsedSpace : 3.89 TB
UnoptimizedSize : 11.77 TB
SavedSpace : 7.88 TB
SavingsRate : 66 %
OptimizedFilesCount : 8301
OptimizedFilesSize : 11.7 TB
OptimizedFilesSavingsRate : 67 %
InPolicyFilesCount : 8301
InPolicyFilesSize : 11.7 TB
LastOptimizationTime : 5/3/2013 9:45:20 AM
LastOptimizationResult : 0x00000000
LastOptimizationResultMessage : The operation completed successfully.
LastGarbageCollectionTime : 4/27/2013 2:53:36 AM
LastGarbageCollectionResult : 0x00000000
LastGarbageCollectionResultMessage : The operation completed successfully.
LastScrubbingTime : 3/9/2013 6:19:10 AM
LastScrubbingResult : 0x00000000
LastScrubbingResultMessage : The operation completed successfully.

[johnlong]

It just keeps getting better for me. I'm guessing as older veeam deduped jobs are removed windows dedup gets better and better. That coupled with the fact that my data change rate is relatively small.

Volume : R:
VolumeId : \\?\Volume{2abc0d3c-e0e0-4285-a50c-dc718d40641f}\
Capacity : 6.36 TB
FreeSpace : 2.22 TB
UsedSpace : 4.15 TB
UnoptimizedSize : 15.49 TB
SavedSpace : 11.34 TB
SavingsRate : 73 %
OptimizedFilesCount : 436
OptimizedFilesSize : 15.42 TB
OptimizedFilesSavingsRate : 73 %
InPolicyFilesCount : 436
InPolicyFilesSize : 15.42 TB
LastOptimizationTime : 7/19/2013 8:45:26 AM
LastOptimizationResult : 0x00000000
LastOptimizationResultMessage : The operation completed successfully.
LastGarbageCollectionTime : 7/13/2013 3:37:21 AM
LastGarbageCollectionResult : 0x00000000
LastGarbageCollectionResultMessage : The operation completed successfully.
LastScrubbingTime : 7/3/2013 9:28:02 AM
LastScrubbingResult : 0x00000000
LastScrubbingResultMessage : The operation completed successfully.

[Starman]

Im going to cut and paste an issue and warning on using M$ Dedupe, I'll alter it to make more sense here. Nerveless, I have used it since 2012 came out and now Im ripping it out.

____________

Our issue seems to revolve more around Server 2012 and dedupe. I've run DFS for many years now with no problems. I migrated several stores to Server 2012 with Dedupe and they ran fine for a while and then DFS started screwing up hard. Several (not all) started to not replicate with odd issues revolving around an error such as "share has no data" (not exact). My fix at the time was to stop DFS since we were migrating off. This issue only occurred on sever 2012 with dedupe, if that folder had 6 targets in 6 different offices, server 2008R2 replicated DFS between them fine. Our shar browsing rebounding Started a few months back. A user starts drilling into a share and it kicks them back to the root continuously. Last night I unoptimized one of the servers (removed Dedupe) and I can tell by just using the Windows gui that its running 10 times faster already. This server has 3 drives with a good 300 gigs deduped with a savings of around 75 gigs. I have to wait to hear from users as if this fix's the rebounding of share browsing to their roots. This may possibly be an issue on the Dell Iscsi based Eualogics but the DFS error happened when they were on direct connected 6 gig SAS Dell MD3200's with massive I/O. At this time I'm going to dump Dedupe on the other 4 NAS servers that I have and consider Dedupe nothing but trouble. On a side, note, there is very little on the web about issues like this. Hours and hours of searching with very little info. Its my thought that this is new and caused after longer term use.

Its important to note that using Vcenter, Vcops and the Equalogic SAN manger there were absolutely no issues going on with that virtual machine (like high IO, latency or anything else). The 2012 server itself presented no issues in any way via events and such.

Since removing Dedupe, the virtual machine and shares have worked perfectly.