SRV 2012 Dedup does nothing on backup copy job volume

[tfloor]

Hi,

I have a Physical Machine with B&R on our DR site that has an QNAP ISCSI NAS mounted.
The backup Copy jobs are landed on that NAS.

I've enabled Server 2012 R2 Deduplication on the ISCSI Volume.

Settings are:
- Deduplicate filder older than 3 days
- No File Exclusions
- Enable Throuput Optimization every day between at 12:00 (duration 5 hours).


I've wait 2 weeks but nothing is deduped.

As you see here:
PS C:\Users\Administrator> Get-DedupStatus b: | fl


Volume : B:
VolumeId : \\?\Volume{82bcbf46-a252-4c5a-9106-ae7315675fad}\
Capacity : 36.25 TB
FreeSpace : 31.28 TB
UsedSpace : 4.97 TB
UnoptimizedSize : 4.97 TB
SavedSpace : 0 B
SavingsRate : 0 %
OptimizedFilesCount : 2
OptimizedFilesSize : 3.78 TB
OptimizedFilesSavingsRate : 0 %
InPolicyFilesCount : 4
InPolicyFilesSize : 3.97 TB
LastOptimizationTime : 17-7-2014 16:18:32
LastOptimizationResult : 0x8056533D
LastOptimizationResultMessage : The operation was cancelled.
LastGarbageCollectionTime : 5-7-2014 02:45:06
LastGarbageCollectionResult : 0x00565302
LastGarbageCollectionResultMessage : There are no actions associated with this job.
LastScrubbingTime : 12-7-2014 03:46:07
LastScrubbingResult : 0x00000000
LastScrubbingResultMessage : The operation completed successfully.


Any idea if this should work?
It looks like it's doing nothing, or timed out.

Kind regards,

Tristan.

[veremin]

What is copy interval specified for a backup copy job? How often it runs, in other words? What are the retention settings? Can you also check the status of dedupe job, using Get-DedupeJob commandlet? Thanks.

[andersonts]

You should also check the compression settings in the backup copy job...if the file is compressed then Windows 2012 won't likely be able to do anything with it. You might want to review the following post if you haven't already: http://forums.veeam.com/veeam-backup-re ... 02-15.html

[tfloor]

What is copy interval specified for a backup copy job? How often it runs, in other words? What are the retention settings? Can you also check the status of dedupe job, using Get-DedupeJob commandlet? Thanks.

The Backup copy job "Copy every: 1 day starting at 4:00
Restore points to keep: 720
GFS:
weekly : 0
Montkly: 0
Quarterly: 0
Yearly : 1

I'm still playing to understand the best GFS settings. I know what i want but translate it to GFS will takes some time to see the results.


Get-dedupjob gives me Always a 0% progress.

[tfloor]

You should also check the compression settings in the backup copy job...if the file is compressed then Windows 2012 won't likely be able to do anything with it. You might want to review the following post if you haven't already: http://forums.veeam.com/veeam-backup-re ... 02-15.html

The Storage settings of the backup copy job are:

- Enable inline data-deduplication : yes
- Compression: Auto


The storage settings of the backup jobs are:

- Enable inline data-deduplication : yes
- Compression: Optimal
- Optimize for : LAN Target

[veremin]

Get-dedupjob gives me Always a 0% progress.

I believe that MS community might be the better place to address issues related to Windows deduplication mechanism. Just a wild guess - if my memory serves me well, Windows 2012 deduplication can process roughly 2 TB of data per day on a single volume, the space currently occupied on the given volume is bigger than 4 TB, so, deduplication job being scheduled to run on daily basis might just come across the said limitation. Thanks.

[tfloor]

hmmm.. maybe deduplication sounds better than it is for us.

[veremin]

Also, it seems that files larger than 1TB can't be deduplicated. Thanks.

[tfloor]

Also, it seems that files larger than 1TB can't be deduplicated. Thanks.

i See, but i'm only interesting in dedup the big files
Otherwise i don't need dedup.

[veremin]

My point was that your files appear to be bigger than supported threshold (1 TB), and, thus, cannot be deduped. So, there is little we can do in that case. Thanks.

[hans_lenze]

LastOptimizationResultMessage : The operation was cancelled.

Are you sure that you allow for enough time to let the dedup process finish?

I've set the standard background dedup job with the following settings:
- Background optimization for files older than 1 day
- Changed the scheduled task with: set-dedupschedule back* -memory 75 -priority high

I choose background optimization because of the hourly schedule and stops automatically when an other process requires IO. By altering the memory and priority the dedup process can claim more resources for the job when it runs so the deduplication of files completes faster. This way, most of the idle time will be used for deduplication.
The backup and backup copy jobs use the 'dedupe friendly' compression setting and inline deduplication is enabled.

Code: Select all

PS C:\Windows\system32> Get-DedupSchedule back* | fl


Name                     : BackgroundOptimization
Enabled                  : True
Type                     : Optimization
Days                     :
Start                    :
DurationHours            :
StopWhenSystemBusy       : True
Memory                   : 25 %
Priority                 : Low
InputOutputThrottleLevel : Low
ScheduledTask            : \Microsoft\Windows\Deduplication\BackgroundOptimization
Full                     :
ReadOnly                 :



PS C:\Windows\system32> Set-DedupSchedule back* -Memory 75 -Priority high
PS C:\Windows\system32> Get-DedupSchedule back* | fl


Name                     : BackgroundOptimization
Enabled                  : True
Type                     : Optimization
Days                     :
Start                    :
DurationHours            :
StopWhenSystemBusy       : True
Memory                   : 75 %
Priority                 : High
InputOutputThrottleLevel : Low
ScheduledTask            : \Microsoft\Windows\Deduplication\BackgroundOptimization
Full                     :
ReadOnly                 :

Result on the offsite DR repository:

Code: Select all

PS C:\Windows\system32> Get-DedupVolume | fl


Volume                   : D:
VolumeId                 : \\?\Volume{d337fd9d-74fb-4564-a9b0-a17da131a269}\
Enabled                  : True
UsageType                : Default
DataAccessEnabled        : True
Capacity                 : 18.19 TB
FreeSpace                : 12.71 TB
UsedSpace                : 5.48 TB
UnoptimizedSize          : 16.66 TB
SavedSpace               : 11.18 TB
SavingsRate              : 67 %
MinimumFileAgeDays       : 1
MinimumFileSize          : 32768
NoCompress               : False
ExcludeFolder            :
ExcludeFileType          :
ExcludeFileTypeDefault   : {edb, jrs}
NoCompressionFileType    : {asf, mov, wma, wmv...}
ChunkRedundancyThreshold : 100
Verify                   : False
OptimizeInUseFiles       : False
OptimizePartialFiles     : False

The biggest files in this repository are about 2,3TB and there are eight of them.


One important note: don't forget to manually reformat the partition using the /L flag and the correct NTFS blocksize before enabling deduplication! Big files on the repository may end up in a read-only state if you don't. This is not an issue for a DR repository but very annoying on the primary repository when using reverse incremental backups.

[tfloor]

Are you sure that you allow for enough time to let the dedup process finish?

It is cancelled by windows himself because there is no progress.
But are you telling me that you have deduplication successfully with big files (2,3TB) ?
My biggest files are 3 TB at the moment. That's the VBK (seeding) on the backup copy offsite location.

[hans_lenze]

Yes, deduplication works for me.

But: I use several jobs to copy all the files to the DR repository (because a single job didn't work before patch4) so there is a bigger change of finding similar blocks between jobs. And I have a number of full VBK files (one every week for three months, one every month for a year, one every year for five years). Since VBK files have a lot of similar data this really boosts my dedup ratio.

If you use a single job and have just a single restore point on the DR site, a next to nothing dedup ratio would make sense. The dedup ratio should increase when you start to add more (full) backups to the repository.
How many restore points do you have on the DR repository?

[romura]

There's another thread on this topic: http://forums.veeam.com/veeam-backup-re ... 17785.html. From my experience, backup copy jobs and dedup volumes do NOT work well with each other.

[veremin]

The problem described in the referenced topic seems to be a different one. That problem is more about dedupe appliances not copping with backup copy job transformation well, meanwhile, the OP's one is about Windows deduplication mechanism not deduping at all. Thanks.

[tfloor]

Yup so at the moment i ended with disabling dedup, since i have no success with it.

[veremin]

Tristian, have you tried to post the very same issue on MS Technet communities? Chances are, they have better clues on what might prevent dedupe jobs from running. Thanks.