Inline deduplication

[andreas2012]

Hi,

I have a Windows 2016 server with a NTFS share that I have enabled deduplication on.
Then I have created a backup job that back up several virtual machines, and then I have enabled what you can see in the image below
https://ibb.co/fuyyup

Then I have several physical servers in a protection group, and added a these to a backup job, but here I am not able to select deduplication, is it not supported ?
https://ibb.co/gSs2LU



Btw, I first created a ReFS volume, and thought I could enable deduplication on this, but since I have version 1607 of Windows 2016, it is not supported, so the option is grey.
I have not been able to download version 1803 since I dont have access to it from MS download VLSC.... so am I missing something here? could I download it from somewhere... because when Veeam say they support it, well they could also pinpoint the requirements... Or am I wrong ?

Thanks for reply.

/Regards
Andreas

[csydas]

Hi,

I have a Windows 2016 server with a NTFS share that I have enabled deduplication on.
Then I have created a backup job that back up several virtual machines, and then I have enabled what you can see in the image below
https://ibb.co/fuyyup

I'm not sure this is ideal you're basically trying to deduplicate already deduplicated data, and you're missing out on potential space savings. From what I recall, you either want to do the inline deduplication for smaller backup files, or enable the storage deduplication and let it handle the dedupe process at the storage level across all of the data content instead of per file.

By deduplicating the files and putting them on a deduplicating server, you're basically just running the dedupe process across some already heavily unique segments, and you get far fewer returns.

Instead, I think you want to disable the inline and just let the storage handle the dedupe.

[andreas2012]

Hi,

Well If I have read the article correct I should enable Inline dedup also when dedup is used on the windows 2016 box

https://www.veeam.com/blog/data-dedupli ... veeam.html

Ref:
****
But the most interesting case happens if the administrator wants to put Veeam backups in a repository which resides on a disk volume with Windows Data Deduplication enabled. For those cases, we have several important requirements to ensure these two technologies work together perfectly. An administrator must make Veeam aware of the upcoming Windows Data Deduplication so it behaves accordingly, not trying to do the whole job itself. It’s recommended to set up the Backup Job properly: Navigate to its settings in Storage -> Advanced -> Storage tab and make sure that inline data deduplication is enabled, compression level is set to “None”, storage optimization is selected with “LAN target”
***

So am I mistaking ?

Regards
Andreas

[csydas]

That is a little curious, as I hadn't read that article. Thanks for sharing. No, you're not mistaken, it's just that Veeam has some seemingly conflicting recommendations.

Maybe Veeam has run internal tests and found that there are still some space saving benefits with Windows Dedupe, but for their Dedupe Appliances (e.g., DataDomain as per this KB: https://www.veeam.com/kb1956), Veeam recommends turning off inline dedupe.

From just a purely theoretical perspective, in my mind running dedupe on already deduped files doesn't make sense -- the algorithm has less data to work with and to me it would seem reasonable that a file which has already had its duplicate data stripped out increases its uniqueness when observed by the storage deduplication.

Maybe someone from Veeam can explain the discrepancy? Is there a reason that Inline-Dedupe is recommended for Windows Dedupe Repositories but not for a proper Deduplication Appliance?

[andreas2012]

Right, well lets wait and see if they comment.

But my main question was regarding physical servers in a protection group, and added a these to a backup job, but here I am not able to select deduplication, is it not supported ? Ref link to images i main post...

/R
Andreas

[csydas]

I'm assuming no if it's not there, and the user guide for agent seems to confirm it. I would assume it's because the process runs inside the VM itself and isn't able to get a look at the blocks like a hypervisor backup does.

[skrause]

A few quick things about Windows Deduplication:

1) It is not inline, it is post-process and can only run one operation per Windows volume at a time (but can run operations on multiple volumes at once.) So for best results you would want to have multiple volumes with dedupe enabled to speed up processing rather than one giant volume.
2) It is not a block-level or file-level deduplication. It uses "chunklets" which kind of split the difference.
3) While using Windows dedupe on already compressed and deduplicated files does not give you the best space savings, it does still work pretty well. We have a lot repository volumes with dedupe turned on and we usually get around 2:1 space savings on average.

[RockemSockem]

Gang, leaving Veeam's inline deduplication enabled for NTFS deduplicated repos is fine. Look at it this way - it's fewer writes to disk for each backup cycle, and it's less work that the Windows post-process engine has to do to hash blocks and add them to the chunk store if Veeam has already finished some of the work. I've seen no problems in our environment leaving inline deduplication enabled. No, your reduction rates won't be as substantial by percentage saved; but you're completing part of the work up front, so it's a shared benefit.

Also, the proper way to disable compression for deduplicated storage (whether inline or post-process architecture) is to configure the *repository* to decompress before writing, not to disable compression in the jobs. If you have a distributed backup infrastructure (specifically separate proxies and repositories) and disable compression in the job, you are eliminating network transfer efficiencies that the job-level compression provides which can then affect your transfer rates and thus your job processing rates.

One additional point - if you use your deduplicated repo as a source for *backup copy jobs* and your copy repo is *not deduplicated*, be sure to change the compression level on the copy job to an explicit compression level and *do not leave it on auto*. Otherwise, your copy will have no compression.

Cheers!

[foggy]

Veeam B&R uses much larger block size (512 KB for LAN Target), so its inline deduplication doesn't affect deduplication rates of storage appliances/Windows dedupe.