Repo dedupe question

[alex1002]

Good day,
I am setting up a Windows 2016 test repo. I want to setup dedupe for it. But there's different options. General files, virtualized backup storage etc. What's the best for veeam.
Thank you

[david.buchanan]

Hi Alex,

Normally you'd just select "General Files". The virtualized backup option simply as additional extensions to the exclusion list none of which are Veeam extensions (Excluding .tmp of course ).

By default .ebd and .jars files are excluded. The Virtualized option adds a bunch more, bin,vsv,slp,xml,tmp, etc. This is the only difference between the two that I'm aware of.

You can find out more about Dedup in Server 2016 here if it helps: https://blogs.technet.microsoft.com/aus ... available/

I'd recommend you exclude a few additional items manually such as vbm extensions as I've had cases where dedup has caused issues with these files which resulted in having to run a full backup.

I'd also suggest you don't enable "Partial files" and "In use" files being dedup'd. These are not on by default but just in case you do.

Thanks,
David.

[Mike Resseler]

Hi Alex,

Windows server 2016 dedupe actually brings lots of major enhancements to deduplication and really becomes interesting... There are still limitations compared to hardware dedupe boxes but that is expected

Anyway, I would chose backup storage (I still need to start testing the moment v9.5 releases...) but in the end, it really doesn't matter what you choose.
General files gives you these options:
- Background optimization
- Default optimization policy: Minimum file age = 3 days, Optimize in-use files = No, Optimize partial files = No
Backup option gives you this:
- Priority optimization
- Default optimization policy: Minimum file age = 0 days, Optimize in-use files = Yes, Optimize partial files = No
- "Under-the-hood" tweaks for interop with DPM/DPM-like solutions

The only one I personally would be scared off is that "under-the-hood" tweaking as I have no idea what it does for the moment. But, if you take the general setting, then you can easily change all of these options by using PowerShell or mostly in the UI also.

So you can start with the general files, and then tweak it to the same settings as backups (without those under the hood tweaks )

Hope it sheds some light
Mike

[alex1002]

Thank you for advice
It seems I was able to save 2TB so far.

[Mike Resseler]

Alex,

I can't say if it is much or not... 2TB out of a 100 would be low, 2 TB out of 4 seems better so...

[alex1002]

Sorry I got 14tb

Sent from my Nexus 6P using Tapatalk

[Mike Resseler]

Alex,

That seems rather low... But it might increase the next days. With windows dedupe it can take some time in the beginning. Keep an eye on it and look if it doesn't give you issues if it doesn't succeed in time to get everything deduped. If that happens, change the minimum file age day to a higher number and then gradually lower it until zero

Cheers
Mike

[alex1002]

Alex,

That seems rather low... But it might increase the next days. With windows dedupe it can take some time in the beginning. Keep an eye on it and look if it doesn't give you issues if it doesn't succeed in time to get everything deduped. If that happens, change the minimum file age day to a higher number and then gradually lower it until zero

Cheers
Mike

Thanks for the advise. I'll keep you posted. Just curious how does windows still manage to save space. I taught.veeam did it's own compression and dedupe magic.

Sent from my Nexus 6P using Tapatalk

[mkretzer]

We did a test with backup settings to normal (not optimized for Dedup) with Per-VM backup Files and got 24 % space saving...

[Mike Resseler]

@Alex: We do indeed have compression and dedupe but there are differences. We do this per backup file and windows per volume. This is just one example. There is also different algorithm and so on and so forth

@mkretzer: What are your settings for Dedup if I may ask? (See above for the settings I'm looking for )

Thanks
Mike

[mkretzer]

We used the virtualized backup setting in TP5. Because that's what veeam backups are I think?

Gesendet von meinem Pixel XL mit Tapatalk

[Mike Resseler]

True...

But one of the things is that it has some under the hood tweaks specifically for DPM... And since I don't know what those are... I basically use the other settings and apply them manually