MS Dedup usable again in bigger environments thanks to V9?

[mkretzer]

Our backup repo is 200 TB big and hosts 160 TB of backups right now. We use MS storagespaces to span one volume over multiple storage LUNs.
Backup type is traditional incrementals with weekly full.

Our original plan was to use MS dedup but we are well over 64 TB volume size and also most of our backups are bigger than 1 TB.

Am i correct that with V9 we could perhaps use MS dedup if we use <64 TB LUNs, Scale-Out Repo and Per-VM backup files? Does someone use these features together with ms dedup?

[Gostev]

Hi,

In theory, the answer is YES. We definitely kept Windows Server dedupe in mind to play along with these features.

In practice, I honestly don't think Server 2012 dedupe will be able to even touch your backup footprint due to poor dedupe processing performance. However, Server 2016 adds major improvements around specifically dedupe scalability. We have done some testing on TP4, and the results are quite impressive.

So, my advise is that you don't waste time now, but come back to this with Server 2016 release.

Thanks!

[mkretzer]

Thank you, i worried that this is the case. Is that because there is a fixed dedup performance limit (i read something about 100 GB per day) or because it is single-threaded?

Will Veeam support 2016 directly after release?

[nielsengelen]

Currently (as 2016 isn't out yet) it isn't supported however looking at our history we have always been really fast with delivering full support for new releases (vSphere & Microsoft).

[Gostev]

Yes, single threaded is the issue.

[Delo123]

Create multiple <64TB thin volumes from your Storage pool in 2012R2 and you are good to go (remember to format with /l) , since each volume has it's own dedupe jobs running you can paralyze by this.
Our biggest VBK (after using per VM backup chains) is 4,5TB, savings are great

PS C:\Windows\system32> get-dedupstatus
FreeSpace SavedSpace OptimizedFiles InPolicyFiles Volume
--------- ---------- -------------- ------------- ------
50.37 TB 63.47 TB 242 240 J:
43.24 TB 98.89 TB 355 355 G:
52.75 TB 64.95 TB 440 440 I:
16.04 TB 268.78 TB 737 756 H:

Guido

[Gostev]

Surprised you're not seeing any issues with that 4.5TB VBK, at least we saw them in our lab, even with slightly smaller files... I can only hope you will not discover that backup file is borked when you actually need to recover from it!

[Delo123]

Which issues did you see?

We create daily replica's from backup files and also copy them away and have never seen an issue with the actual files,
however we do have issues replicating big vm's in general and have a Ticket open for that (Ticket 1687344) since some time, but i don't think it's any how related.

[Gostev]

I don't remember the specific issues by now, as these tests were done a few years ago.
But I do know that Server 2016 dedupe for example simply ingores such files completely.

[Delo123]

Didn't know Server 2016 will actually ignore larger files, too bad.....
In that case again I would like to request the feature to split up backup files in chunks of filesize x.

[Gostev]

There is the corresponding discussion already > Thoughts on per-VM backup files: per-disk split

[Delo123]

I noticed that, thx! But per vm, per disk doesn't make sense per se... A vm can be 10gb or 10tb. A disk can be 1gb or 20tb... That's why the only logical option os to split backup files in fixed extents which has multiple benefits, not only for dedupe but also for copy jobs etc.... Think about copying a 5tb backup file and then have it failing at 99% or have 50 100gb files, if one of the 50 failes, one can quickly copy it again... I thought of using a vtl, but somehow that's also just a workaround...

[nunciate]

What types of backups do you perform for your weekly fulls if any? Do you do active fulls for a lot of VMs?
We have over 100Tb in backup storage but I use individual volumes between 10 and 20 Tb each. Most have 2012 Dedupe turned on.
I am looking forward to being able to use scale-out repositories once we upgrade to v9.
The majority of my backups utilize a weekly Active Full. I try to keep my VBK files down to between 500Gb to 1Tb in size where possible. Where not I do reverse incremental jobs to non-deduped storage.
I have seen a dramatic amount of space savings with my Active Full backups on 2012 storage with Dedupe.

[mkretzer]

That's why the only logical option os to split backup files in fixed extents which has multiple benefits, not only for dedupe but also for copy jobs etc....

I also think this might be a great idea. I do not know what the effect on veeam dedup would be but this could be a feature especially for 3rd-party dedup like MS dedup!