Best Practice for MS Server 2012 DeDup Repo

[conradblack]

Heya,

I'm looking to setup Veeam on a Windows 2012 Server with Windows Server 2012 deduplication on the backup repository volume. I'm curious about the guidelines/limitations when it comes to using instant VM recovery and SureBackup from a dedupe'd volume. Is this supported/recommended by Veeam? If so, are there any things that I would need to be aware of before setting it up?

Thanks

[Shestakov]

Hi,
Windows server 2012 deduplication is supported.
Please review the topic for to achive more information about enabling deduplication on the repository.
Thanks.

[conradblack]

Just to confirm it is supported for both Instant VM Recovery and SureBackup?

[Gostev]

Yes, however performance of those may not be acceptable for obvious reasons.
Generally, we do not recommend deduplication on a primary backup storage.

[Shestakov]

Confirmed.
Note, that to improve performance, you can set OS to dedupe only files that are older than a specified interval.
For instance, if you want to make instant recovery from the last daily backup files, set to dedupe files older than 7 days only.
Thanks.

[onthax]

Been following the changes with server 2012 R2 deduplication over the last year with interest and noted that microsoft have a stack of new best practices for deduplicated backup stores (for DPM)

https://technet.microsoft.com/en-us/lib ... 91438.aspx

Was wondering if there were any Veeam recommendations on wether to use "General Purpose file server" or "Hyper-V" as the -usage type when provisioning a new deduplicated volume?

Also wether to use "LAN Target" for 512KB blocks or "WAN Target" for 256KB Blocks.

MS recommendation is to format the volume with -AllocationUnitSize 64KB

[veremin]

Kindly, search through the posts provided above for more information regarding Windows 2012 deduplicated repository. Thanks.

[lando_uk]

2012 Dedupe didn't worth for us with copy jobs, as once retention is met the vbk transforms caused a NTFS error and the jobs failed, causing the copies to increase and never get pruned.
We went through Dell and Veeam support with this and got nowhere, so now dedupe is disable for copy jobs.

[namiko78]

If it is complaining about file system limitations, you have to format the drive with the /L switch for large file size records. this is a MS recommendation. also keeping the drive defragmented helps avoid this issue.

[lando_uk]

Hi

This issue I has a year ago now (ticket 00533881) - maybe things have changed, I've not really kept track of the issue and just accepted the limitation.

I have no hope of formatting this volume as its full, but maybe we'll buy a new repository soon so I can try again with the /l option. But I'm not holding my breath that it fixes it, I just don't think it works with large (2TB+) vbk's.

[namiko78]

They say that files above 1TB are not good candidates for deduplication, but i've had some success with it, assuming the files are defragged properly. (formatting with /L may not be required, it apparently helps if fragmentation is bad) I'm only doing testing with large files over the past month or 2, and it's crapped out several times, but i started over with /L and a good defragmenter, and it's working now.

Going to give it a few months of telling the copy job to make it's monthly full's and see how things work. My uncompressed VBK is 5.1 TB, and it reports 74% deduplication ratio. Once i get a few more of these from the GFS settings, the savings should go way up.

[onthax]

Kindly, search through the posts provided above for more information regarding Windows 2012 deduplicated repository. Thanks.

Hi Eremin,

I read all of this thread before but it is looking at server 2012, where as there were a number of changes in windows dedup with 2012 R2 so i was after R2 specific information.

also this information requested isn't in the thread so i was hoping it would get attention, instead of in a 9 page thread on 2012.

Are there any veeam recommendations on -usagetype or allocation size?

[Brad.Morris]

I have just built a Windows 2012 R2 Repository server and seeing great results on Windows Dedup. Seems to be a lot quicker doing the deduping. But I have to say I am not backing up to the repo yet. So far it just holds old backup copy data that I cannot mount back into B&R due to the bug.

[veremin]

So far it just holds old backup copy data that I cannot mount back into B&R due to the bug.

Does your backup copy job chain contain GFS restore point, and thus you can't map a job to it? If so, please, be aware that the issue is addressed in the upcoming update 2. Thanks.

[Vitaliy S.]

Are there any veeam recommendations on -usagetype or allocation size?

I don't believe we have any recommendations for this parameter, but let me check it with our QC team. If I have any reply on this, I will post back.

[lightsout]

I'll give my feedback on 2012 dedup best practices.

1. Format the disk using the command line "/L" for "large size file records".
2. Also format using 64KB cluster size.
3. Use Windows 2012 R2. Apply all patches as some rollups have improvements to dedup.
4. Use Active full jobs with incrementals.
5. Turn Veeam's compression off and use the "LAN" block size. Veeam's deduplication can stay on. This gave best overall space savings for me.
6. If possible, spread your active full backups over the entire week. I have a script do it if you're interested.
7. Modify the garbage collection schedule to run daily rather than weekly.
8. Try to keep your VBK files below 1TB in size - Microsoft doesn't official support files bigger than this. Large files take a long time to dedup and will have to be fully reprocessed if the process is interrupted. I've had 4TB VBK process fine, it just take a long time!
9. Use multiple volumes, where possible. Windows dedup is single threaded, but it can process multiple volumes at once. Although bigger volumes mean better dedup ratios!
10. Configure your dedup process to run once a day, and for as long as possible.

[tsightler]

The above recommendations from lightsout are excellent and line up perfectly with my own recommendations for Windows dedupe volumes. Windows dedupe can be quite effective, but you must be very careful that you follow the guidelines set forth above.

[jadams159]

Lightsout, killer post, thanks!

[Mike Resseler]

Lightsout,

Killer recommendations. I have one comment on number 8. I fully agree with the 1TB specification but since 2012 R2 it is actually supported for 2TB . But out of experience, keep it with the 1TB is indeed the best recommendation

Thanks

Mike

[alanbolte]

Mike,

Do you have a source for the 2TB limit? I was never able to find one.

[GlebGo]

Very useful for sure.

But I'm agree with Alan, it's a challenge to find the official statement from MS about 1TB/2TB limits. I will really appreciate if anybody could share the official source link.

[GlebGo]

By the way, I found one on MSDN: https://msdn.microsoft.com/en-us/librar ... s.85).aspx

[alanbolte]

I've seen that before for the 1TB file size limit, I just haven't seen any statement about 2TB, or clarifying whether maximum supported file size depends on OS or whether you're using /L etc. Clearly you get best results by following the best practices in this thread, but official support statements are always nice.

[push3r]

I'll give my feedback on 2012 dedup best practices.

1. Format the disk using the command line "/L" for "large size file records".
2. Also format using 64KB cluster size.
3. Use Windows 2012 R2. Apply all patches as some rollups have improvements to dedup.
4. Use Active full jobs with incrementals.
5. Turn Veeam's compression off and use the "LAN" block size. Veeam's deduplication can stay on. This gave best overall space savings for me.
6. If possible, spread your active full backups over the entire week. I have a script do it if you're interested.
7. Modify the garbage collection schedule to run daily rather than weekly.
8. Try to keep your VBK files below 1TB in size - Microsoft doesn't official support files bigger than this. Large files take a long time to dedup and will have to be fully reprocessed if the process is interrupted. I've had 4TB VBK process fine, it just take a long time!
9. Use multiple volumes, where possible. Windows dedup is single threaded, but it can process multiple volumes at once. Although bigger volumes mean better dedup ratios!
10. Configure your dedup process to run once a day, and for as long as possible.

Thank you very much for this best practice.

One question though, what's the logic for using the Incrementals with weekly Active Full? Wouldn't the Weekly Active Full be resource intensive and time consuming on both the production SAN/NAS and the Veeam repository? Why not just use Reverse Incremental? Also, why not Incrementals with Monthly Active Full?

Thank you.

[veremin]

Why not just use Reverse Incremental?

Because reversed incremental mode produces random I/O (not recommended for dedupe targets) in contrast to sequential write of active full/incremental.

Also, why not Incrementals with Monthly Active Full?

Even though incremental chain will be longer in that case, it might be also a way to go.

Thanks.

[lightsout]

What he said.

Monthly fulls would work well, but it is just a long chain which if corrupted can lose you a lot of backups.

[Shestakov]

Push3r,
The reason why reversed incremental backup method is marked as "slower" one is explained here.
And as for me, you are correct, monthly full is enough in lots of cases. The weakness of longer chains is a bit longer RTO rather than lower reliability. Usually 30-points long chain is pretty reliable, especially if you use Surebackup job.
Thanks.

[push3r]

Thank you guys for the reply.

I get it now with regards to Reverse Incremental having random IO and not good for DeDup. I will use regular incremental instead.

As for the best practice recommendation of Weekly Full, I think I can live with Monthly Full because weekly would requires 4 Full Backup files and the IOPs strain on the production SAN/NAS.

From my testings using this best practice, Windows 2012 R2 DeDup is reporting around 50% DeDup Rate. Really awesome technologies from Microsoft and Veeam with minimal cost and pain.

[push3r]

On a slightly different note, what settings are optimal for pushing Backup Copy Jobs to DeDup Target? I am pushing these to a Co-Location Site with 100Mbps WAN Site-to-Site VPN.

From my testing, I have these Veeam Settings:

For Target:
- "Enable Inline data dedup" box: Checked
- Compression Level: Dedup-friendly

Should I use "None" for Backup Copy Job too?

For Data Transfer:
- Through built-in WAN accelerators.

I am wondering, with 100Mbps WAN connection, Direct would be better? But Bandwidth saving is not as good as through WAN Accelerators? I don't have any Bandwidth Quotas limitation; so, it seems Direct would be better?

With those test settings, I am seeing around 55% DeDup rate. Just wandering if I can even improve on this.

[lightsout]

I turned off compression for my backup copy jobs too, as that goes to a HW dedup device.