Best Practice for MS Server 2012 DeDup Repo

[push3r]

I turned off compression for my backup copy jobs too, as that goes to a HW dedup device.

But I am sending my Backup Copy Jobs across the WAN. Any suggestions?

[lightsout]

That is the downside in that it does create more LAN traffic. I do get higher dedup ratios on my storage without it though.

Given you've got WAN acceleration though, I would suspect leaving it off is ok.

[veremin]

Should I use "None" for Backup Copy Job too?

Why not to leave compression as is, but enable "Decompress backup data blocks before storing" option in the setting of the given repository? This way the data will cross the wire in compressed state, and will be decompressed at the repository's site.

[push3r]

v.Ermin,

Thanks for the tip. I forgot about those settings in the Repository.

Is there some sort of best practice pdf from Veeam for stuff like this? Useful information scattered everywhere in the forum.

[Shestakov]

I am wondering, with 100Mbps WAN connection, Direct would be better? But Bandwidth saving is not as good as through WAN Accelerators? I don't have any Bandwidth Quotas limitation; so, it seems Direct would be better?

If you have Enterprise Plus Edition, I would suggest to use WAN Acceleration, it can give up 50x traffic savings, what`s especially useful for 100 Mbps bandwidth and big data transmission.
Thanks.

[push3r]

Yes, I have Ent Plus License. I did some more reading and saw that 100Mbps and below should use WAN Acceleration. Thanks.

[conradblack]

Heya,

I have some inquiries about using Windows Server 2012 deduplication alongside Veeam.

I've read Veeam documentation, and I am under the impression that it is recommended to not use reverse incremental/synthetic fulls with Windows deduplication. I'm curious about forever forward incremental. When the max amount of restore points is met, the oldest increment gets merged into the full vbk.. isn't this the same process as making a synthetic full? Is this process supported with dedupe volumes?

Also, using virtual full synthetic to tape, would I still be placed with these limitations? I'm assuming as much, since the files that are being synthesized into a full backup to tape are being read from a dedupe volume. I ran a few tests with synthetic full from a dedupe volume, while having "Transform previous backup chains into rollbacks" selected, and the performance was terrible. I'm guessing the performance will not improve that much without transforming the previous backup chain?

What is the best practice with Server 2012 dedupe and Veeam? I'd really like to use synthetic fulls in some capacity with Windows dedupe, whether it be reverse incremental or periodic synthetic fulls with forward incremental.

Thanks

[foggy]

Hello, I've merged your post into existing discussion with some hints/recommendation on using Windows deduplication on the backup repository. However, you definitely have to expect performance degradation during synthetic activity (think random I/O) on any deduplicating storage.

[conradblack]

Hello, I've merged your post into existing discussion with some hints/recommendation on using Windows deduplication on the backup repository. However, you definitely have to expect performance degradation during synthetic activity (think random I/O) on any deduplicating storage.

Thanks.

Does this apply as well to forever forward incremental when the oldest increment is merged into the full backup file?

[foggy]

Yes, though forever forward is less I/O intensive than reverse incremental, it still does synthetic activity during the merge phase.

[conradblack]

Yes, though forever forward is less I/O intensive than reverse incremental, it still does synthetic activity during the merge phase.

Thanks, that clears things up.

I should have asked last post, but would virtual full synthetic to tape be similar in I/O performance to regular synthetic fulls?

[veremin]

It should be similar in terms of I/O. Though, I don't really think you have to worry about that, because vsb is just a tiny file consisting of pointers.

[namiko78]

Should i create my RAID with 64k block sizes as well, same as the file system? (default is 256k)

[foggy]

Steven, please review this discussion, should help.

[bjdboyer]

Any updates on "lightsout" procedure with V9? Can you still use windows dedup with scale-out repositories?

Bill

[lightsout]

I'm upgrading too, but at least in concept, I would say yes. Bitlooker should be turned on though!

[foggy]

Currently there're no contraindications to using Windows deduplication on scale-out repository extents.

[lightsout]

I guess the other interesting one is per-VM VBKs. I don't see this increasing the de-dup ratio, but more smaller VBK is probably a better situation than fewer very large VBKs due to the processing time for each time.

[asris]

We're currently implementing v9. We have a Windows2KR2 proxy with a repository on a 2TB volume that is dedupe enabled.

When I create a backup job and point it to that repository can I just use the defaults settings? Or is there a preferred configuration for that type of repository?

[foggy]

You can find settings recommended for Windows dedupe repository in this thread.

[asris]

Thanks Foggy. I found the KB2023 which recommends formatting the volume with the /L switch and allocation size of 64K which is similar to what i've seen on this thread. I also don't see any mention of Storage compatibility settings or Backup job settings. For me, I only select to "decompress before storing". It would be great for Veeam to optimize the setting choices when adding a Windows dedupe repository.

Anyhow, our setup consists of backup jobs storing on a 2TB Windows volume with 7-14 restore points. I have a backup copy job that copies those backups to an onsite Data Domain. I have a another backup copy job that copies the backups to an offsite Data Domain.

So is it even worth it to enable the Deduplication on the Windows volume? My only concern is its affect on Instant VM recovery performance.

[foggy]

I also don't see any mention of Storage compatibility settings or Backup job settings.

You can find other recommended settings in this thread as well, for example, in this post.

So is it even worth it to enable the Deduplication on the Windows volume? My only concern is its affect on Instant VM recovery performance.

Indeed, having deduplication on the primary storage does not comply with our general recommendations.

[Choodee]

Great thread. What is the recommended setting for creating a new backup repository on a windows server 2012 dedup volume? Should the options below be checked?

Align backup file data blocks
Decompress Backup Data Block before storing

Thanks

[Gostev]

No, since Windows uses variable block size for dedupe.
Yes, otherwise you will get no dedupe at all.

[Choodee]

I just wanted to summarize what I found useful in this thread for others who are combing through. Thank you everyone for their contribution.

I read through the whole thing and summarized to the following important points.
veeam-backup-replication-f2/best-practi ... 2-165.html

Window Server 2012 Dedup Best Practice:

Format the Windows Server Volume by using 'Large File Records'
https://www.veeam.com/kb2023

1. It is important to format the volume with large NTFS File Record Segment (FRS) (4096 bytes instead of 1024 by default) as you could face NTFS limitation errors in future. You could verify FRS size with the following command:
fsutil fsinfo ntfsinfo <volume pathname>

Command to reformat NTFS volume with larger FRS (/L):
format <volume pathname> /L

2. Use maximum NTFS allocation unit size(cluster size) of 64Kb.
Considering option mentioned in pt. 1, two commands should be united into one:
format <volume pathname> /A:64K /L

3. Avoid growth of files more than 1TB;
4. For Backup jobs preferred backup method is Forward Incremental with Active Full backups enabled. (Forever Incremental will also work since synthetic fulls are automatically created when the last retention point is met and rolled up).

Veeam Settings:

Backup Repository:
No - Align backup file data blocks
Yes - Decompress Backup Data Block before storing

Backup/Backup Copy/Replication Jobs:
Enable inline data deduplication - Yes
Exclude swap file blocks from processing - Yes
​Compression Level - None
Storage Optimization - LAN target

Veeam Notes:
Format the disk using the command line "/L" for "large size file records".
Also format using 64KB cluster size.
Use Windows 2012 R2. Apply all patches as some rollups have improvements to dedup.
Use Active full jobs with Incrementals or Forever Incrementals.
Turn Veeam's compression to "None" and use the "LAN target" block size. Veeam's inline deduplication can stay on.
Windows Server Dedup Notes:
Modify the garbage collection schedule to run daily rather than weekly.
Try to keep your VBK files below 1TB in size - Microsoft doesn't official support files bigger than this. Large files take a long time to dedup and will have to be fully reprocessed if the process is interrupted.
Use multiple volumes, where possible. Windows dedup is single threaded, but it can process multiple volumes at once. Although bigger volumes mean better dedup ratios!
Configure your dedup process to run once a day, and for as long as possible.

[HussainMahfood]

Hi,

I have Dedupe-friendly compression in a daily job backup setting to MS WIndows 2012 R2 De-duplication as best practice.

My question if I change this to High compression would I still benefit from deduplcation or not?



Note: I know that High compression will effect 10x higher CPU usage which I do not care about

[foggy]

The higher compression level, the lower deduplication ratio you will get for the backup files.

[veremin]

The deduplication will still be there, though its rates are likely to be affected negatively. So, I don't think it's worth it. Thanks.

[lightsout]

Agreed. I think Windows' dedup does better without any compression from my testing.

[HussainMahfood]

As I would need to offload these files later to a tape backup.

if these files are uncompressed in the deduplication then will be offloaded uncompressed to backup to tape job