Windows Deduplication Repository; SS vs Hardware Raid

[drnash21]

We have been using a Dell R720XD+MD1200 Windows 2012R2 server with deduplication for a backup repository for some time with success. Unfortunately as the amount of systems we are backing up has grown we have outgrown the capacity and performance of the current repository. We have purchased another enclosure and more disks to expand capacity and i wanted to address the raid setup to fix the performance issues. Currently we are using the H710+H810 raid cards in Raid 6. Unfortunately the write speed of this isn't great obviously. This is causing dedup jobs to run 2-3 days and backup windows are starting to be missed without all backups completing. I was planning on moving to a LSI 9207-8i/e hba with Windows storage spaces setup with a pci ssd as cache to improve the performance of the repository.

Before doing so could anyone share there experience in the performance of storage spaces vs hardware raid as a deduplication backup repositiory?

[nmdange]

I can't speak to Parity Storage Spaces performance, I've only used Mirror spaces. I've heard that parity performance is pretty terrible. With Storage Spaces Direct, it's supposed to be better, but that is a very different setup hardware wise.

I use RAID 50 for my backup repositories and I get good performance. The write performance with RAID 50 is going to be better than RAID 6. You can still sustain multiple drive failures with RAID 50, but it depends on which drives fail.

Also you might be better of moving to Windows Server 2016 and using ReFS. Actually deduplication is also a lot faster in WS 2016 too, as long as they fix the data corruption issue

[drnash21]

Thanks for the input. I have the license and was planning on moving to 2016 as well once the data corruption issue was fixed. My understanding was that dedup didn't work on ReFS. Has that changed?

[Gostev]

I can confirm parity storage spaces performance is pretty bad, I would not recommend it for primary backup repository - but rather for secondary one (archival) repositories only.

Dedupe does not work on ReFS, but fast cloning prevents duplication in the first place (aka spaceless full backups).

[Mike Resseler]

I can only agree with Gostev around the ReFS story. It is a great story but MSFT has some fixes to make. But since I am a Monday morning reader of Gostev's digest I have read that MSFT is making progress and getting things fixed so fingers crossed.

For deduplication, if you would decide not to go for S2D and ReFS but want to go back to deduplication in server 2016, there are a lot of differences there also. You can change priorities of the dedup process, assign cores, amount of memory so tweaking it to your needs has really improved and when you have a beefy system you can go far now. It still comes with downsides (and now needs fixes also, something MSFT is also working on) such as the TB limit per file but I would certainly not write it off yet and do some testing with it

My 2 cents
M/

[DonZoomik]

I believe SS Parity would have usable (not great performance) in right conditions. I'm planning a large backup repository based on a SuperMicro 60-disk chassis with 6 NVMe SSDs.
My plan:
* 64TB volumes with maximum column size (17)
* Increasing chunk size (512K, 1M...)
* SSD write cache with 100GB Write Cache per volume (6*2TB SSD with high DWPD such as Intel P3700)
* Setting IsPowerProtected flag to allow relaxed caching and flushing. While it increases corruption risk on power failure, I'd say its a reasonable compromise with dual power rails with independent UPS and a generator.
With 10TB disks and a few SSDs I'd expect total write throughput of at least 3GB/s (conservative) over 10 or so volumes. While not great, it's reasonable and likely to choke SANs and FC/10G. And price-performance-density-power ratio would be pretty good.

Some guy did some testing here http://vault-tec.info/post/129360770011 ... sis-part-1
With some tuning... it could work but only at big scales with you start to max out networking.

[Gostev]

Please do let us know how it goes! We're all ears for sure

[skrause]

One way to alleviate the long deduplication runs is to use smaller volumes in Windows. Windows dedupe is single threaded per windows volume.

Of course, it is a bit messy with internal drives on the same RAID controller but if you are in a situation where currently you have say, one 36TB "E:" disk in Windows, you separate it into three 12TB volumes "E:" "F:" and "G:" your machine will be able to be more efficient on the CPU/RAM utilization. The amount of reads/writes by the controller might still be an issue and dealing with more repositories in Veeam is a bit more fiddly, but you might find better performance.

[Gostev]

Windows dedupe is single threaded per windows volume.

This is no longer the case with Windows Server 2016.