RAID6 or RAID10?

[marama]

Hi!

I know this one has been asked a lot, but I couldn't find a best practice for my case.
So I have a Dell with H700 and 8 HDD slots, + 2 MB SATA connectors. That's supposed to the Backup-Server. I have 6x3TB WD RED (rather slow) drives, 2x4TB RE4 (fast) + Budget for other 2 HDDs of my choice.
Data to be backup-ed: ~2TB, ~80GB Daily change. Backup-Server should have as much GFS backup copies as possible.
I also have an offsite Veeam with 1x4 TB WD RED. This should be for disaster recovery, so basically I just need 1-2 copies.
I'd prefere VmWare ESXi as host.

My first guess was to set up:
- array with RAID1 (2x4TB fast = 4GB): VMFS, for Veeam VM and VMDK backup jobs (incremental/reverse incremental?). Backups of 30 VMs, 2-3 times a day.
- array with RAID6 (6x3TB slow = 12GB), set it up as RDM and use it for backup copies, I should be able to get 6 full backups. Copy jubs running 1xdaily.
- 2xSATA: haven't thought of any use

I could also buy 2 more 3TB drives and make 8x3TB RDM RAID6, and have the VMs and backup jobst on 1x4TB SATA drive (probably silly not to use RAID for this, but since it's just a backup-server VM, I can afford the disaster on this SATA part)

My concernes are: RAID6 gives me lot of diskspace for GFS, but am concerned about the transformation speed. Since it's for backup copies and wouldn't affect the production, I guess I could live with RAID6, am I right? Are we atalking about >12h of transformation? Everybody talks of RAID10 instead of RAID6, but if RAID6 would work for me (don't know what to expect from transformation times), then I'd like to have the extra space.

Any thoughts?

TIA

[Gostev]

Yes. Besides, Backup Copy transformation has 33% lighter IOPS load than "classic" reversed incremental backup, so RAID-6 may be fine (depending on your daily changes amount, of course). I recommend you simply try it out.

[marama]

Hi!

Testing - I would try to avoid if possible, and reach out for other peoples experience.

I also don't have 100% understanding of Backup copy process - am not sure why it needs any transformation at all if it only takes the full backup from backup job and simply copies it 1:1.

So would following make any sense?
- 1x4TB SATA (no RAID) WD RE4 for storing backup jobs (incremental or reverse incremental)
- 8x3TB RAID6 WD RED: GFS copies only

Since I am also doing an offsite backup copy, I could take the risk of having no-RAID for backup jobs. Can I completely avoid transformation on RAID6 (backup copy jobs)? For what I have in mind I could also write a batch file to copy the full backups from SATA to RAID - but would it be slower performance then the Veeam process?

TIA

[veremin]

I also don't have 100% understanding of Backup copy process - am not sure why it needs any transformation at all if it only takes the full backup from backup job and simply copies it 1:1.

In fact, the backup copy job doesn't copy backup file as a whole, but rather synthetically creates required restore points in "target" location from VM data in source backup repositories. Once the specified retention is reached, backup copy job starts transforming on daily basis. So to say, the oldest restore point (full backup, .vbk) will be moved forward once a day.

Thanks.

[averylarry]

RAID 6 should easily, easily do 80Gb/day with little difference from RAID 10. I personally would not recommend choosing a method that uses transform -- just use reverse incremental all the time, unless you have an extremely small backup window outside of a weekend.

[veremin]

I personally would not recommend choosing a method that uses transform -- just use reverse incremental all the time

There must be slight misunderstanding as the discussion is related to backup copy job that has only one method - transforming oldest restore point (.vbk) on regular basis. However, as mentioned above, this activity creates 33% less I/O operations than reversed incremental backup does.

Thanks.

[averylarry]

[cffit]

I'm setting up a new VEEAM server with internal storage. I have 16x1.2TB 10K SAS disks and I'm trying to figure out the following:

RAID 6 or RAID 10? I could use the extra space of RAID 6, but how much better for a backup repository is RAID 10? The writes are better I believe, but with RAID controllers these days is the parity calc that detrimental with RAID 6?

Stripe Size? Beings the backup files are all very large, would I go with the largest strip size for my array as possible? And then match that in Windows when I configure the disk?

[tsightler]

Sequential write performance is generally not a problem with RAID6, actually it will likely be faster than RAID 10. Random write performance is where RAID6 suffers vs RAID10 because of the I/O penalty that comes from stripe rewrite, far more so that the parity calc.

RAID controllers attempt to compensate by having battery backed caches, because then they can "wait" to flush stripes to disk, ideally during otherwise idle times, or at least they can cache multiple blocks written to the same stripe before performing the stripe rewrite, saving IOPs. Many modern RAID controllers can use SSD writeback which can improve writes tremendously as long as you don't overrun the cache.

It's quite difficult to calculate the final difference between RAID10 vs RAID6 because of the significantly varying effectiveness of the various caching techniques employed. Just looking at it from a raw drive perspective, a blended mix of 50/50 read/write mix I/O with large blocks, a load pattern similar to merge/synthetic operations, RAID 6 should have a roughly 60% reduction in total available IOPS vs RAID10, however a strong RAID controller with large caching can narrow this gap significantly, I've seen cases where the overall gap is only 20-25%.

From a real world perspective, the vast majority of clients take a maximum capacity approach to RAID on their backup disk, and thus RAID 6 (or 60) is by far the most common scenario I see. Large stripe sizes usually provide the best performance, with sizes of 128K or 256K being the most common I see. Make sure you have either mulitple jobs running, or you use per-VM chains (the latter is preferred for absolute best performance) to fully utilize the available IOPS.

[cffit]

So given this is for a VEEAM repository, and thus it will be mostly writing large files to disk, RAID6 would provide about the same performance as RAID10 in that use case?

The RAID controller we are using is an onboard in a HP Proliant DL380 G9 P44ar. Here are the performance specs below. Any guidance in addition to what you wrote above with this added info would be appreciated. Thanks for taking the tim to advise.

12Gb/s SAS (1200 MB/s theoretical bandwidth per physical lane) on ProLiant Gen9
 x8 6Gb/s SAS physical links (compatible with 6Gb/s SATA)
 2 GiBytes 72-bit wide DDR3-1866MHz flash backed write cache provides up to 12.8GB/s maximum cache bandwidth
 PCI Express Gen3 x8 link width
 Read ahead caching
 Write-back caching

[VladV]

RAID6 with 16 disk is risky because of slow rebuild rates. Try striping multiple RAID6 like 2 x 8 disk RAID6.

Also, when choosing stripe size, I would go with at least half of the write size Veeam uses which I think depends on how you set up the target datastore. The goal is to have (in case of RAID6 and 8 disks) at least 6 stripes per write.

[cffit]

So is having 2 RAID6 sets together what is called RAID60 then?

This would be local storage on the server, so would I go with the 1024KB option? Then those backups on local storage go to VEEAM cloud connect over a WAN.

My options for "Strip Size / Full Stripe Size" on the array are:

1024 KiB / 8 MiB
512 KiB / 4MiB
256 KiB / 2MiB
128 KiB /1024 KiB
etc

[VladV]

In theory a strip size of 64K with local disk setting (1MB block) should be ideal. But testing it beforehand should be your best bet as the involvement of multiple streams from per VM backup will surely skew the results. Also, I don't know how the controller handles multiple IO generated by the small strip size and it may become a bottleneck.

You can use this guide for testing. https://www.veeam.com/kb2014

[tsightler]

Personally, given that setup, I'd probably just do RAID 6. I personally wouldn't worry too much about RAID rebuild times because your individual drives are still fairly small. In traditional RAID rebuild performance is mostly limited by the throughput of individual disk, and with the drives you have, I'd expect at least 100MB/s as a baseline performance (likely slightly more) which is only ~4 hour rebuild time. Rebuild times can become much bigger factor with 4/6/8TB drives, in my lab I have a RAID6 with 4TB/7.2K RPM SATA drives and it takes around 12 hours to rebuild at about 100MB/s, which I still think is pretty acceptable, but 8 TB drives would start getting close to 24 hours.

A safe baseline is the 64K stripe as it's a well tested and deployed size and should perform well in a large cross-section of use cases, both backup and restore. Sure, you can tweak to get slightly better performance, especially for synthetic/merge operations, but the exact perfect settings will vary based on whether you use the default block/compression or decide to change these options, as well as whether you choose to use per-VM or just try to run one big job in a traditional way. Because of this there's no universal truth for the perfect setup, but the 64K stripe size is a good balance that will work well for pretty much any setting.

[cffit]

Tom - before you mentioned " Large stripe sizes usually provide the best performance, with sizes of 128K or 256K being the most common I see." But you think 64KB now is the way to go? Just making sure I understand this. I did read in another article that since NTFS can only go 64KiB that that's the size to use for the array stripe.

[tsightler]

64K is a very solid default that will work reasonably well for pretty much any setup with more than 8 drives. If I had less drives in a RAID set, I'd definitely want a bigger stripe size. I still consider 64K a large stripe as a lot of lower end cards and NAS style arrays don't even do anything bigger than 64K, which is why I see this a lot as well. For example, I'm pretty sure the Cisco C3160, which is quite a common target in my user base, defaults to 64K.

But many modern arrays default to bigger stripes of 128K/256K and these can lead to better performance in certain circumstances, especially with lots of parallel tasks (also common in my user base, which is customers with 1000s of VMs). That's why it's such a difficult question to answer, it really matters exactly how you are going to setup your jobs. For example if I knew that you were going to use per-VM and wanted to run 8 tasks concurrently and you'd use the default job settings (local disk), I'd almost certainly recommend 256K stripes. If you were going to use traditional backup files, with many VMs in a single job, with only one job running at a time, then the 64K block size is almost certainly better.

In summary, I'd consider 64K to be a conservative value that is almost impossible to go wrong with, while 256K might work better for some cases, but worse for others. What you definitely don't want is some type of setup with 8k stripes (and yes I've seen that, even 4K). If you want the best answer you should do the testing linked in the KB article above with the different stripe sizes, or, even better, follow the guide below.

https://www.veeam.com/wp-veeam-backup-r ... mance.html