RAID level considerations/rebuild times

[jamcool]

Have a large disk enclosure using to backup to and trying to consider if should do RAID 5, 50, 6, 60 (I would consider RAID 10 but it results it too little usable disk space). Anybody have any experience with/knowledge of RAID 50 and RAID 60 and with spinning disk and was the rebuild any faster than older RAID 5 or RAID 6? If I cannot get faster rebuild than a week, I am consider just re-creating the array and logic disk and do active full backup over weekend and rely on my backup copies if I should need to restore something.

My setup
The enclosure has 35 x 12 TB (7,200 RPM) disk in it. I have done hardware level RAID 5 but switched to RAID 6 so have the 2 disk failure protection. My recent problem is that when I did lose a drive, the rebuild time was 3 weeks and my backups suffered as performance was impacted with some backups not completing. I almost wanted to delete the logical disk and start with fresh backups after the first week having to stop/start jobs running too long or failed backups because performance was so bad. An active full backup takes 3 days (I am doing REFS and Synthetic full backups that take about 12 hours normally). I have matching unit in another datacenter that I do a backup copy to so have redundant backups.

[HannesK]

Hello,
even if RAID50 would be faster than RAID60, I would never go for it for safety reasons. RAID 5 and 6 are not an option for 35 disks.

So from my point of view, the main question is, how large your RAID-sets are and what hardware you are using? 6+2, 8+2? Rebuild time around a week for a system that are under permanent load sound more or less "normal" depending on the hardware you use (I heard similar times from customers).

What kind of RAID-controller and other hardware is it? Really 35 disks (or is it a typo vs. 36)?

Best regards,
Hannes

[Seve CH]

Hi Jamcool,

35 HDs in a single "classic" RAID set is IMHO too much. Not only rebuild time, but random writes should be painful... I would go to smaller ones (8+2 disks RAID6) and concatenate or stripe over them (if you do stripping, you will have less flexibility).

Depending on your array, it might support DDP (Netapp) or ADAPT (Dell). IBM/Lenovo doesn't sell their system anymore (AFAIK). Check if it is supported.

I use Dell and rebuild times are measured on hours, not days or weeks for pools of over 200TB. I won't go for anything else.

Regards

[jamcool]

Thank you both for your replies. It is an HPE D6020 disk enclosure connected to a single server with external disk controller via mini-SAS cables (one per drawer). There are two drawers and each one holds 35 hard drives (12 TB drives). Both drawers are full. We allocated two drives per drawer as hot spares and did RAID 6 on 33 drives. Things have been very good for over 2 years but recently we added in some backup jobs that run hourly and when we lost a drive, the system is too busy to rebuild (taking weeks not days and is slower). One of our backup jobs generates about 160 TB of used space (2 weeks of daily backups with full backups on weekend and incremental in-between) so need big disk and thus why did all 33 drives in a single logical disk and not do RAID 10.

My question is RAID 50 or RAID 60 any faster for rebuilds than older RAID 5 or RAID 6?

[HannesK]

yes, it's faster to do a rebuild of 8 or 10 disks, than for 35 disks.

I could not imagine a configuration of 35 disks in one raidgroup

[Seve CH]

It seems that you have a JBOD connected to a PCI RAID controller. The functionality will depend on that controller.

I do not know exactly how it will behave, but I suppose that only having half of the array degraded/rebuilding will be faster (RAID 60). You won't need to read every single bit of the array to do the rebuild.

In your case, I would go for one of these:

• Smaller RAID6 groups (11 disks: 9 + 2) and stripping at the OS level (storage spools with no redundancy and 2 columns or more). You will have a bit less capacity due to the increased number of parity disks, but rebuilds should be faster (days, not weeks) and you will have almost all IOPS available. This is what I was planning when considering to buy Apollo servers. Please note that this will be quite inflexible for future expansion.

• The intermediate: Several RAID60 groups done via hardware and just concatenate at OS level (storage pools, no redundancy, 1 column). Depending on your I/O pattern, you won't have all IOPS available (idle raidgroups while others are busy).

• RAID60 at the controller level and rebuilds should be half your current rebuild time. I do not know if you can configure the RAID6 part of the RAID60 to do RAID0 over several RAID6 groups.

I think the first 2 options should have the fastest rebuild times.

As HannesK said, I won't go for RAID5x. When a disk fails, there is increased probability to have a second one failing due to the higher rebuild load.

Regards