[V13] Software RAID support

[prodatix]

Regarding software RAID or (even VROC RAID). Please help me understand why this feature was taken out of Rocky Linux for the Veeam Software Appliance. We build backup servers for our customers, and when we install Rocky Linux on our servers to test, the software RAID works great (same for Proxmox). However, in order to sell our backup servers now, we have to add a RAID card, adding several hundred dollars to the price. Just confirming if this will be allowed back in (software or VROC RAID) in a future release? I am looking for some understanding before we start selling our new backup servers with VSA. Thank you for the help (and I apologize if this was answered in another forum that I missed).

[Gostev]

This feature was taken out because while both software RAID and XFS separately by itself are rock solid technologies, we found that their combination does not work reliably under heavy load and results in I/O lockups that can only be fixed with reboot. For this reason we will not support this combination in our software appliances, since they will become the base of many pre-built physical appliance offerings soon just like you said. And once such appliances ship to customers, the only way to fix the issue would be to come back to every single one with a hardware RAID card and wipe/rebuild everything, which is a complete catastrophe from both support costs and customer experience perspectives.

Therefore please do plan on installing hardware RAID controller (ideally with BBWC) in all your VSA-based backup appliances.

[pirx]

That's unusual. The combination of xfs and mdraid is pretty stable if configured properly (stripe size etc).

[Gostev]

In addition to customer environments the issue has been observed by one of our storage partners who [used to] rely on this combination for their appliance. Being storage folks I'm sure they know how to properly configure this

[HannesK]

Hello,
for VROC: it's also too error prone and also more or less deprecated by Intel.

We have seen zero technical issues with BBWC RAID controllers. Stability, simplicity and the options to troubleshoot are important for backup

Best regards
Hannes

[ashleyw]

This sounds a perfect use case for OpenZFS! Myself and other are hoping this will be considered for incorporation into a future release of JeOS/VSA.
My own testing with XFS on top of mdraid did work but significantly added to the complexity of the deployment and didn't deliver the speeds/reliability that we were getting out of OpenZFS striping across multiple sets of vdevs to form a pool.
We use commodity Dell PowerEdge servers crammed with the spinning rust drives at sweet spot pricing (currently using 24x16TB devices) booting off an NVMe drive with OpenZFS right now in Veeam V13.
I believe the Dell server has a variant of the cheap HBA330 card in it (same card I use in my homelab).
You could roll with a hyper-converged architecture where you run KVM hypervisor (to avoid VMware licensing) and PCI pass through the HBA to a rocky VM running OpenZFS, then a separate VM for VSA. In a VMware environment the proxies need to have the vmdks mounted so you'd run the proxies in the main VMware cluster to ensure you could use hotadd.
We used to run our backup servers like this but we had a cluster update with significantly more resources so it was easier and better for us to run an OpenZFS box as tin rather than as a VM.

[prodatix]

This feature was taken out because while both software RAID and XFS separately by itself are rock solid technologies, we found that their combination does not work reliably under heavy load and results in I/O lockups that can only be fixed with reboot. For this reason we will not support this combination in our software appliances, since they will become the base of many pre-built physical appliance offerings soon just like you said. And once such appliances ship to customers, the only way to fix the issue would be to come back to every single one with a hardware RAID card and wipe/rebuild everything, which is a complete catastrophe from both support costs and customer experience perspectives.

Therefore please do plan on installing hardware RAID controller (ideally with BBWC) in all your VSA-based backup appliances.

Thank you for the answer. Is there a preferred or recommended RAID card hardware list? I have the one from Red Hat and Rocky, but am now concerned that the VSA may only like certain brand or versions. Any direction would be appreciated.

[HannesK]

Hello,

Veeam JeOS is 99% plain Rocky Linux and there are no kernel modifications (the other 1% are backports / minor bugfixes in random libraries). There is no brand / version recommendation from our side as we don't modify anything in that area.

This link has a list of known working controllers and covers all major brands. If you use something else successfully, please let us know and we will expand the list.

Best regards
Hannes