[V13] Software RAID support

[prodatix]

Regarding software RAID or (even VROC RAID). Please help me understand why this feature was taken out of Rocky Linux for the Veeam Software Appliance. We build backup servers for our customers, and when we install Rocky Linux on our servers to test, the software RAID works great (same for Proxmox). However, in order to sell our backup servers now, we have to add a RAID card, adding several hundred dollars to the price. Just confirming if this will be allowed back in (software or VROC RAID) in a future release? I am looking for some understanding before we start selling our new backup servers with VSA. Thank you for the help (and I apologize if this was answered in another forum that I missed).

[Gostev]

This feature was taken out because while both software RAID and XFS separately by itself are rock solid technologies, we found that their combination does not work reliably under heavy load and results in I/O lockups that can only be fixed with reboot. For this reason we will not support this combination in our software appliances, since they will become the base of many pre-built physical appliance offerings soon just like you said. And once such appliances ship to customers, the only way to fix the issue would be to come back to every single one with a hardware RAID card and wipe/rebuild everything, which is a complete catastrophe from both support costs and customer experience perspectives.

Therefore please do plan on installing hardware RAID controller (ideally with BBWC) in all your VSA-based backup appliances.

[pirx]

That's unusual. The combination of xfs and mdraid is pretty stable if configured properly (stripe size etc).

[Gostev]

In addition to customer environments the issue has been observed by one of our storage partners who [used to] rely on this combination for their appliance. Being storage folks I'm sure they know how to properly configure this

[HannesK]

Hello,
for VROC: it's also too error prone and also more or less deprecated by Intel.

We have seen zero technical issues with BBWC RAID controllers. Stability, simplicity and the options to troubleshoot are important for backup

Best regards
Hannes

[ashleyw]

This sounds a perfect use case for OpenZFS! Myself and other are hoping this will be considered for incorporation into a future release of JeOS/VSA.
My own testing with XFS on top of mdraid did work but significantly added to the complexity of the deployment and didn't deliver the speeds/reliability that we were getting out of OpenZFS striping across multiple sets of vdevs to form a pool.
We use commodity Dell PowerEdge servers crammed with the spinning rust drives at sweet spot pricing (currently using 24x16TB devices) booting off an NVMe drive with OpenZFS right now in Veeam V13.
I believe the Dell server has a variant of the cheap HBA330 card in it (same card I use in my homelab).
You could roll with a hyper-converged architecture where you run KVM hypervisor (to avoid VMware licensing) and PCI pass through the HBA to a rocky VM running OpenZFS, then a separate VM for VSA. In a VMware environment the proxies need to have the vmdks mounted so you'd run the proxies in the main VMware cluster to ensure you could use hotadd.
We used to run our backup servers like this but we had a cluster update with significantly more resources so it was easier and better for us to run an OpenZFS box as tin rather than as a VM.

[prodatix]

This feature was taken out because while both software RAID and XFS separately by itself are rock solid technologies, we found that their combination does not work reliably under heavy load and results in I/O lockups that can only be fixed with reboot. For this reason we will not support this combination in our software appliances, since they will become the base of many pre-built physical appliance offerings soon just like you said. And once such appliances ship to customers, the only way to fix the issue would be to come back to every single one with a hardware RAID card and wipe/rebuild everything, which is a complete catastrophe from both support costs and customer experience perspectives.

Therefore please do plan on installing hardware RAID controller (ideally with BBWC) in all your VSA-based backup appliances.

Thank you for the answer. Is there a preferred or recommended RAID card hardware list? I have the one from Red Hat and Rocky, but am now concerned that the VSA may only like certain brand or versions. Any direction would be appreciated.

[HannesK]

Hello,

Veeam JeOS is 99% plain Rocky Linux and there are no kernel modifications (the other 1% are backports / minor bugfixes in random libraries). There is no brand / version recommendation from our side as we don't modify anything in that area.

This link has a list of known working controllers and covers all major brands. If you use something else successfully, please let us know and we will expand the list.

Best regards
Hannes

[wjordan]

Hi Hannes,

It seems like this issue is closed and will not be addressed, but I would like to mention the scenario I now find myself in. Our Supermicro servers are using VROC without NVMe backplanes for M.2 drives as the boot drives. VROC is not used for actual data storage drives on repositories. As far as my hardware person has mentioned this means that there isn't hardware RAID available to be used.

For the Veeam Backup and Replication appliance (which from my understanding should not be used as a repository), would it be possible to at least allow Intel VROC RAID, or will it always be classed as full software RAID and therefore not supported? Obviously, I have to imagine that potential stability issues would be better than being forced to use non-redundant drives for a VSA with such a hardware setup.

Given I planned to use Linux for my VBR server, is there currently a plan to be able to install VBR software on a non-appliance Linux server? Are there any other alternatives that might better suit this setup that don't involve a Windows VBR server?

Regards,
-Walt

[HannesK]

Hello,
and welcome to the forums.

We explicitly added a statement about Intel VROC as "unsupported" because it doesn't work. That was done based on customer feedback some time ago. As you say that you have hardware, did you test it and does it work?

If it works, can you maybe share a bit more about your hardware and how the RAID is configured? We don't have that hardware and my understanding is, that Intel deprecated it and replaced it with graid. In best case, use the live ISO https://www.veeam.com/kb4761 and collect hardware information with the veeam_hw_collect command and share the output.

If it doesn't work, feel free to repeat the test with a Red Hat / Rocky Linux 9.6 which might have some new drivers. I'm specifically saying 9.6 because we will upgrade the appliance to 9.6 with the next version (13.1).

Given I planned to use Linux for my VBR server, is there currently a plan to be able to install VBR software on a non-appliance Linux server?

no, there are no plans for "standalone installation" on "bring your own operating system" simply because there are some custom backports that do not exist in standard Linux distros (mainly Microsoft dotnet)

Best regards
Hannes

[aakash.rajwani]

Hello,

Does this mean we should not be using Veeam Backup & Replication (Community Edition) to backup VMs hosted on a Hyper-V server built on Intel VROC (NVME SSDs)?

[HannesK]

Hello,
no, because that's a different scenario. Veeam Backup & Replication does not care what hypervisor hardware is used.

This topic is about the Veeam Software / Infrastructure Appliance installed on physical hardware.

Best regards
Hannes