[RELEASE] Managed Hardened Repository ISO by Veeam

[Gostev]

We're excited to move our managed Hardened Repository project to the next step and upgrade its status from Community Preview to experimentally supported. This means hardened repositories provisioned from this ISO build are now officially supported for use in production environments, and you can open support cases normally in case of any issues (experimental support SLA disclaimer applies only to issues with the ISO Installer and the Configurator Tool specifically).

We would like to thank the community for a huge interest in this project and valuable feedback. The Community Preview has been downloaded over 650 times in less than a month and helped us to identify a few major hardware support issues which have all been fixed in the current build. Further, some requests have already been implemented, such as autogenerated password simplification.

Build number
0.1.17

Build status
Public build for production use, subject to Veeam Experimental Support. This build will be maintained and will support upgrading to later versions.

System Requirements
• Veeam Backup & Replication 12.2 or later.
• Physical server or virtual machine on RHEL compatibility list with at least 2 disks of at least 100GB each.
• Using a VM is not recommended due to vastly increased attack surface (hypervisor) and inability to access backups in case of a hypervisor host outage.

Features and Capabilities
Managed Hardened Repository is delivered as bootable ISO to dramatically simplify provisioning experience while eliminating the need for any Linux expertise. The OS is pre-hardened out of the box with all advanced security settings already applied. Further, on-going management costs are reduced thanks to both hardened repository components as well as the base OS updates provided directly by Veeam.

ISO Installer
• Simplified base OS installer that allows only keyboard language selection, time setting and network configuration.
• Installer requires at least 2 separate disks (at least 100GB each). Smallest disk will be used for the OS with secure partitioning required by DISA STIG applied automatically. All other disks will form a logical volume (LVM formatted with XFS) to use the entire capacity in a single repository.
• Installation to non-UEFI systems (BIOS) is blocked for security considerations.
• Internet access is not required for the installation.

Pre-Hardened Base OS
• DISA STIG security profile is applied to the base OS automatically.
• SSH is disabled by default but can be enabled temporarily using the configuration TUI.
• Time shift protection is enabled by default: the network time service (chrony) is pre-configured to ignore significant time changes during startup.

Hardened Repository Configurator Tool
• Simplified network settings configuration (all settings are available via nmtui)
• HTTP proxy settings (for downloading updates and to access external object storage)
• Change hostname
• Change password for vhradmin user
• Temporarily enable SSH to enable Veeam Backup & Replication to establish the initial connection.
• Update OS and Veeam components (dnf update is leveraged under the hood)
• Reset time shift protection
• Logout, reboot, shutdown
• Automatic logout after 10min

Known Issues and Limitation
The following known issues and limitation apply to the current version and will be addressed in a later release.

ISO Installer
• Re-installing the base OS while keeping backups is not current possible as all disks will be erased during redeployment. The OS repair option will be delivered in the next ISO update. If you need to do this sooner, check with us if we already have a pre-release build.
• Current sudo permissions for the veeamsvc user allow to install additional packages that are signed by a trusted key.
• The Installer does not discard systems with UEFI Secure Boot disabled for now (this is on purpose not to create barriers for testing).
• The Help button does not function. Please see the documentation at https://helpcenter.veeam.com

Hardened Repository Configurator Tool
• The License Agreement wizard and files are incomplete.

Support
This build is now a subject to Veeam Experimental Support. This means this build is officially supported for use in production environments and in case of issues you can open support cases directly with our Customer Support, just with no SLA guarantees. Please do feel free to keep sharing your feedback and experiences directly in this forum thread, but also help us build Support expertise with this new offering by opening support cases for all technical issues.

Documentation & License
Documentation for hardened repository is provided in the Veeam Backup & Replication User's Guide.
No license file is required to install and use this build.

Download
Go to either of the official download locations: in the Customer Portal or trial downloads on our web site.
In there, click Additional Downloads > Extensions and Other > Veeam Hardened Repository ISO

Unofficial server compatibility list
Last updated: Dec 11th 2024

The following is the list of all tape libraries that were reported by your fellow forum members, customers and partners as working fine with the ISO.
If you have successfully deployed the ISO on a server configuration not listed below, please share it in this topic and we will update the list.

Cisco C3260M5 < 2x UCS S3260 Dual Raid Controller based on Broadcom 3316 ROC; 2x Cisco Ethernet Converged NIC XXV710-DA2 (rebranded Intel NIC, dual port with 4x 25Gbit in LACP mode)
Dell R550 < PERC H755 RAID Controller in RAID5, BOSS-S2 with dual 480GB M.2, Broadcom 57414 10/25Gb OCP NIC 3.0
Dell R730xd < PERC H730P mini RAID controller
Dell R740xd < (Debranded EMC DP4400), Perc H730P Mini Embedded 153TB raid 60, Boss-s1 230GB
Dell R750 < PERC H755 RAID Controller RAID6; RAID1 BOSS-S2 (2xSSD 450GB); Broadcom NetXtreme Gigabit, Intel Ethernet 10G 4P X710-T4L-t
Dell R750xs < no further details known
Dell R760 < PERC H965i Front for DATA / BOSS-N1 for OS, Broadcom Adv. Dual 25Gb
Dell R760xd2 < PERC H755 for DATA / BOSS-N1 for OS; Broadcom Adv. Dual 25Gb Ethernet
Dell R760xs < PERC H755 controller 6 x 12TB SATA / BOSS-N1 with 2 x 480GB SSD drives
Dell PE T430 < PERC H730 controller
Dell T640 < PERC H730P controller
HPE Alletra 4140 < HPE SR932i-p Gen 11, NS204i-u Gen 11, Broadcom BCM 57416 10GbE 2p BASE-T OCP3, Melanox MLX MCX623106AS 100GbE 2p QSFP56
HPE DL360G10 < HPE Smart Array P408i-a SR Gen10
HPE DL380G10 < E208i-p and P408i-a SR controllers
HPE DL380G11 < HPE Smart Array P408i-p SR Gen10
HPE ProLiant XL450 Gen10 < PE Smart Array P408i-p SR Gen 10 (for HDD); HPE NS204i-p GEN10+ Boot Controller (for RAID1 NVME boot device)
HPE Apollo 4510 Gen10 < PE Smart Array P408i-p SR Gen 10 (for HDD); HPE NS204i-p GEN10+ Boot Controller (for RAID1 NVME boot device)
Lenovo SR630V3 < LSI MegaRAID Tri-Mode SAS3508
Lenovo SR650V3 < ThinkSystem M.2 NVMe 2-Bay RAID Enablement Kit; ThinkSystem RAID 940-16i 4GB Flash PCIe Gen4 12 Gb Adapter
Supermicro X11SPI-TF < LSI MegaRAID SAS 9361-8i

[mecki]

Only reinstall will erase all disks, future updates are possible without loss of backup data, is that right? So the next installer will get a repair/update option? Is it planed to migrate an existing installation with already installed e.g. Ubuntu to this solution preserving the backup data?
CU
Martin

[HannesK]

Hello,
"install" deletes all disks. "repair" in the next version keeps existing data (it only deletes the smallest disk). Future updates are planned to be possible without losing backup data, yes.

Migrations from other Linux distros (like Ubuntu) might work as long as the smallest disk is used for the operating system and there are separate disks for the backup data. As there are many ways how customers could have set up their environments before, it's hard to say right now whether migrations could become a supported scenario.


Best regards
Hannes

[DaStivi]

hi, is there a way of expanding the data-volume? for example if disks are added (or in my test-szenario, the virtual disk is expanded?)

[HannesK]

Hello,
An option in the Hardened Repository Configurator to add more disks is planned. The thinking is the following: if a customer starts in 2024 with let's say 12 disks, then he will hopefully have enough disk space for the next 12 months before he adds 12 more disks. Just expanding volume size (like making a virtual disk larger) is currently not planned, because Hardened Repository should run on physical machines.

Technically it's possible to expand / adding disks by getting root even today (that is for customers who know Linux and documentation on such operations is not planned because the risk of doing it wrong is too high).

Best regards
Hannes

[gmajestix]

Just to give a feedback. Installation works flawlessly on Cisco UCS S3260 with 56 disks and 2 RAID controllers. We followed the recommended settings in the document https://www.veeam.com/resources/wp-rans ... veeam.html and splited the disks between RAID controllers so that each RAID controller has 28 disks. Installation created an LVM volume over both RAID60 arrays.

[mcz]

As there are many ways how customers could have set up their environments before, it's hard to say right now whether migrations could become a supported scenario.

What if you just added a "test possible migration" to the wizard? Probably the vast majority just had a XFS-setup that could be reused without any issues. If it was not supported, we could come back here, post the config and then you decide if you wannt cover that scenario or not.

[HannesK]

thanks @gmajestix for reporting working hardware. I added a new section in the initial post from Gostev "Hardware configurations that were reported "working fine" by customers and partners" where 9 systems are listed.

@mcz: a "test" option would cost a big amount of resources for something that is not "repair" of the supported ISO. The current plan is to support a few things out of the box and everything else would require some manual work by the customer (I assume people who had such configurations are probably able to do it themselves).

[mecki]

Dell PowerEdge R750 2xXeon Silver 4309Y 2.8GHz, 64GB, Broadcom NetXtreme Gigabit, Intel Ethernet 10G 4P X710-T4L-t, RAID1 BOSS-S2 (2xSSD 450GB), PERC H755 RAID Controller RAID6 18TB, 7x 4TB
working perfect, no performance test yet due to 1Gb Network at the moment.
Thx for the nice work,
Martin

[dloseke]

Dell PowerEdge R550: Single Intel Xeon Gold 5317, 2x 16GB RDIMM, PERC H755 RAID Controller in RAID5 with 8x 12GB 7.2k 12Gb SAS Drives, RAID1 BOSS-S2 with dual 480GB M.2, Broadcom 57414 10/25Gb OCP NIC 3.0

I'm utilizing 40Gb QSFP+ to 4x 10Gb SFP+ breakout DAC's to uplink to our Dell PowerSwitch S4112-T Tor switches but I do plan on probably replacing the breakout cables with 100Gb QSFP28 to 4x 25Gb SFP28 breakout DAC's down the road if I find that my slower breakout cables are no longer necessary for uplinking to our core switch stack. Initial results are quite good though I haven't performed any testing aside from the initial setup and turning it loose on the backup schedule last week.


[HannesK]: thanks, I added the server to the list

[biohazrd]

An issue that I've encountered while deploying this is that over slower links (think WAN/site-to-site VPN) using IPMI/iDRAC/iLO often fails to start the x server before it times out. There is an option you can specify in Grub to increase this timeout:

Code: Select all

inst.xtimeout=1200

I'm going to guess that this method of deployment for users with remote sites will likely be a common occurrence, so upping the timeout by default might be helpful for some users.

[albertwt]

@HannesK & @Gostev,
Thank you for making this extra effort to improve the Immutable ISO installation.

What does it mean by Unofficial server compatibility list?

Will that become Official at some stage to be fully supported or it will stay as it is based on the Community testing report?

[marco.horstmann]

@albertwt We have our Veeam Ready Program and as far as I know we supplied the iso to the server vendors to check and certify their servers with us. This takes a while. The unofficial list is like what community has reported to us as working solution.

The already certified solutions you would find here:
https://www.veeam.com/partners/alliance ... -appliance

[vmtech123]

Currently installing a Dell R660xs with multiple FC and network connections. I'll report back the results after.

[vmtech123]

Currently installing a Dell R660xs with multiple FC and network connections. I'll report back the results after.

Well, It was not as smooth as I had hoped, Network bond set up and working.

After the drives didn't want to allow me to continue, I ended up reformatting and it seemed to work. It's now stuck at Creating XFS on /dev/mapper/datavol-hostname

I'm assuming it has to do with multipathing but can't do anything else at this point. When it booted it did say 5 drives, when there is 1 local NVMe and one FC drive with 4 paths.

[gmajestix]

Per document https://helpcenter.veeam.com/docs/backu ... ml?ver=120:

Limitations

The solution has the following limitations:

Updates to the server occur automatically at 8:00 AM of the configured time zone. However, the server will not automatically reboot. Additionally, there are no notifications about required reboots. To make sure that all updates are installed and applied properly, reboot the hardened repository regularly.

The ISO does not currently have a “repair mode”. If the operating system volume is lost due to a RAID failure or similar issue, you will need to install another supported Linux-based operating system and mount the storage volume or volumes to access your data.

Only hardware RAID controllers are supported.
Only internal / direct attached volumes with a hardware RAID controller with write-back cache are supported.
iSCSI or Fibre Channel LUNs provisioned to the server are not supported.

Wireless network connections are not supported.


I hope in the future also iSCSI and FC is supported.

[chris.childerhose]

I hope FC is supported now since that is typically all we use for storage.

[Outlaw]

Hello

Do you plan to publish ISO based on other Linux? In the commerce, usually buy support for one distro. I am personally interested in using SLES

[HannesK]

sorry for the delay...

@biohazrd: what happened without that parameter? Did it fail over to text mode? If yes... did text mode work? In general, a minimum bandwidth might be something for the system requirements in future because all packages need to be transferred over network anyway (because the setup does not require internet access and will not try to download from the internet)

@albertwt: the user guide has a support statement

All hardware must be on the Red Hat compatibility list.

If this and the other things in the system requirements are met, then it's supported from Veeam side.

@vmtech123: I will come back to you directly. But yes, not supported as of today. Thanks for the feedback!

@chris.childerhose: no. Only internal disks will be supported short term. Once we see how that goes, we can think about expanding support.

@gmajestix: FC is under consideration. iSCSI not because iSCSI is much more complex to configure with initiators, IP addresses, authentication...

@Outlaw: if you are interested in Linux, then the ISO is not for you The ISO is for people who want Veeam "doing everything" without the need to care about the OS themselves. Please use SLES if you prefer SLES. It's supported.

[chris.childerhose]

Thanks, Hannes. I will run this ISO through its paces with a test box with FC connected to see. Otherwise, we will need to rethink our offering of the VHR and how we deploy it.

[Outlaw]

@Outlaw: if you are interested in Linux, then the ISO is not for you The ISO is for people who want Veeam "doing everything" without the need to care about the OS themselves.

It's not the answer
I'm interested HardRepo (Linux or another OS, it doesn't matter). If Linux, then "doing everything" with corporate compliance

[HannesK]

I'm not sure I get the "corporate compliance" part... does your company force software / hardware vendors (storage quorums, firewalls, filers, VMware VCenter, etc.) to only use SLES if they deliver an appliance? If yes, then I guess we cannot help there. Either one maintains SLES oneself, or one can use whatever operating system we use (Rocky Linux in this case).

[jayt]

Installed on Dell R740xd (Debranded EMC DP4400), Boss-s1 230GB, Perc H730P Mini Embedded 153TB raid 60. Working pretty great, because it grabs all storage I had to remove all the extra stuff like NVME, IDSM, etc. Only issue is I may need to gain root so I can update the ca-certs, can't connect to update server at the moment.

[HannesK]: thanks, I added the server to the list

[HannesK]

Hello,
and welcome to the forums. Thanks for reporting back!

Yes, the mix of spinning disk and flash is something we have in mind to address in the Hardened Repository Configurator. For the IDSM... is that SD cards? Because SD-cards should be ignored by the setup and if it's an SD-card, then I would like come back to you directly to figure out why that happened.

Best regards
Hannes

[jayt]

Thanks! Yeah the IDSDM is a dual sd card module in raid 1. I think they put them in all of the dell emc systems, vxrail, data protection, etc. It presents itself as USB storage and it'd kind of annoying, you get the same issue in Windows, tries to loop itself in as a regular HD. I won't be at that site for another week and the module is pulled out so can't verify the issue until then.

For the ca-certificates, guessing my only option is to get root on the device through grub?

[HannesK]

Hello,
sorry, yes, you can get root through grub (in future we plan to have an option with four-eyes authentication directly in Hardened Repository Configurator)

- change the "ro" to "rw" and add "init=/bin/bash"
- set a password for root
- "touch /.autorelabel" to make SELinux happy
- reboot
- relabel happens after reboot and then another reboot happens

What do you mean "can't connect to the update server"? We consider adding fallback to HTTP because it has no security impact because the packages are signed anyway. But for the official update servers, there are official certificates in place, so that should not be a problem. Or do you mirror repository.veeam.com to on-prem and you like to use HTTPS?

Best regards
Hannes

[Outlaw]

I'm not sure I get the "corporate compliance" part... does your company force software / hardware vendors (storage quorums, firewalls, filers, VMware VCenter, etc.) to only use SLES if they deliver an appliance? If yes, then I guess we cannot help there. Either one maintains SLES oneself, or one can use whatever operating system we use (Rocky Linux in this case).

Hello

I just asked if there are plans to publish ISO based on other distros. I need a short answer, yes or no

[HannesK]

Hello,
the answer was / is "no".

I just try to understand why customers ask such things, because we try to build things customers ask for

Best regards
Hannes

[chris.childerhose]

Rocky is what they are doing for this ISO. So short answer is no but future maybe.

[Gostev]

I highly doubt... there's no point, just like VCSA does not come on different flavors of Linux even after many years of its existence