[PREVIEW] Managed Hardened Repository ISO by Veeam

[Gostev]

Update 2024-10-29:Locking this topic as we're moving to the next stage of this project.
Huge thanks to everyone who invested their personal time into testing the Community Preview!

Update 2024-10-24: as we are preparing for release, adding the preview versions to Veeam Backup & Replication will start to fail. You can continue testing whether it works to install the ISO, but errors around adding the installed system to the Veeam Console is expected to fail.

We're extremely excited to present you Community Preview of the new delivery method for Veeam Hardened Repository in a form of bootable ISO which dramatically simplifies the provisioning experience while completely eliminating the need for any Linux expertise. Further, deploying hardened repositories from this ISO reduces on-going management costs thanks to both the hardened repository components as wells as the base Linux OS updates provided directly by Veeam.

The goal of this Community Preview is to collect initial feedback from our users and partners as we're considering making this an officially supported offering next month, therefore your engagement is highly appreciated and will directly impact our decision to move forward. Just share your thoughts and feedback with us directly in this thread.

In addition, the feedback on this Community Preview will directly impact available delivery options for VBR on Linux in V13 and whether we will provide the ISO delivery method in particular. Which makes this the most important technology preview in the history of Veeam!

Build number
0.1.17 (bugfix)
0.1.16 (bugfix)
0.1.15 (initial release)

Build status
• It is a Technology Preview: We do not recommend using this in production environment because the preview builds are not supported by our Customer Support.
• Upgrading to the officially supported version (available later this year depending on feedback on this preview) will require redeploying your repository from a new ISO.
• The base OS update functionality in the Hardened Repository Configurator tool will therefore remain functional for Technology Preview installs only until the supported version is released.

Features and Capabilities
The following functionality is available for testing in the Community Preview build.

ISO Installer
• Simplified base OS installer that allows only keyboard language selection, time setting and network configuration.
• Installer requires at least 2 separate disks (at least 100GB each). Smallest disk will be used for the OS with secure partitioning required by DISA STIG applied automatically. All other disks will form a logical volume (LVM formatted with XFS) to use the entire capacity in a single repository.

Pre-Hardened Base OS
• DISA STIG security profile is applied to the base OS automatically
• SSH is disabled by default
• Time shift protection is enabled by default: the network time service (chrony) is pre-configured to ignore significant time changes during startup.

Hardened Repository Configurator Tool
• Simplified network settings configuration (all settings are available via nmtui)
• HTTP proxy settings (for downloading updates and to access external object storage)
• Change hostname
• Change password for vhradmin user
• Temporarily enable SSH (to allow Veeam Backup & Replication to establish initial connection)
• Update OS and Veeam components (dnf update is leveraged under the hood)
• Reset time shift proteciton
• Logout, reboot, shutdown
• Automatic logout after 10min

System Requirements
• Veeam Backup & Replication 12.2 or later.
• Physical or virtual machine (physical recommended for reduced attack surface)
• At least 2 disks of at least 100GB each.

Known Issues and Limitation
The following known issues and limitation apply to the Community Preview build only and will be addressed in the final release.

ISO Installer
• Current sudo permissions for the veeamsvc user allow to install additional packages that are signed by a trusted key.
• The Installer does not discard systems with UEFI Secure Boot disabled (this is on purpose not to create barriers for preview testing)
• The Help button does not function. Please use the included manual.

Hardened Repository Configurator Tool
• The License Agreement wizard and files are incomplete.
• Some texts and error messages are not final.

Support
Please do NOT contact Veeam Customer Support for assistance with this Community Preview. Instead, share your feedback and experiences directly in this thread. If you experience technical issues, please create a dedicated topic for each issue with the [VHRISO] tag at the start of the topic name. Be sure to include log files using a file sharing service of your choice (see the Troubleshooting section of the user manual on how to get logs).

Documentation & License
The user manual and step-by-step installation and first logon video are included in the download location (see below).
No license file is required to install and use the Community Preview build.

Download
1. Go to https://www.veeam.com/beta/vhriso
2. Log in with your Veeam credentials
3. Additional password for the download is VHRISO

[HannesK]

As it comes to feedback, we are interested in the following information in particular:

1) Thinks you liked

2) Things you didn’t like

3) Features you're missing from the Configurator

4) Hardware you used to deploy a hardened repository with this ISO:
- Vendor
- Model & Generation
- RAID controller
- Network card model & speed
- Disk configuration (e.g. 2x 200GB RAID 1 for OS and 12x10TB RAID 60 for data)

5) If the installation failed, please provide logfiles via a file sharing service of your convenience. See the troubleshooting section of the user manual on how to get logs.

[javichumellamo]

Hi
1) Thinks you liked
Simple and straightforward process to deploy

2) Things you didn’t like
missing Admin menu to mantain appliance once deployed (e.g. adding new LUNs to stg pool without need of advanced Linux skills)

3) Features you're missing from the Configurator
'Advanced mode' for LVM setup (option to configure SSD cache)


4) Hardware you used to deploy a hardened repository with this ISO:
Virtual machine

5) If the installation failed, please provide logfiles via a file sharing service of your convenience. See the troubleshooting section of the user manual on how to get logs.

[HannesK]

Hello,
thank you for the feedback!

Adding storage is something we have in mind and it's the main reason why LVM is used. I also noted the SSD cache as feature request

Best regards,
Hannes

[bcravn]

Moderator edit: this system works fine with a later version of the ISO

4) Hardware you used to deploy a hardened repository with this ISO:
- Vendor : HPE
- Model & Generation: ProLiant XL450 Gen10 / HPE Apollo 4510 Gen10
- RAID controller: HPE Smart Array P408i-p SR Gen 10 ( for mechanical drives) / HPE NS204i-p GEN10+ Boot Controller (for raid1 NVME boot device)
- Network card model & speed: 1G for the test
- Disk configuration : HPE boot device 2 x nvme | 38 x 10 TB SAS-HDD for storage ~ 300TiB

5) If the installation failed, please provide logfiles via a file sharing service of your convenience. See the troubleshooting section of the user manual on how to get logs.
Install failed, logfiles provided. Properly due to OS would be installed on NVME device

Logfiles and a few screenshots has been uploaded to:
https://cnm.sharefile.com/public/share/ ... 8221907e42
We did get the iso working in a VM, and that went just fine. But we often use hardware like the ProLiant XL450 Gen10 / HPE Apollo 4510 Gen10 .

[Adrian1980]

Hello,

if i locked me out by 3 times failed login, how I can reset the password?
The link to the KB in the PDF points to an 404.

[HannesK]

@bcravn: thanks for reporting and uploading the logs. We look into and if there are more questions, I might come back to you directly.

@Adrian1980: I will send you the draft of the KB article directly and work that we get the KB article online

[tpayton]

Veeam KB4663:How to Reset Hardened Repository ISO Appliance Account Lockout has been published.

[efd121]

First install using Dell R730xd with PERC H730P Mini raid controller. Three logical volumes, vDisk0 mirrored 120GB SSD. vDisk1 and vDisk2 configured as raid-6 with twelve 1.6TB drives each. I need to run through the process again to see if I can get an Active/Backup bond setup with two NICs.
I was expecting to see two data volumes after the install, but they were combined into a single 32TB volume after the system was added to Veeam. Is this expected behavior or will the new ISO always combine the larger volumes?
Is the data evenly split between both logical volumes or is it written to just one until its full before writing to the other?
Dave

[HannesK]

Hello David,
thanks for testing!

For networking: bonding can also be configured in the advanced networking section in the Hardened Repository Configurator. A re-installation should not be needed.

Yes, this is documented in the first post of this topic: all volumes except the operating system volume are combined to one large logical volume. That makes repository management easier. It also avoids situations where customers configured multiple scale out backup repository extents on the same machine.

or is it written to just one until its full before writing to the other

this is what we do today because we format with help of the installer. The plan is to add striping to distribute load and improve performance. The only downside would be, that a failure of one RAID would lead to the loss of all data. RAID6 vs RAID60.

Best regards,
Hannes

[arngrimurm]

I guess this is going to be licensed in the future?

[HannesK]

We don't license our backup infrastructure components, you get to use as many as you need. We only license protected workloads. There are no plans for now to change that.

[efd121]

The plan is to add striping to distribute load and improve performance. The only downside would be, that a failure of one RAID would lead to the loss of all data. RAID6 vs RAID60.

I like this especially with new repositories but the reason I’m asking is when it comes time to upgrade my existing repos. Using the old ISO I have 8 systems (2 SOBRs) with 2 106TB volumes each and was hoping once the repair option was available, I would be able to reload the OS leaving the data volumes alone. If that won’t be an option, I will have to evacuate the extents and totally rebuild each one.

[HannesK]

ah okay... for the "repair mode" we plan to keep whatever exists and mount everything to /mnt/veeamrepository-XX - that should solve that issue.

[padair]

For purposes of testing I tried to install this ISO in 2 virtual environments.

When installing in VMware it seemed to go ok and let you get past the disk selections "done" part.

When installing in Proxmox it seems to crash in UEFI mode always. Never gets to the setup screens. Using SeaBios it loads but has a few different options at the start and then doesn't move past the disk selections and you have to hit "return" as the done does nothing. At the main page the "begin installation" button is always grayed out.

Any advice on how to test this ISO virtually on Proxmox?

UPDATE! You can ignore this post... unless you are specifically looking to test with Proxmox. I got it working using the following.
OVMF Bios
Add to SCSI disks. I used first SCSI:0 as 128GB and SCSI:1 as 2TB.
Add a default EFI disk
Machine to be i440fx
SCSI controller to be VMware PVSCSI

Boots as it does on VMware now where it presents only the 1 install option and does not display the warnings about preview and pre-selects the kb language etc.

[efd121]

One anomily I ran into is configuring a bond for the networking during the initial installation. It doesn’t present the NICs to add to the bond unti l click on the question mark of the Choose a Connection Type dialog box. I tried this on multiple servers and its inconsistent if it displays the NICs. Before I realized clicking on the question mark worked, I was able to configure the bond post install. Its not a big deal either way but I wanted to pass it along in case anyone else runs into this issue.
Dave

[efd121]

ah okay... for the "repair mode" we plan to keep whatever exists and mount everything to /mnt/veeamrepository-XX - that should solve that issue.

When the repair option is added I would like to see it as the default when booting from the ISO instead of the full wipe like the previous ISO.
I came very close to wiping one of my repositories when I needed to reinstall and almost missed the initial screen.
Dave

[HannesK]

Hello,
@padair: first, welcome to the forums. Second: UEFI (Secure boot) is a requirement. If you select RHEL in VMware, that is set up automatically. With BIOS, it does not work (it was designed to only work with UEFI to be able to enforce Secure Boot later. The documentation has the scenario covered). My Proxmox has VirtIO SCSI single controller (not VMware PVSCSI). I have this setup:



@efd121 : Yes, the network cards visibility in the BOND configuration is a thing in RHEL & Rocky in general. I have seen that also when creating the Red Hat 9 & Rocky 9 installation blog posts. In the video recording, it was the first time since five times or so that the NICs were available. I did not find out the trick with the question mark though. Thanks for sharing. In the past I just clicked into the wizard again and with the second try usually the network cards show up.

Repair mode / Installation: in the current ISO, it wipes nothing until one clicks "begin installation". But yes, it does not hurt to make "repair" the default option.

Best regards
Hannes

[padair]

Thanks @HannesK

I've been able to get it installed on Proxmox ok now.

I also am reviewing it on VMware more actively now and the hard repository install went like clockwork, I got logged in, completed the rest of the basic setup and turned on the ssh to enable the veeamsvc account. I then logged into a Win Server 2022 with Veeam B&R 12.2 installed. When I go to add the linux hardened repository this is the errors I receive. Not sure what the issue is.

The message box has 7 lines in it. The first 3 have green check marks to the left and say
Starting infrastructure item update process
Connecting server via SSH
Checking if Veeam components are supported by Linux server

Then the next 4 lines have red X's beside them and say
Installing Installer service Error: Error: Rocky Linux powered by Veeam - BaseOS...
Installing Installer service Error: Error: Rocky Linux powered by Veeam - BaseOS...
Failed to save Linux server: Single-use credentials require Veeam Data Mover service installed.
Infrastructure item save failed Error: Single-use credentials require Veeam Data Mover service installed.

I checked and the data mover service is installed and running on the V B&R server. This server has other linux hardened repositories already added but they were built manually with Ubuntu.

Any advice?

thanks, Paul

[padair]

Ignore the above. Figure it out. We use different subnets for NAS / backup machines. Even though the V B&R has multiple IPs and access to regular network and the NAS vlan the VHR didn't like existing on that network alone. We could test the login with putty and it would work but the V B&R install kept failing. Once we moved it all to the main lan everything worked fine.

[sheru]

Hi,

Is there any plan to include iSCSI LUN support for the repo volume within VHRISO in near future?

[Andreas Neufert]

No, currently not in this first release.

In general mounted external storage has the disadvantage that we can not ensure the immutability as someone can just go to the storage and delete the volume.

[HannesK]

Hello,
@padair : I cannot reproduce the problem and wrote an email to you to get more details.

@sheru: welcome to the forums! No, iSCSI is not planned, because it's much more complex than SAS or FibreChannel. Direct connected SAS & FC should "just work", while iSCSI needs separate network configuration, authentication etc.. For now, only internal disks are supported, to avoid Veeam support having to troubleshoot external storage.

Best regards,
Hannes

[Gustav]

This is great news making life much easier for Linux noobs like me.

However, do you plan to add 2FA?
If not:

• Why not?

• Will there be an option for (manually) installing this?

At least for a normal Ubuntu install, this is not at all difficult:
https://github.com/GustavBrock/Veeam.Li ... entication

[HannesK]

Hello,
the current focus is to remove SSH logins completely, which only leaves local console access.

Manual installation of packages is prohibited because otherwise it becomes complicated to support.

Best regards,
Hannes

[hhls]

Will it be possible to upgrade to the official/supported version later?

[HannesK]

No upgrades will be provided from this build. The goal for now is to get feedback about hardware compatibility and usability. Later beta versions will very likely be upgradeable.

[lampshade]

Will something like this ever be officially supported by support down the road? It's a great option for end users.

[Gustav]

the current focus is to remove SSH logins completely, which only leaves local console access.

Manual installation of packages is prohibited because otherwise it becomes complicated to support.

I understand the current (primary) focus, but are you confident that local console access only will be acceptable in a large enterprise-style installation?
As an example, and we are a small shop only, one of our HR servers is physically located 45 km away from our office ...

[Gostev]

Will something like this ever be officially supported by support down the road? It's a great option for end users.

Yes. The plan is to make this a supported offering under our experimental support policy as the next step, potentially later this month already - but this will depend on the level of interest to the preview (number of downloads) and overall feedback on the Community Preview (like number of installation issues reported with different hardware).

As soon as we declare experimental support, you will be able to open support cases and get assistance on this offering directly from our Customer Support.