[PREVIEW] Managed Hardened Repository ISO by Veeam

[HannesK]

I understand the current (primary) focus, but are you confident that local console access only will be acceptable in a large enterprise-style installation?
As an example, and we are a small shop only, one of our HR servers is physically located 45 km away from our office ...

the current assumption is, that enterprise-style installations will have remote consoles for hardware monitoring and remote console access. Does that apply to you, or what type of hardware do you use that does not have a remote console?

[Gostev]

I understand the current (primary) focus, but are you confident that local console access only will be acceptable in a large enterprise-style installation?
As an example, and we are a small shop only, one of our HR servers is physically located 45 km away from our office...

I believe a better direction would be to understand the specific needs for performing SSH logins in the first place, and extend the Configurator Tool functionality accordingly. As there is a history of SSH vulnerabilities which show why disabling SSH completely is a good idea for "hardened" repositories.

[BrianBuchanan]

ah okay... for the "repair mode" we plan to keep whatever exists and mount everything to /mnt/veeamrepository-XX - that should solve that issue.

Feedback on the mount point name. Since it's a mount point for a specific filesystem, I liked OpenMediaVault's decision to use /srv/dev-disk-by-uuid-{uuid} because it's obviously the mountpoint for the filesystem with that UUID (also found in /dev/disk/by-uuid). I don't know the reason they went with /srv/ but I think /mnt/ is just as valid. Over all it makes mounting filesystems consistent and very easy to keep track of.

[chris.childerhose]

Hi,

Is there any plan to include iSCSI LUN support for the repo volume within VHRISO in near future?

We use a lot of FC storage, so I am waiting on the next build to test that with some servers to see, as I know the first time the ISO came out, it did not work well with FC storage.

[chris.childerhose]

1) Thinks you liked

- Straightforward installation - simple interface and options being minimal is nice, making it less prone to mistakes like disk configuration
- Having VMware tools in this build is much nicer as you can properly shutdown or reboot the system without having to log in and use the configurator tool

2) Things you didn’t like

- Nothing, as this ISO made installation a breeze and was very smooth

3) Features you're missing from the Configurator

- Maybe an option to update the VMware Tools installation? I am unsure if the Update All option would do this or if that is just for OS updates

4) Hardware you used to deploy a hardened repository with this ISO:

- Used a VM for this installation (2), which went smoothly. We are waiting on a beta release to test the physical hardware, and we hope the FC storage will work better.

[Gostev]

We use a lot of FC storage, so I am waiting on the next build to test that with some servers to see, as I know the first time the ISO came out, it did not work well with FC storage.

This is not the same ISO as you tested Chris... officially, this public build is the third preview stage already, while as Veeam Vanguard you were in the very first stage (private).

[HannesK]

@BrianBuchanan : I'm not 100% sure I understood the feedback, but I try to answer With when re-installing the operating system with "repair mode", then the installer would not know where something was mounted before. It would just keep everything except the smallest disk and mount that to /mnt/... . My understanding is, that there is no reason against /mnt/.

@chris.childerhose: Is it possible that you mean the Ubuntu-based ISO from last year? If you have seen any FC issues with any of the Rocky-Linux based ISOs (the ones from the last weeks), then please provide more details (see "send feedback" section of the user guide). Even though we plan to keep FC unsupported for the beginning, I expect it to work fine. We don't plan any changes around FC between now and the next beta.

VMware tools: same as any other software, it would be updates automatically (or with the "update all" button).

[chris.childerhose]

Sounds good, Hannes. I will test the current ISO with FC-attached storage and see how it works. I know the previous Ubuntu ISO did not work too well with it. I cannot provide details right now as I need to test with FC storage, which I will do after this week (12.2 upgrades this week ).

[Entropy]

Not in a position to test as we only have our production VHR but will be very interested in this.

We rolled our own bare metal Ubuntu LTS VHR and it would be great if we could "upgrade" that (wipe and replace OS but keep the repo data intact) with this product when it's ready.

OS is on its own disk (hardware RAID 1 controller, Supermicro equivalent of Dell BOSS), repo on Broadcom hardware RAID 60.

How is UPS support? Hardest part of setting up our VHR for this Linux noob was configuring NUT for UPS comms/shutdown/email alerts and Supermicro's Super Doctor for emailing hardware alerts. We'd need to switch to the firewalled IPMI for alerts instead with this ISO as it would lack SuperDoctor support - we leave the Supermicro IPMI disconnected currently as it does not support 2FA. Console access only.

[Gostev]

For hardware monitoring, our plan is to integrate with Redfish API down the road.

[Gostev]

@bcravn thank you, this issue lead us to discovering a bug in our logic which would be impossible to run into in our QA labs... we will provide you an updated ISO soon to confirm that it is indeed resolved.

[HannesK]

@Entropy:
upgrades from Ubuntu: yes, that "upgrade path" is something mentioned earlier in the thread via "repair mode".
UPS support: I put it on the feature request list. How do you connect to the UPS? serial, USB, network?

[LekkerRobert]

Just giving this a shot on some old desktop hardware lying around rather than as a VM. Veeam RockyLinux fails on install with a hang on "multipath settings" for the two disks (sata and nvme). Trying a RockyLinux 9 ISO on the box has no issues whatsoever. I assume there is something in the installation scripting that is causing the issue. I have seen posts on RockyLinux forums regarding similar issue when using Kickstart to build boot ISO's.

[HannesK]

@LekkerRobert: what is the exact error message? Is it "Stopped cancel waiting for multipath siblings of xxx"? We have seen that in earlier builds on one hardware, but not since quite some time. To get around the error, there is probably a workaround, but we want to try to fix the root cause. I will reach out to you via email to get more details.

Yes, kickstart is used in the background.

[aschm590]

Just wanted to say that I am excited about this project and plan to use it when it becomes available for production use.

[flow90]

Hello, I have an issue during the installation on a Dell server that contains an NVMe RAID 1 and a SAS RAID 60.

On the installation destination tab, I get the message "Kickstart insufficient".
I suspect it might be related to my disks that were already partitioned in the previous Ubuntu installation, but even after deleting everything with the gdisk command, the same message is still present and I can't go ahead with this deployment

[HannesK]

Robert: thanks for confirming the error message and the hardware logs. I will come back to you & the forums once I have more details

@flow90: Thanks for raising the issue! The installer should be able to deal with all kinds of pre-installed operating systems. "kickstart insufficient" is something we had in earlier (internal) builds and it should have been fixed. Maybe it's related to NVMe drives, where we work on a fix. Your situation is definitely interesting and I will reach out to you via email for logs upload.

[ccox@griffinnet.com]

HI everybody,

Running into the "Cancel waiting for multipath siblings of sdx" error.
I have tried a few HW things
- firmware updates
- unplug the optical drive no change
- unplug sas cables from raid card gets rid of multipath error but Plymouth never gets past Basic System
- clear raid card config and build new virtual disks gives "Cancel waiting for multipath siblings of sda sdb sdc" and hangs

Testing on a Dell PE T430 on Red Hat Compatibility list 7.0.- 7.x
2x Xenon E5-2603 v3
96GB ECC DDR4
PERC H730
4x SAS 2TB Raid 10
2x SATA 512 Raid 1
2x SATA 256 Raid 1

UEFI set
Secure Boot is set
TPM 2.0 is set

[HannesK]

@ccox@griffinnet.com : thanks for testing. We are looking into this already and I might come back to you via email. I will forward the information you provided internally.

Very likely there is a workaround: if you edit the the "Install Hardened Repository (deletes all data)" boot menu entry and you add "inst.nompath" to the boot options, then it should work.

I will update the thread once we have more information on how to get it working "out of the box"

[ccox@griffinnet.com]

Any guidance on where to place "inst.nompath" I have tried at the end of both the boot options, as a third entry at the end and with and without the '=' and the only change was kernel panic hardlock when I added it to every option

[bluj83]

I am also stuck at the "Cancel waiting for multipath siblings of sda sdb sdc". I could use guidance as well on the inst.nompath

Fully updated on Firmware and BIOS.
Hardware:
Dell T640
1x Intel Xeon Silver 4114
PERC H730P
64GB RAM (2 x 32 GB sticks)
4 x SATA SSD 480GB in Raid 5 (Storage)
4 x SATA HDD 7.2K in Raid 5 (Storage)
2 x SATA SSD 500GB in Raid 1 (OS Drive)

UEFI set
Secure Boot set
No TPM module

[HannesK]

I don't have a 100% confirmation, but the colleague who has seen the error before believes to remember, that he put it in the "linuxefi" line. Like this:



press "e" in the Rocky Linux boot menu and after edit continue to boot with F10

[bluj83]

Darn, that does not resolve it for me. Still stuck.

[HannesK]

okay, I hope to get a build soon where this is fixed and I will send it to you for testing once I have it.

[bluj83]

Awesome!

Thank you!

[HannesK]

Hello,
quick update for everyone. I just sent out new ISO builds to the following testers

@flow90 - Kickstart Inufficient
@LekkerRobert @ccox@griffinnet.com @bluj83 - Cancel waiting for multipath siblings
@bcravn - installation failed without specific error message

I will provide updates as soon as I get news.

Best regards,
Hannes

[Gustav]

the current assumption is, that enterprise-style installations will have remote consoles for hardware monitoring and remote console access. Does that apply to you, or what type of hardware do you use that does not have a remote console?

That's what I guessed. But we have only a straight HPE server with no physical or virtual keyboard. 2FA works so well in other areas (ie. Azure) and also for our current installation.

I believe a better direction would be to understand the specific needs for performing SSH logins in the first place, and extend the Configurator Tool functionality accordingly. As there is a history of SSH vulnerabilities which show why disabling SSH completely is a good idea for "hardened" repositories.

The need would be to free smaller non-enterprise-style-setups like ours from the additional costs for remote keybord add-ons which, by the way, will add additional security concerns.

There are pros and cons for the various methods but, as it is so easy to add 2FA, that would leave the choice for console access to the operator's preferences and options.

[Gostev]

We don't want to provide options that worsen security while having SSH Server up and listening certainly does. This offering is called "hardened repository" for a reason and we want to be able to stand by its name.

Also, keep in mind that those with Linux expertise, which likely applies to you as you need SSH enabled for remote management, have full flexibility to build, secure and manage their own [less] hardened repositories. I mean, there are already over 100'000 hardened repositories like that out there! While this pre-built offering is specifically designed for folks without Linux or security expertise, and as such we don't want to provide them with opportunities to misconfigure.

[ccox@griffinnet.com]

NEW ISO UPDATE
Still getting stuck on multipath with and without inst.nompath.
I will try and find a different raid card(older) and maybe some non-sas drives - can't complain about multipath if there are no multipath disks, right?

[HannesK]

Hello,
I would suggest waiting on the multipath issue. We focus on the T640 for now because it's RHEL 9 certified hardware. Once that is solved, I would assume it solves it also for older hardware.

Best regards,
Hannes