[RELEASE] Managed Hardened Repository ISO by Veeam

[fezzebru]

Hello HannesK / Gostev -
We have installed our large storage arrays and are beginning the POC, we would like very much to work together to try to assist you on this for multipath and SAN and large volumes. Next week we will start a 3x SOBR with 400TiB luns (400TiB just seems like a good number?) with 2 servers, so 2.4PiB (2x3x400TB) total, total array is 4.2PiB. We have 1 of them in 2 different locations.
We have POC for 30 days and can do multiple things to your liking.

Is there a way to try this and work together? We are willing to dedicate FTE to project for the betterment of the community and ourselves.

[HannesK]

@cmcawood welcome to the forums! I think there is good news... the next version of the allows to get root with four-eyes approval. Limiting sudo permissions to specific commands has challenges around these commands being mis-used. Instead of giving a "false sense of protection", the "root option" will make clear that one has full permissions now.

@fezzebru: thanks for the offer. Supporting SAN / multipathing storage is out of scope for V13, but I will reach out to you via email to look into some details how multipathing configurations could look like (different vendors have different settings from what I have seen so far)

[cmcawood]

Hi Hannes

That sounds awesome! Now the only question is... when can we get the new release

Looking forward to it!

C

[HannesK]

the Veeam Software Appliance was announced for Q3 and Veeam Data Platform V13 for Q4 at VeeamON

[dloseke]

Can confirm that the VHR ISO was able to be deployed to a Dell PowerEdge T160 with a 480GB RAID 1 BOSS-N1 card for the OS and (3) cabled 8TB drives in RAID 5 using a PERC H755 card for the repo. Processor is an Intel Xeon E-2468 2.6G 8C/16T and 1x 32GB DIMM. Makes for a nice little platform actually.

I will note that I learned something with this deployment in that I used a local keyboard and mouse when deploying the VHR ISO from a thumb drive, but then we placed the server on-site and tried to complete the deployment using the IDRAC console, the virtualized Dell IDRAC keyboard and mouse were blocked from use. I had to redeploy the VHR via the IDRAC to get past this. It's possibly noted somewhere that USBGuard is enabled in the VHR, but I wasn't aware of it until I ran into the issue, and others I've talked to about it weren't aware of this restriction as well. I'm assuming that devices present when the OS is deployed are whitelisted by default, so if deploying locally, I'd suggest noting that any out-of-band management (IDRAC, ILO, CIMC, IPMI, etc) be present and mounted up when deploying the ISO to prevent this issue in the future. Otherwise, I'm not sure if there's any inherent security risk in doing so, but it appears that rule matching can be enabled with USBGuard to allow any keyboard and mouse connected to be utilized.

https://unix.stackexchange.com/question ... -and-mouse
https://access.redhat.com/discussions/2 ... fd599242cd


HannesK: thank you, added to list

[mengl]

I have recently installed a few more repository servers. In the process, I have noticed two further potential improvements.
1. It would be nice to have a quick menu item to display the current network configuration, especially regarding DHCP IP (on a temporary installation network).
2. NTP should be configurable without a functioning network. The use case is to install a server in the lab and use the network configuration of the customer environment (IP + NIC teaming). Since the NTP servers cannot be tested successfully in this way, the setup wizard seems to ignore the settings completely and leaves it at Rocky Linux default. A later configuration via the VHR GUI is unfortunately not possible, as the interface does not offer the option of configuring an NTP pool.

Do you also have plans to support a split network configuration? Eg to have a a 10G adapter for management/updates and a ~100G adapter as direct p2p connection to SAN proxy servers.

By the way: So far no problems on various HPE servers
DL345 Gen11: Smart Array p932, MCX6314LX 10/25G ns204i-u boot device
ML350 Gen10: Smart Array p408 + SAS expander, HP Ethernet 10Gb 2-port 530SFP+ Adapter, ns204i-u boot device
DL380 Gen10: Smart Array p408
Will have a HPE Alletra 4120 next days

HannesK: thank you, added to list

[mengl]

HPE Alletra Storage Server 4120 installed also successful. Same NIC/controller components as the dl345 in my previous post.

At the moment the update repository is returning HTTP 403. Can somebody have a look on it?

Code: Select all

Updating all
Errors during downloading metadata for repository 'baseos':
- Status code: 403 for https://repository.veeam.com/hardened-repository/rocky/9/latest/x86_64/repodata/repomd.xml (IP: 108.138.26.22)

[crojatti]

Same error on "udpate all" here

Code: Select all

Errors during downloading metadata for repository 'baseos':
- Status code: 403 for https://repository.veeam.com/hardened-repository/rocky/9/latest/x86_64/repodata/repomd.xml (IP: 108.138.26.22)

[HannesK]

sorry, I didn't get email notifications on the last posts... thanks for reporting new hardware working and I added them to the list

I can confirm the repository problem and work on it.

@mengl, comment / question on
1. Which details would you want to see and what is the scenario? We show the IP address today at the login screen and also in the network settings
2. the V13 software appliance should allow that, but let me check. Thanks!

[HannesK]

@mengl @crojatti : thanks for reporting the issue. It works now and we will review our monitoring.

[mengl]

@mengl, comment / question on
1. Which details would you want to see and what is the scenario? We show the IP address today at the login screen and also in the network settings

Basically the output of "ip address" and "ip route" would be handy as a quick overview.
For troubleshooting a test button like ping default gateway/dns (as ESXi has) or a "ip neighbor" view could also be helpful. Use case is when you have a large environment with separate network team and some form of network authentication to be able to better locate the source of a misconfiguration.
Alternatively access the same shell as via ssh on the console.

Update is now working fine again!

[HannesK]

thanks, I added the commands to the feature request.

In V13, the workaround will be to "elevate to root" (with four-eyes authorization approval) and then one can get shell access and run every command. But I agree, that having it directly available like in ESXi makes it much more user friendly.

[massimiliano.rizzi]

Hello and good day,

a quick update here in order to report a successful fresh installation using the hardware below:

==================================================
- Lenovo ThinkSystem ST250 V3 server (Machine Type 7DCE) running the most recent firmware updates
- Boot Drive -> ThinkSystem M.2 RAID B540i-2i SATA/NVMe Enablement Kit, using a couple of 240GB M.2 SATA SSDs
- LUN for the Veeam repository -> RAID 5 as the RAID level for 4 disk drives backed by a ThinkSystem RAID 9350-8i 2GB Flash PCIe 12Gb Adapter
- 1x Broadcom 5720 1GbE RJ45 2-Port PCIe Ethernet LOM bonded using 802.3ad bonding
==================================================

As can be seen looking at the the specs, this configuration is targeted towards typical SMB environment.

Thanks!

Massimiliano

MOD: thanks, added to list

[MOBO]

when the repair option arrives , will it be able to upgrader/replace VHR that is installed following the blog posts of Hannesk from 2023 or will i need to start over ?

Hello,
"repair" as it was developed today (internal builds) only works for systems installed with a Hardened Repository ISO. But we are evaluating alternative options that require manual user interaction to allow migrations.

Best regards
Hannes

any news/updates on this ?

[chrisss]

Hi,

what are the plans for the managed hardened repository iso After the release of v13? I read in this thread that there will be a migration path. Will the iso be updated/supported after the release of v13 or will it be discontinued in favor of v13?

Why i am asking, we are still using legacy socket based licenses and i read somewhere in this forum, that the v13 Appliance will require VUL licenses. Does this also applies to the new v13 hardened repository?

[Gostev]

We don't license backup infrastructure components.

[HannesK]

@MOBO: no, there are no news on migrating from other Linux distros to Hardened Repository ISO / V13 software appliance ISO. There are many possible combinations how a customer could have set it up and baking that into a product and supporting that migration path is complex. Technically, one could abuse the repair (or also the install option) by modifying the ISO (the kickstart / .ks file) in a way that it does only format the smallest disk and then mount the partitions manually afterwards. But we cannot support such a hack directly.

[Souko]

Hey guys, I get this message when I use the "Update All" option.

"Repository baseos is listed more than once in the configuration"

Then nothing else happens. Any help would be appreciated. Thank you.

[HannesK]

Hello,
we are working on that message already. Please ignore it for now.

Best regards
Hannes

[1dna]

Hello,

we have experinced that our newly Linux Hardend (by the veeam iso) is crashing or disconencting from the vbr every monday at 8:00 approx without anything in the logs where the server has to be rebooted.

Case: 03436719

we have decided to starte migrate the data back to preserve as much data as remotely possible. ( and evaluvate if we should create our own linux instead ) as i lost trust in this iso (even so, i like the project alot)

is this something that is known ? as its very limited our logging options is not really helping us much.

[HannesK]

Hello,
can you please check the case number? They start with 07...

Yes, updates happen at Monday 08:00 am. But we did not see the issues internally (we tested them in a staging environment before) and need to find the too cause.

That sounds like an issue reported also in other thread we are investigating.

Best regards
Hannes

[HannesK]

update on the "disconnected hardened repositories": the current state of investigations is, that the Network Manager update caused a firewall reload. The Hardened Repository role (the software component that is installed on any Linux machine / on the Hardened Repository ISO) creates firewall rules dynamically during service start. That's why a reboot solves the problem. The bug was fixed in future versions.

[hhls]

Hi,
We have experienced multiple repository disconnect at different customers last week and this Monday. Reboot resolved the connection.
I think the solution would be no automatic update.

[HannesK]

@Souko and all: the "Repository baseos is listed more than once in the configuration" is also solved. You need to hit the "update all" button multiple times and wait a bit (I rebooted one of my machines) and then it shows the "nothing to do" eventually.

@hhls: for Hardened Repository ISO as it is today it will stay "automatic", but the V13 ISO will have improvements with the "Veeam Updater" where one can configure when and how to update what.

[dloseke]

I had the same entry when I tried top perform an Update All before rebooting. Once I rebooted and performed an Update All, it updated normally. Rebooted again for good measure and it's working fine. That said, I do appear to have a duplicate entry in GRUB now but I'm not going to worry much about it.

[mengl]

Basically the output of "ip address" and "ip route" would be handy as a quick overview.

Based on recent experiences with a failed SFP it should also include some counters like CRC errors, link flaps,..

Also an iperf server that one could temporary enable via menu would be very helpful for troubleshooting. Just to have some load to make sure the problem is gone after SFP replacement or if we have to look further at the other side, cables,..