[PREVIEW] Managed Hardened Repository ISO by Veeam

[chris.childerhose]

Quick suggestion: Update the OP to let everyone know the new 0.1.16 is available. I tried the download link and got the ISO to test, but many will wonder.

[padair]

Hey Veeam-ers...

I just installed the latest 0.0.16 and the install seemed to go straight forward, reboot, logged in, changed password, enabled ssh.

I then went to the VB&R server we are using to add the new hardened repository and I'm getting in the message window during the "apply" portion of the New Linux Server window. I tried twice and rebooted in-between. Same result.
(All lines in black have green check marks to the left, the ones in red have red X at the left of it.)

Starting infrastructure item update process
Connecting to server via SSH
Checking if Veeam components are supported by Linux server
Resolving backup server certificate's thumbprint Error: Remote failed with message: /opt/veeam/deployment/veeamdeploymentsvc: Failed to get server fingerprint: there is no certificate at /opt/veeam/deployment/certs/cert.p12
Uninstalling Installer
Installing Installer service Error: Error: Execution timeout has been exceeded.
Resolving backup server certificate's thumbprint Error: Remote failed with message: /opt/veeam/deployment/veeamdeploymentsvc: Failed to get server fingerprint: there is no certificate at /opt/veeam/deployment/certs/cert.p12
Uninstalling Installer
Installing Installer service Error: Error: Execution timeout has been exceeded.
Failed to save Linux server: Single-use credentials require Veeam Data Mover service installed.
Infrastructure item save failed Error: Single-use credentials require Veeam Data Mover service installed.

Anyone else come across this using 0.0.16?

[chris.childerhose]

I will be testing this out tomorrow and report back if I have the same issues or not. It tends to go well and smoothly for me. LOL

[HannesK]

@padair: is it possible that we talk about similar network configurations like earlier today (via email)? Did you test a plain Linux (Rocky, Ubuntu, whatever you like) with the same network settings and did you try to add that as Hardened Repository? How does your setup look like? One network card or multiple? The errors do not sound like an ISO issue, but there is one scenario I like to check tomorrow.

[HannesK]

@mcz: thanks for confirming that it worked with the 0.1.16. All drivers are in the Linux kernel. If not, then the system won't be supported (not on Red Hat compatibility list). Adding kernel modules (drivers) is technically possible with Linux, but there are no plans to support such an operation with the Veeam Hardened Repository ISO. The concept is a bit different in Linux than in Windows.

@padair: I tested my last idea I had why it failed for you. Without further details on the setup, I would suggest to wait for the "beta 1" build and then you can open a Veeam support case to check the logs. One of the main goals of the ISO preview is to check whether it installs on various hardware. Adding it to Veeam Backup & Replication is more or less irrelevant because that's "regular Veeam software" that works the same on Red Hat, Rocky, Rocky provided by Veeam (the ISO) etc..

@chris.childerhose: I updated the build number in the initial post, thanks.

In general: adding the ISO to VBR will fail soon as we are preparing the "beta 1" (except 0.1.16 without internet). Also, all updates will fail.

Big thank you for everyone who tested, especially those who tested on hardware!

[chris.childerhose]

No problem Hannes. Just easier when you know there is a new version.

[mcz]

All drivers are in the Linux kernel. If not, then the system won't be supported (not on Red Hat compatibility list)

Two years ago I did setup a hardened repository on a HPE ML 350 server with a 10 Gigabit NIC. I had to install the RHEL drivers manually to have the 10 Gigabit running. That's actually why I was asking. So somehow the driver wasn't automatically installed and most likely not within the kernel?

I mean if there was a way to install it manually and you couldn't, that would be a bit...sad...

Thanks!

[HannesK]

the plan is, that either we integrate them directly in the ISO, or the hardware won't be supported.

Customers should not need to worry about such things. We work on Veeam Ready articles that show which configurations work to avoid customers buying hardware that is unsupported. I guess we will also have an "unofficial compatibility list" sooner or later like we have it for tape devices and object storage. That's why I encourage everyone to post which hardware worked

- Vendor
- Model & Generation
- RAID controller
- Network card model & speed
- Disk configuration (e.g. 2x 200GB RAID 1 for OS and 12x10TB RAID 60 for data)

[mkretzer]

Hello,

three questions:
- Can we recreate the LVM so that striping is used? Our backend storage "likes" the incoming data to be striped to utilize all storage system cores.
- How often are OS updates applied? Is this job aware?
- Can we install our EDR software agent?

Markus

[HannesK]

Hello Markus,

1) technically yes. We cannot stop you from getting root and in the end, nobody will notice. Please keep in mind, that we only support internal disks in the beginning to avoid that our support needs to troubleshoot FibreChannel. Technically I expect it to work fine.
2) the first version applies updates at 8am Mondays of the configured time zone. No reboots are done. There are also no notifications (it's mentioned in the limitations). The next step (no timeline) would be to have a "reboot window". "job awareness" is something I have in mind on the longer run, but that is not planned in detail yet.
3) No. Any installed security software would result in "not supported". We want to avoid having the same problems like on Windows. The majority of escalations I get on my table these days have to do with security software breaking something because not everything was excluded. The worst thing I have seen was 9 security products / components from 5 vendors managed by different teams at the customer

Best regards,
Hannes

[mkretzer]

Hello Hannes,

"Job awareness" would be one of the main benefits of this solution for us.

Markus

[chris.childerhose]

Hello Markus,

1) technically yes. We cannot stop you from getting root and in the end, nobody will notice. Please keep in mind, that we only support internal disks in the beginning to avoid that our support needs to troubleshoot FibreChannel. Technically I expect it to work fine.

Best regards,
Hannes

If you need help to work through the FibreChannel testing let me know as that is all we use typically for storage. I can make recommendations for local storage if that is a hard requirement. I will gladly help work through the FC testing though.

[HannesK]

If you need help to work through the FibreChannel testing let me know as that is all we use typically for storage

if you would post your test results in in this thread, that would be perfect

Server
- Vendor
- Model & Generation
- Fibre Channal / SAS HBA (vendor, number of HBAs, dual port...)
- local RAID controller if used
- local disk configuration if used (e.g. 2x 200GB RAID 1, 480 GB BOSS card etc.)
- Network card model & speed

Storage
- Vendor
- Model & generation
- connection type: SAS / FC
- Connection: direct / switched fabric

[Gostev]

Build 0.1.17 is now available for download, this addresses all remaining known issues.

[chris.childerhose]

Awesome. Did not have the chance to test 16 build so will test this one instead.

[JaySt]

i read the docs, saw this:
"If access to Veeam update servers is blocked, then GPG keys will expire at some point in time. If the
GPG keys expired, then updates won’t be possible anymore and re-installation of the system is
needed."

is this a preview/beta thing or behaviour? can't imagine this being OK.

[HannesK]

Hello
public key and certificate expiration is a normal thing in general. They need to be renewed before they expire or packages that use new signature keys cannot be verified anymore. We talk about years here and the new keys would be delivered via the package manager months ahead.

Technically one could also add new keys after everything expired with a functionality to import new keys.

Now my question: what would be the scenario to not update a Hardened Repository for months or years? That would also mean not having security updates for long time.

Best regards
Hannes

[JaySt]

let's say i deploy the repository in a secure segment of the network. Take measures of some kind to mark it as "very secure". No outbound internet access and the repository is deemed secure untill it isnt, customers call.
Depending how fast things expire (and you mention it could be years, but that actually depends on the point in time things are isntalled/used.), at the time i -do- see the need to update, i just dont want any expiry process to result in the need to reinstall the repository.
Or is that an abnormal stance? To me, it just seems not right in principle to not abe able to import keys and continue.

I totally get the wrong take in not updating for so long, it's not about that.

[DaStivi]

hi, installer of 0.1.17 hangs on "configuring addons" for me...

[HannesK]

@JaySt: sure, the plan is that Hardened Repository does not need direct (or via HTTP poxy) internet access. If someone installs an ISO that is years old with expired keys, that would create issues. Newer ISO versions would have newer signing keys integrated. If you never update, then no new keys are needed. The new keys are only needed, if you decide after months / years that the system should get updates while the keys expired before.

@DaStivi: I remember that issue without ever being able to reproduce it and I see two options: Either you retry installation and see how that goes or you send logs to me and we investigate (troubleshooting -> export logs section in the user guide). Please do also provide information about the hardware configuration and a screenshot.

[DaStivi]

i've deleted all the partitions manually with fdisk before (re-) installation might have been a issue with existing partition information?! after manually cleaning the disks the installer ran through now! (just testing on a vm)

[HannesK]

I think that existing partitions are not the reason. We had bugs around existing partitions only in very early builds. These are all fixed in non-public builds and if you can somehow reproduce it, then we would like to look into it.

[padair]

@padair: is it possible that we talk about similar network configurations like earlier today (via email)? Did you test a plain Linux (Rocky, Ubuntu, whatever you like) with the same network settings and did you try to add that as Hardened Repository? How does your setup look like? One network card or multiple? The errors do not sound like an ISO issue, but there is one scenario I like to check tomorrow.

@HannesK - thinking back now, I recall I did have to play with networking in order to get it to install on the original 0.15 preview. Our network has a separate VLAN for backup purposes. That VLAN also does not have internet access. The VHR has a single IP on the separate VLAN IP and the update process does fail. The V B&R host has multiple network interfaces, one having access to internet thru the regular corp network (and my RDP access to it) and also one for a separate servers VLAN(vmware) and the VLAN for NAS/Backups where the VHR resides.

I will put the VHR .16 install on the regular corp network with internet access, perform an update then put it back to the NAS/backup VLAN and try to re-added it again.

[padair]

@HannesK - My memory was correct. (ODD!) As soon as I placed the .16 VHR ISO install on the regular network with internet access, performed the "Update All" on the GUI menu, then placed it back on the backup VLAN the V B&R server was able to add it as a repository without issue.

I see there is a 0.17 released now. I'll try that right now as well.

[Gostev]

Locking this topic as we're moving to the next stage of this project > [RELEASE] Managed Hardened Repository ISO by Veeam
Huge thanks to everyone who invested their personal time into testing the Community Preview!