[RELEASE] Managed Hardened Repository ISO by Veeam

[Gostev]

I thought DAS is no different from internal disks (in terms how OS sees both). So if they are connected at the time of ISO installation, they should just become a part of the pool. Or am I completely wrong here?

[Dynamic]

Ah, if this is the case - perfect. Thanks.
Hadn't yet the chance to test it (with additional DAS shelfs), just want to be prepared.

[Gostev]

Let's wait for Hannes to confirm

[Hauke]

Is there any chance to allow the Keyboard from iDRAC to work?
Currently it's blocked, since it connects by USB.

"Device is not authorized for usage" in the console.

https://imgur.com/a/YRA2BFb

[Gostev]

Unfortunately not, USBGuard is a DISA STIG requirement.

[Hauke]

I just discovered that connecting another physical keyboard or using a different USB port for it also renders the system inaccessible. Interesting.

[mdwophil]

Thanks Hannes for your answer regards the Blacksite.

I have another question, in a similar direction like VK-DMX:
A customer has some Windows Repositories with local integrated RAID disks and also some external SAS connected Shelfs (DAS). Currently they are integrated within VBR as SOBR extends.

If this customer is planning to switch from these Windows Repository to the VHR approach, it's not a problem to use the internal Disks for Capacity for sure.
But how we could add the other DAS Shelfs, to also consume them as a SOBR? Is there any option, something on the roadmap or should we go with another Distribution like Ubuntu?

Thanks and Best regards, Markus

What is that external DAS - a JBOD, or an external RAID subsystem like a Dell PowerVault ME5 SAN (or the equivalent from someone else, like an HPE MSA)? And what is the connection from the DAS to the VHR server?

VHR doesn't do software RAID, so a JBOD DAS needs to be connected via a RAID controller somehow - like a Dell PERC RAID controller with external connectors. A RAID subsystem just needs an HBA, but be mindful that VHR also (currently) doesn't do multipath.

Additionally, the VHR ISO doesn't support more than one volume for backup storage: the installation ISO doesn't ask any questions about storage, and the text menu admin interface has no options for that. I believe the installer simply takes all disks (except the smallest, that one becomes the OS disk), bundles them into an LVM volume group, and creates a single logical volume from that. The mount point for that logical volume is /mnt/veeam-repository01.

Caveat: I have done exactly 2 installations of a VHR - a PoC install and a Production install

For your case, I think you'll want to go with something more "generic" like Rocky Linux 9.x, AlmaLinux 9.x, or Ubuntu LTS for your customer. That will give you the option to configure multipath, softraid, or anything else you need. The biggest thing I would suggest is turn of SSH when you don't need it, and turn it on only when you do via the remote KVM feature of your sever (Dell iDRAC, HPE iLO, etc).

[Dynamic]

@mdwophil:
It's a DELL PowerVault MD1420 series shelf, like mentioned SAS attached, an no SAN Device.
This DAS is managed by RAID Controller (PERC H840P) in the server.

An internal RAID (PERC H730P+) controller manages the internal Disks within the server itself.

The customer would be fine, if the VHR create a logical volume from these two different Arrays - or also if it's create 2 seperate volumes.
Thanks

[HannesK]

@Dynamic: yes, external shelves / JBODs are considered as DAS (direct attached storage). It works the same like internal disks where the RAID controller is part of the server and just the disks are external.

@Hauke: all keyboards that were connected during installation should work. Adding new ones afterwards is blocked by USBguard as mentioned before. iDRAC virtual keyboards worked for other customers in the past and I cannot explain why it should not work.

[Hauke]

Dell iDRAC is connecting its keyboard only during an active session, so looks like to get it running it's needed to install that server by using iDRAC.
There is also an option "Keyboard/Mouse Attach State" in iDRAC, "Attached, Detached, Auto". Switching this from (default) Auto to Attached could also help.

Otherwise I had no issues with VHR ISO on this device, PowerEdge R760xs with BOSS for OS. Installation worked perfectly, all needed drivers included.

[HannesK]

yes, the options you mention seem to be a good explanation. thanks for sharing

[mattskalecki]

@Dynamic: yes, external shelves / JBODs are considered as DAS (direct attached storage). It works the same like internal disks where the RAID controller is part of the server and just the disks are external.

@Hauke: all keyboards that were connected during installation should work. Adding new ones afterwards is blocked by USBguard as mentioned before. iDRAC virtual keyboards worked for other customers in the past and I cannot explain why it should not work.

Is it possible to regain keyboard functionality after disconnecting the keyboard? I missed the note on USBguard and want to login to perform a reboot to complete the linux system updates.

[HannesK]

The same keyboard should continue to work. If that's not available anymore, then using repair mode is probably the easiest way if the server does not have a remote console. If you have Linux knowledge, then using "live boot" and configuring USBguard manually could also work (that's an untested scenario)

[AlexandreD]

Hello,

Quick question: I don't see an option in the main menu for configuring SNMP.
Is there another way to configure SNMP or is this not allowed?

Thank you

Alexandre

[Gostev]

Right, Veeam never offered dedicated SNMP settings on individual backup infrastructure components. Have you considered configuring SNMP in VBR server that uses this hardened repository instead?

[fezzebru]

We are a service provider and will have multiple PB's in a multiple location, it concerns me that it seems to create a single large LVM, is there a suggested maximum size limit? That's a rather large fault domain and we have had an issue with a superblock in the past.
A hardened repo server is expected to have 3-4x 400TiB luns so want to make sure it scales. I'd love to chat with you all this directly. Ideally his would be done via boot from SAN and the volumes will be presented from a monolithic SAN.

[HannesK]

Hello,
the suggested side is maximum 1PB which is the RHEL recommended maximum. But there is no hard limit as XFS can be much larger.

What you ask for totally makes sense and I add you +1 for the "manage storage" feature request. As you mention LUNs, I also added you +1 to "support multipathing" request.

"Boot from SAN" would probably need modifications to the installer itself. We have no request for this so far and when I remember troubleshooting boot from SAN back in the time, I'm not convinced that is something Veeam support eventually should troubleshoot.

Best regards
Hannes

[bgenner]

Hi, Where can I download the latest ISO for the LHR? Thanks!

[gordon.svk]

https://www.veeam.com/download_add_pack ... epository/

[Gostev]

The link is actually in the first post of this thread

[mvalpreda]

I know everyone is talking about doing this with Dell/HP/etc and their associated RAID cards. Curious if there is a way to set up software RAID with this ISO for a machine that is JBOD.

[HannesK]

Hello,
no, hardware RAID is a system requirement. Hardware RAID is proven to be fast (write cache) and stable, which is the reason why it's required.

With a JBOD, the ISO would install the operating system on the smallest disk and create one spanned logical volume with LVM without any redundancy with the other disks.

Best regards
Hannes

[Gostev]

mdraid has stability issues with XFS in particular, so it's a no-go

[albertwt]

https://helpcenter.veeam.com/docs/backu ... ml?ver=120

This is great news and truly simplifies the Immutable backup deployment.

Thanks, team, for sharing this ISO file.

May I know if we can install some agents in this Linux server like:

1. System monitoring software
2. EDR or AntiVirus team

[HannesK]

Hello,

Short answer: no.

Longer answer: You should not add any additional (monitoring) software as it increases attack surface and the risk of introducing vulnerabilities. Furthermore, (misconfigured) security software is one of the most popular root causes of support cases around backup reliability.

Better built-in monitoring is something we have in mind to add to this offering in future. Adding 3rd party software is however not supported. Technically "you can" if you have Linux knowledge, but it's impossible to predict what will happen when we post future updates for the ISO. So you're much better off just using a Linux distro of your choice and managing its full lifecycle yourself.

Best regards
Hannes

[keksbert]

Hello Hannes, thank you for this clarification. Could you please mention this statemant explicitly in the beginning of the thread.

Adding 3rd party software is not supported.

[HannesK]

Hello Keksbert,
sure. I put it in red in the "support" section.

Best regards
Hannes

[fezzebru]

Hello,
the suggested side is maximum 1PB which is the RHEL recommended maximum. But there is no hard limit as XFS can be much larger.

What you ask for totally makes sense and I add you +1 for the "manage storage" feature request. As you mention LUNs, I also added you +1 to "support multipathing" request.

"Boot from SAN" would probably need modifications to the installer itself. We have no request for this so far and when I remember troubleshooting boot from SAN back in the time, I'm not convinced that is something Veeam support eventually should troubleshoot.

Best regards
Hannes

Thank you, this checks out nicely, is it 1PB per LUN or total LVM size?

[Gostev]

Resulting XFS volume size, no matter of the storage configuration underneath.

Honestly, most likely it is just some random number set decades ago. I wager someone at Red Hat was tasked to put some "definitely safe" number for Technical Support to use. Or they were simply technically unable to test larger volumes back then, plus "no one will ever need more memory anyway (c)"

Further, I'm pretty sure I heard that we have some customers with volumes larger than 1PB at this time and that they are not seeing any issues. Which makes sense as 1PB is 3 orders of magnitude below architectural maximum for XFS volumes.

[HannesK]

Hello,

I talked to Red Hat and the reason for that 1PB value is "operational reasons".

Yes, one can go higher and there is no technical limit that stops customers at 1PB. We have customers using more than 1PB for XFS (but not many because we also see customer preferring "manageable chunks" as mentioned for example here)

The main challenge would be that there is currently no online repair. Once there is online scrub / repair for XFS, that value would probably be increased for RHEL.

Best regards
Hannes