How do you monitor your hardened repo?

[YoMarK]

Good day everyone,

The question is is in the title, but some more information about our environment.
So, we've setup our new Veeam hardened repo in "full paranoid" mode. Reason why: we want as little attack surface as possible.

Basically it's a HDS Fibre channel SAN, with a HPE server connected directly using Fibre channel. On the server we've installed Ubuntu 20.4 LTS with XFS reflink and using hardened repo functionality from Veeam. By the way, hardened repo functionality on Linux with XFS reflink works more than great in our initial tests.
SAN has no network connectivity, and the HPE server has no ILO connected.
Linux has firewall enabled, and ONLY ports necessary for Veeam are allowed incoming/outgoing(although it adds firewall rules on it's own, but that's a different story).
So no SSH, no HTTP(out/in) no DNS, no NTP(we will have to trust hardware clock), nothing.
The only attack surface from the network/remotely are the Veeam agents.

So I have a box there that even we cannot access remotely. So if we want to run Ubuntu patches, we have to connect a physical keyboard, then open some firewall ports, and then run the updates.

But how do we monitor hardware failures? A disk, a power supply or something else could fail and of course I want to know it as soon as possible.
SMART doesn't seem to work for SAN volumes(of course a volume consists of multiple disks).

The best plan we have to place a webcam on the SAN and server, and check then regularly for red or orange lights.
But maybe someone has some some other ingenious solution.

My question is: how do you monitor you airgapped stuff or hardened repo?

Thank you in advance!

--Mark

[HannesK]

Hello,
at least one other person is using a webcam (which was mentioned some weeks ago in the forum digest)

as you disconnected everything, SNMP traps / email alarms don't work (which would only need outgoing ports)

Best regards,
Hannes

[mdiver]

IMHO there can not be a fully sufficient solution here. It's like wanting to transport information from the inside to the outside of a singularity - which is physically impossible - as far as we know today...

We usually try to go the oob-management way (ILO, iDRAC, etc.). Therefore one has to make sure the network used for the BMC is fully separated - no routing at all - from the productive and even backup/management network.

All communicating software agents that run on the host itself - even if read only - could in theory have a vulnerability. So they would contradict the solution.
The camera surveillance is possible but to me sounds not very practical.

We are currently working on another path leveraging VBRs stack itself. I'll follow up once we have something reliable here.

[Steve-nIP]

If we're thinking as paranoid as possible here, maybe a small PC directly attached to the iLO dedicated port with an IP KVM (without the USB plugs attached, display out only). Then you could put that IP KVM on the normal network to see the display of the iLO logged in, or have a script probing SNMP on iLO, and have that output on screen.

Either way, that KVM won't be a risk at all if the USB plugs aren't connected, as it won't be able to interact with the machine in any way, and even in the wildest dreams of someone who could send dodgy EDID data to crash a graphics card driver somehow - well, it would only be that monitoring PC affected.

If you then wanted to automate an alert from that image output, that can fairly easily be done with something like an AutoIT script running on a third machine watching the KVM output, monitor for pixel colour change, which can then generate an email or whatever.

[YoMarK]

Tnx @HannesK!

We usually try to go the oob-management way (ILO, iDRAC, etc.). Therefore one has to make sure the network used for the BMC is fully separated - no routing at all - from the productive and even backup/management network.

Thank you for your insight.
This something we already have, but then I have to wonder if a hacker cannot compromise the network stack(routing), and when they do, they have access to ILO(several security issues lately) and it's possibly game over. I do not necessarily manage the network and/or are responsible for keeping OOB management up to date.
I can surely understand why you would go this route though, because it's fairly safe but still manageable.

We are currently working on another path leveraging VBRs stack itself. I'll follow up once we have something reliable here.

Interesting, tnx!

@Steve-nIP: very interesting idea. A small Pc connected only to SAN management and ILO (status pages) with a KVM switch(only connected to VGA) would give me far more information then red/orange lights from a webcam.
However, i'm wondering if I can reliably work around HTTP session timeouts(I used AutoIT scripts in the past ). I will try do make something work or at least test some things.

[stvoglio]

How can I install SNMP for Linux server (Ubuntu 20.04) in hardened mode,
i.e. the server can only use iso and has no internet access.

Thank you.

Stefano

[tkonzal]

Any progress here Veeam?

[mkretzer]

What progress do you need? This is by design.
As Hannes stated above we monitor our hardened repo via webcam which we check two times a week for red LEDs.
Disk space is monitored via Veeam API (you could use Appliances like Sexigraf for that (https://www.sexigraf.fr/).

[Gostev]

Since with V13 release the managed hardened repository ISO v2 is deprecated, we will not be further enhancing it.

However, here's the related thread with some updates as it pertains V13 appliances.
Basically, this request is high on our radar but most likely we won't start from SNMP.

[matteu]

Hello,
I totally agree with YoMark here.

How can we know we have an issue with disk ? power ? or other hardware stuff when using the managed hardened repository ?

Best practices say management interface should be unplug so the only way is to use the OS. However the appliance strictly says nothing should be installed on it.

Just plug the cable 1 / month and check the management web interface and unplug it again ? It's not really good solution :/
Thanks for your answer.

[mkretzer]

Problem is that all these agents introduce new security risks. Also, not all server hardware can be monitored via standard SNMP linux tools.

That was the main reason why we use the camera solution. It provides 100 % an airgap for monitoring of the hardware. Currently i am validating AI tools to analyze the image. We use a Frigate video server which can be connected to home assistant which then can periodically send an image from the camera to the AI tools of your choice (we use Gemini, there is a free tier available).

Here are two results (normal operation and error):
"The visible status lights (like the one at the top left of the Hitachi rack and on the servers to the right) appear to be green, which usually signifies normal operation. Green LED: 2 Yellow LED: 0 Red LED: 0"
"There is a red alarm LED illuminated on the bottom-left of the Hitachi unit, next to the logo. This typically indicates an error or fault condition that requires attention. Green LED: 1 Yellow LED: 0 Red LED: 1"

This result can be parsed again.

[Gostev]

Good point. Although the communication direction will be purely outgoing from software appliance, we will of course not have any optional functionality enabled by default so that customers who built creative air-gapped monitoring solutions do not get their attack surface increased.

[mkretzer]

Thank you Anton. Which hardware can be monitored by your new solution?
Even outgoing communication can cause a risk when the server which the appliance is talking to is compromised, so most likely we will keep being "creative"

[Gostev]

Any hardware that implements Redfish API, so most modern hardware.

[Marijn]

• Please do set up NTP on your repository. Hardware clocks can wander and this will cause timeshift detection errors.
  Also make sure to use NTP servers outside your production domain so that when those get compromised/fail your backup infra will still know what time it is. The wrong time on a host makes MFA not work.....
• Connect the ILO/Idrac/etc and set up a readonly account for a Redfish API capable application. CheckMK works great for us. If you have your oob ports in a seperate network, segmented or behind a firewall, that allows you to really reduce the attack surface whilst still enabling you to do remote management. Of course you have seperate logins that are not used anywhere else in the production environment.

bluntly put: disks will fail. The chance that a hacker exploits a bug in the ILO/iDrac that you kept up-to-date, secured with passwords and in a segmented network is a lot lower.
In this case you need to protect your data against the bigger threat, i.e. disk failures.