How do you monitor your hardened repo?

[YoMarK]

Good day everyone,

The question is is in the title, but some more information about our environment.
So, we've setup our new Veeam hardened repo in "full paranoid" mode. Reason why: we want as little attack surface as possible.

Basically it's a HDS Fibre channel SAN, with a HPE server connected directly using Fibre channel. On the server we've installed Ubuntu 20.4 LTS with XFS reflink and using hardened repo functionality from Veeam. By the way, hardened repo functionality on Linux with XFS reflink works more than great in our initial tests.
SAN has no network connectivity, and the HPE server has no ILO connected.
Linux has firewall enabled, and ONLY ports necessary for Veeam are allowed incoming/outgoing(although it adds firewall rules on it's own, but that's a different story).
So no SSH, no HTTP(out/in) no DNS, no NTP(we will have to trust hardware clock), nothing.
The only attack surface from the network/remotely are the Veeam agents.

So I have a box there that even we cannot access remotely. So if we want to run Ubuntu patches, we have to connect a physical keyboard, then open some firewall ports, and then run the updates.

But how do we monitor hardware failures? A disk, a power supply or something else could fail and of course I want to know it as soon as possible.
SMART doesn't seem to work for SAN volumes(of course a volume consists of multiple disks).

The best plan we have to place a webcam on the SAN and server, and check then regularly for red or orange lights.
But maybe someone has some some other ingenious solution.

My question is: how do you monitor you airgapped stuff or hardened repo?

Thank you in advance!

--Mark

[HannesK]

Hello,
at least one other person is using a webcam (which was mentioned some weeks ago in the forum digest)

as you disconnected everything, SNMP traps / email alarms don't work (which would only need outgoing ports)

Best regards,
Hannes

[mdiver]

IMHO there can not be a fully sufficient solution here. It's like wanting to transport information from the inside to the outside of a singularity - which is physically impossible - as far as we know today...

We usually try to go the oob-management way (ILO, iDRAC, etc.). Therefore one has to make sure the network used for the BMC is fully separated - no routing at all - from the productive and even backup/management network.

All communicating software agents that run on the host itself - even if read only - could in theory have a vulnerability. So they would contradict the solution.
The camera surveillance is possible but to me sounds not very practical.

We are currently working on another path leveraging VBRs stack itself. I'll follow up once we have something reliable here.

[Steve-nIP]

If we're thinking as paranoid as possible here, maybe a small PC directly attached to the iLO dedicated port with an IP KVM (without the USB plugs attached, display out only). Then you could put that IP KVM on the normal network to see the display of the iLO logged in, or have a script probing SNMP on iLO, and have that output on screen.

Either way, that KVM won't be a risk at all if the USB plugs aren't connected, as it won't be able to interact with the machine in any way, and even in the wildest dreams of someone who could send dodgy EDID data to crash a graphics card driver somehow - well, it would only be that monitoring PC affected.

If you then wanted to automate an alert from that image output, that can fairly easily be done with something like an AutoIT script running on a third machine watching the KVM output, monitor for pixel colour change, which can then generate an email or whatever.

[YoMarK]

Tnx @HannesK!

We usually try to go the oob-management way (ILO, iDRAC, etc.). Therefore one has to make sure the network used for the BMC is fully separated - no routing at all - from the productive and even backup/management network.

Thank you for your insight.
This something we already have, but then I have to wonder if a hacker cannot compromise the network stack(routing), and when they do, they have access to ILO(several security issues lately) and it's possibly game over. I do not necessarily manage the network and/or are responsible for keeping OOB management up to date.
I can surely understand why you would go this route though, because it's fairly safe but still manageable.

We are currently working on another path leveraging VBRs stack itself. I'll follow up once we have something reliable here.

Interesting, tnx!

@Steve-nIP: very interesting idea. A small Pc connected only to SAN management and ILO (status pages) with a KVM switch(only connected to VGA) would give me far more information then red/orange lights from a webcam.
However, i'm wondering if I can reliably work around HTTP session timeouts(I used AutoIT scripts in the past ). I will try do make something work or at least test some things.

[stvoglio]

How can I install SNMP for Linux server (Ubuntu 20.04) in hardened mode,
i.e. the server can only use iso and has no internet access.

Thank you.

Stefano