Veeam hardened repository and UPS

[AndreB]

Hello, I would like to integrate a hardened repository. I have a DL380 Gen10 available for this purpose.

I actually wanted to use the Veeam ISO on a Rocky basis.

However, I can't find a way to integrate, for example, apcupsd to enable a graceful shutdown in the event of a power failure.

I could also control it externally via the ILO interface, but that would also open up a potential security vulnerability. I would like to avoid that.

Although the RAID controller has a battery, an unfortunate shutdown should still be avoided.

As an alternative, I would otherwise use Ubuntu 22.04, but that would obviously require more effort on my part.

How did you solve this?

Regards
André

[mkretzer]

Just my POV: We did not install any UPS agents. To me a UPS agent is another attack risk as most of the time it does things via the network.
Also, if your repository crashes because of a power failure in the middle of a backup the last backup has to re-read anyway and thus the likelihood of data corruption is not that high as no file actively written is used at a later time again.

But as you said, this depends on good raid controllers with working battery backup or disabled caches.

[AndreB]

The UPS agent is just outgoing traffic that queries the UPS via SNMP. I don't see any potential for attack there.

[mkretzer]

Yes, but in theory the snmp "clientside" can be attacked when something compromises your UPS or if traffic is intercepted.
Also, are you sure the UPS agent only communicates to the UPS? I know alot of agents work the other way around for shutdown commands (because they are time critical and are not good for polling).

[AndreB]

An unregulated shutdown is pretty bad. Alternatively, I could initiate a shutdown via IPMI. However, this would require the ILO to be connected to the network again. It could be in its own VLAN, but it would still be connected to the network.

I could also deactivate all other ILO components.

[mkretzer]

Is it really? Sorry, but after 1,5 years of monthly Veeam Server Bluescreens (central Veeam system, Veeam SQL, all our Repos) because Windows could no longer handle our Repo load on our 1,3 PB of ReFS filesystems i am quite jaded.
We had not one issue with any backup after all that. Filesystems like NTFS, ReFS and XFS should handle crashes without any issue on good hardware, especially since no backup file that was written at the time got used after the crash.

[HannesK]

Hello,
adding packages to Hardened Repository ISO is unsupported. Technically it's possible, but you need to figure out on your own and that's why I would suggest to go with Ubuntu.

We have a feature request tracked to support uninterruptable power supplies, but no timeline to implement it.

I would agree with Markus: proper hardware should have no problems with power outages. That's what battery backed RAID caches are built for.

Best regards
Hannes

[AndreB]

OK, you've convinced me.
I'll run it without UPS support and possibly add more later if it's supported. The security and lower overhead of a ready-made system aren't worth it to me.

[vbussiro]

Just keep it as hardened (and simple) as possible.
Even if the unthinkable corruption occurs, you should have another backup (maybe also immutable) offsite, for these "environment problem", unlikely to have simultaneous power outage

[AndreB]

Yes, that's exactly how it's done.

1. Veeam hardened repository
2. NAS at another location
3. S3 cloud service
4. Weekly offline backup to HDD

[mkretzer]

Or 4. Tape
It still has its uses.

[AndreB]

I also think tape is the better choice.
However, I can't turn everything inside out and throw it away. I haven't been here very long, and there are still major projects to be done :
But I think we're pretty well positioned with this.

[Entropy]

Previous UPS "feature requestor" fir the Rocky ISO here.

If you are concerned enough about power quality and security the UPS can talk to the Hardened bare metal repo via USB cable. This is how we did ours (Ubuntu). Dedicated UPS in our case.