What are the to-do list for creating a Hardened Linux Repository?

[AlexLaforge]

Hi,

I am protecting 10 VMWare ESXi VM's with Veeam BR. The target is a Windows repository.

I am considering to create a new Hardened Linux Repository. However, I am absolutely not a Linux man!

Do you have resources where I can find specific instructions to create such a Linux system?
I bet that just installing a fresh Ubuntu distribution is not enough... so I guess that some specific initial configuration & security measures have to be applied before using this Linux system as a Hardened Linux Repository.

Thank you for your help, tips and guidance

[HannesK]

Hello,

I bet that just installing a fresh Ubuntu distribution is not enough

you might lose that bet

Just go with a minimal installation. No GUI. Youtube or whatever you prefer has videos for sure how to install the distribution of your choice. There should also be blog posts on basic installation.

I recommend XFS to be able to use block cloning.

If you go with Ubuntu, you might like this community tool https://github.com/tdewin/veeamhubrepo

Did you also see the FAQ: post402811.html#p402811 ?

Best regards,
Hannes

[nitramd]

Hello Alex.

Here are several URLs that may be of help:
https://help.ubuntu.com/community
https://ubuntu.com/server/docs
https://askubuntu.com/help

If you stick with it you'll be a Linux man in no time.

Hope this helps and good luck.

[Rascii]

Is it correct that it can only function as a Repository and not a Proxy as well?

Ah, I found the answer in my searching.

Q: Can I install any other Veeam roles (proxy, tape server, WAN Accelerator) on a Hardened Repository server?
A: No. The Hardened Repository is an exclusive role per machine.

[SkyDiver79]

I am considering to create a new Hardened Linux Repository. However, I am absolutely not a Linux man!

Me too.

I have now made several installations, here is my own tutorial:

- Install fresh Ubuntu Server 20.04
- In the installer I build the network bond, set the IP and let the SSH service install.
- Format and mount the disk
--> sudo mkfs.xfs -b size=4096 -m reflink=1,crc=1 /dev/sdb
sudo mkdir /data
sudo mount /dev/sdb /data
sudo nano /etc/fstab
/dev/sda1 /data xfs rw,noatime 0 0
- authorize user
--> sudo chown -R veeamrepouser: veeamrepouser /data
sudo chmod 700 /data
- update system
--> sudo apt update
sudo apt upgrade
sudo reboot
- install Agent over VBR Console (Adding Linux Server)
- setting Firewall
--> sudo ufw default deny incoming
sudo ufw default deny outgoing
sudo ufw allow from *VBRServerIP* to any port 6162 proto tcp
optional: sudo ufw allow ssh
sudo ufw enable

That's all, I'm always open for improvements

[AlexLaforge]

Thank you A LOT to all of you guys until there!
Each of your posts is useful to me.

Obviously, it won't be easy for me to craft all your tips and advice together to obtain the "ultimate bulletproof" todo-list, but it will help for sure.
As you can imagine, my main concern is about security, as I'm not a Unix man, nor fond of networking stuff

Has someone here already used a penetration-testing service/company/person that I can trust?
... cause I'm not fan of paying the first self-called "ethical hacker" on Fiverr !

[SkyDiver79]

Hi,

I had my installation checked by a Linux specialist.
He said SSH on with 2 factor authentication, (I had ssh disabled) and he does not like the concept of Ubuntu with the sudo rights. But that goes then too deep into the topic Linux.
It is also important to secure the IPMI (iLO, iDRAC and so on.).

[richardgr88]

Hi,

I'm with you - the Linux hardened repository offers some great benefits but also like you Linux isn't my bread and butter. I believe Gostev mentioned in a post that came around either in one of the digests or just general release info for V11 is that an official guide/step by step/deployment mechanism was coming to help when it comes to deploying one of these.

Anyone know if this is available yet or coming soon? I've just upgraded one of our servers from 6 x 14TB to 12 x 14TB drives and lets just say Windows isn't going back on it.

Thanks in advance

[Gostev]

The quick step-by-step summary just a few posts above is pretty perfect.

If you're looking for something very comprehensive, then I'd recommend this guide > https://www.starwindsoftware.com/blog/v ... ory-part-1

[tdewin]

I had my installation checked by a Linux specialist.
He said SSH on with 2 factor authentication, (I had ssh disabled) and he does not like the concept of Ubuntu with the sudo rights. But that goes then too deep into the topic Linux.
It is also important to secure the IPMI (iLO, iDRAC and so on.).

In any case, please make sure to remove sudo rights after you have added the linux server to VBR. Otherwise the datamover process still has sudo capabilities (if that is the way how you elevated during installation) and this is of course a potential security issue

[dandav]

One thing that all these howto's miss is if you're running Ubuntu you really need to replace timesyncd with ntpd. Timesyncd is by default designed to accept ANY time response and adjust the system clock, while this is great for desktops or systems with no RTC (e.g. many ARM boards), it's not so great in situations when time synchronisation is an attack vector.

ntpd by default will error when the returned time from an NTP server differs against the system clock by more than 1000s. RTC's are plenty accurate nowadays so you shouldn't need to check for time updates very often, so if you make ntpd run at boot then once a week using the -q flag (this makes ntpd check, update then exit) then the NTP attack vector takes a very long time to execute. As an example, an attacker would need to be able to spoof NTP requests on your network for seven weeks just to move your clock forward by two hours.

Also, use IP addresses for your configured time servers, this way compromised DNS cannot affect your timesync.

[Gostev]

This is a great suggestion! Although if hackers manage to take over the NTP server itself, then you still have a problem...

[dandav]

I agree, but it people should also be reminded that for 99% of cases the immutable Linux repository isn't a replacement for immutable off-site backups.

In my eyes the immutable Linux repository isn't really about guaranteeing the integrity of my data (immutable S3 backup copies do that), it's about saving time. In the event of an attack, the hardened Linux repository just decreases the chance that I will be forced to restore over a slow WAN link.

[Gostev]

It's a good way to think about it, as one thing the hardened repository cannot protect against is a malicious insider.

[dandav]

Also, ntpd should never be used with one server only, so a single compromised NTP server should not be able to affect the system date. The recommended minimum number with ntpd is four and it will use consensus to detect a problematic time server. With ntpd configured correctly an attack against the system clock will either need local system access or transport-layer access to your network for a VERY long time.

I don't think that completely disabling NTP checks is a good solution either, if you don't like automated NTP updates then a better option would be to use something like ntpq in a scheduled task to check and alert if the time difference reaches your chosen offset without actually updating the RTC. Either way you should be monitoring and alerting somebody in the event of an unexpected time shift or large offset, this has the added benefit of also alerting in the case of an attacker with physical system access who can alter the clock in BIOS.

[Gostev]

The recommended minimum number with ntpd is four and it will use consensus to detect a problematic time server.

The main problem with that is the reality... I mean, I would be surprised to learn that even 1% of all environments out there follow this best practice

Anyway, the main point here is that disabling NTP checks lets you completely forget about this attack vector, while in any other case you have to think hard about it. And then, it comes down to your skill level. I mean, anyone can disable NTP check, it is super easy. Also, most people should be able to acquire and install a GPS time dongle, if the precise time is important. But how many users have skills to secure their NTP deployment, and resources to align it with the best practices? Probably not a lot.

[dandav]

GPS time is ideal, however obtaining a GPS signal in a datacentre isn't always possible. Precise time can be important for other hardening factors such as Time-based One Time Passwords (TOTP has the added bonus that broken time will remove the ability for an attacker to login and delete things, even from the local console).

[rschols]

We recently completed this setup with Debian 10 on bare metal using a combination of the above checklists.
So far so good!

Two big concerns remain:

1. With out of band management disabled and all services on the Linux server disabled, it becomes extremely difficult to be alerted to hardware failure. I'm concerned about a faulty disk in the raid array being detected too late. Some minimal monitoring will need to be setup, and I wonder what would be the most secure way.

2. A hacker or malicious insider can easily disable the immutable config in B&R and wait X days for backups to become erasable.
I had expected the "number of days of immutability" setting would be made on the hardened Linux host in a config file read-only to the veeam service account.

[Mildur]

1)
My server have ILO or IDRAC.
I will allow Port 587 to send out notification mails about the hardware.

Source: ILO
Destination: Internal Mail Server or dedicatet smtp relay for Veeam
Port: 587

This looks safe for me.

2)
That would be a good way, if the config is not on the vbr server itself. There could be some improvement after the first release.

A important thing for admins todo, is to audit the access logs and changes in the backup jobs. A linux hardened repo doesn‘t take this tasks away. If you audit the access to the server, you can take action before something bad happens

[mkretzer]

1. With out of band management disabled and all services on the Linux server disabled, it becomes extremely difficult to be alerted to hardware failure. I'm concerned about a faulty disk in the raid array being detected too late. Some minimal monitoring will need to be setup, and I wonder what would be the most secure way.

We installed a dedicated camera in front of the Rack which we check regularly. Also RAID 6 + 3 Hotspares and we are thinking about RAM hotspares. Yes, we are paranoid.

[soncscy]

>2. A hacker or malicious insider can easily disable the immutable config in B&R and wait X days for backups to become erasable.
I had expected the "number of days of immutability" setting would be made on the hardened Linux host in a config file read-only to the veeam service account.

I think this falls under the risk of "if you get admin/root access" personally. If I understand you correctly, you want the "hardened' aspect of the hardened repo to be separate from VBR, which it already is afaik and handled by the service on the repo.

But admins need a way to administer, so for me makes sense that it's something you can configure from the VBR server (but I would __LOVE__ a "hardcore" mode where once you turn it on, the service on the repo refuses to turn it off and anything that goes into this repo is immutable, no taksiesbacksies, and nothing short of a volume wipe would change this. Can we make this a thing Veeam? Please?"

But, since we don't have hard core immutability mode, you have to monitor the VBR server. This is simple with some powershell scripting

[Gostev]

I was thinking about something like this, but quickly realized that in case of unrestricted backup server access and unlimited time available to perform the attack, there are just too many other ways for a hacker to leave you without backups. From simplest ones like changing the backup file encryption password, to more complex ones involving the replacement of some software modules. So, keeping your backup server off domain and protected with MFA remains as important as before.

[rschols]

1)
My server have ILO or IDRAC.
I will allow Port 587 to send out notification mails about the hardware.

[...]

This looks safe for me.

We disconnected the iLO port to make sure a hacker cannot control the machine from iLO. Otherwise we are back down to simple username+password to own the whole machine.
Or were you able to set up iLO to just send out smtp but not permit login?

It would be convenient to set up snmp on the Linux OS so that we can poll with our network monitoring tools. But this would require opening a port and running snmp software and I don't know how secure snmp implementations are.

I am considering setting up smtp on the Linux OS. Postfix has a very good reputation for being secure, and we would set it up for outgoing only. Less convenient than snmp, but probably most secure option.

[Mildur]

ILO or IDRAC Management Ports will be protected by hardware Firewall. Traffic will be blocked. We will change the ilo Ports to an alternative port, not the default 443.
It will only be used to send out mail notifications. For OS updates, it can temporarly be opened.

As an alternative, i‘m thinking about placing a Postfix/Monitoring server in the same network segment like our backup repo server. This server will also be protected by firewall, the only ports open are from this monitoring server to our main mailsystem with port 25.
Network Traffic from the production networks to this server will be all blocked.

We always have a second copy of our data, in our case a S3 Storage with Object Lock in a second offsite Location. For us, this will be enough security to use the ilo port to send out the notification mails.

[rschols]

We are running into a serious problem with fast clone. This is something we will add to our checklist for building a hardened repo.

When you move existing backup chains to the new immutable repository and map your copy jobs to that new location, you still need to run Active Full or "Defragment and compact full backup file" on these jobs to get fast clone working.

Active Full is not an ideal option for large copy jobs going across the internet — in our case it would take 20+ days.

But you cannot enable "Defragment and compact full backup" if GFS is turned on. And you cannot disable GFS if immutable is turned on.

So if you are moving existing backup chains to your new storage, run "Defragment and compact full backup" for each job as needed BEFORE you switch on immutability on the repository.