Hardened Repository Brainstorming

[FrankHa]

I get that the main idea is to apply the immutable flag on Linux file system level. Transport Service receives backup files as veeam user and spawns Immutability Service with full root rights to set and remove flags according to configurations. I do not think full root rights are really necessary for this, you could allow "chattr +i" for veeam user and and leave "chattr -i" to another very restricted user (or maybe root), without spawning the service for "chattr -i" from transport service.

There is also a veeam deployment service running as root, with a corresponding open firewall port. I get that there needs to be a way to deploy software components and updates which require root rights, but to a certain degree this defeats the point of veeam using single use credentials for setting up veeam hardened repository. Lets say an attacker succeeds in cracking VBR console, he can't use single use credentials and the attack possibilities via transport service are limited, but veeamdeployment service would still be a juicy attack vector. Wouldn't it be wiser to start this service and open this port only on demand (with local server access, of course)?

Veeam does open ports in the firewall itself on demand, which is nice, but the rules Veeam is setting could be much tighter. In my case, I think Veeam Transport Service only needs access to Proxy and Gateway Servers and it does not need IPv6 at all (yet).

Altogether, I really like the concept of hardened Linux Repositories. Works good so far.

Any opinions?

[HannesK]

Hello,
which problem do you try to solve in the first scenario? Agree, there are multiple ways to implement it, but the service running as root has not reachable via network.

For the second topic, I guess there is a misunderstanding on how it works. The picture below should make clear, that the service that is listening on the network has no root permissions.



IPv6 is optional, correct.

Best regards,
Hannes

[FrankHa]

In first paragraph, I relate to something I see as suboptimal which is the fact that the process which unsets immutability is triggered by by Veeam outside the repository server. It is far fetched that this is somehow exploitable, but I would see it as security improvement if unsetting immutability is triggered by a service which just starts with appropriate rights at boot and does not listen to transport service. This can potentially make a difference if there is a huge yet undiscovered issue with transport service. And yes, I acknowledge that seperating immutability service is already nice.

In regards to the updater, this picture does not tell me something new. Again, nice that the root processes do not directly listen to 6120/TCP, but if there is a vulnerability in the Backup Server, it can potentially send a malicious update which is then forwarded and installed with root rights, providing an attacker root privileges on the hardened repository. For me it would be totally acceptable if I just open the port 6120/tcp on demand (and only to backup server, not world wide as veeam defaults) - I guess I will just try it and post an update if I run into an issue with that.

[HannesK]

which is the fact that the process which unsets immutability is triggered by by Veeam outside the repository server

where did you find that information? Unsetting the immutability is an internal process of the Hardened Repository. It works completely without backup server

but if there is a vulnerability in the Backup Server, it can potentially send a malicious update which is then forwarded and installed with root rights,

no, that's impossible as long as certificates are considered as secure and the attacker does not have access to Veeam's signing key. An attacker would have to sign that malicious update with Veeam's key. It's the subtitle of the slide

[FrankHa]

> where did you find that information? Unsetting the immutability is an internal process of the Hardened Repository. It works completely without backup server
https://helpcenter.veeam.com/docs/backu ... ml?ver=120

..."The immutability service runs with root permissions as a child process of the transport service."

Sure, if everything is done correctly, this works fine. I am more concerned about what can potentially happen if not.

> no, that's impossible as long as certificates are considered as secure

Why would they be more secure than lets say an ssh keypair? There is one access (ssh) completely cut off because it is more secure than encryption, but I should rely on encryption in regards to the other access. That is somewhat inconsistent. Yes, signing software components/updates is absolutely necessary, but on demand, manual confirmation by an administrator would still be an improvement when it comes to security (and I know this comes with extra work).

[FrankHa]

Backups seem to work just fine when limiting port 6162 to backup server and turning off 6160 completely. As expected, veeam still opens necessary further ports for transport service on demand (unfortunately world wide open).

[HannesK]

The immutability service runs with root permissions as a child process of the transport service

that's correct. But it's not a security problem, because the parent service can only send file names and immutability times through the pipe to the child process. So in worst case, an attacker could set more immutability than desired. Reducing immutability is not an option, because the immutability service checks for that. The parent process cannot execute random commands through the child process.

Why would they be more secure than lets say an ssh keypair?

I got lost here Software packages (DEB, RPM, whatever) are signed with digital signatures. SSH plays no role here. SSH can be disabled, yes. Actually it's recommended.

manual confirmation by an administrator would still be an improvement

Yes, you can manually reduce access and open ports only on demand. But it's something complicated to enforce with tens of thousands of customers using it with very different Linux skill levels. Some running only one or a few Hardened Repositories. Others running hundreds of them and sometimes without automation on these hosts for security reasons. Opening ports manually on 100+ machines for an update is nothing we want to enforce

We even have customers, where there is no administrator at the Hardened Repository. The "locking it down" section on this blog post. The only thing one can do in such a setup is "shutdown" and "reboot". No way to open ports.

[FrankHa]

> not a security problem, because the parent service can only send file names and immutability times through the pipe to the child process.

I hope this is correct.

> I got lost here Software packages (DEB, RPM, whatever) are signed with digital signatures. SSH plays no role here. SSH can be disabled, yes. Actually it's recommended.

Exactly, it is recommended to turn off SSH because this is more secure than good encryption. And as far as I am aware, OpenSSH has a more convinving history in regards to security than veeam - I am not trying to be mean.
Yes indeed, there are some aspects that require encryption and I don't argue against code signing at all. However, the less I have to trust anyone, the better.

> Opening ports manually on 100+ machines for an update is nothing we want to enforce

I would not enforce this either, but I also would not pretend it would be best to leave the firewall untouched. For some it is, sure.

> The only thing one can do in such a setup is "shutdown" and "reboot"

I am familiar with that approach and and I like it. An attacker with local access can boot his own system on the repository server or just physically destroy the storage. Therefore this is also something that certainly does not fit every use case, nevertheless, veeam suggests it for a good reason.

[Technogod]

When I upgraded from V11 to V12. I had to switch to single use credentials. I used this document https://nolabnoparty.com/en/veeam-v11-d ... epository/ to delete the files in the Hardened Repository. The process to change the permissions on the existing Backups was so convoluted. I gave up and started new Backups.

[RubinCompServ]

Sure, if everything is done correctly, this works fine. I am more concerned about what can potentially happen if not.

So you're concern is that, if it's set up insecurely, it will be insecure?