Feature request: Linux Proxies with minimal sudo

[christian.naenny]

The following case was presented to the Veeam Support as case # 04909494

The goal is to use as little sudo commands as possible for security reasons. Right now (11.0.0.837 P20210507) the installation and operation of a Linux Proxy uses way too many sudo commands. In fact, Veeam asks for the sudo blanko checque which our Unix engineering team does only grudgingly approve of!
Here's what we did: on the Linux Server that was to serve as Linux Proxy we configured a user for the Linux Proxy which could login with SSH keys. Then we started the installation of the Veeam Proxy from the Veeam Console without any sudo commands allowed. All sudo attempts were logged and with each new attempt we gradually gave the proxy user more commands he was able to execute with sudo.
The same approach was taken for the operation of the Linux Proxy for File Level Restores.
In the end we had a working Linux Proxy with a defined set of sudo commands added to the sudoers configuration. But in our opinion most of the commands absolutely do not need to be executed as root!

Are we the only ones that don't like the sudo blanko checque that is basically requested?

Resulting set of working sudoer configuration:

Code: Select all

# commands necessary for the installation of the Veeam Linux Proxy
linux_user ALL = NOPASSWD: /bin/whoami
linux_user ALL = NOPASSWD: /bin/uname *
linux_user ALL = NOPASSWD: /bin/arch
linux_user ALL = NOPASSWD: /bin/firewall-cmd --version
linux_user ALL = NOPASSWD: /usr/sbin/iptables *
linux_user ALL = NOPASSWD: /bin/ls *
linux_user ALL = NOPASSWD: /bin/rm * /tmp/VeeamAgent*
linux_user ALL = NOPASSWD: /bin/rm * /opt/veeam/*
linux_user ALL = NOPASSWD: /bin/rm * /home/veeamlnx_proxy/*
linux_user ALL = NOPASSWD: /bin/touch /tmp/VeeamAgent*
linux_user ALL = NOPASSWD: /bin/touch /opt/veeam/*
linux_user ALL = NOPASSWD: /bin/rmdir /opt/veeam/*
linux_user ALL = NOPASSWD: /bin/mkdir * /opt/*
linux_user ALL = NOPASSWD: /bin/[ -d /tmp/ ]
linux_user ALL = NOPASSWD: /bin/chmod * /tmp/VeeamAgent*
linux_user ALL = NOPASSWD: /bin/chmod * /opt/veeam/*
linux_user ALL = NOPASSWD: /bin/cp * /home/veeamlnx_proxy/* /tmp/*
linux_user ALL = NOPASSWD: /bin/cp * /home/veeamlnx_proxy/* /opt/veeam/*
linux_user ALL = NOPASSWD: /tmp/VeeamAgent*
linux_user ALL = NOPASSWD: /bin/ps *
linux_user ALL = NOPASSWD: /bin/tar * /opt/veeam/*
linux_user ALL = NOPASSWD: /opt/veeam/transport/veeamtransport *

# commands necessary for the operation of the Veeam Linux Proxy
linux_user ALL = NOPASSWD: /bin/test -e /var/tmp
linux_user ALL = NOPASSWD: /bin/test -e /var/log/VeeamBackup
linux_user ALL = NOPASSWD: /bin/touch /tmp/*_vblkid
linux_user ALL = NOPASSWD: /bin/id -u
linux_user ALL = NOPASSWD: /bin/cp *
linux_user ALL = NOPASSWD: /sbin/blkid
linux_user ALL = NOPASSWD: /bin/chmod * /tmp/*_vblkid
linux_user ALL = NOPASSWD: /bin/tar
linux_user ALL = NOPASSWD: /bin/cat /proc/self/mountinfo
linux_user ALL = NOPASSWD: /bin/mount
linux_user ALL = NOPASSWD: /tmp/*
linux_user ALL = NOPASSWD: /bin/rm * /tmp/*_vblki
linux_user ALL = NOPASSWD: /usr/sbin/lvs
linux_user ALL = NOPASSWD: /bin/test -e /tmp/*
linux_user ALL = NOPASSWD: /bin/cat /tmp/Veeam.Mount*

[HannesK]

Hello,
probably you are not the only one. On the other hand: what are you trying to protect from root? A Linux proxy usually is a clean / minimal installation machine that has no data that needs protection.

Even if you would only give VeeamAgent sudo permissions (because it needs them for example to mount with Hot-Add mode): that binary could do everything then.

In any case: reducing required permissions where possible is a valid feature request.

Best regards,
Hannes

[christian.naenny]

It basically boils down to the fact that if the Linux machine is somehow compromised, a blanko sudo permission would give an attacker the permission to do whatever they want! Yes, the binary of the VeeamAgent run with sudo could do anything. But in this case we have to trust Veeam to do the right thing and make this binary as secure as possible. But as you can see in the sudoers file we created there is way too many commands that would perfectly fine as the Veeam Linux user.

[HannesK]

to do whatever they want

to get the password / ssh key, the backup server must be compromised first. taking over the proxy with no data versus doing something to the backups sounds unrealistic to me

to do whatever they want

that's what I'm interested in. a proxy usually is an empty machine. whatever an attacker does, it doesn't affect any data on that machine (or I missed something)

[PTide]

@christian.naenny,

May I ask why did you use NOPASSWD option? I am asking because we've eliminated NOPASSWD requirement long ago, and I am worried that there might be a bug related to it.

Thanks!

[tsightler]

Hi Christian, one question. If an attacker has the ability to install software that runs as root then don't they already have effective permission to do whatever they want? This one is pretty impossible to get around from what I can see. I've often thought about how we could put more effort into addressing these sudo permissions as I agree we run too many commands this way, however, it's quite difficult to see the real value from a security perspective when the account needs permissions to install software, which would allow bypassing of anything anyway. It's also more complex than it at first seems because, in some of the environments I work with, even very simple commands that I would consider trivial are blocked from being run by regular users.

[christian.naenny]

May I ask why did you use NOPASSWD option? I am asking because we've eliminated NOPASSWD requirement long ago, and I am worried that there might be a bug related to it.

In our environment we don't allow the ssh login by root with a password. Also, except the security administrators nobody is supposed to know the root password for our servers. Therefore the login user needs to issue the command with "sudo $command" and not be asked for the password. But maybe I don't fully understand all the necessary details in this case...

[tsightler]

While it's possible to configure sudo to require entering the root password when elevating to root, the more typical configuration these days is to have sudo request that the user enter their own password to authenticate immediately before elevating. The idea behind this is that, even if the shell access is somehow achieved via some method that didn't require a password (an application vulnerability for example), the user cannot elevate without also knowing the compromised users password. When using NOPASSWD, the risk is much higher since any compromise of the unprivileged account immediately gives the attacker access to any sudo commands that user can run without acquiring/knowing anything else.

Normally what I see in the field, and what I generally recommend, is that access by the veeam user to SSH on the proxy should enforce the use of public key based authentication only, but then that user should also have a strong password set that is required to run sudo commands. This way, the account never uses the root password and, if an attacker somehow manages to access this user account, which is a high barrier since they would need to compromise SSH public key auth, or find some other method to get a shell under that user account context, they will then also still need to know the users password to be able to execute sudo commands. Since SSH is configured to require public key auth for the user, the users password isn't useful for logging in remotely via SSH and I usually recommend disabling access for that account on the console as well.

If an attacker has both the public key authentication information needed to login as the user via SSH, and also has that user account password to enable him to run sudo, then you've almost certainly already lost in that case regardless.

[christian.naenny]

Hello Tom, thank you very much for your explanation! I totally forgot about this option, although I should have known...

[christian.naenny]

Since neither personal nor technical users do have a password on our systems, all authentication on the system is done via SSH keys. Thus, only the owners of the private keys have the possibility to log on to the system via SSH. If the user has to execute a command as root or another user, exactly and only this command is entered as a sudo rule (whitelisting) - everything else is not allowed by default. However, since the technical user of Veeam requires the following rule according to Veeam:

Code: Select all

veamtechuser ALL = (ALL) NOPASSWD: ALL

there is automatically a "second" technical user with root permissions on the system. This does not conform to our user concept, nor is it even necessary for operation that the technical user of Veeam is allowed to execute ALL (!) commands as root and thus has complete access to the system down to the Linux kernel.
For most commands executed by the Veeam tech user, a sudo as root is not necessary and therefore superfluous. In addition, the user could kill the system without any problems, e.g. by removing system-relevant libraries via package manager (apt, yum). A clean assignment of user permissions (chown/chmod) to certain directories would already be sufficient and make most of the sudo rules obsolete. Just think about all the filesystem operations (mv, cp). Why are they done with sudo? Why is stuff copied into the /tmp directory with sudo? Sorry but to me this looks like sloppy programming. Everything in /tmp belongs to the user who copied it into /tmp. We could even give you your own filesystem for /home/veeamtechuser. Most of the operations could be done in this home directory.

Summary:
The sudoers option NOPASSWD is necessary in an environment where password authentication is disabled for all users. However, a second technical user with a blank sudo check is not a solution in a high-security IT environment.

[Gostev]

Why is stuff copied into the /tmp directory with sudo? Sorry but to me this looks like sloppy programming.

There's some of that for sure, as Linux interaction engine is literally 15 years old (Veeam FastSCP times)!
Fine tuning for minimum possible privileges was not exactly the priority for our few developers back then

[tsightler]

Summary:
The sudoers option NOPASSWD is necessary in an environment where password authentication is disabled for all users. However, a second technical user with a blank sudo check is not a solution in a high-security IT environment.

I appreciate this statement, but what it lacks is any detail on what approach would somehow be acceptable in this case. The Veeam proxy services need root to do their job, at least for transport modes like hotadd and Direct SAN, because they require access to functions like scanning SCSI devices, mounting volumes, writing to volumes directly, etc.

To install these services that require root, or to be able to at least run them as root, Veeam will need root level access to perform the initial install. If the account has sudo permissions to install something that has root access, it has all the access needed to bypass any restrictions. Also, if you give sudo access to something the account as permissions to install, then you've effectively given full access to that user.

We could even give you your own filesystem for /home/veeamtechuser. Most of the operations could be done in this home directory.

If I can write to this, and you give me sudo to run any single command in there, than I can sudo whatever I want, no security would be added. Using sudo is only secure to run commands that are root owned which means you'd have to be willing to manually install at least something. Are you willing to perform a manual install of software to overcome this? Indeed, we could reduce the total number of commands that use sudo, but it would add zero security unless the ability to install software was also blocked, which means manual steps for deployment and upgrades. Is this acceptable?

For hardened repositories, we use this approach today, you have to give the account temporary access to sudo, so that it can install/update the components which run as root, and then disable that access. This adds administrative overhead, but it's an acceptable overhead to protect backups. In general proxies are not quite as rich of a target, but of course, there are potential attack vectors if a user manages to get access, although, properly managed the only likely way is by compromising the Veeam server, which means there's already significant trouble afoot.

BTW, if you're happy with just using NBD transport mode then you can work around this today by running the proxy in an unprivileged LXC container. NBD mode doesn't require any mounting/unmounting or access to SCSI/FC commands, so it will work this way. Basically, Linux thinks it has root, but it's only a root account in the container, on the host the "root" processes run as a regular user. One of the features I'd like to see is to allow NBD mode proxy without root since that should be easily possible, and it may happen one day, but that day is not right around the corner. But, for other transport modes, it's difficult to see how it could be addressed without requiring manual software install. There's some things that could maybe done for upgrades, but that first install would almost certainly have to be manually, or at least sudo allowed temporarily.

[christian.naenny]

I appreciate this statement, but what it lacks is any detail on what approach would somehow be acceptable in this case.

There is a difference between asking for a blank sudo for everything and trying to achieve the shortest list of absolutely needed sudo commands! We absolutely understand that some operations such as mounting a snapshot need root permissions.
Operating systems such as Oracle Solaris even know the concept of Role Based Access Control.
Here's a quote from the Oracle page about RBAC:

RBAC uses the security principle of least privilege. Least privilege means that a user has precisely the amount of privilege that is necessary to perform a job.

Honestly, after analyzing all the commands that were executed with sudo during the installation and operation of the proxy we felt that it was not even tried to achieve this principle of least privilege.

[tsightler]

I'm fully aware of RBAC and the concept of least privilege and I agree with you that there is a logical difference between a blank sudo for everything and trying to achieve the shortest list possible. However, achieving the shortest list possible doesn't seem a very worthy goal if any of the commands in that list allow the user to install software that runs as root.

So while I agree with you that there is a logical difference between these two things, from an actual security perspective I don't believe there is any practical difference. If an account has the ability to install software that runs as root, without a password, then it can install software that does whatever it's wants, so what practical security value is restricting that account from also running, for example, cp or mv, via sudo?

Of course we could have created our own tool, maybe "veeaminstaller" or something, put it wherever, and asked to have it run via sudo, that would be only one entry in sudoers, but that wouldn't be least privilege, that would be most privilege. Indeed today, if sudo is enabled, we simply run every command with sudo, because that at least provides a solid log of every action performed.

I'm happy to hear any thoughts on how to do this in a way that is compliant with your environment and that actually improves security, or is it just about keeping the list short or running the commands via sudo only when actually needed? We could almost certainly cut down on the commands required for running, but for installing, the actual sudo list wouldn't shrink a lot, and would still provide anyone with access to the account the ability to attain full permissions on the system, so nothing significant would be gained from a security perspective, which is why it's never been a focus.

[christian.naenny]

No, I don't think it's actually about keeping the sudoers list as short possible. I think we also have to see the issue with time in mind. I'm using the analogy of opening and closing a door. Of course you need sudo to install the software. But actually why didn't you just create packages (RPM, DEB etc.) that can be installed by a single, defined command? Of course that command needs to be run as root! And then there is one binary that needs to be run as root for the operations such as mount.
This is like opening and closing a door. Giving the Veeam technical user a blank sudo without password is like leaving the gate open at all times! Of course we need to trust Veeam to do the right thing in creating the installation package and the executable. But if we could not trust Veeam then we would have a much bigger problem because we trust you guys to handle our Insurance (Backups)! And by the way, an installation package and a binary could be audited if Veeam agreed to it...

[tsightler]

Of course you need sudo to install the software. But actually why didn't you just create packages (RPM, DEB etc.) that can be installed by a single, defined command? Of course that command needs to be run as root! And then there is one binary that needs to be run as root for the operations such as mount.

Largely the answer to this is because the code being used here is mostly the same code that has been used for Veeam Linux datamovers for ages, and that code was written to work across pretty much any Linux system, not just specific distros. Of course we could make RPMs and DEBs for each distro we support, actually I suggested as much above, but I asked if it would be OK if these had to be deployed manually, at least the first time, because, otherwise, no security is added, it's just a reduction in sudo commands.

I would say that over time it's logical to reduce sudo permissions, but, if it does not actually improve security, then what is the real value added? Above you noted that having an account with blank sudo is bad for a highly secure environment. My point is that having an account that can install software is just as bad for a highly secure environment. The time for an attacker to gain access to root is largely the same.

Of course we need to trust Veeam to do the right thing in creating the installation package and the executable.

But it's not about trusting Veeam, it's about securing the account that Veeam is using. If that account has the ability to install software without a password, then it is effectively an administrator, regardless if it's is only a single command in sudo that gives me that access. Veeam may install only trusted packages, but an attacker will install whatever they want.

Now, if you agree that the initial installation could be performed manually (or at least with sudo temporarily enabled, like we do with hardened repo), then you and I are largely on the same page.