can't get hardened repository working

[wishr]

Right, but you can also manually set that configuration and face the same issue, especially keeping in mind it's a NIST recommendation nowadays.

[pesos]

Ok… but we didn’t do that… and still had the problem… so out of the box the issue exists on newer versions of Ubuntu when using the instructions provided by veeam… which is what support confirmed for us in the original ticket…

[wishr]

Correct, that's why the instructions will be updated accordingly, as mentioned before.

[pesos]

Not sure if there is possibly a language issue here but this statement from your earlier post is misleading and contradicts what you most recently wrote. The distro and versions DO indeed matter because until you update the documentation (which frankly should have been done months ago) anyone running the current Ubuntu ver and following instructions will hit this issue. Sorry not sorry but getting kinda sick of being told repeatedly for weeks by support that I must be doing something wrong (then eventually being vindicated), then hitting the same issue months later after having spent a lot of time (and downtime) helping chase down the root cause in veeams software just to have it not actually implemented in multiple updates since - and on top of it not even having docs updated to at least alert people to that fact.

“Ubuntu is supported for sure. Moreover, the distro does not matter much. What matters here is the user rights configuration because you can get into this issue using any distro in the aforementioned circumstances.

Thanks”

Obviously one can get one’s self into trouble by going off the reservation and doing things outside of the provided documentation. That is completely irrelevant. What is relevant is that the existing combination of veeam code and documentation being followed to the letter leads to a broken and unusable repository and for some reason neither has evidently been updated to reflect this for months.

[wishr]

Sad to hear you had such an experience and thanks for the feedback.

[pesos]

To be clear, I am and continue to be a long time veeam fan and evangelist. However I feel some of the support changes of late have been for the worse.

[emcclure78]

Hello,

Is this still currently an active issue? I had just setup a hardened repository yesterday using Ubuntu 20.04.3 LTS. The version of Veeam running is the one originally mentioned in this thread. If there's a resolution I'd love to know what it is as I'm unable to navigate that folder in Linux and seem to have the same issues posted here.

Thanks.

[pesos]

Yes it is very much still an issue with no code fix yet even though I helped veeam determine the bug back in July! More shocking is that documentation has also not been no properly updated nor is there any warning when you download updates (applying an update re-breaks things).

I’m not at the computer now but can try and get you the exact steps to manually fix after Christmas. Basically after the veeam code runs against the repository, it leaves permissions in a borked state - so you have to temporarily give your repouser account sudo rights again, then run a chmod and a chown command against the repo folder. The syntax is here:
https://www.starwindsoftware.com/blog/v ... ory-part-1

Let me know if that helps!

[pesos]

Don’t forget to take away sudo after it’s working (and we also then disable ssh).

[emcclure78]

Hmm. So I followed the steps you mentioned and I'm still having the same issue. When I try to run this command:
/mnt/veeamrepo/backups/.... lsattr it fails. Heck if I try to auto tab past /mnt/veeamrepo it doesn't bring anything up at all.

However when I run this command: sudo chown -R locveeam:locveeam /mnt/veeamrepo
It tells me that the operation is not permitted on the one test backup I created which shows a full path of /mnt/veeamrepo/backups/Immutability Test/Immutability Test.vbk and then also shows some lock files as well.

So am I borked here? Should I just reinstall Linux and then after connecting the Veeam backup server to the hardened repository, I should run those commands again for the permissions before running a test backup?

I appreciate the steps for after Christmas. Very shocking that this hasn't been resolved yet.

[pesos]

Hmm, it seems like you might be seeing something a bit different than I was. Might not hurt to rebuild with an older ubuntu distro to avoid the permission issue altogether.

[emcclure78]

Will do. Should Ubuntu 20.04.2 work, or should I go for 20.04.1 or something even older?

[pesos]

I am a Linux noob TBH - I’m not sure when the design change was introduced that reacts poorly with veeam’s logic…

[emcclure78]

Ok no problem. I'll start with 20.04.2 and if I have issues I'll go to 20.04.1 and so on. I'll post here to list what works/doesn't work for me.

[pesos]

Found this snippet from my original ticket with support:

“ Many other debian distros have shown to not encounter this issue, specifically older version of Ubuntu such as version of 19. The main cause of this issue is when the user's umask is 077. If you notice that the user account when created in the Linux server has the umask of 077 then you would need to follow the same steps.”

[emcclure78]

Thanks for the info. I've downloaded Ubuntu 20.04.2 and .1 as well as 19.04 as well. I'll install them all and see what I get.

[emcclure78]

Hmm so I think I've figured out a couple of things, if someone can correct me if I'm wrong that'd be great.
From looking at this link: https://nolabnoparty.com/en/veeam-v11-h ... lity-pt-1/ it was posted in April of 2021. Veeam Backup & Recovery was available on February 24th of 2021 according to this: https://www.veeam.com/download-version. ... b=previous and the version available was 11.0.0.837 P20210525
Doing a quick google search on when Ubuntu 20.04.3 was released shows that came out on August 26th, 2021. So these directions were written with 20.04.1 which was released on August 6th, 2020 or 20.04.2 which was released on February 4th 2021.

I dug around and downloaded the previous versions of Ubuntu to test this on. We're currently using the version of Veeam B&R that was available when the directions were created.
I'm getting the same error I mentioned, with either version. The only thing I'm going to do differently, is when I have attached the Veeam B&R server to the hardened repository, I'm going to re-do the sudo permissions for the user, temporarily of course and see if that improves anything for me. The other issue I have, is unfortunately I cannot download updates for my server, as internet access is blocked to it and I'm not sure when/if that will be available, so I'm not sure if there's an update that could fix my issue.

I'm curious, has anybody else tried this with a non-Ubuntu server and had success? If so, which one, which version, and what steps did you follow? I will update this as soon as I complete my setup again.

[emcclure78]

So I'm still having the issue where I can't perform the lsattr command that's listed on this page: https://nolabnoparty.com/en/veeam-v11-h ... tion-pt-2/. I can get to /mnt/veeamrepo/ but after that I can't get anything. Tabbing fills out nothing. If I know what's in there then nothing happens. I re-ran the # sudo chown -R locveeam:locveeam /mnt/veeamrepo/
# sudo chmod 700 /mnt/veeamrepo commands after I configured the Veeam repository and added the backup repository and still no change. Even did a reboot. I did run a test backup which is fine and at least from the Veeam console it will not allow me to delete it.

So how good am I? I can't verify anything on the Linux machine, and this should be 20.04.1 or 2. Not quite sure exactly where to get that info. Is this secure? Safe? Working properly? Do I need to open a ticket? Get the latest patches on Ubuntu somehow?

[HannesK]

Hello,
I just tested Ubuntu 20.04.3 LTS and the umask default value is 0022. That means to me, that the issues emcclure78 has are different from pesos.

documentation on the umask issue: https://www.veeam.com/kb4250 which is linked in the user guide for NIST 800-171 security profile reflects the umask requirement. We are also working on more documentation for the user guide, but that got delayed due to vacation season.

general note on umask 077 / 0077: in short, administrators have to take care of that manually. Automatic fixes for the problem from Veeam side could also create customer complaints.

Longer explanation: The easiest workaround that might come into ones mind to change the owner (chown) of the required files. But that would add security risks. Example: the Veeam transport service runs as user "repo". An attacker finds a way to exploit the transport service and has gained access to the server with permissions of the user "repo". If the transport service files would be owned by "repo", he could now replace them. As Linux has different ways of setting permissions (POSIX permissions, ACL permissions, SE-Linux context permissions), it's error-prone for us to try to copy all kind of permissions an administrator might have set. So we require that the necessary files are readable for the "repo" user. Whether that's achieved by umask or manual configuration is up to the administrator.


on emcclue78 questions:

It tells me that the operation is not permitted

that sounds like everything works fine. the operation not permitted probably comes from the immutable backup files

So am I borked here?

not sure - what do you try to do? the installation of hardened repository does not involve lsattr commands. the file system just needs to support it and if you use the recommended XFS, then everything is fine (also ext3 / 4 which are common just work).

installation of hardened repository has the following steps:
1) have a normal user (I call it the "repo" user)
2) have a path for backup data where user from step 1 has write access to. my favorite way (because easy) is that the "repo" user owns that path (chown).
3) add the server with single-use credentials with the "repo" user. use "su" to temporary get root permissions (sudo is possible, but should only be used temporary. su is easier). select the path from step 2.
4) done - start backing up

Best regards,
Hannes

[pesos]

Here we go again.

Patched due to the security vulnerability.

Apparently this issue STILL hasn't been fixed in the veeam code. @#%*&!$!!

Not only that, but the workaround that got things working again in my last two cases doesn't appear to be working this time around - new backups are failing.

It's constantly one step forward two steps back with this solution, honestly.

[HannesK]

Hello,
I repeat: it will stay as it is because changing the behavior has side-effects mentioned above.

access for the Veeam repository user is required for the following files:

Read and execute:
/opt/veeam/
/opt/veeam/transport/
/opt/veeam/transport/certs/
/opt/veeam/transport/certs/client/

Read only:
/opt/veeam/transport/certs/cert.p12
/opt/veeam/transport/certs/client/cert_*
/opt/veeam/transport/VeeamTransportConfig

Execute only:
/opt/veeam/transport/veeamagent


Best regards,
Hannes

[pesos]

Hi Hannes, I appreciate the willingness to help, but I have to say I do not really appreciate the attitude that's been received regarding this issue, nor the crummy support that resulted in WEEKS of downtime on our original case, long-delayed (MONTHS!) documentation updates, etc.

Shouldn't have to be a linux guru to make this work, and for a feature that's been shouted from the rooftops as THE THING TO HAVE V11 for and that it's the greatest thing since sliced bread, I feel more attention should have been paid to this.

In reading through your post above, I do not recognize any of the paths you listed so I'm not sure if you are discussing someone else's issue. The problem we've encountered is that the actual repo directory itself (/mnt/reponame) gets messed up every time we update.

I believe I was able to fix it now using
chown repouser.repouser /mnt/veeamrepo
chmod 700 /mnt/veeamrepo
(again not a linux guy, so I don't know why the /mnt/ part above was left out of the veeam documentation; maybe that works on certain distros, shrug)
(plus manually adding and removing repouser from sudo group before and after running the transport update)

[HannesK]

Hello,
hmm, I though it's still about the umask 077 issue... which issue is it about, if it's not umask?

The default umask is 022 in Ubuntu. If the umask setting was changed, then the side effects have to be fixed manually.

Best regards,
Hannes

[pesos]

Again, I'm not a linux guy, so I'm working with terms that are fairly unfamiliar and doing my best to learn as I go.

All I know is that with the first case, it took weeks to find a resolution and the resolution was to run the commands I've listed above regarding the repository folder (I'm not familiar with the various paths that you listed which appear to be veeam system file paths, not data paths).

We've never changed any umask settings manually or on purpose - from what I gleaned from the two prior cases with support, Veeam assumed certain settings from previous distros that are no longer standard in newer ubuntu distros which is the core of the problem.

[HannesK]

the configuration is done for Ubuntu in /etc/login.defs

Code: Select all

UMASK           022

I have no idea why the defaults in your installation are different than in all my Ubuntu installations. By setting the value to 022, the problem should be solved permanently (at least it's working fine for many customers, because otherwise the forums would be full of questions around this)

[pesos]

If that's true, it's news to your support agents. They were very clear that things are different on newer ubuntu distros vs. older (hence the problem occurring at all).

I'm happy to take a look at the value. If we do change it, do I then need to rerun any of the steps mentioned above?

[HannesK]

The setting is only relevant for new roll-outs and upgraded. So the setup must work already to see the effect at the next upgrade. If your environment behaves like mine, then once applied "it just works". No need for any manual "tuning" in future.

[pesos]

mine is also umask 022

[EcoboostPerformance]

Following issue...

veeam-backup-replication-f2/linux-harde ... 82320.html

I have a similar issue and have offered veeam some possible solutions to get around this problem permanently. As setting the UMASK to 022 for the root user is against Center for Internet security standards and shouldn't be done. I was sent here for the reasoning.