can't get hardened repository working

[pesos]

We've set up an ubuntu server per the instructions here:
https://nolabnoparty.com/en/veeam-v11-h ... lity-pt-1/
https://www.starwindsoftware.com/blog/v ... ory-part-1

Have created and recreated the repository partition a few times now and it never works. I am unable to create a subfolder within the repo from veeam (agent invoke failed with non retryable error
permission denied
failed to create directory '/veeamrepo/test/'
agent failed to process method [FileSystem.DirectoryCreate}

If I try to just continue creating the repository using the /veeamrepo root without a subfolder I get "file system behind the provided path is not XFS: /veeamrepo"

[Mildur]

Is the User („single-use credentials“) you have used to configure the backup repo the owner of the path on the Linux System?
Are you using Veeam Agent V5.0?

[pesos]

I am using build 11.0.0.837 p20210525, so whatever that pushes to the linux box is what's installed.

The local account (same one used for single use creds) was set up as the owner of the path, per the blogs linked above
sudo chown -R locveeam:locveeam /mnt/veeamrepo
sudo chmod 700 veeamxfsrepo01-02

thanks!

[wishr]

Hi John,

When configuring any Veeam components and software we recommend relying only on the official documentation posted on our website.

Since you are experiencing a technical issue, please open a case with our technical support team directly, as troubleshooting is not possible through the forums (as mentioned when you click "New Topic").

Thanks

[pesos]

No problem, will open a case. Just wanted to make sure I wasn't doing something really linux-noob-stupid.

[wishr]

The only thing that looks strange to me is that the path is different between the two lines you posted above.

[pesos]

yeah that 2nd line was a copy paste from the blog. it had the right path when i ran it on my actual machine

[wishr]

Got it What does the below line show in the output?

Code: Select all

ll /mnt/

[pesos]

[wishr]

At the first sight, it looks fine so I would suggest continuing the investigation with our tech. support engineers. Probably there is an issue with the mount or XFS configuration.

[pesos]

case 04851782

Unable to get hardened repository working. Have rebuilt from scratch and still no luck. Once we actually try to start running backups, veeam attempts to create a subfolder for the backup files and fails:

7/9/2021 1:56:48 PM :: Error: Permission denied
Failed to create directory '/mnt/hardrepo1/nextcloudHR/'
Agent failed to process method {FileSystem.DirectoryCreate}.

Have gone through the exact configs and steps taken with support and they've verified it all looks correct.

[Natalia Lupacheva]

Hi John,

I've moved your post to the existing thread to keep similar issues together.
I see you have quite a long discussion with Support and already tried several ways to figure out how to fix this failure. So please keep working with Support team to beat this problem.

Thanks!

[pesos]

Thanks Natalia. Finally a breakthrough with support today after a 3rd rebuild of the linux server.

Turns out it's looking like it's a bug in the Veeam software that reverts the /mnt/repofolder ownership/perms back to root once it's added in the veeam console... which in turn prevents the repo user veeam leverages from being able to create subfolders for actual job storage. We caught it in the act this time, reverted ownership to the repo user account and then we were able to run jobs. Support is going to repro in their lab and look at sorting out where the ownership is getting reversed.

[soncscy]

Hi John,

I've set up quite a few of these, and never had this issue, so I'm not quite sure that it's accurate to say that the software does it.

What distribution are you using? An exact link to the download you used would be great, since even the same distribution release might be a different branch.

[pesos]

It’s the current/latest version of Ubuntu server. Just passing along what we experienced and support is leaning towards after multiple repros. Will definitely follow up here once they can repro in their own lab and provide more detail.

[pesos]

Update:

“The issue is being looked at by our tier 3 support team as well as our research and development teams, so the way we were configuring the repository initially is not the cause of the permissions denied failures. There is likely an issue with the default umask setting for users on specific distros of Linux. When Veeam is deploying the components to the repository the permission end up being reverted to root 755. The good news is that at this time the issue is being actively worked on for a more sustainable resolution, the unfortunate part is that at this time to only solution to what we noticed is manual correction as we did on our call.”

[Gostev]

By current/latest version of Ubuntu, do you mean Ubuntu 21.04?

[pesos]

Hi Anton,

It's 20.04.2 LTS

[Gostev]

Thanks, I wonder if this is a change in .2 because normally V11 should have received plenty of testing with .1...

[pesos]

I asked if there was a recommended distro and heard back that:

"Many other debian distros have shown to not encounter this issue, specifically older version of Ubuntu such as version of 19. The main cause of this issue is when the user's umask is 077. If you notice that the user account when created in the Linux server has the umask of 077 then you would need to follow the same steps."

[tomaskalabis]

same problem here.
ubuntu 20.10
vbr 11.0.0.837
i have 3 backup jobs to linux repo all failing at: 23.08.2021 10:33:34 :: Failed to pre-process the job Error: Permission denied
Failed to create directory '/veeam/HOSTING - Backup ALL to XFSsecure-repo/'
Agent failed to process method {FileSystem.DirectoryCreate}.

[wishr]

Hi Tomas,

Have you already opened a ticket? If so, could you please share your case ID, as requested when you click "New topic"?

Thanks

[pesos]

Support should be able to get you the specific steps to fix the permission-breaking that veeam is doing. I didn’t note the exact steps but here is the basic idea:

“ I have good and unfortunate news regarding what we discovered on our webex for the Linux repository. The issue is being looked at by our tier 3 support team as well as our research and development teams, so the way we were configuring the repository initially is not the cause of the permissions denied failures. There is likely an issue with the default umask setting for users on specific distros of Linux. When Veeam is deploying the components to the repository the permission end up being reverted to root 755. The good news is that at this time the issue is being actively worked on for a more sustainable resolution, the unfortunate part is that at this time to only solution to what we noticed is manual correction as we did on our call. ”

[pesos]

Looks like this is still not fixed. We updated to build 1261 and the hardened repo is broken again. Have opened a new case.

[wishr]

Hi John,

For now, the solution remains the same - you should modify the access rights as was previously suggested by our technicians. We'll discuss with the teams improvements for the next versions to negate the issue for the environments where umask 077 being set on the user sudo level. We did not want to increase the security risks by changing the owner of the certificates folder upon components deployment/update - that's why such behavior exists.

Thanks

[pesos]

Since the root issue is not being addressed, are these steps now fully/properly documented somewhere?

[wishr]

Sure, the documentation will be updated to reflect that.

[pesos]

Great - do you know when? The original case was back in July. Might be good to also warn about the particular distros that exhibit the problem so that people can avoid them. We had been told at the time by veeam among others that Ubuntu was supported.

[wishr]

Currently, we expect it to go live in the upcoming weeks.

Ubuntu is supported for sure. Moreover, the distro does not matter much. What matters here is the user rights configuration because you can get into this issue using any distro in the aforementioned circumstances.

Thanks

[pesos]

Hmm that contradicts what support told us, which is that older Ubuntu versions are not affected.