Host-based backup of Microsoft Hyper-V VMs.
pesos
Expert
Posts: 205
Liked: 17 times
Joined: Nov 12, 2014 9:40 am
Full Name: John Johnson
Contact:

can't get hardened repository working

Post by pesos »

We've set up an ubuntu server per the instructions here:
https://nolabnoparty.com/en/veeam-v11-h ... lity-pt-1/
https://www.starwindsoftware.com/blog/v ... ory-part-1

Have created and recreated the repository partition a few times now and it never works. I am unable to create a subfolder within the repo from veeam (agent invoke failed with non retryable error
permission denied
failed to create directory '/veeamrepo/test/'
agent failed to process method [FileSystem.DirectoryCreate}

If I try to just continue creating the repository using the /veeamrepo root without a subfolder I get "file system behind the provided path is not XFS: /veeamrepo"
Mildur
Product Manager
Posts: 8549
Liked: 2223 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: can't get hardened repository working

Post by Mildur »

Is the User („single-use credentials“) you have used to configure the backup repo the owner of the path on the Linux System?
Are you using Veeam Agent V5.0?
Product Management Analyst @ Veeam Software
pesos
Expert
Posts: 205
Liked: 17 times
Joined: Nov 12, 2014 9:40 am
Full Name: John Johnson
Contact:

Re: can't get hardened repository working

Post by pesos »

I am using build 11.0.0.837 p20210525, so whatever that pushes to the linux box is what's installed.

The local account (same one used for single use creds) was set up as the owner of the path, per the blogs linked above
sudo chown -R locveeam:locveeam /mnt/veeamrepo
sudo chmod 700 veeamxfsrepo01-02

thanks!
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: can't get hardened repository working

Post by wishr »

Hi John,

When configuring any Veeam components and software we recommend relying only on the official documentation posted on our website.

Since you are experiencing a technical issue, please open a case with our technical support team directly, as troubleshooting is not possible through the forums (as mentioned when you click "New Topic").

Thanks
pesos
Expert
Posts: 205
Liked: 17 times
Joined: Nov 12, 2014 9:40 am
Full Name: John Johnson
Contact:

Re: can't get hardened repository working

Post by pesos »

No problem, will open a case. Just wanted to make sure I wasn't doing something really linux-noob-stupid.
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: can't get hardened repository working

Post by wishr »

The only thing that looks strange to me is that the path is different between the two lines you posted above.
pesos
Expert
Posts: 205
Liked: 17 times
Joined: Nov 12, 2014 9:40 am
Full Name: John Johnson
Contact:

Re: can't get hardened repository working

Post by pesos »

yeah that 2nd line was a copy paste from the blog. it had the right path when i ran it on my actual machine
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: can't get hardened repository working

Post by wishr »

Got it :) What does the below line show in the output?

Code: Select all

ll /mnt/
pesos
Expert
Posts: 205
Liked: 17 times
Joined: Nov 12, 2014 9:40 am
Full Name: John Johnson
Contact:

Re: can't get hardened repository working

Post by pesos »

Image
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: can't get hardened repository working

Post by wishr »

At the first sight, it looks fine so I would suggest continuing the investigation with our tech. support engineers. Probably there is an issue with the mount or XFS configuration.
pesos
Expert
Posts: 205
Liked: 17 times
Joined: Nov 12, 2014 9:40 am
Full Name: John Johnson
Contact:

[MERGED] hardened repository not working

Post by pesos »

case 04851782

Unable to get hardened repository working. Have rebuilt from scratch and still no luck. Once we actually try to start running backups, veeam attempts to create a subfolder for the backup files and fails:

7/9/2021 1:56:48 PM :: Error: Permission denied
Failed to create directory '/mnt/hardrepo1/nextcloudHR/'
Agent failed to process method {FileSystem.DirectoryCreate}.

Have gone through the exact configs and steps taken with support and they've verified it all looks correct.
Natalia Lupacheva
Veteran
Posts: 1143
Liked: 302 times
Joined: Apr 27, 2020 12:46 pm
Full Name: Natalia Lupacheva
Contact:

Re: can't get hardened repository working

Post by Natalia Lupacheva »

Hi John,

I've moved your post to the existing thread to keep similar issues together.
I see you have quite a long discussion with Support and already tried several ways to figure out how to fix this failure. So please keep working with Support team to beat this problem.

Thanks!
pesos
Expert
Posts: 205
Liked: 17 times
Joined: Nov 12, 2014 9:40 am
Full Name: John Johnson
Contact:

Re: can't get hardened repository working

Post by pesos »

Thanks Natalia. Finally a breakthrough with support today after a 3rd rebuild of the linux server.

Turns out it's looking like it's a bug in the Veeam software that reverts the /mnt/repofolder ownership/perms back to root once it's added in the veeam console... which in turn prevents the repo user veeam leverages from being able to create subfolders for actual job storage. We caught it in the act this time, reverted ownership to the repo user account and then we were able to run jobs. Support is going to repro in their lab and look at sorting out where the ownership is getting reversed.
soncscy
Veteran
Posts: 643
Liked: 312 times
Joined: Aug 04, 2019 2:57 pm
Full Name: Harvey
Contact:

Re: can't get hardened repository working

Post by soncscy »

Hi John,

I've set up quite a few of these, and never had this issue, so I'm not quite sure that it's accurate to say that the software does it.

What distribution are you using? An exact link to the download you used would be great, since even the same distribution release might be a different branch.
pesos
Expert
Posts: 205
Liked: 17 times
Joined: Nov 12, 2014 9:40 am
Full Name: John Johnson
Contact:

Re: can't get hardened repository working

Post by pesos » 1 person likes this post

It’s the current/latest version of Ubuntu server. Just passing along what we experienced and support is leaning towards after multiple repros. Will definitely follow up here once they can repro in their own lab and provide more detail.
pesos
Expert
Posts: 205
Liked: 17 times
Joined: Nov 12, 2014 9:40 am
Full Name: John Johnson
Contact:

Re: can't get hardened repository working

Post by pesos » 1 person likes this post

Update:

“The issue is being looked at by our tier 3 support team as well as our research and development teams, so the way we were configuring the repository initially is not the cause of the permissions denied failures. There is likely an issue with the default umask setting for users on specific distros of Linux. When Veeam is deploying the components to the repository the permission end up being reverted to root 755. The good news is that at this time the issue is being actively worked on for a more sustainable resolution, the unfortunate part is that at this time to only solution to what we noticed is manual correction as we did on our call.”
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: can't get hardened repository working

Post by Gostev »

By current/latest version of Ubuntu, do you mean Ubuntu 21.04?
pesos
Expert
Posts: 205
Liked: 17 times
Joined: Nov 12, 2014 9:40 am
Full Name: John Johnson
Contact:

Re: can't get hardened repository working

Post by pesos »

Hi Anton,

It's 20.04.2 LTS
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: can't get hardened repository working

Post by Gostev »

Thanks, I wonder if this is a change in .2 because normally V11 should have received plenty of testing with .1...
pesos
Expert
Posts: 205
Liked: 17 times
Joined: Nov 12, 2014 9:40 am
Full Name: John Johnson
Contact:

Re: can't get hardened repository working

Post by pesos » 1 person likes this post

I asked if there was a recommended distro and heard back that:

"Many other debian distros have shown to not encounter this issue, specifically older version of Ubuntu such as version of 19. The main cause of this issue is when the user's umask is 077. If you notice that the user account when created in the Linux server has the umask of 077 then you would need to follow the same steps."
tomaskalabis
Novice
Posts: 5
Liked: never
Joined: Oct 19, 2016 6:36 pm
Full Name: Tomas Kalabis
Contact:

Re: can't get hardened repository working

Post by tomaskalabis »

same problem here.
ubuntu 20.10
vbr 11.0.0.837
i have 3 backup jobs to linux repo all failing at: 23.08.2021 10:33:34 :: Failed to pre-process the job Error: Permission denied
Failed to create directory '/veeam/HOSTING - Backup ALL to XFSsecure-repo/'
Agent failed to process method {FileSystem.DirectoryCreate}.
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: can't get hardened repository working

Post by wishr »

Hi Tomas,

Have you already opened a ticket? If so, could you please share your case ID, as requested when you click "New topic"?

Thanks
pesos
Expert
Posts: 205
Liked: 17 times
Joined: Nov 12, 2014 9:40 am
Full Name: John Johnson
Contact:

Re: can't get hardened repository working

Post by pesos » 1 person likes this post

Support should be able to get you the specific steps to fix the permission-breaking that veeam is doing. I didn’t note the exact steps but here is the basic idea:

“ I have good and unfortunate news regarding what we discovered on our webex for the Linux repository. The issue is being looked at by our tier 3 support team as well as our research and development teams, so the way we were configuring the repository initially is not the cause of the permissions denied failures. There is likely an issue with the default umask setting for users on specific distros of Linux. When Veeam is deploying the components to the repository the permission end up being reverted to root 755. The good news is that at this time the issue is being actively worked on for a more sustainable resolution, the unfortunate part is that at this time to only solution to what we noticed is manual correction as we did on our call. ”
pesos
Expert
Posts: 205
Liked: 17 times
Joined: Nov 12, 2014 9:40 am
Full Name: John Johnson
Contact:

Re: can't get hardened repository working

Post by pesos »

Looks like this is still not fixed. We updated to build 1261 and the hardened repo is broken again. Have opened a new case.
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: can't get hardened repository working

Post by wishr »

Hi John,

For now, the solution remains the same - you should modify the access rights as was previously suggested by our technicians. We'll discuss with the teams improvements for the next versions to negate the issue for the environments where umask 077 being set on the user sudo level. We did not want to increase the security risks by changing the owner of the certificates folder upon components deployment/update - that's why such behavior exists.

Thanks
pesos
Expert
Posts: 205
Liked: 17 times
Joined: Nov 12, 2014 9:40 am
Full Name: John Johnson
Contact:

Re: can't get hardened repository working

Post by pesos »

:shock:
Since the root issue is not being addressed, are these steps now fully/properly documented somewhere?
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: can't get hardened repository working

Post by wishr »

Sure, the documentation will be updated to reflect that.
pesos
Expert
Posts: 205
Liked: 17 times
Joined: Nov 12, 2014 9:40 am
Full Name: John Johnson
Contact:

Re: can't get hardened repository working

Post by pesos »

Great - do you know when? The original case was back in July. Might be good to also warn about the particular distros that exhibit the problem so that people can avoid them. We had been told at the time by veeam among others that Ubuntu was supported.
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: can't get hardened repository working

Post by wishr »

Currently, we expect it to go live in the upcoming weeks.

Ubuntu is supported for sure. Moreover, the distro does not matter much. What matters here is the user rights configuration because you can get into this issue using any distro in the aforementioned circumstances.

Thanks
pesos
Expert
Posts: 205
Liked: 17 times
Joined: Nov 12, 2014 9:40 am
Full Name: John Johnson
Contact:

Re: can't get hardened repository working

Post by pesos »

Hmm that contradicts what support told us, which is that older Ubuntu versions are not affected.
Post Reply

Who is online

Users browsing this forum: No registered users and 29 guests