cannot create '/opt/veeam'

[Zew]

Sigh, this is so sad. Something so insanely simple and I can't find the answer, and I have to create a support case.

Case #05530602

Pre-knowledge: Copy Backups started failing, I knew the server was old, so it was updated to the latest build of Debian 11 - bullseye.
Task: Go to update the linux server under Veeams managed servers.
Problem: Fails to install data mover.
Reason: mkdir: cannot create directory ‘/opt/veeam’: Permission denied
Cause: apparently changes in V11 (cool man)
Research: veeam-backup-replication-f2/problem-set ... 75144.html

Suggestion by Hannes: "You need temporary sudo permissions for installing the persistent data mover (new in V11)"

I tried that, and it didn't work issue persists. I even logged in as the user account specified in Veeam, and create a folder via "sudo mkdir /opt/poop" and it created the folder just fine.

So please, will someone enlighten me as to what I did wrong? I'd love to know the answer here....

[Zew]

It's not that I didn't do some reaserach, all I found was these Veeam Blog post...
https://community.veeam.com/blogs-and-p ... s-for-1165

"For Linux servers, Veeam Data Movers can be persistent or non-persistent. Non-persistent Veeam Data Mover is uploaded and removed each time Veeam Backup & Replication addresses a server.

For Veeam Data Mover to be persistent, you must specify an account with root or equivalent to root permissions when adding a Linux server. Persistent Veeam Data Movers are required for the following backup infrastructure components:

Hardened (immutable) repositories.
Backup proxies."

Ok... "you must specify an account with root permissions" what does this mean... the root account itself? The only other alternative is full sudo rights.
"or equivalent to root permissions" OK what does this mean? Where can I find more information as to these limited permission entail, and how do I configure them?

"For other backup infrastructure components based on Linux servers, Veeam Data Movers can be persistent or non-persistent. If you do not want to provide root or equivalent to root permissions, specify an account with non-root permissions. In this case, Veeam Data Movers will be non-persistent. Veeam Backup & Replication will upload and start Veeam Data Movers through the SSH connection when Veeam Backup & Replication addresses the server.

*NOTE*

In Veeam Backup & Replication version 11, persistent Veeam Data Movers are required for Linux servers with the backup proxy role assigned. When you upgrade to Veeam Backup & Replication version 11, Veeam Backup & Replication automatically updates all Linux backup proxies and installs Veeam Data Movers on them. Veeam Backup & Replication does not automatically update Linux servers that have other roles. If you want to install Veeam Data Movers on such servers, open the Edit Linux Server wizard for the necessary server and click Finish."

It's that final part that I'm trying to do here and it fails with the error I stated in my first comment in this form post. What gives? even this doesn't link to, or provide any information as to what permissions I'm missing or how to apply them.

[PetrM]

Hello,

It looks like a technical issue and I don't think we can troubleshoot it effectively over the forum posts. Also, I'm not sure that it's "insanely simple", especially taking into account that manual creation of folder under the same account works fine. It would be best to continue working with our support team, please keep in mind you can escalate the support case if you feel that it's not on the right track.

So far, you can double check the requirements for repository.

Thanks!

[Zew]

Thank you for the response, so far I have no got a response from the support team at all.

I will however note that the backup copy jobs have actually started working again, simply by having updated the Linux server, and going into the managed server properties and letting it finish, though again, it finishes with a error that the data mover service could not be installed on the target managed linux server. So the severity at this time can be reduced from what it was initially assigned as.

[PetrM]

Thanks for update. I'll ask our support team to pay more attention to your case.

Thanks!

[Zew]

Just an update.

1) As stated before the backup copy jobs utilizing the manage server in question, has started working again from updating the linux managed server, and going through the managed server properties wizard.

2) When going through the Managed Server Wizard it would note that the transport Service is not installed, and it would try to install it.

3) it always failed to install the transport role, stating "Unable to create '/opt/veeam'". Permission denied.

4) Logging in to the managed linux server with the no root account and running "mkdir /opt/veeam" fails with permission denied. however running "sudo mkdir /opt/veeam' worked without issue.

5) I decided (with the support personal present), to enable root to be able to SSH in, added the root credential into Veeam, and used it for the managed server properties. Using the root account was able to install the transport service.

6) Changing the managed server properties back to the non-root user, shows the transport service is installed, and I figured it would be fine but now the server wizard error with the following instead... "Unable to create /opt/veeam/upload/%datastamp%" exactly like the OP in the first link I shared in my initial post.

This is a extremely annoying issue....

[PetrM]

Hello,

As far as I see, support engineer is actively working on this case and should come back with further action plan soon. May I ask you if you tried already to follow this instruction to configure hardened repository? Maybe single-use credentials would be an option instead of steps 5 and 6 from the workflow above?

Thanks!

[Zew]

That was what was suggested by the tech on the case. Will investigate this as an option, and report back when time permits.

Thanks again.

[Zew]

Since the jobs seem to be completing successfully even though this data mover service fails to install, is that normal? Is the Data Mover a required service for jobs to complete?

[PetrM]

Hello,

It seems to be strange, Data Movers are required for jobs to complete successfully. In general, this component is responsible for the whole data flow including retrieving data from source and sending data from a source to a target. In case of repository, the target Data Mover gets data from the source Data Mover and writes data to a backup file. You can find more information about Data Mover on this page. I recommend to clarify with your support engineer the reason of Data Mover installation failures.

Thanks!

[Zew]

"I recommend to clarify with your support engineer the reason of Data Mover installation failures."\

That's what I've been trying to do, but as mentioned in this thread and by the support engineer, I've only been given a potential work around "try using a hardened repository" which is cool and all and if I was doing a new install, I'd def do this, however once I realized it wasn't just "that easy" to do, this work around option has lost interest by me.

I wanted a root cause analysis, but I'm not getting it. As i stated in the OP:

"Pre-knowledge: Copy Backups started failing, I knew the server was old, so it was updated to the latest build of Debian 11 - bullseye."

The Managed Linux server was in use for YEARS, the original build was Debian Jesse, and backup started failing looking at the logs on the exact day I created this thread, and support case.. "7/13/2022 6:00:47 AM :: Unable to access target repository Error: Server does not support diffie-hellman-group1-sha1 for keyexchange".

This was the error that started happening. I COULD have simply attempted to figure out this root problem, which I'm unsure how this started happening considering the Linux managed Server was not updated, I'm not sure, either way what I ended up doing, again was simply updating this managed linux server from Debian Jesse, to Debian Bullseye. Following the update guidance from Debian directly.

After this I had to rescan the managed linux server, it complained again about key exchange, and simply going through the managed server properties in Veeam, the wizard fixed this, however the same wizard for some reason doesn't detect that the data mover service is installed and tries to create it from scratch, maybe I dunno. If the data mover is installed why is it trying to create folders that don't exist? I don't know I don't have the magical white papers from Veeam to tell me exactly how this is suppose to work. Otherwise I'd have things I could try to do actual root cause analysis.

If you'd like to tell your support guy to "do better", please help a guy out here. I'm gettin' hung to dry.

[PetrM]

Hi Aemilianus,

Apologies if I'm missing some important details but in fact it's a bit challenging to work with technical issues over the forum posts. I fully agree with your concerns, we need to perform RCA in order to find a suitable solution. I will ask our support team to escalate your case.

Thanks!

[Zew]

Oh, for anyone interested, we all gave up on this problem and accepted as "it is what it is".

I love IT work. If anyone finds themselves in the same boat, I can only suggest learning to ignore the problem and work around.

I am happy, happy thoughts make happy people.

[Olgn]

Hi, i'm having this issue on a old server and a new server did you guys find out anything out about this or?

[Zew]

Sorry no resolution was ever determined. Simply keep updating (both the target OS and the Veeam software and components). No jobs were affected.