Backup-Copy to hardened repo fail after update

[Tom_LeFx]

Hi,

I am pretty sure I messed up something, so please bear with me here:
I have two repos, one on-site (Centos, not hardened) and one off-site (Centos, hardened after the Update to v12, but - as far as I am aware - not with single-use-credentials, because I never set that up myself - hardened means immutable in that case)

They worked well so far, but today I decided to follow the update advise and installed the patch 20230718 to my v12 1420 - I agreed to update all components during the installation, because it's a really small setup and I figured it wouldn't hurt.
Unfortunately, since then, my backup-copy jobs fail with "Cannot connect to target backup repository"

When I go to "Backup infrastructure" and try to rescan my offsite repo, I get the following errors "Failed to import backup path /[PATH] Details: Permission denied POSIX: Failed to create or open file [PATH]
I assume it has something to do with the fact, that the update somehow changed the credentials/permissions of the backup user or something along those lines.

I probably missed something in the documentation - but I really hope I can fix this. Can someone please point me towards the right direction for this?

[Tom_LeFx]

Ok - i investigated a bit more and it is a bit strange:

If i go into the backup-repo-folder and check the backup-copy-VBKs, they only have the user "root" listed with access rights.
If I try to chown them to the backup-user, I get the information, that this operation is not allowed (because they are immutable, I assume?)

I still added the backup-user to the wheel group to make it a sudoer again and then applied the access rights to the folders themselves. This seemingly worked and removed the errors when rescanning the repository.
It all still feels a bit weird and I don't know if I unnecessarily watered down the access protection. So please point me towards the documentation I should have followed.

[Mildur]

Hello Tom

and one off-site (Centos, hardened after the Update to v12, but - as far as I am aware - not with single-use-credentials, because I never set that up myself - hardened means immutable in that case)

Cannot be hardened in V12 if you don't use single-use-credentials (or user root). Single-Use credentials are a requirement to have a hardened repository.

Please see this kb article to take over the ownership and reconfigure the repository as a hardened repository:
https://www.veeam.com/kb4348

If the kb doesn't help, open a case with our customer support team and let them help you.
All technical issues must be solved via a support case. We cannot solve those over forum posts.

Best,
Fabian

[Tom_LeFx]

"Cannot be hardened in V12 if you don't use single-use-credentials (or user root). Single-Use credentials are a requirement to have a hardened repository."

Well, that's what I thought - but I only use a regular user account on the machine.
Your linked KB article also mentions this: "In Veeam Backup & Replication, a Hardened Repository refers to the use of single-use credentials or immutability, or a combination of both."
It clearly says "or" in that case - so my files being marked as immutable should be sufficient to be marked as hardened.

My question here is really - why did installing an update make it necessary to reconfigure any ownership at all?
While I would love to open a support case, I can't, because I run on community edition.

That's why I hoped to be pointed towards a proper update installation documentation. I did not expect that a "small" patch within the v12 cycle would have such fundamental effects.

Right now I seemingly "fixed" it by adding my backup user to the sudoer group - but I have a feeling that this might not be the right way to do things.
Your KB article helps, as it explains how I could change ownership of immutable files - my main issue is still, that I would like to know what the proper/correct way is for the ownership situation. It confused me, that the immutable files had "root" as the only owner account, while I setup my backup jobs to use a specific backup-user-account


Edit:
I continued to read through the main documentation - it seems appropriate that the immutable files have only access permissions by root, as the immutability service runs with root priviliges itself and sets that flag after receiving instructions from the transport service. So far, so logical.
It still leaves me with the question: Why did I need to change anything in terms of access rights after installing the veeam update? Is there a documentation on how to properly update a veeam infrastructure with hardened repos that covers the part where I need to do anything in terms of access rights?

Edit2:
In your linked KB article i found this statement:
"A non-root Account or Private Key that has sudo or su elevate capabilities with Elevate account privileges automatically enabled.
During the upgrade to Veeam Backup & Replication 12, the persistent non-root account will be converted to a single-use credential."
Ok - this explains the situation I started with - so my user got basically turned into a single-use credential setup. Interesting.
Still - it would be really great, if there was a proper "how to install updates on hardened repositories" because the documentation here does only mention that very, very briefly and the patch notes itself only tell you to run the installer, tick "update components" and you're good to go...which is definitely not the case.

[Mildur]

Hi Tom

Well, that's what I thought - but I only use a regular user account on the machine.
Your linked KB article also mentions this: "In Veeam Backup & Replication, a Hardened Repository refers to the use of single-use credentials or immutability, or a combination of both." It clearly says "or" in that case - so my files being marked as immutable should be sufficient to be marked as hardened.

In V11, a hardened repository could use "user credentials". Which is not possible in V12. Single Use credentials is required. After an upgrade from v11 to v12, additional steps are required if the user "root" was used for the hardened repository:
https://helpcenter.veeam.com/docs/backu ... ml?ver=120

My question here is really - why did installing an update make it necessary to reconfigure any ownership at all?

When updating a V12 environment with a Patch or major version, hardened repositories can automatically updated without any interaction on the hardened repo server itself. If changes on the hardened repository are required, then something isn't right with the installation.
https://helpcenter.veeam.com/docs/backu ... ml?ver=120

While I would love to open a support case, I can't, because I run on community edition.

Of course. It's also documented in our forum rules that you can open a support case as a free product user:
veeam-backup-replication-f2/rules-of-po ... -t755.html

Best,
Fabian

[Tom_LeFx]

Hello Tom


Cannot be hardened in V12 if you don't use single-use-credentials (or user root). Single-Use credentials are a requirement to have a hardened repository.

Please see this kb article to take over the ownership and reconfigure the repository as a hardened repository:
https://www.veeam.com/kb4348

If the kb doesn't help, open a case with our customer support team and let them help you.
All technical issues must be solved via a support case. We cannot solve those over forum posts.

Best,
Fabian

I have tried to raise a support ticket, but unfortunately it was closed after two months without resolution, because currently the capacity is not there for free support (which I can understand)
My backups are working at the moment (everything runs well, health checks pass, booting the backups for test spinups works as well) - but I'd still like to get to the bottom of this