Linux hardened repo not connectable after VBR migration

[cpfleger]

I've setup a physical Linux repo (let's call it reposerver; running Ubuntu 20.04 LTS) and hardened it for immutability.

Setup a directory for backup usage (let's say REPO01 at /REPO01), did all the other stuff.
Ran a Backup Copy Job against it - which worked and using lsattr -a /REPO1/* showed that immutability was set correctly.

Later I needed to change my VBR server (before VM, afterwards a vm but different platform)- migrated the latest .BCO backup of the old VBR (let's call it vbr01) onto a new VBR server with same name, same release (11.0.0.837 - the lates image with the first patch) and IP address.

Migration of .BCO went though without any problem.

When I now try to rescan the Linux system VBR tells me

"Failed to rescan repository REPO Error: Product components installed on reposerver have older version (11.0.0.837) than the current product version (11.0.0.837) and must be updated."

ooookay - older version but it is the same one? Anyway - let's try to update.

Asked for a Single-use credential for hardened repository ... gave it one but now:

"Failed to upgrade host components. Error: 'Invalid package type Nfs' "

What the heck?
Anyone an idea?

Regards!

PS: no, I haven't opened up a support case yet. Looks like a bug to me?

[Mildur]

One time after upgrading to V11, I had to use the root user (temporary enable and disable again) for updating the components. I had a support case for that. I am not sure if it‘s the same bug in your case.

It‘s best to open a case. Technical issues always need a case
If it‘s a new bug, veeam should now about that.

[cpfleger]

https://ibb.co/dGkRDvc
https://ibb.co/qFx1mQf

Interestingly Details view in Components update says that it would truly be different versions?
Fine for me - but how to teach the Linux OS to accept the update?

btw. Case #04810788

Regards!

[tsightler]

11.0.0.825 is RTM while 11.0.0.837 is GA. Seems like one VBR server was installed with RTM ISO while other was installed with GA ISO or upgraded with Day 0 patch. Maybe you should just upgrade the server and then you wouldn't have a mismatch?

[cpfleger]

What happened to my answer here??? It disappeared?

[Gostev]

Looks like it was never posted, because the moderator activity log for today is very short (Sunday!) and it does not show any topic or post deletions.

[cpfleger]

Hmyeah, layer 8 thing then.

Thx Anton!

[tsightler]

Hmyeah, layer 8 thing then.

Maybe not as I've definitely noticed a little bit of weirdness in the forum the last couple of days. Just now I was replying to a post and it suddenly made me login again, which was weird enough, but then it told me the forum I selected did not exist, which is, well, unlikely, since I was replying to an active thread!

[Mildur]

Me too

[cpfleger]

In German language we say "Wunder gibt es immer wieder..." - there are always magical/mystery moments or wonders to experience

Regards!

[Gostev]

Aber alles hat eine Erklärung

[cpfleger]

Da!
I'd love to speak Russian as well as you obviously German, Anton! Hope we meet sometimes in future somewhere again!

[NightBird]

Same error here :'( ==> Failed to upgrade host components. Error: 'Invalid package type Nfs'

From 11a P20211123 to 11a P20211211 no way to upgrade the Linux hardened repo (Ubuntu LTS 20.04)

[Mildur]

Hi Nightbird

Update should work, but you need to enable SSH and give the "single use credentials"-user sudo permission to do the upgrade of the proxy.

1) Enable SSH

Code: Select all

sudo systemctl start ssh

2) Add Repo User to Sudo

Code: Select all

sudo usermod -aG sudo repouser

3) Update Backup Repo in VBR Console with Single Use Credentials
Server Components Upgrade - User Guide (veeam.com)

4) Remove Repo User from Sudo

Code: Select all

sudo deluser repouser sudo

5) Disable SSH

Code: Select all

sudo systemctl stop ssh

If you have tried that, but it is not working, could you open a support case?

[NightBird]

It's the case my veeam user is in the sudo group (same as at the moment of installation).

ssh started
I don't understand why it is not working

I will open a support case

[Magz]

Hi
I have the exact same problem. Did Veeam support manage to fix this?

[TiredCustomer]

Literally my first post after being a Veeam customer for years.

Similar case here. Same error, but we were not migrating anything like the OP. In our case we simply installed the Veeam Cumulative Patch 20220302 for Veeam B&R 11a. After it installed successfully, it asked to update existing managed servers. We had two hardened linux repositories (one POC and one Prod, same OS. POC is a VM, Prod is physical). The POC one updated fine, but the Prod one keeps having this error: "Failed to upgrade host components. Error: 'Invalid package type Nfs'".

Support case has been opened (#02588103). It is a week old now. Replies I've been getting from support are generic, asking to basically try the same steps again. The second engineer that picked up the case hasn't even read what the previous one was recommending and just repeated the same recommendation, wasting another day. The latest response from last night is literally a copy-paste from Mildur above to the point where they even copied "Update should work" words. Yeah, right! Stating that would definitely help us

It's obvious that SSH needs to be enabled and any half-decent systems engineer can figure out that the username running the update needs appropriate permissions. I've also tried using persistent credentials instead of a single-use ones to no avail. If the error is related to SSH, why not say "Connectivity issues", or if it's related to permissions, why not say something that would indicate that in the error message? Instead it is very mysterious.

This is literally the only thread I could find mentioning the same error. It's possible that the issue is on our end and is not Veeam's direct fault. I can only suspect that the lack of replies from other users regarding solutions implies that they found something themselves and have not updated this thread with a 'fix'. Alternatively, if other reasons for this error, that is more than a year old, are known to Veeam, it is surprising that it hasn't been added to their internal KBs and hasn't been provided by support yet. In any case, I expected more effort and attention from engineers assigned to the case. Not even reading through previous notes when handed over existing ticket is unacceptable.

For reference, our immutable repository is running on Rocky Linux 8.5. If I figure this out myself, whether it is our own mistake or Veeam's bug, or if I receive a solution from support, I'll post it here.

[HannesK]

Hello,
and welcome to the forums.

Question: did you ever use / add that host before in VBR? I just heard from support, that the likely reason is that it is a repurposed server.

I found some cases with the same error message where database edits were the solution. So that's something support needs to do.

Best regards,
Hannes
PS: to escalate cases, there is the "talk to manager" button in the support portal.

[TiredCustomer]

The Prod Linux server with hardened repository was purposefully built for this, it was not repurposed and it was not converted. The immutable repository worked fine for months until this cumulative patch was installed.

[TiredCustomer]

My apologies - I misunderstood the question about using the same server in VBR before. I was thinking about the server from the OS perspective, not the hardware. Yes, this hardware used to be a different Windows server in Veeam. It was removed and all its backups deleted from the inventory before the new Linux OS was installed afresh.

From all the comments and from the support ticket it seems that our case is not singular, and it's possible that even after removing the old Windows server from the inventory some of its remnants remain in the database, which happened in our case too. The reason for this is unknown fully as explained by support.

The fix for us was to identify the id of the old server, and manually remove entries related to it from the database. I'm posting the queries below for completeness, but the advice would be to have guidance from support on this topic as it deals with manually modifying the database. Proceed at your own risk.

- Download and install Microsoft SQL Management studio or other SQL management software if it is not installed on the Veeam server yet.
- Take the backup of the 'VeeamBackup' database and take Configuration Backup from Veeam UI before proceeding further.
- Knowing the hostname of the old server, the ID can be found with this database query:

Code: Select all

select id from dbo.PhysicalHosts where name = '{hostname}'

where {hostname} is the hostname if the old server.

- If unsure about the old hostname, you can just list all the hostnames saved in Veeam, which includes the old ones with this query:

Code: Select all

select name from dbo.PhysicalHosts

and look through the list to find the correct one.

- The query to remove old records from the database to fix the issue is:

Code: Select all

delete from dbo.HostComponents where physical_host_id = '{old_host_id}' and type != '0'

where {old_host_id} is the ID of the old server found with the help of the first query.

Hope this helps someone else with this rare issue who might stumble on this single thread on the Internet about it.

[KLEHMAN]

Running the SQL above fix also worked for me with a similar issue! My error said "invalid packagetype restoreproxy". I wanted to make sure to post this for others in my shoes.

I uninstalled using this link,
https://www.veeam.com/kb4298

Then ran the sql commands

Then basically just went to "properties" on the server in B&R, and let in install the components.

Thanks!

[JasonC]

Got a similar issue after a cumulative patch but not with a specifically hardened server...

Upgrade of Veeam B&R from 11.0.1.1261 P20220302 to 11.0.1.1261 P20230227 completed successfully on server with no issues.

Following this the component update for the associated Linux Proxy/Repository fails with the error:
"Failed to upgrade host components. Error: 'Invalid package type DeployerSvc'

Anyone got any ideas - done a reasonable amount of obvious troubleshooting on this already (too much to detail here)

Case #06023209

[Ben.online]

I had the same issue with a LHR that was a Windows repo on the same hardware with the same name.
After upgrade to V12 the LHR could be edited succesfull but kept saying it needed to be upgraded. That failed with "Failed to upgrade host components. Error: 'Invalid component type Nfs' and later RestoreProxy.

Because the name is the same there is no old host ID
With support we found that the nfs role was still linked to the hostid

Take backups of DB

#MOD: SQL code removed

Than use that id to find all components for that host
#MOD: SQL code removed

Then delete the role
#MOD: SQL code removed

and rerun the wizard in Veeam

[Mildur]

Hi Ben

Thank you for providing an update.
Please do not provide SQL queries here in the forum or other public places. SQL queries are only supported with guidance from our support team.
They may work for one case, but could make things worse in other situations.
If you leave your case number here in this topic, other forum members with a similar issue can share your case number within their support case. Our support engineer will then be able to look up the SQL queries from your case and make sure that they can be used for the new case.

Thank you,
Fabian

[Ben.online]

Case #07175412
Hi Fabian,

The queries were non specific (not runnable as posted) and more or less a copy of those posted by TiredCustomer so i did not see any harm in it.
It gives a basic understanding of the databasetables involved.

Kind regards