Upgrade of VBR to 12.1 with Hardened Repo

[d.artzen]

Hello together,

I have a short question regarding the upgrade of the VBR to 12.1 in combination with a Linux Hardened Repo. We are currently using the latest 12.0 of VBR. In another thread post498961.html#p498961 Mildur explained to me, that starting with 12.0 there is no need for the veeam account on the Hardened Repo to have root priveleges nor SSH enabled anymore, because of the new veeamdeployerservice that is installed on the linux. Now in the Release Notes of the new 12.1 on page 39 it says, that it is once again necessary to enable SSH and give sudo rights to the veeam user.

Is this an error in the documentation or do we really need it for the upgrade from 12.0 to 12.1?

Best regards
Daniel

[HannesK]

Hello,
yes, unfortunately SSH is required to upgrade to 12.1. That was not planned and in Mildur's answer was correct until an unexpected corner case came up.

We are evaluating options to fix that in a cumulative update for 12.1 and do the upgrade without SSH requirement.

Best regards,
Hannes

[d.artzen]

Hello Hannes,

thank you for the fast response. I just wanted to make sure, this is not a real problem for me, just a minor inconvenience.

Best regards
Daniel

[Gostev]

Over 90% chance it should be addressed in the very first patch. We didn't know it was possible until a couple of days ago. First wrongly assumed it's not possible but apparently the functionality to handle such special cases is there - just never tested or approved for use. And at a first sight chances are good that it works.

So those for whom it is a major inconvenience can just wait for the first patch. There's even a note suggesting this possibility in the Release Notes (see the Upgrade section).

[d.artzen]

Hello again,

so the upgrade worked fine so far, just one thing is a bit strange. We have two Hardened Repos with Ubuntu 22.04 LTS. The second one was installed in "Minimized Mode", which has not been any problem so far. After the upgrade, if I go into the properties of this server in the VBR console and click through the wizard and press "Apply" it gives me a warning "Failed to open deployer service management port" at the beginning and a warning "Failed to close deployer service management port", but everything else is green. The first Hardened Repo Server (which was installed in "Standard Mode") did not give this warning.
The servers are identical with the only difference being the "Minimized Mode" on the second server. Could this be an issue here? Or any other ideas, what the issue could be here?

Best regards
Daniel

[Gostev]

Sounds like some local firewall management components are missing in Min mode.

[d.artzen]

Just to give a quick update: Even with the warnings our scheduled backup copy jobs to that repo were successful without any warnings this night.
But I wanted to the warnings in the properties of the repo gone and since "Minmal Mode" was not originally planned for our hardened repos, I ran the "unminimize" command on the server to bring it back to a standard Ubuntu installation. After completion and a reboot I clicked through the properties of the server in the VBR console and now all warnings are gone. So it really seems that 12.1 does something different (or has other requirements) than 12.0 in this scenario, since I know that previous updates/upgrades with this repo went without any warnings.
I just haven't found anything about this in the documentation or release notes.

[Gostev]

It does indeed, in order to further reduce the attack surface by minimizing the constantly listening ports to only one. Check out the Hardened Repository section in the What's New in 12.1 document for additional details.

[pesos]

Happy New Year everyone! Just wanted to check in on this as we have an existing 12.0 setup with hardened repo and we are looking at setting up another HR shortly. For now best to wait on the 12.1 patch (or just know we have to temp allow SSH on the one to be upgraded)?

[HannesK]

Hello,
I guess it does not really matter. It becomes more of a challenge if you have hundreds of Hardened Repositories and you need to open a ticket to get SSH enabled. If you control everything yourself, I see no reason to wait.

Best regards,
Hannes

[BackItUp2020]

I made the mistake of upgrading to 12.1 on a whim and ran into the same issue. Had to do a late-night datacenter trip and do all of the fun stuff (KVM up, Grub PW, enable admin, grant rights, etc., etc.).

Twas quite the exercise!

[Gostev]

Sorry to hear... you should take note of the Upgrade Checklist in the Release Notes documents in future, we try really hard to remember and mention all the impacting changes there.

[BackItUp2020]

Ooh I definitely new better

[pesos]

Thanks guys. Sorry didn’t respond back sooner but just realized I’m not getting email updates to tickets from this forum anymore for some reason. Is there a rough eta on the first patch when we expect this issue to be resolved? Thank you!

[Gostev]

It should be available in a few days hopefully, as we have reached the RC stage on Friday.

[pesos]

Patched and upgraded painlessly. Thanks guys!

[cdw23]

So I enabled ssh and sudo on the VHR to upgrade from 12.0 to 12.1 and had one minor misleading issue. When I used the "test credentials" button for the connection to the VHR it always failed ... but the credentials did work for the actual upgrade.

[Gostev]

So I enabled ssh and sudo on the VHR to upgrade from 12.0 to 12.1

To be clear, this should no longer be needed when using build 12.1.1.56 (or later) to upgrade to 12.1

[jeremyrogers]

I am encountering a similar issue in a lab test of the upgrade process from 12.0. Using a freshly downloaded iso (VeeamBackup&Replication_12.1.1.56_20240116) I get a component update failure calling out the lack of SSH credentials on our test VHR systems.

Updating from 12.0.0.1423 to 12.1.1.56. Working with a member of the Lighthouse team on it, but no official case # as of yet.

[HannesK]

Hello,
that should not be and I guess the fastest way to get an answer why it happens is a support case. Please post the case number that I can look into it.

I can see the issue also in one of my labs (worked fine in another lab) from 12.1 to 12.1.1 which we just started investigating. Maybe yours is the same. I will come back once I have an answer.

Best regards,
Hannes

[HannesK]

EDIT: the upgrade process was updated to remove open port 6160 requirement in VeeamBackup&Replication_12.1.1.56_20240127.iso

Hello,
without having a case number for your situations, I would guess that you configured the host-firewall yourself (or used the community built ISO of Hardened Repository / hardening script from Github). There is a bug in the firewall rules management. You probably cannot connect to port 6160 from the backup server.

Opening port 6160 manually on Hardened Repository or disabling the firewall entirely during upgrade solves it.

Best regards,
Hannes

[jeremyrogers]

In our case the VHRs are using the RedHat 8.8 distro & the DISA STIG was configured during the Anaconda setup phase + the post-setup grub hardening steps & fapolicyd rules to permit execution of the Veeam installer/agent/transport services. No manual tweaks to firewalld aside from whatever rules the Veeam installer pushes

HannesK's guess proved to be correct - there was only a single port entry in the firewalld config for port 6162/tcp. ADDING 6160/tcp allowed the update wizard to do its deed WITHOUT supplying SSH credentials.

I'm running some more tests before updating the case (Veeam Support - Case # 07106167), but so far all seems well. Thanks!

[HannesK]

Hello,
good to hear, that it works.

Yes, the SSH error is wrong and we are discussing a proper error message.

Best regards,
Hannes

[perjonsson1960]

Folks,

Where can I get the latest iso for B&R (is it VeeamBackup&Replication_12.1.1.56_20240116) mentioned above?
When I downloaded, I ended up with an iso called VeeamDataPlatform_23H2_20240118, and I don't think that is the right one. Please point me to the correct iso.

PJ

[d.artzen]

You can get the latest iso from here: https://www.veeam.com/download-version.html
Also you can find the link the newest versions (and KB-article for patches) in this stickied thread: veeam-backup-replication-f2/current-version-t9456.html

[perjonsson1960]

Thanks. I found the correct iso here: https://www.veeam.com/kb4510

PJ

[perjonsson1960]

Bottom line: When upgrading from 12.0.0.1420 P20230718 to 12.1.1.56 P20240116, do I need to enable SSH for the hardened repo or not?

PJ

[HannesK]

EDIT: the upgrade process was updated to remove open port 6160 requirement in VeeamBackup&Replication_12.1.1.56_20240127.iso

No. Only port 6160 needs to be opened manually if you use a host-firewall.

[perjonsson1960]

Thanks!

[HannesK]

we updated the ISO. Now it "just works". No SSH. No manual firewall configuration.

VeeamBackup&Replication_12.1.1.56_20240127.iso is the new file name