Your experiences with VSA/VIA-environments updated to v13.1.x

[Didi7]

Since the release of VSA or VIA v13.0.x we had no significant problems, also not during upgrades. Everything went smooth and systems rebooted properly.

At the day when we decided to upgrade to v13.1.x we were not at the latest updates of the current week, instead we decided to upgrade to v13.1.x directly. The upgrade did not went totally smooth. We had to repeat upgrades on several machines, some went fine, some did not went fine. It took several hours to complete the update for every machine in our infrastructure.

In the upcoming days weekly updates did not went smoothly as well, we had to interfer each time.

Probably because we always had troubles with our physical Hardened Repository systems. Every time updates had been installed, systems were no longer reachable and we had to restart them, some of them the hard way, as 'Sign in' wasn't possible.

These 11 Hardened Repository systems are up-to-date now, are reachable via Ping and in the infrastructure but the Console on 8 of 11 machines can not be used. Sign in is not possible. It is probably always the same behavior ... as soon as updates are installed, all 11 sytems do not communicate externally, they have to be rebooted manually and in the end most of them can not be operated Console-wise.

The rest of the machines are virtual VMware machines and they work properly. Sign in is possible. We checked every system today.

We would be interested to hear from your update experiences, from similar effects in your environments. Are there users with 'Sign in' problems on the Veeam Host Management console on physical Veeam Hardened Repository systems, e.g. you cannot press ENTER or RETURN to sign in? Do updates for v13.1.x run smoothly on your Hardened Repository systems?

[vnikiforov]

Hello,
Please avoid creating the same questions in different threads, as I already stated previously - what needs to be done is to create a support case.

Moderator edit: removed speculative content

[Didi7]

Hello vnikiforov,

you know we intended to get some experiences from other environments by creating this new thread. Are there environments with Veeam Host Management consoles, where login is no longer possible, as the console does not allow to sign in anymore.

As login is no longer possible, we cannot activate Web- or SSH-access as well. How should we generate logs in this case?

Of course we could open a support case, but currently we simply see no possibility to get access to the management of those system.

May we ask, if internal issue MODERATOR REMOVED is related to login issues in the Veeam Host Management console?

As we need to move forward, we seriously consider moving backup data from those backup repository system to another virtual VHR system and reconstruct the original system, just in case VHR reconstruction with the VIA ISO fails and we need to reconstruct those systems from scratch.

[vnikiforov]

Hello,
That is fair, as you can't login.

Moderator edit: removed speculative content

[Didi7]

I see, but the question that raises up here, is it really related to the NIC? It might be related to the NIC as well, but as those systems are still reachable from the VBR server, the communication works somehow and currently we would be able to move backup repository data somewhere else.

To us it looks more like a problem with the Veeam Host Management console and the keyboard driver.

It would be a total disaster, if communication from outside (the VBR server) would be disrupted as well, that is still not the case.

[vnikiforov]

Moderator edit: removed speculative content

[HannesK]

Hello,
I edited answers above and removed the issue number. First I like to repeat, that technical issues should go through support. Even if login is not possible, support will come up with ideas to investigate (e.g. using the live boot system).

As far as I understand, the problem is with signing into the the terminal user interface of the Host Management Console (the Host Management Console also exists as Web UI on port 10443). For the terminal user interface, the most common issue is usbguard. I have not heard any issues related to the 13.1 update, but that is something support can check. Network disconnects after upgrade that require a reboot to get networking again are not normal. That is something support should check.

The Host Management Console web interface on port 10443 is disabled per default on Hardened Repository systems. SSH is disabled per default on all appliances. From that side, everything seems to "works as designed".

I don't see any reason to re-install any of your systems from the description above. Support should check what the problem is and if needed, escalated it to R&D internally. There is no "wide spread issue" known for Hardened Repository upgrades that 8 out of 11 systems are unusable after upgrades. Something seems to be environment-specific.

Finally my question: what is your support case number

Best regards
Hannes

[Didi7]

Hello vladimir,

I read your last post before it was moderated. I appreciate your feedback and rest assured we are fully aware that you might not be in a position to provide a solution.

As we use a standard HPE DL380 Gen11 server with HPE certified hardware (fully updated BIOS, firmware and Intelligent Provisioning), we would not have expected to have a situation, where we are even no longer able to sign in to the Veeam Management Console! That's a basic feature, that must work.

The JeOS should at least work rock-solid on certified server systems from the usual server suppliers and yes it is a great challenge that such systems work on those different server systems, but we are talking about keyboard support and a login screen, still a basic feature. We are forced to hard reset these systems since the upgrade from 13.0.x and they worked so perfectly under the former 13.0.x. We would have never imagined that such basic features are blocked now.

It is comparable with a hardened Windows installation, where you are no longer able to login, as you might have removed access via RDP or whatever and you can do nothing about it.

Should people realize that systems might get inaccessible after an upgrade, maybe they think twice about upgrading or maybe change the underlying hardware, e.g. using a virtual VIA-based Hardened Repository, as long as e.g. VMware VIA-based VMs are working properly and reliable. Yes, new potential ways to comprimise the backup repository system!

In the end, the question might be ... is a virtual VMware, Hyper-V or Nutanix VIA-based Hardened Repository with accessible backups better, as a physical VIA-based Hardened Repository, that is inaccessible, at least from the console and also from anywhere else, as Web-console and SSH is disabled by default.

We are lucky, that the systems are currently still accessible from the VBR-server, so that repository data can be pulled.

Let's see how this ends now here!

[Didi7]

Hello Hannes,

Quote: The Host Management Console web interface on port 10443 is disabled per default on Hardened Repository systems. SSH is disabled per default on all appliances. From that side, everything seems to "works as designed"

Exactly and that also means we are unable to login to the WebGUI and reboot the system from there.

**********************************

Quote: I don't see any reason to re-install any of your systems from the description above. Support should check what the problem is and if needed, escalated it to R&D internally. There is no "wide spread issue" known for Hardened Repository upgrades that 8 out of 11 systems are unusable after upgrades. Something seems to be environment-specific.

10 systems based on HPE DL380 Gen11 server (8 of 10 still work at the login screen) => BIOS, Firmware and Intelligent Provisioning up-to-date and identical. 1 system based on Alletra 4120 server (doesn't work at login screen) => BIOS, Firmware and Intelligent Provisioning up-to-date.

But BIOS, Firmware and Intelligent Provisioning did not change before the upgrade. HW-changes were done afterwards in order to fix the SW problems with JeOS. We even downgraded one system and the problem remained.

Before we open a case, we will move backup repository data to a virtual VHR-system, so that in case of a disaster, we are still able to access the backup data of this system.