[BETA] [HPE] Configuring VME fails on Linux VSA

[ctalbot]

I can't seem to get past the error below.

I'm seeing this repeat in the logs several times before failing:

Code: Select all

2025-12-03 12:32:07.7108 00026 [2444] ERROR | [RetryHelper]: Failed "SSH connection to the cluster host 10.30.156.20". Error: "error:03000098:digital envelope routines::invalid digest": Interop+Crypto+OpenSslCryptographicException: error:03000098:digital envelope routines::invalid digest
2025-12-03 12:32:07.7108 00026 [2444] ERROR | [RetryHelper]:    at Interop.Crypto.RsaSignHash(SafeEvpPKeyHandle pkey, RSASignaturePaddingMode paddingMode, IntPtr digestAlgorithm, ReadOnlySpan`1 hash, Span`1 destination)
2025-12-03 12:32:07.7108 00026 [2444] ERROR | [RetryHelper]:    at System.Security.Cryptography.RSAOpenSsl.TrySignHash(ReadOnlySpan`1 hash, Span`1 destination, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding, Boolean allocateSignature, Int32& bytesWritten, Byte[]& signature)
2025-12-03 12:32:07.7108 00026 [2444] ERROR | [RetryHelper]:    at System.Security.Cryptography.RSAOpenSsl.SignHash(Byte[] hash, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding)
2025-12-03 12:32:07.7108 00026 [2444] ERROR | [RetryHelper]:    at System.Security.Cryptography.RSA.SignData(Byte[] data, Int32 offset, Int32 count, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding)
2025-12-03 12:32:07.7108 00026 [2444] ERROR | [RetryHelper]:    at Renci.SshNet.Security.Cryptography.RsaDigitalSignature.Sign(Byte[] input)
2025-12-03 12:32:07.7108 00026 [2444] ERROR | [RetryHelper]:    at Renci.SshNet.Security.KeyHostAlgorithm.Sign(Byte[] data)
2025-12-03 12:32:07.7108 00026 [2444] ERROR | [RetryHelper]:    at Renci.SshNet.PrivateKeyAuthenticationMethod.Authenticate(Session session)
2025-12-03 12:32:07.7108 00026 [2444] ERROR | [RetryHelper]:    at Renci.SshNet.AuthenticationMethod.Renci.SshNet.IAuthenticationMethod.Authenticate(ISession session)
2025-12-03 12:32:07.7108 00026 [2444] ERROR | [RetryHelper]:    at Renci.SshNet.ClientAuthentication.TryAuthenticate(ISession session, AuthenticationState authenticationState, String[] allowedAuthenticationMethods, SshAuthenticationException& authenticationException)
2025-12-03 12:32:07.7108 00026 [2444] ERROR | [RetryHelper]:    at Renci.SshNet.ClientAuthentication.Authenticate(IConnectionInfoInternal connectionInfo, ISession session)
2025-12-03 12:32:07.7108 00026 [2444] ERROR | [RetryHelper]:    at Renci.SshNet.ConnectionInfo.Authenticate(ISession session, IServiceFactory serviceFactory)
2025-12-03 12:32:07.7108 00026 [2444] ERROR | [RetryHelper]:    at Renci.SshNet.Session.ConnectAsync(CancellationToken cancellationToken)
2025-12-03 12:32:07.7108 00026 [2444] ERROR | [RetryHelper]:    at Renci.SshNet.BaseClient.CreateAndConnectSessionAsync(CancellationToken cancellationToken)
2025-12-03 12:32:07.7108 00026 [2444] ERROR | [RetryHelper]:    at Renci.SshNet.BaseClient.ConnectAsync(CancellationToken cancellationToken)
2025-12-03 12:32:07.7108 00026 [2444] ERROR | [RetryHelper]:    at Veeam.Vbf.Common.Helper.Retry.RetryHelper.<>c__DisplayClass4_0.<<ExecuteWithRetryAsync>b__0>d.MoveNext()
2025-12-03 12:32:07.7108 00026 [2444] ERROR | [RetryHelper]: --- End of stack trace from previous location ---
2025-12-03 12:32:07.7108 00026 [2444] ERROR | [RetryHelper]:    at Veeam.Vbf.Common.Helper.Retry.RetryHelper.ExecuteActionAsync[T](Func`2 asyncAction, String description, ILogger logger, LogLevel logLevel, CancellationToken cancellationToken)

[cody.ault]

Hey Carlos,

What OS is your hypervisor host running?

[ctalbot]

Hey Cody, it's running Ubuntu 24.04 as part of a 3-node VME cluster on 8.0.11.

[cody.ault]

Can you collect the log bundle and send the .\backup\plugins\hpemorpheusvme folder?

[EvgenyBaev]

Hello, ctalbot

Could you please do a few tests on the host 10.30.156.20.

1) Check OpenSSH version:
ssh -V
and provide the output.

2) Check the presence of ssh keys in /etc/ssh folder:
/etc/ssh/ssh_host_ed25519_key.pub
/etc/ssh/ssh_host_ecdsa_key.pub

3) Execute for those keys:
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
ssh-keygen -lf /etc/ssh/ssh_host_ecdsa_key.pub
and share the output

It could be an old fingerprint algorithm used by OpenSSH. Normally sha256 should be used by default in OpenSSH 6.8 and above.
It's from 6.8 release notes:
Fingerprints now have the hash algorithm prepended. An example of
the new format: SHA256:mVPwvezndPv/ARoIadVY98vAC0g+P/5633yTC4d/wXE
Please note that visual host keys will also be different.

If the algorithm is different from sha256 (e.g. MD5) we can have such issues.

FYI: SSH credentials are used only in the beta version and will be removed in the 1.0 release.

[ctalbot]

Code: Select all

root@hvm-host01:~# ssh -V
OpenSSH_9.6p1 Ubuntu-3ubuntu13.11, OpenSSL 3.0.13 30 Jan 2024
root@hvm-host01:~# ls -l /etc/ssh/ssh_host_*
-rw------- 1 root root  505 Aug 14 14:05 /etc/ssh/ssh_host_ecdsa_key
-rw-r--r-- 1 root root  177 Aug 14 14:05 /etc/ssh/ssh_host_ecdsa_key.pub
-rw------- 1 root root  411 Aug 14 14:05 /etc/ssh/ssh_host_ed25519_key
-rw-r--r-- 1 root root   97 Aug 14 14:05 /etc/ssh/ssh_host_ed25519_key.pub
-rw------- 1 root root 2602 Aug 14 14:05 /etc/ssh/ssh_host_rsa_key
-rw-r--r-- 1 root root  569 Aug 14 14:05 /etc/ssh/ssh_host_rsa_key.pub
root@hvm-host01:~# ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
256 SHA256:mNgGu4OEdKpDNZbGo63va2daIgVhkMj5lB+yJpHCZs8 root@hvm-host01 (ED25519)
root@hvm-host01:~# ssh-keygen -lf /etc/ssh/ssh_host_ecdsa_key.pub
256 SHA256:/VEVKNRn1jpc+wDx6Tx/VIbfRKLGXIxl8lnFJnYdmUs root@hvm-host01 (ECDSA)

[EvgenyBaev]

Hello, Carlos

Thank you. The output seems correct.

Could you please do a few more things:
1) Execute the same commands on the other nodes and compare the output. On my lab I see that OpenSSH version is slightly newer (13.14). I don't think it can be a reason but I would check just in case.
2) Share the content of /etc/ssh/sshd_config (via email to Cody). Or check if it's the same as on other nodes or different.
3) Check the output of "journalctl -u ssh" and /var/log/auth.log on problematic node when trying to add HPE server to VBR. Would be good to share it with us.
4) We can also check supported ssh algorithms on all nodes and compare:
sshd -T | grep -i algo

[ctalbot]

1) Looks like there is a slight variation on the version:

Code: Select all

root@nested-gfs2-01:~# ssh -V
OpenSSH_9.6p1 Ubuntu-3ubuntu13.14, OpenSSL 3.0.13 30 Jan 2024
root@nested-gfs2-01:~# ls -l /etc/ssh/ssh_host_*
-rw------- 1 root root  513 Oct 14 11:21 /etc/ssh/ssh_host_ecdsa_key
-rw-r--r-- 1 root root  181 Oct 14 11:21 /etc/ssh/ssh_host_ecdsa_key.pub
-rw------- 1 root root  411 Oct 14 11:21 /etc/ssh/ssh_host_ed25519_key
-rw-r--r-- 1 root root  101 Oct 14 11:21 /etc/ssh/ssh_host_ed25519_key.pub
-rw------- 1 root root 2602 Oct 14 11:21 /etc/ssh/ssh_host_rsa_key
-rw-r--r-- 1 root root  573 Oct 14 11:21 /etc/ssh/ssh_host_rsa_key.pub
root@nested-gfs2-01:~# ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
256 SHA256:C400/COTkSTHyOsYAkWZEDfQxY2uVjEvZbbmjU53MIk root@nested-gfs2-01 (ED25519)
root@nested-gfs2-01:~# ssh-keygen -lf /etc/ssh/ssh_host_ecdsa_key.pub
256 SHA256:D3eqWwAWDHKQVsxUSsf1caKLS/LAckpyVxfPI15ssYQ root@nested-gfs2-01 (ECDSA)

2) sshd_config is the same on the working and non-working node (10.30.156.16 & 10.30.156.20)
3) I emailed the contents to Cody.
4) on working node 10.30.156.16:

Code: Select all

root@nested-gfs2-01:~# sshd -T | grep -i algo
gssapikexalgorithms gss-group14-sha256-,gss-group16-sha512-,gss-nistp256-sha256-,gss-curve25519-sha256-,gss-group14-sha1-,gss-gex-sha1-
kexalgorithms sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
casignaturealgorithms ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
hostbasedacceptedalgorithms ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
hostkeyalgorithms ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
pubkeyacceptedalgorithms ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256

on non-working node 10.30.156.20:

Code: Select all

root@hvm-host01:~# sshd -T | grep -i algo
gssapikexalgorithms gss-group14-sha256-,gss-group16-sha512-,gss-nistp256-sha256-,gss-curve25519-sha256-,gss-group14-sha1-,gss-gex-sha1-
kexalgorithms sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
casignaturealgorithms ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
hostbasedacceptedalgorithms ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
hostkeyalgorithms ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
pubkeyacceptedalgorithms ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa

Looks like the non-working has ssh-rsa in the list whereas the other does not.