Linux Guest Processing: no matching key exchange method found

[daisho]

Hi,

for some reason since today a backup job cannot log in to one of my Ubuntu (18.04.3 LTS) servers anymore.
Veeam job shows:

Unable to perform guest file system indexing: Failed to login to host: '10.0.0.60', port: 22, user: 'web', elevation to root: 'yes', autoSudo: no, use su if sudo fails: no, auth type: 'Password', host name: nginx, IPs: [10.0.0.60].

auth.log on the affected machine when testing the credentials shows:

Oct 4 12:52:02 web sshd[4187]: Unable to negotiate with 10.0.0.70 port 52289: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1 [preauth]
Oct 4 12:52:05 web sshd[4189]: Invalid user plex from 10.0.0.70 port 52290
Oct 4 12:52:05 web sshd[4189]: pam_unix(sshd:auth): check pass; user unknown
Oct 4 12:52:05 web sshd[4189]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.0.0.70

Other Ubuntu machines (same patch level) have no problems with login for guest processing, all with default open-ssh config.
Also I don't get why after the failed key exchange negotiation a different user (for another machine) is used ('plex' here) instead of the one which is configured ('web' in this case).

So first a key exchange failure is posted, then Veeam tries to log in with a wrong user that doesn't exist on the system (and is not configured in the backup job for that machine)?

Anyone got into a similar problem?

Thanks, daisho

[PTide]

Hi,

Which VBR version are you running? Please also post the result of executing this line on Ubuntu:

Code: Select all

sshd -T | grep kexa

In addition to that, kindly try to edit the job, re-add the proper user in the guest processing settings for that machine, and see if it works.

Thanks!

[daisho]

Could not load host key: /etc/ssh/ssh_host_rsa_key
Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Could not load host key: /etc/ssh/ssh_host_ed25519_key
kexalgorithms curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1

List of algorithms is exactly the same as on another machine which does not have that issue.
Already did the re-applying of credentials, I even removed the VM itself from the job and also the credentials and added them again without success.
I now also completely re-created the whole backup job including credentials for that machine, still the same behavior (have to say, it is a Virtual Machine backup job if that matters).

Machines should be on the same level, though the failing one is a web server (with nginx and apache onboard) if that matters for some reason.
Is there a way (or need) to remove cached ssh credentials on Veeam side? (I remember that for windows ssh client one can e.g. remove known_hosts entries from .ssh folder to "refresh" if the key has changed on server side, not sure if that exists for Veeam (or vsphere?) somewhere).

Connecting using windows ssh client is no problem on all machines.

I also do not really understand why it tries to login then with a different credential which is not configured for that VM - it should not do that at all

[PTide]

Since you've already tried all basic troubleshooting steps, and Windows machines can log in just fine, I'd suggest to contact our support team directly as I suspect that there might be a problem with VBR's database (the rogue "plex" user). Once you open a case please post your case ID in this thread.

Thanks!

[daisho]

Hi, I created a case now for this issue (03832825), thank you for your support in this.

[FvdAa]

Similar issue here.

Backup of RHEL/CentOS 7 systems is working fine. Added a RHEL 8 system today, but got the same error when scanning the host.

Turns out that VBR is using ancient key exchange algorithm. RHEL 8 doesn't accept the 'diffie-hellman-group1-sha1' algorithm out of the box.

Adding it to the KexAlgorithms of SSHd made it work, but I think Veaam should upgrade their security algorithms.

Probably your systems received an update that also disabled those algorithms and now VBR can no longer connect.

[daisho]

The thing is, algorithm look fine on the linux machines and I have like 3 Ubuntu machines that are on same update level and all have same SSH config, and 2 work fine but the 3rd does not.
For me it looks like there is maybe some issue with cached credentials somewhere. Veeam itself thinks it is indeed connecting with the correct user (at least when I check the UI after a test) but on SSH server side log a wrong user is displayed that tried to connect (and there is no other process connecting to that machine, so I can sort that out).



Support told me to try alternative Renci SSH protocol by adding the registry DWORD32 to HKLM\SOFTWARE\Veeam\Veeam Backup and Replication\UseRenciSSH as REG_DWORD and set it to 1.
It looks like the setting will be read out immediately, otherwise to be sure just restart the Backup & Replication service.

That worked for me apparently.