My bad, it actually easily happens with normal ssh clients as well. I had some local configuration that hid the issue.

To reproduce:

1. Wipe the host from ~/.ssh/known_hosts
2. ssh -o HostKeyAlgorithms=ssh-rsa,ecdsa-sha2-nistp256 rhel9.example.com

This will consistently fail until you connect without that option so that the host keys can be stored.

RSA signatures with SHA1 (ssh-rsa) are deprecated in RHEL9 so this is a bug in ThinLinc not supporting RFC 8332 [1]. You can workaround this by setting crypto policy to LEGACY or DEFAULT:SHA1 if you need to use legacy algorithms like this.

[1] https://datatracker.ietf.org/doc/html/rfc8332

ThinLinc supports the new ones as well, I guess we just forgot to add them in the host key exchange.

However, it has little bearing on this bug. RHEL 9's sshd gladly supports ecdsa-sha2-nistp256 and there are no issues performing the host key handshake. The fact that it crashes later if there is an unsupported algorithm that's supported by the client is most definitely a bug in RHEL 9.

It's one thing to require modern algorithms. It's an entirely different thing to refuse clients that *also* support older algorithms.

Oddly enough, this doesn't happen with either Fedora 34, or Fedora 35.

Fedora 34:

> debug3: receive packet: type 81
> debug3: client_global_hostkeys_private_confirm: verify RSA key 0 using default sigalg
> Learned new hostkey: RSA SHA256:+eCw/M0ODWI+rC5JrHDvTSdhStAZrsdHWJN5qNitG+A
> debug3: hostkeys_foreach: reading file "/home/ossman/.ssh/known_hosts"
> debug3: host_delete: ECDSA key already at /home/ossman/.ssh/known_hosts:522
> Adding new key for localhost to /home/ossman/.ssh/known_hosts: ssh-rsa SHA256:+eCw/M0ODWI+rC5JrHDvTSdhStAZrsdHWJN5qNitG+A
> debug1: update_known_hosts: known hosts file /home/ossman/.ssh/known_hosts2 does not exist

Fedora 35:

> debug3: receive packet: type 81
> debug3: client_global_hostkeys_private_confirm: verify RSA key 0 using default sigalg
> Learned new hostkey: RSA SHA256:JDcS3DiIoOuRNT5yUyuA9k5F5EaMOZExaMWVAz6Q1hA
> debug3: hostkeys_foreach: reading file "/home/ossman/.ssh/known_hosts"
> debug3: host_delete: ECDSA key already at /home/ossman/.ssh/known_hosts:527
> Adding new key for samuel to /home/ossman/.ssh/known_hosts: ssh-rsa SHA256:JDcS3DiIoOuRNT5yUyuA9k5F5EaMOZExaMWVAz6Q1hA

Fedora 34 has: OpenSSH_8.7p1, OpenSSL 1.1.1n  FIPS 15 Mar 2022
Fedora 35 has: OpenSSH_8.8p1, OpenSSL 3.0.2 15 Mar 2022

Both systems are running the DEFAULT crypto policy.

So, I guess this is an issue with the specific combination of OpenSSH 8.7 and OpenSSL 3?

It also works with RHEL 9 if the modern algorithms are also mentioned. So the mere presence of "ssh-rsa" is not enough to trigger the bug. It needs to be the only alternative for RSA keys.

Unfortunately, ThinLinc has no settings to control this, so it is unfortunately not a workaround from the client side of things. :/

I'm hitting this issue with RHEL 9 minimal install when connecting using Fedora 36 with OpenSSH_8.8p1, OpenSSL 3.0.2 15 Mar 2022, using https://infosec.mozilla.org/guidelines/openssh.html#modern crypto settings on the client
HostKeyAlgorithms ssh-ed25519-cert-v01,ssh-rsa-cert-v01,ssh-ed25519,ssh-rsa,ecdsa-sha2-nistp521-cert-v01,ecdsa-sha2-nistp384-cert-v01,ecdsa-sha2-nistp256-cert-v01,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256

KexAlgorithms curve25519-sha256,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256

MACs hmac-sha2-512-etm,hmac-sha2-256-etm,umac-128-etm,hmac-sha2-512,hmac-sha2-256,umac-128

Ciphers chacha20-poly1305,aes256-gcm,aes128-gcm,aes256-ctr,aes192-ctr,aes128-ctr

I can connect and authenticate 
/var/log/messages shows me systemd[1]: Started Session 21 of User x.
Followed by /var/log/secure sshd stating that sessions opened for user, followed by fatal: mm_answer_sign: sign: error in libcrypto

If i revert to Fedora default crypto, no errors.

I find it odd that OpenSSH negotiating and agreeing upon a common set of algorithms, Ciphers and MACs should end up with unexpectd disconnect with a somewhat cryptic and unexpected libcrypto error

(In reply to Thomas Juberg from comment #6)
> https://infosec.mozilla.org/guidelines/openssh.html#modern crypto settings
> on the client
> HostKeyAlgorithms
> ssh-ed25519-cert-v01,ssh-rsa-cert-v01,ssh-ed25519,
> ssh-rsa,ecdsa-sha2-nistp521-cert-v01,ecdsa-sha2-nistp384-cert-
> v01,ecdsa-sha2-nistp256-cert-v01,ecdsa-sha2-nistp521,
> ecdsa-sha2-nistp384,ecdsa-sha2-nistp256

This sounds like very outdated configuration as it does not list the SHA2 variants of RSA keys at all, which are present and standardized for several years:

https://datatracker.ietf.org/doc/html/rfc8332

Please, add to this line the following arguments

   rsa-sha2-512-cert-v01,rsa-sha2-256-cert-v01,rsa-sha2-512,rsa-sha2-256

and remove the following ones:

   ssh-rsa-cert-v01,ssh-rsa

FYI I opened https://github.com/mozilla/infosec.mozilla.org/pull/162

*** Bug 2095656 has been marked as a duplicate of this bug. ***

This bug is not fixed in RHEL-9.1 and it deserves further investigation. Relevant part of log from RHEL-9.2:

Dec 15 06:58:46 localhost sshd[4321]: debug1: server_input_global_request: rtype hostkeys-prove-00 want_reply 1
Dec 15 06:58:46 localhost sshd[4321]: debug3: server_input_hostkeys_prove: sign RSA key (index 0) using sigalg default
Dec 15 06:58:46 localhost sshd[4321]: debug3: mm_sshkey_sign: entering
Dec 15 06:58:46 localhost sshd[4321]: debug3: mm_request_send: entering, type 6
Dec 15 06:58:46 localhost sshd[4318]: debug3: mm_request_receive: entering
Dec 15 06:58:46 localhost sshd[4318]: debug3: monitor_read: checking request 6
Dec 15 06:58:46 localhost sshd[4318]: debug3: mm_answer_sign: entering
Dec 15 06:58:46 localhost sshd[4318]: fatal: mm_answer_sign: sign: error in libcrypto
Dec 15 06:58:46 localhost sshd[4318]: debug1: do_cleanup
Dec 15 06:58:46 localhost sshd[4318]: debug1: PAM: cleanup
Dec 15 06:58:46 localhost sshd[4318]: debug1: PAM: closing session
Dec 15 06:58:46 localhost sshd[4318]: pam_unix(sshd:session): session closed for user root

Also affects podman with some docker.io images: 

`podman pull docker.io/library/postgres:15.1-alpine` hangs on CentOS Stream, OK on other distributions. Other images from docker.io were fine literally 1 hour since this comment, now have the same problem.

After a minute, outputs:

WARN[0060] Failed, retrying in 1s ... (1/3). Error: initializing source docker://postgres:15.1-alpine: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp 44.205.64.79:443: i/o time

And shows `sshd[6321]: fatal: mm_answer_sign: sign: error in libcrypto` in the journal.

Nevermind, looks like the podman issue is unrelated. My bad.

For me, upgrading to openssh-8.7p1-28.el9 and running update-crypto-policy --set LEGACY   sadly did NOT fix the problem.

It is intended to be fixed in 9.2

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (openssh bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2554