Bug 2088750 - sshd drops connection with "fatal: mm_answer_sign: sign: error in libcrypto"
Summary: sshd drops connection with "fatal: mm_answer_sign: sign: error in libcrypto"
Keywords:
Status: NEW
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: openssh
Version: 9.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Dmitry Belyavskiy
QA Contact: Marek Havrila
URL:
Whiteboard:
: 2095656 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-05-20 12:11 UTC by Pierre Ossman
Modified: 2022-11-22 11:03 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker CRYPTO-7287 0 None None None 2022-05-23 09:31:58 UTC
Red Hat Issue Tracker RHELPLAN-122655 0 None None None 2022-05-20 12:15:42 UTC

Description Pierre Ossman 2022-05-20 12:11:36 UTC
Description of problem:

When connecting with the ThinLinc client (which uses ssh), sshd in RHEL 9 just drops the connection right after authentication. The client gets a TCP reset, and we see this in the journal on the server:

> May 20 09:13:37 lab-96.lkpg.cendio.se sshd[15387]: Accepted password for cendio from 10.47.1.240 port 43536 ssh2
> May 20 09:13:37 lab-96.lkpg.cendio.se systemd-logind[872]: New session 20 of user cendio.
> May 20 09:13:37 lab-96.lkpg.cendio.se systemd[1]: Started Session 20 of User cendio.
> May 20 09:13:37 lab-96.lkpg.cendio.se sshd[15387]: pam_unix(sshd:session): session opened for user cendio(uid=1000) by (uid=0)
> May 20 09:13:37 lab-96.lkpg.cendio.se sshd[15387]: fatal: mm_answer_sign: sign: error in libcrypto
> May 20 09:13:37 lab-96.lkpg.cendio.se sshd[15387]: pam_unix(sshd:session): session closed for user cendio
> May 20 09:13:37 lab-96.lkpg.cendio.se sshd[15391]: fatal: mm_request_send: write: Broken pipe
> May 20 09:13:37 lab-96.lkpg.cendio.se systemd[1]: session-20.scope: Deactivated successfully.
> May 20 09:13:37 lab-96.lkpg.cendio.se systemd-logind[872]: Session 20 logged out. Waiting for processes to exit.
> May 20 09:13:37 lab-96.lkpg.cendio.se systemd-logind[872]: Removed session 20.


Version-Release number of selected component (if applicable):

openssh-server-8.7p1-8.el9.x86_64


How reproducible:
100%

Steps to Reproduce:

1. Default install of RHEL 9
2. Default install of ThinLinc
3. Connect

(can't reproduce this with a manual run of ssh yet. trying to find the exact trigger for easier testing)


Actual results:

Client reports connection reset.


Expected results:

Connection succeeds.


Additional info:

This worked fine in the beta. As does Fedora 35, which has the same version of OpenSSH. So this is something that is specific to RHEL 9 GA.

Bugzilla entry for ThinLinc:

https://www.cendio.com/bugzilla/show_bug.cgi?id=7933

Comment 1 Pierre Ossman 2022-05-20 12:49:55 UTC
My bad, it actually easily happens with normal ssh clients as well. I had some local configuration that hid the issue.

To reproduce:

1. Wipe the host from ~/.ssh/known_hosts
2. ssh -o HostKeyAlgorithms=ssh-rsa,ecdsa-sha2-nistp256 rhel9.example.com

This will consistently fail until you connect without that option so that the host keys can be stored.

Comment 2 Jakub Jelen 2022-05-23 06:23:00 UTC
RSA signatures with SHA1 (ssh-rsa) are deprecated in RHEL9 so this is a bug in ThinLinc not supporting RFC 8332 [1]. You can workaround this by setting crypto policy to LEGACY or DEFAULT:SHA1 if you need to use legacy algorithms like this.

[1] https://datatracker.ietf.org/doc/html/rfc8332

Comment 3 Pierre Ossman 2022-05-23 06:51:02 UTC
ThinLinc supports the new ones as well, I guess we just forgot to add them in the host key exchange.

However, it has little bearing on this bug. RHEL 9's sshd gladly supports ecdsa-sha2-nistp256 and there are no issues performing the host key handshake. The fact that it crashes later if there is an unsupported algorithm that's supported by the client is most definitely a bug in RHEL 9.

It's one thing to require modern algorithms. It's an entirely different thing to refuse clients that *also* support older algorithms.

Comment 4 Pierre Ossman 2022-05-23 07:01:07 UTC
Oddly enough, this doesn't happen with either Fedora 34, or Fedora 35.

Fedora 34:

> debug3: receive packet: type 81
> debug3: client_global_hostkeys_private_confirm: verify RSA key 0 using default sigalg
> Learned new hostkey: RSA SHA256:+eCw/M0ODWI+rC5JrHDvTSdhStAZrsdHWJN5qNitG+A
> debug3: hostkeys_foreach: reading file "/home/ossman/.ssh/known_hosts"
> debug3: host_delete: ECDSA key already at /home/ossman/.ssh/known_hosts:522
> Adding new key for localhost to /home/ossman/.ssh/known_hosts: ssh-rsa SHA256:+eCw/M0ODWI+rC5JrHDvTSdhStAZrsdHWJN5qNitG+A
> debug1: update_known_hosts: known hosts file /home/ossman/.ssh/known_hosts2 does not exist

Fedora 35:

> debug3: receive packet: type 81
> debug3: client_global_hostkeys_private_confirm: verify RSA key 0 using default sigalg
> Learned new hostkey: RSA SHA256:JDcS3DiIoOuRNT5yUyuA9k5F5EaMOZExaMWVAz6Q1hA
> debug3: hostkeys_foreach: reading file "/home/ossman/.ssh/known_hosts"
> debug3: host_delete: ECDSA key already at /home/ossman/.ssh/known_hosts:527
> Adding new key for samuel to /home/ossman/.ssh/known_hosts: ssh-rsa SHA256:JDcS3DiIoOuRNT5yUyuA9k5F5EaMOZExaMWVAz6Q1hA

Fedora 34 has: OpenSSH_8.7p1, OpenSSL 1.1.1n  FIPS 15 Mar 2022
Fedora 35 has: OpenSSH_8.8p1, OpenSSL 3.0.2 15 Mar 2022

Both systems are running the DEFAULT crypto policy.

So, I guess this is an issue with the specific combination of OpenSSH 8.7 and OpenSSL 3?

Comment 5 Pierre Ossman 2022-05-23 07:42:33 UTC
It also works with RHEL 9 if the modern algorithms are also mentioned. So the mere presence of "ssh-rsa" is not enough to trigger the bug. It needs to be the only alternative for RSA keys.

Unfortunately, ThinLinc has no settings to control this, so it is unfortunately not a workaround from the client side of things. :/

Comment 6 Thomas Juberg 2022-05-24 21:44:37 UTC
I'm hitting this issue with RHEL 9 minimal install when connecting using Fedora 36 with OpenSSH_8.8p1, OpenSSL 3.0.2 15 Mar 2022, using https://infosec.mozilla.org/guidelines/openssh.html#modern crypto settings on the client
HostKeyAlgorithms ssh-ed25519-cert-v01,ssh-rsa-cert-v01,ssh-ed25519,ssh-rsa,ecdsa-sha2-nistp521-cert-v01,ecdsa-sha2-nistp384-cert-v01,ecdsa-sha2-nistp256-cert-v01,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256

KexAlgorithms curve25519-sha256,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256

MACs hmac-sha2-512-etm,hmac-sha2-256-etm,umac-128-etm,hmac-sha2-512,hmac-sha2-256,umac-128

Ciphers chacha20-poly1305,aes256-gcm,aes128-gcm,aes256-ctr,aes192-ctr,aes128-ctr

I can connect and authenticate 
/var/log/messages shows me systemd[1]: Started Session 21 of User x.
Followed by /var/log/secure sshd stating that sessions opened for user, followed by fatal: mm_answer_sign: sign: error in libcrypto

If i revert to Fedora default crypto, no errors.

I find it odd that OpenSSH negotiating and agreeing upon a common set of algorithms, Ciphers and MACs should end up with unexpectd disconnect with a somewhat cryptic and unexpected libcrypto error

Comment 7 Jakub Jelen 2022-05-25 08:05:42 UTC
(In reply to Thomas Juberg from comment #6)
> https://infosec.mozilla.org/guidelines/openssh.html#modern crypto settings
> on the client
> HostKeyAlgorithms
> ssh-ed25519-cert-v01,ssh-rsa-cert-v01,ssh-ed25519,
> ssh-rsa,ecdsa-sha2-nistp521-cert-v01,ecdsa-sha2-nistp384-cert-
> v01,ecdsa-sha2-nistp256-cert-v01,ecdsa-sha2-nistp521,
> ecdsa-sha2-nistp384,ecdsa-sha2-nistp256

This sounds like very outdated configuration as it does not list the SHA2 variants of RSA keys at all, which are present and standardized for several years:

https://datatracker.ietf.org/doc/html/rfc8332

Please, add to this line the following arguments

   rsa-sha2-512-cert-v01,rsa-sha2-256-cert-v01,rsa-sha2-512,rsa-sha2-256

and remove the following ones:

   ssh-rsa-cert-v01,ssh-rsa

Comment 8 Jakub Jelen 2022-05-25 08:14:24 UTC
FYI I opened https://github.com/mozilla/infosec.mozilla.org/pull/162

Comment 10 Dmitry Belyavskiy 2022-06-21 14:40:23 UTC
*** Bug 2095656 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.