Bug 2060232

Summary: openssl-3.0.4-14 does not allow ssh-rsa host keys for ssh
Product: Red Hat Enterprise Linux 9 Reporter: Brian Stinson <bstinson>
Component: opensshAssignee: Dmitry Belyavskiy <dbelyavs>
Status: CLOSED DUPLICATE QA Contact: Marek Havrila <mhavrila>
Severity: medium Docs Contact: Jan Fiala <jafiala>
Priority: medium    
Version: 9.0CC: dbelyavs, fge, griffint, jafiala, jjelen, jwboyer, mhavrila, pizhang, rduda, rpbikker, tscherf
Target Milestone: rcKeywords: Reopened, Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-06-07 09:05:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
sshd -T on the F35 server
none
"ssh -G F35-server" on the CS9 client
none
"ssh -vvv F35server" form CS9 client none

Description Brian Stinson 2022-03-03 02:40:39 UTC
With openssl-3.0.4-5 I could connect to RHEL/CentOS Linux systems that have sha256 ssh-rsa host keys.

Using the latest, openssl-3.0.4-14 I am not able to connect to hosts that only present ssh-rsa host keys, failing with an error message:

ssh_dispatch_run_fatal: Connection to 10.19.208.80 port 22: error in libcrypto

Hosts that present multiple keys (ssh-ed25519 in addition to ssh-rsa) connect properly, but show the following error during initial connection:

client_global_hostkeys_private_confirm: server gave bad signature for RSA key 0: error in libcrypto

The hosts that I tried all had sha256 ssh-rsa keys.

Comment 6 Dmitry Belyavskiy 2022-03-04 11:45:09 UTC
*** Bug 2060855 has been marked as a duplicate of this bug. ***

Comment 8 Dmitry Belyavskiy 2022-03-14 08:52:04 UTC
*** Bug 2063610 has been marked as a duplicate of this bug. ***

Comment 9 Radek Duda 2022-03-14 10:43:07 UTC
Workaround: update-crypto-policies --set LEGACY

Comment 10 Dmitry Belyavskiy 2022-03-24 17:00:33 UTC
Closing this as the behavior is intentional

Comment 11 Jakub Jelen 2022-03-24 19:29:19 UTC
The behavior is intentional, but the error message should be improved on the openssh side. It should fail with the normal "no common hostkey algorithms" rather than "error in libcrypto".

Normally, this would be handled by the crypto policies, which OpenSSH follows, but this does not apply for hostkey algorithms because setting them from c-p makes them ordered, which is not the intention and breaks later ordering based on local keys in known_hosts.

I proposed a change for OpenSSH to handle this in https://bugzilla.mindrot.org/show_bug.cgi?id=2924 three years ago.

Other option might be using the non-public openssl API to detect the rsa-sha1 support at runtime to fail cleaner.

Reopening and moving to OpenSSH as I believe this deserves some more attention.

Comment 13 Dmitry Belyavskiy 2022-03-24 20:48:26 UTC
I agree with your evaluation, but reduce the priority. Thanks!

Comment 14 Thorsten Scherf 2022-03-25 09:25:07 UTC
(In reply to Jakub Jelen from comment #11)
> The behavior is intentional, but the error message should be improved on the
> openssh side. It should fail with the normal "no common hostkey algorithms"
> rather than "error in libcrypto".

As discussed with Simo, I've opened the following BZ for the error message improvements:

https://bugzilla.redhat.com/show_bug.cgi?id=2068423

Comment 15 Dmitry Belyavskiy 2022-04-07 13:26:29 UTC
*** Bug 2073009 has been marked as a duplicate of this bug. ***

Comment 16 Terry Griffin 2022-05-28 03:54:38 UTC
Just FYI, one of the services that suffers from this error is AWS CodeCommit, when trying to clone a git repository using an SSH URL on RHEL 9.

I've confirmed that the 'LEGACY' workaround solves the problem.

Comment 17 Reinier Bikker 2022-05-28 22:02:03 UTC
I'm getting this error on a Centos Stream 9 host when connecting to Fedora 35 hosts. They are running the same version of openssh (8.7p1), with exactly the same crypto-policies for opensshserver, openssh and openssl. The ssh_host_rsa_keys on the Fedora hosts were newly generated under the DEFAULT crypto-policies (3072 bits). I don't know of any way to check whether sha1 is used for signing the keys or not. The only obvious difference is that Fedora 35 uses libcrypto from openssl-libs 1.1.1n and CS9 uses openssl-libs 3.0.1.

Furthermore, this error isn't purely cosmetic. It seems to stop the UpdateHostKeys process, as in that the ECDSA host keys also presented by the Fedora hosts are never added to the known_hosts file.

Comment 18 Jakub Jelen 2022-05-30 07:01:39 UTC
(In reply to Reinier Bikker from comment #17)
> I'm getting this error on a Centos Stream 9 host when connecting to Fedora
> 35 hosts. They are running the same version of openssh (8.7p1), with exactly
> the same crypto-policies for opensshserver, openssh and openssl. The
> ssh_host_rsa_keys on the Fedora hosts were newly generated under the DEFAULT
> crypto-policies (3072 bits). I don't know of any way to check whether sha1
> is used for signing the keys or not. The only obvious difference is that
> Fedora 35 uses libcrypto from openssl-libs 1.1.1n and CS9 uses openssl-libs
> 3.0.1.
> 
> Furthermore, this error isn't purely cosmetic. It seems to stop the
> UpdateHostKeys process, as in that the ECDSA host keys also presented by the
> Fedora hosts are never added to the known_hosts file.

You have probably some configuration that enforces ssh-rsa signature type somewhere. Please, share your complete client and server configuration and debug logs if you want somebody to have a look into your issue more. Fedora 35 supports RSA with SHA2 so there is no reason why it should not work with default/sane configuration.

Comment 19 Reinier Bikker 2022-05-30 11:26:37 UTC
Created attachment 1885062 [details]
sshd -T on the F35 server

Comment 20 Reinier Bikker 2022-05-30 11:28:35 UTC
Created attachment 1885075 [details]
"ssh -G F35-server" on the CS9 client

Comment 21 Reinier Bikker 2022-05-30 11:29:51 UTC
Created attachment 1885077 [details]
"ssh -vvv F35server" form CS9 client

Comment 22 Jakub Jelen 2022-05-30 12:06:10 UTC
Oh, yeah. It was bug in OpenSSH <8.9 servers: https://bugzilla.mindrot.org/show_bug.cgi?id=3375

Would be great if we would have this backported or rebased finally, but please open a separate bug for this as it is not completely related to the original issue.

Comment 23 Dmitry Belyavskiy 2022-05-30 12:28:13 UTC
I think this patch (or at least, my original version of this patch) is implemented in our code.

Comment 28 Red Hat Bugzilla 2023-09-15 01:52:29 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days