*** Bug 2069495 has been marked as a duplicate of this bug. ***

It's not enough to "disable" ssh-rsa* in libcrypto, they should be removed from the defaults from lists of the supported algos in ssh client and server both (directives hostkeyalgorithms hostbasedkeytypes casignaturealgorithms pubkeyacceptedkeytypes).
The reason behind that is, it tries to find the algo in the specified order that the both ends support. If you just disable support of them in the libcrypto without removing from the lists (what's done now), if there are no other supported by the both ends algos with higher prio, it will pick this one, but connection will fail, while if it's not on the list, it will not be chosen.
So, without updating the lists, you should expect your users to experience random ssh connection failures to different host (one client can connect to host A, but cannot to host B, while another client can connect to host B, but can connect to A), depending on versions and the orders in their configs.

Ideally, you would need to check if the algo specified in ssh config is supported by libcrypto. IDK if it's possible to do that and if so, how easy it would be.

But please, at least remove the disabled in libcrypto algorithms from the ssh defaults to prevent client/server handshake from choosing them and let the handshake a chance to pick supported algo at least by default.

*** Bug 2088916 has been marked as a duplicate of this bug. ***

*** Bug 2088386 has been marked as a duplicate of this bug. ***

Let me try again, since my bug on this issue https://bugzilla.redhat.com/show_bug.cgi?id=2088916
was peremptorily closed. It seems that ssh_keysign is puking on a key that was generated by your very
own sshd-keygen.target:


[root@localhost ssh]# cat /usr/lib/systemd/system/sshd-keygen.target 
[Unit]
Wants=sshd-keygen
Wants=sshd-keygen
Wants=sshd-keygen
PartOf=sshd.service

Just remove "rsa" if it is no longer supported.


Please take a look at #2088916 again. Why are you generating a key that your own code
refuses to sign?  I show in that bug specific steps to generate an inscrutable error
message. I wasted a lot of time debugging this before realizing that ssh-rsa keys
simply aren't working any more.

(In reply to Andrew Schorr from comment #5)
> Please take a look at #2088916 again. Why are you generating a key that your
> own code
> refuses to sign?  I show in that bug specific steps to generate an
> inscrutable error
> message. I wasted a lot of time debugging this before realizing that ssh-rsa
> keys
> simply aren't working any more.

The RSA key is still ok. The SHA1 signatures are not ok. Please, provide complete debug log if you want somebody to look into your issue. My assumption is that you have "ssh-rsa" somewhere in some configuration file or you are connecting to some legacy system which does not support SHA2 signatures with RSA, which I already mentioned in the previous bug.

Either I'm losing my mind, or running "/usr/libexec/openssh/sshd-keygen rsa" on a CentOS Stream 9
system (as it is invoked by sshd-keygen.target via sshd-keygen@.service) results in an ssh-rsa key,
not an SHA2 variant. I just did it, and here's what I get:

[root@ti138 system]# cat sshd-keygen@.service 
[Unit]
Description=OpenSSH %i Server Key Generation
ConditionFileNotEmpty=|!/etc/ssh/ssh_host_%i_key

[Service]
Type=oneshot
EnvironmentFile=-/etc/sysconfig/sshd
ExecStart=/usr/libexec/openssh/sshd-keygen %i

[Install]
WantedBy=sshd-keygen.target
[root@ti138 system]# /usr/libexec/openssh/sshd-keygen rsa
[root@ti138 system]# ls -l /etc/ssh
total 584
-rw-r--r-- 1 root root     578094 Feb 22 07:32 moduli
-rw-r--r-- 1 root root       1921 Feb 22 07:32 ssh_config
drwxr-xr-x 2 root root         51 May 23 23:05 ssh_config.d
-rw-r----- 1 root ssh_keys   2578 May 24 08:03 ssh_host_rsa_key
-rw-r--r-- 1 root root        554 May 24 08:03 ssh_host_rsa_key.pub
lrwxrwxrwx 1 root root         23 May 13  2012 ssh_known_hosts -> ../conf/ssh_known_hosts
-rw------- 1 root root       3669 Feb 22 07:32 sshd_config
drwx------ 2 root root         51 May 23 23:05 sshd_config.d
[root@ti138 system]# cat /etc/ssh/ssh_host_rsa_key.pub 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDb2UjEJfzljkCiCvgy68U4kYhOwrbEQt2IoY9N4RLniqSKcAgmmPVnVlYEUPAoGVwAPlQoPp8N1etJgsJ8DV0FEwVIqwtoDkCCH48CmxOmQVj16UpwSbSpq3DvtQKYweQ+ziqf0mMci4dOfb2NyUQ6C5BZsxPPOBmgaqXFFETBYMaCqxRistkIHd/nLnVkRjni72Vy4PPQ8oxMibwRGokY9FLX1KyUgP9raSwy8RrEvNLFNzNGw7F1NzQMEkTQpnzXynGPhbwMBbDSTHii3rERTp/CkUruOzm18XWPSq22FJ6ROSjy8vd+swmEvc0dNp2CNAuzNkQmXoCmT1jgAqJs/cGqN7ViFC+qR8ioZne+RfevhIVKIUp0GXYmQ8+hZ2iPnaI+Z5SOgm0On8D/RZRuSaNmT0vhORNsPYZSYtK/3Ypj/AWSEmznGli6iYWfoucaRP01jHqZbRe/36pXJDZTKPt9JSieWPcj8AnzCPTOEcm47v5SYuzH64bihV0JlZM= 
[root@ti138 system]# cat sshd-keygen.target 
[Unit]
Wants=sshd-keygen
Wants=sshd-keygen
Wants=sshd-keygen
PartOf=sshd.service

So is the system supposed to be generating an ssh-rsa key by default? That seems weird
if they're no longer supported. Or is the idea that you're doing that for people
who are going to set the crypto policy to add back support for SHA1 RSA keys?

The ssh-keygen man page says:


     -t dsa | ecdsa | ecdsa-sk | ed25519 | ed25519-sk | rsa
             Specifies the type of key to create.  The possible values are “dsa”, “ecdsa”, “ecdsa-sk”, “ed25519”, “ed25519-sk”, or “rsa”.

             This flag may also be used to specify the desired signature type when signing certificates using an RSA CA key.  The available RSA sig‐
             nature variants are “ssh-rsa” (SHA1 signatures, not recommended), “rsa-sha2-256”, and “rsa-sha2-512” (the default).

It appears to me that it's actually producing ssh-rsa, not rsa-sha2-512, so this seems like a bug
in ssh-keygen. Is there some other option to ssh-keygen in addition to -t rsa that's required
to select rsa-sha2-512 instead of ssh-rsa?

But the main point remains that the error messages are really bad and offer no useful guidance about what might be going wrong.

Please, check RFC 8332 [1]. The key type is not the same as a signature type. The key itself does not have anything to do with the digest. It comes into the play only when the signature is created and the algorithm is negotiated during the connection.

In any case, we should try to make the error message more clear.

https://datatracker.ietf.org/doc/html/rfc8332

OK, fair enough. But I don't think that affects the point I think I demonstrated (but maybe I'm
hallucinating): the ssh-keygen -t rsa command is
producing a key that is not accepted by the openssh keysign code. Why is that?

Also, is there some way to examine a key labeled ssh-rsa to identify whether
it's actually ssh-rsa (SHA1) or rsa-sha2-256 or rsa-sha2-512? How can one
tell them apart? Sorry if this is a stupid question.

Can you provide the debug log and affected server and client configuration?

The key is ssh-rsa always. There are no labels on them. The difference is only in the way how it is used by the client and servers to create signature (and verify it on the other hand indeed, but this side does not have any control of what the signer did).

Copying from bug 2088916:

Steps to Reproduce:
1. In a freshly installed CentOS Stream 9 qemu kvm, create /etc/ssh/sshd_config.d/99-local.conf with these contents:
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostbasedAuthentication yes
PasswordAuthentication no
Then create /etc/ssh/ssh_config.d/99-local.conf with these contents:
HostbasedAuthentication yes
EnableSSHKeysign yes
2. systemctl restart sshd
3. ssh -n localhost uptime

Actual results:
[root@localhost ssh]# ssh -n localhost uptime
ssh-keysign: sshkey_sign failed: error in libcrypto
ssh_keysign: no reply
sign using hostkey ssh-rsa SHA256:RUgzE2VqS2tcUzf1UM+x6UQX2GzQF25mL+4HPn+lScU failed
root@localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,hostbased).

[root@localhost ~]# ssh -v -n localhost uptime
OpenSSH_8.7p1, OpenSSL 3.0.1 14 Dec 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Reading configuration data /etc/ssh/ssh_config.d/99-local.conf
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Reading configuration data /etc/ssh/ssh_config.d/99-local.conf
debug1: Connecting to localhost [::1] port 22.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa_sk type -1
debug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_ed25519_sk type -1
debug1: identity file /root/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.7
debug1: compat_banner: match: OpenSSH_8.7 pat OpenSSH* compat 0x04000000
debug1: Authenticating to localhost:22 as 'root'
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes256-gcm MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:54MUE5WrCw0N8jfadIPyz5PezFb0QvitCSjcFlUHh68
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'localhost' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: /root/.ssh/id_rsa 
debug1: Will attempt key: /root/.ssh/id_dsa 
debug1: Will attempt key: /root/.ssh/id_ecdsa 
debug1: Will attempt key: /root/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /root/.ssh/id_ed25519 
debug1: Will attempt key: /root/.ssh/id_ed25519_sk 
debug1: Will attempt key: /root/.ssh/id_xmss 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256,webauthn-sk-ecdsa-sha2-nistp256>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,hostbased
debug1: Next authentication method: gssapi-with-mic
debug1: No credentials were supplied, or the credentials were unavailable or inaccessible
No Kerberos credentials available (default cache: KCM:)


debug1: No credentials were supplied, or the credentials were unavailable or inaccessible
No Kerberos credentials available (default cache: KCM:)


debug1: Next authentication method: hostbased
debug1: userauth_hostbased: trying hostkey ssh-ed25519 SHA256:RwdE2ivivfuECyGCYfJQL3bFjr82J+IRA2JGHdM57Pg
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,hostbased
debug1: userauth_hostbased: trying hostkey ecdsa-sha2-nistp256 SHA256:54MUE5WrCw0N8jfadIPyz5PezFb0QvitCSjcFlUHh68
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,hostbased
debug1: userauth_hostbased: trying hostkey ssh-rsa SHA256:JmPpNluivxIVVtgozzkc35z7wH3asKWddp3WcKQUtsM
ssh-keysign: sshkey_sign failed: error in libcrypto
ssh_keysign: no reply
sign using hostkey ssh-rsa SHA256:JmPpNluivxIVVtgozzkc35z7wH3asKWddp3WcKQUtsM failed
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/id_rsa
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ecdsa_sk
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: Trying private key: /root/.ssh/id_ed25519_sk
debug1: Trying private key: /root/.ssh/id_xmss
debug1: No more authentication methods to try.
root@localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,hostbased).
[root@localhost ~]#

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (openssh bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8375

https://bugzilla.redhat.com/show_bug.cgi?id=2209096#c11 contains the explanation why this approach was chosen.