RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2073009 - ssh-agent signing failed for 2048 SSH256 key with default crypto-policy
Summary: ssh-agent signing failed for 2048 SSH256 key with default crypto-policy
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: openssl
Version: CentOS Stream
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Dmitry Belyavskiy
QA Contact: BaseOS QE Security Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-04-07 13:13 UTC by Josh Boyer
Modified: 2022-04-11 18:50 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-04-11 18:49:57 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-118256 0 None None None 2022-04-07 13:28:43 UTC

Description Josh Boyer 2022-04-07 13:13:40 UTC 
Description of problem:

Trying to git pull from code.engineering.redhat.com with a local 2048 bit RSA key results in this error:

[jwboyer@zod penumbra-baseline-data]$ git pull
sign_and_send_pubkey: signing failed for RSA "id_rsa-redhat" from agent: agent refused operation

code.engineering.redhat.com has multiple host keys, but the one used for this exchange is ED25519.

In /var/log/messages, I see:

Apr  7 08:49:04 zod ssh-agent[8876]: error: process_sign_request2: sshkey_sign: error in libcrypto

If I switch the crypto-policy to LEGACY and start a new ssh-agent, things work.  A debug run of just ssh with LEGACY shows:

[jwboyer@zod penumbra-baseline-data]$ ssh -A -v jboyer.redhat.com
OpenSSH_8.7p1, OpenSSL 3.0.1 14 Dec 2021
debug1: Reading configuration data /home/jwboyer/.ssh/config
debug1: /home/jwboyer/.ssh/config line 10: Applying options for code.engineering.redhat.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /home/jwboyer/.ssh/config
debug1: /home/jwboyer/.ssh/config line 10: Applying options for code.engineering.redhat.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Connecting to code.engineering.redhat.com [10.19.208.64] port 22.
debug1: Connection established.
debug1: identity file /home/jwboyer/.ssh/id_rsa type -1
debug1: identity file /home/jwboyer/.ssh/id_rsa-cert type -1
debug1: identity file /home/jwboyer/.ssh/id_dsa type -1
debug1: identity file /home/jwboyer/.ssh/id_dsa-cert type -1
debug1: identity file /home/jwboyer/.ssh/id_ecdsa type -1
debug1: identity file /home/jwboyer/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/jwboyer/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/jwboyer/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/jwboyer/.ssh/id_ed25519 type -1
debug1: identity file /home/jwboyer/.ssh/id_ed25519-cert type -1
debug1: identity file /home/jwboyer/.ssh/id_ed25519_sk type -1
debug1: identity file /home/jwboyer/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/jwboyer/.ssh/id_xmss type -1
debug1: identity file /home/jwboyer/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.7
debug1: Remote protocol version 2.0, remote software version GerritCodeReview_3.4.4 (APACHE-SSHD-2.6.0)
debug1: compat_banner: no match: GerritCodeReview_3.4.4 (APACHE-SSHD-2.6.0)
debug1: Authenticating to code.engineering.redhat.com:22 as 'jboyer'
debug1: load_hostkeys: fopen /home/jwboyer/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: ecdh-sha2-nistp256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: aes256-gcm MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm MAC: <implicit> compression: none
debug1: kex: ecdh-sha2-nistp256 need=32 dh_need=32
debug1: kex: ecdh-sha2-nistp256 need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:MO5xUAge3Tgbv9hnahccKjARZROZzcpiT/i67/TJ9Ro
debug1: load_hostkeys: fopen /home/jwboyer/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'code.engineering.redhat.com' is known and matches the ED25519 host key.
debug1: Found key in /home/jwboyer/.ssh/known_hosts:154
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: /home/jwboyer/.ssh/id_rsa-redhat RSA SHA256:WyQW2d7UuWz4BnTJdIIMIdg+Zj6JjkRKYakyCV3Rzug agent
debug1: Will attempt key: /home/jwboyer/.ssh/id_rsa 
debug1: Will attempt key: /home/jwboyer/.ssh/id_dsa 
debug1: Will attempt key: /home/jwboyer/.ssh/id_ecdsa 
debug1: Will attempt key: /home/jwboyer/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /home/jwboyer/.ssh/id_ed25519 
debug1: Will attempt key: /home/jwboyer/.ssh/id_ed25519_sk 
debug1: Will attempt key: /home/jwboyer/.ssh/id_xmss 
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/jwboyer/.ssh/id_rsa-redhat RSA SHA256:WyQW2d7UuWz4BnTJdIIMIdg+Zj6JjkRKYakyCV3Rzug agent
debug1: Server accepts key: /home/jwboyer/.ssh/id_rsa-redhat RSA SHA256:WyQW2d7UuWz4BnTJdIIMIdg+Zj6JjkRKYakyCV3Rzug agent
Authenticated to code.engineering.redhat.com ([10.19.208.64]:22) using "publickey".
debug1: pkcs11_del_provider: called, provider_id = (null)
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: pledge: filesystem full
debug1: Requesting authentication agent forwarding.
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0

  ****    Welcome to Gerrit Code Review    ****


Under a DEFAULT crypto-policy, I see:

[jwboyer@zod penumbra-baseline-data]$ ssh -A -v jboyer.redhat.com
OpenSSH_8.7p1, OpenSSL 3.0.1 14 Dec 2021
debug1: Reading configuration data /home/jwboyer/.ssh/config
debug1: /home/jwboyer/.ssh/config line 10: Applying options for code.engineering.redhat.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /home/jwboyer/.ssh/config
debug1: /home/jwboyer/.ssh/config line 10: Applying options for code.engineering.redhat.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Connecting to code.engineering.redhat.com [10.19.208.64] port 22.
debug1: Connection established.
debug1: identity file /home/jwboyer/.ssh/id_rsa type -1
debug1: identity file /home/jwboyer/.ssh/id_rsa-cert type -1
debug1: identity file /home/jwboyer/.ssh/id_dsa type -1
debug1: identity file /home/jwboyer/.ssh/id_dsa-cert type -1
debug1: identity file /home/jwboyer/.ssh/id_ecdsa type -1
debug1: identity file /home/jwboyer/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/jwboyer/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/jwboyer/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/jwboyer/.ssh/id_ed25519 type -1
debug1: identity file /home/jwboyer/.ssh/id_ed25519-cert type -1
debug1: identity file /home/jwboyer/.ssh/id_ed25519_sk type -1
debug1: identity file /home/jwboyer/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/jwboyer/.ssh/id_xmss type -1
debug1: identity file /home/jwboyer/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.7
debug1: Remote protocol version 2.0, remote software version GerritCodeReview_3.4.4 (APACHE-SSHD-2.6.0)
debug1: compat_banner: no match: GerritCodeReview_3.4.4 (APACHE-SSHD-2.6.0)
debug1: Authenticating to code.engineering.redhat.com:22 as 'jboyer'
debug1: load_hostkeys: fopen /home/jwboyer/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: ecdh-sha2-nistp256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: aes256-gcm MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm MAC: <implicit> compression: none
debug1: kex: ecdh-sha2-nistp256 need=32 dh_need=32
debug1: kex: ecdh-sha2-nistp256 need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:MO5xUAge3Tgbv9hnahccKjARZROZzcpiT/i67/TJ9Ro
debug1: load_hostkeys: fopen /home/jwboyer/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'code.engineering.redhat.com' is known and matches the ED25519 host key.
debug1: Found key in /home/jwboyer/.ssh/known_hosts:154
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: /home/jwboyer/.ssh/id_rsa-redhat RSA SHA256:WyQW2d7UuWz4BnTJdIIMIdg+Zj6JjkRKYakyCV3Rzug agent
debug1: Will attempt key: /home/jwboyer/.ssh/id_rsa 
debug1: Will attempt key: /home/jwboyer/.ssh/id_dsa 
debug1: Will attempt key: /home/jwboyer/.ssh/id_ecdsa 
debug1: Will attempt key: /home/jwboyer/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /home/jwboyer/.ssh/id_ed25519 
debug1: Will attempt key: /home/jwboyer/.ssh/id_ed25519_sk 
debug1: Will attempt key: /home/jwboyer/.ssh/id_xmss 
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/jwboyer/.ssh/id_rsa-redhat RSA SHA256:WyQW2d7UuWz4BnTJdIIMIdg+Zj6JjkRKYakyCV3Rzug agent
debug1: Server accepts key: /home/jwboyer/.ssh/id_rsa-redhat RSA SHA256:WyQW2d7UuWz4BnTJdIIMIdg+Zj6JjkRKYakyCV3Rzug agent
sign_and_send_pubkey: signing failed for RSA "/home/jwboyer/.ssh/id_rsa-redhat" from agent: agent refused operation
debug1: Trying private key: /home/jwboyer/.ssh/id_rsa
Enter passphrase for key '/home/jwboyer/.ssh/id_rsa': 


It's unclear why this is happening.  From what I can tell, the server key is compliant with the DEFAULT policy, the key exchange algorithms are compliant, and the local key is compliant

[jwboyer@zod penumbra-baseline-data]$ ssh-keygen -lf ~/.ssh/id_rsa-redhat
2048 SHA256:WyQW2d7UuWz4BnTJdIIMIdg+Zj6JjkRKYakyCV3Rzug jwboyer.redhat.com (RSA)

Help?
Comment 1 Josh Boyer 2022-04-07 13:25:11 UTC 
Another data point.  With a different 2048 bit RSA key to a gitlab server, either policy works fine.

[jwboyer@zod pungi-centos]$ ssh -vA git
OpenSSH_8.7p1, OpenSSL 3.0.1 14 Dec 2021
debug1: Reading configuration data /home/jwboyer/.ssh/config
debug1: /home/jwboyer/.ssh/config line 26: Applying options for gitlab.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /home/jwboyer/.ssh/config
debug1: /home/jwboyer/.ssh/config line 26: Applying options for gitlab.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Connecting to gitlab.com [172.65.251.78] port 22.
debug1: Connection established.
debug1: identity file /home/jwboyer/.ssh/id_rsa-github type 0
debug1: identity file /home/jwboyer/.ssh/id_rsa-github-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.4p1 Debian-5
debug1: compat_banner: match: OpenSSH_8.4p1 Debian-5 pat OpenSSH* compat 0x04000000
debug1: Authenticating to gitlab.com:22 as 'git'
debug1: load_hostkeys: fopen /home/jwboyer/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: aes256-gcm MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:eUXGGm1YGsMAS7vkcx6JOJdOGHPem5gQp4taiCfCLB8
debug1: load_hostkeys: fopen /home/jwboyer/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'gitlab.com' is known and matches the ED25519 host key.
debug1: Found key in /home/jwboyer/.ssh/known_hosts:153
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: /home/jwboyer/.ssh/id_rsa-github RSA SHA256:PT3j1yvgbJ9r4waM5g0NgptXD13tHw98Tl0Zf89y7+s explicit agent
debug1: Will attempt key: /home/jwboyer/.ssh/id_rsa-redhat RSA SHA256:WyQW2d7UuWz4BnTJdIIMIdg+Zj6JjkRKYakyCV3Rzug agent
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256,webauthn-sk-ecdsa-sha2-nistp256>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering public key: /home/jwboyer/.ssh/id_rsa-github RSA SHA256:PT3j1yvgbJ9r4waM5g0NgptXD13tHw98Tl0Zf89y7+s explicit agent
debug1: Server accepts key: /home/jwboyer/.ssh/id_rsa-github RSA SHA256:PT3j1yvgbJ9r4waM5g0NgptXD13tHw98Tl0Zf89y7+s explicit agent
Authenticated to gitlab.com ([172.65.251.78]:22) using "publickey".
debug1: pkcs11_del_provider: called, provider_id = (null)
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions
debug1: Entering interactive session.
debug1: pledge: filesystem full
debug1: client_input_global_request: rtype hostkeys-00 want_reply 0
debug1: client_input_hostkeys: searching /home/jwboyer/.ssh/known_hosts for gitlab.com / (none)
debug1: client_input_hostkeys: searching /home/jwboyer/.ssh/known_hosts2 for gitlab.com / (none)
debug1: client_input_hostkeys: hostkeys file /home/jwboyer/.ssh/known_hosts2 does not exist
debug1: client_input_hostkeys: no new or deprecated keys from server
debug1: Remote: /authorized_keys %u %k:1: key options: command user-rc
debug1: Remote: /authorized_keys %u %k:1: key options: command user-rc
debug1: Requesting authentication agent forwarding.
PTY allocation request failed on channel 0
Welcome to GitLab, @jwboyer!
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow reply 0
debug1: channel 0: free: client-session, nchannels 1
Connection to gitlab.com closed.
Transferred: sent 3124, received 2868 bytes, in 0.1 seconds
Bytes per second: sent 24486.3, received 22479.8
debug1: Exit status 0


[jwboyer@zod pungi-centos]$ ssh-keygen -lf ~/.ssh/id_rsa-github
2048 SHA256:PT3j1yvgbJ9r4waM5g0NgptXD13tHw98Tl0Zf89y7+s jwboyer@obiwan (RSA)
[jwboyer@zod pungi-centos]$ sudo update-crypto-policies --show
DEFAULT
[jwboyer@zod pungi-centos]$
Comment 2 Dmitry Belyavskiy 2022-04-07 13:26:29 UTC 

*** This bug has been marked as a duplicate of bug 2060232 ***
Comment 3 Dmitry Belyavskiy 2022-04-07 13:29:22 UTC 
Sorry, I'm afraid you are misleaded. SHA256 is just a hash used to output the fingerprint of the key.

ssh-keyscan chould show you the peer keys. I expect it is rsa-sha.

ssh-rsa is a separate (and insecure nowadays) signature type, the modern ones are rsa-sha2-*.
Comment 4 Josh Boyer 2022-04-07 13:34:42 UTC 
[jwboyer@zod pungi-centos]$ ssh-keyscan code.engineering.redhat.com
# code.engineering.redhat.com:22 SSH-2.0-GerritCodeReview_3.4.4 (APACHE-SSHD-2.6.0)
code.engineering.redhat.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA1famCORFkEX0L0uZsgfORdwJEP0y3JppUFvgOdLGnjll3vX+Dy8KPTknZGL/Q+hcaaFaIMIiSMHNa+z/285J3KpHc4AKlWtsTEpS7qJXem9Ddmuk0tml+Z2qYgbxJOpkaKeDexOxGLq0qp/s3bH63vu+NPXOZImN0pRrQXT/C3s+J0U9kwRjnQEyaV3rPx8d6OiMp8SnYjkqF4R1X1v40gHM44OFIgW/D0+hmxUuhS3qgn8TVecP6YdxxOYnoJm8994BpPNGh1cYg3cjvqtLKs1I9j5fEgUbunMdDhJA3OpqFpEeEVj9TfVu9Ur3zS1ytBjxhmKYew+tBxkxm5p8+w==
# code.engineering.redhat.com:22 SSH-2.0-GerritCodeReview_3.4.4 (APACHE-SSHD-2.6.0)
code.engineering.redhat.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA9Iah4n4JSZFaXv9Nj7/JO42cOI4lvkgdycEaEJ3PW96HPhWcXyAFboUA1doMvorOOl4Yd/EV+bBwn4lR9fcI8=
# code.engineering.redhat.com:22 SSH-2.0-GerritCodeReview_3.4.4 (APACHE-SSHD-2.6.0)
code.engineering.redhat.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINYZZXmzm14TUL02Qe5SCMw48OfrphoIzi4qXSEK9Hiq
# code.engineering.redhat.com:22 SSH-2.0-GerritCodeReview_3.4.4 (APACHE-SSHD-2.6.0)
# code.engineering.redhat.com:22 SSH-2.0-GerritCodeReview_3.4.4 (APACHE-SSHD-2.6.0)
[jwboyer@zod pungi-centos]$ 


Looks like it has multiple keys, but the key used is ssh-ed25519 ?  I saw bug 2060232 but this looks different.
Comment 7 Dmitry Belyavskiy 2022-04-07 15:50:42 UTC 
The SHA1 signature is forbidden by default in RHEL9/CentOS stream 9. When you switch to LEGACY CP, it starts working. 

Local id_rsa key is not compliant with the default policy.

Regarding gitlab, could you please provide a debug log for that?
Comment 8 Josh Boyer 2022-04-07 16:02:16 UTC 
(In reply to Dmitry Belyavskiy from comment #7)
> The SHA1 signature is forbidden by default in RHEL9/CentOS stream 9. When
> you switch to LEGACY CP, it starts working. 

I understand that, but I cannot determine where in this interaction SHA1 is even being used.  Key exchange, and the keys themselves, do not seem to be SHA1 in any of the logs that I can see.
 
> Local id_rsa key is not compliant with the default policy.

How can you tell?  What commands can I run to determine this is the case on the local client key?

> Regarding gitlab, could you please provide a debug log for that?

See comment #1


Also, if we're going to start failing client keys we absolutely need a kbase linked to the crypto-policy section describing that, and describing how customers can determine if their local keys are non-compliant.  The list in the 9 Beta notes just says RSA keys < 2048 bits and every bit of output I've been able to produce on this system shows the local id_rsa key to be 2048 bits with a SHA256 fingerprint.
Comment 10 Jakub Jelen 2022-04-08 13:26:51 UTC 
(In reply to Josh Boyer from comment #8)
> (In reply to Dmitry Belyavskiy from comment #7)
> > The SHA1 signature is forbidden by default in RHEL9/CentOS stream 9. When
> > you switch to LEGACY CP, it starts working. 
> 
> I understand that, but I cannot determine where in this interaction SHA1 is
> even being used.  Key exchange, and the keys themselves, do not seem to be
> SHA1 in any of the logs that I can see.

Authentication.

> > Local id_rsa key is not compliant with the default policy.
> 
> How can you tell?  What commands can I run to determine this is the case on
> the local client key?

The key is ok. RSA is ok. But the OpenSSH determines based on what is the server if
it supports the SHA2 extension and if it does not, it falls back to SHA-1 signatures,
but they do not work.

The remote server is Apache SSHD, which to mu understanding implemented this extension
last year, so updating the server to something more recent might solve the issue.

https://issues.apache.org/jira/browse/SSHD-1141
https://issues.apache.org/jira/browse/SSHD-1105

> > Regarding gitlab, could you please provide a debug log for that?
> 
> See comment #1
> 
> 
> Also, if we're going to start failing client keys we absolutely need a kbase
> linked to the crypto-policy section describing that, and describing how
> customers can determine if their local keys are non-compliant.  The list in
> the 9 Beta notes just says RSA keys < 2048 bits and every bit of output I've
> been able to produce on this system shows the local id_rsa key to be 2048
> bits with a SHA256 fingerprint.

This is unrelated. The 2k keys are ok, but in reality openssh is happy to use
even smaller keys (#2066882). The problem is really the SHA-1 signature for
public key authetnication that does not work and is forced to openssh by server
not supporting better algorithm.
Comment 11 Jakub Jelen 2022-04-08 13:28:44 UTC 
The gitlab runs standard "OpenSSH_8.4p1 Debian-5" which supports SHA2 extensions without any issues.
Comment 12 Josh Boyer 2022-04-08 14:08:16 UTC 
ACK, thank you.  Simo also pointed me to https://issues.apache.org/jira/browse/SSHD-1141.  I can see the added output connecting to gitlab but missing from code.eng.  With the default being SHA1, that makes sense.  I'm glad client keys are less impacted than I feared.
Comment 13 Simo Sorce 2022-04-11 18:31:59 UTC 
Josh,
is it ok to close this bug?
Seem we determined that the problem is on the server side and we have the crypto policy change as the official way to address this kind of issue.
Comment 14 Josh Boyer 2022-04-11 18:49:57 UTC 
Yes.  If we can somehow get this included in the kbase/release notes so that people can determine it's the server side based on debug output, that would be good.
Note You need to log in before you can comment on or make changes to this bug.