One needs to explicitly do ssh -oPubkeyAcceptedKeyTypes=+rsa-sha2-512 to ssh to host with that kind of host key. This is needed despite the fact we have: PubkeyAcceptedKeyTypes ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01,sk-ecdsa-sha2-nistp256,sk-ecdsa-sha2-nistp256-cert-v01,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01,ssh-ed25519,ssh-ed25519-cert-v01,sk-ssh-ed25519,sk-ssh-ed25519-cert-v01,rsa-sha2-256,rsa-sha2-256-cert-v01,rsa-sha2-512,rsa-sha2-512-cert-v01 ... in /etc/crypto-policies/back-ends/openssh.config. Seems like that configuration is misparsed?
Per Fedora Devel discussion: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/QCZHTG2BHF4LJSSK2OQUCWEXJV4BVODX/
It would be nice to summarize the issue in the bug so I do not have to go through the whole thread on the mailing list figuring out what part leads you to the conclusion that this is a bug in openssh. Posting the whole debug log would also help in debugging the issue. From just the error message, I can only guess that the server did not accept any RSA signature type our client offered. debug1: send_pubkey_test: no mutual signature algorithm Debian 9 has very outdated OpenSSH 7.4p1 without any reasonable bugfixes towards interoperability as far as I know. So this error happens because we removed ssh-rsa (SHA1) signature algorithm from Fedora 33, but Debian supports only that one. So some myth-busting: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/MHM22772KDZGEKDNGHTUN46AR4TLSSZY/ > I am guessing this key was generated with the older ssh-rsa and so the new > boxes won't work unless you force it. I would regenerate the key with a > newer sig :). No, the keys do not contain anything related to digest algorithms. The key is just a key and signature algorithms is negotiated during connect. Regenerating the key with new signature algorithm will not do anything useful. > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/DBL7NG4AESSVORIDGSM3PKBF3CXURBR7/ > My workaround solution was to add to ~/.ssh/config > > Host * > PubkeyAcceptedKeyTypes +rsa-sha2-256,rsa-sha2-512 This does not do anything useful as both of these are already in defaults. Can you clarify what is the problem you are trying to solve in this bug?
The workaround probably worked because it covered up the crypto-policies based configuration. The true workaround is to switch the policy to LEGACY if you need to communicate with system that supports only ssh-rsa.
(In reply to Jakub Jelen from comment #2) > > My workaround solution was to add to ~/.ssh/config > > > > Host * > > PubkeyAcceptedKeyTypes +rsa-sha2-256,rsa-sha2-512 > > This does not do anything useful as both of these are already in defaults. Yes, they are configured, but it has no effect. Users have to re-add on top of the default configuration once more. That's the bug.
> It would be nice to summarize the issue in the bug so I do not have to go > through the whole thread I did not want you to necessarily read that, that was just FTR. I believed the initial comment is self-explaining, sorry. I have a system with 'rsa-sha2-512' host key, and I can not ssh to that system using the default-configured F33 openssh client. I have to explicitly accept the 'rsa-sha2-512' keys (even though it is by default configured as accepted).
For completeness, this is what I'm seeing. I created a new key (which now I'm not so sure was necessary XD ): $ ssh-keygen -t rsa-sha2-512 -b 4096 -f id_git_rsa and updated ssh_config: Host bitbucket.org Hostname bitbucket.org IdentityFile ~/.ssh/id_git_rsa Then, after uploading it to bitbucket.org, when I try a pull, I get. $ git pull The authenticity of host 'bitbucket.org (2406:da00:ff00::22c3:9b0a)' can't be established. RSA key fingerprint is SHA256:zzXQOXSRBEiUtuE8AikJYKwbHaxvSc0ojez9YXaGp1A. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added 'bitbucket.org,2406:da00:ff00::22c3:9b0a' (RSA) to the list of known hosts. git: Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. $ ssh -Tv git OpenSSH_8.3p1, OpenSSL 1.1.1g FIPS 21 Apr 2020 debug1: Reading configuration data /home/asinha/.ssh/config debug1: /home/asinha/.ssh/config line 28: Applying options for bitbucket.org debug1: Reading configuration data /etc/ssh/ssh_config debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug1: configuration requests final Match pass debug1: re-parsing configuration debug1: Reading configuration data /home/asinha/.ssh/config debug1: /home/asinha/.ssh/config line 28: Applying options for bitbucket.org debug1: Reading configuration data /etc/ssh/ssh_config debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug1: Connecting to bitbucket.org [2406:da00:ff00::6b17:d1f5] port 22. debug1: Connection established. debug1: identity file /home/asinha/.ssh/id_git_rsa type 0 debug1: identity file /home/asinha/.ssh/id_git_rsa-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_8.3 debug1: Remote protocol version 2.0, remote software version conker_c123b90d72-dirty conker-3001 debug1: no match: conker_c123b90d72-dirty conker-3001 debug1: Authenticating to bitbucket.org:22 as 'git' debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: ssh-rsa debug1: kex: server->client cipher: chacha20-poly1305 MAC: <implicit> compression: none debug1: kex: client->server cipher: chacha20-poly1305 MAC: <implicit> compression: none debug1: kex: curve25519-sha256 need=64 dh_need=64 debug1: kex: curve25519-sha256 need=64 dh_need=64 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host key: ssh-rsa SHA256:zzXQOXSRBEiUtuE8AikJYKwbHaxvSc0ojez9YXaGp1A debug1: Host 'bitbucket.org' is known and matches the RSA host key. debug1: Found key in /home/asinha/.ssh/known_hosts:2 Warning: Permanently added the RSA host key for IP address '2406:da00:ff00::6b17:d1f5' to the list of known hosts. debug1: rekey out after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: rekey in after 134217728 blocks debug1: Will attempt key: /home/asinha/.ssh/id_git_rsa RSA SHA256:q7cw+NxYxLnmsAcyNXHbYz9JAW2Ript+QTdm4DG7qeA explicit agent debug1: Will attempt key: /home/asinha/.ssh/id_fedora_rsa RSA SHA256:gO+IlRkrD0I72I65sjjbfRDoSq303hK3zM2WmHOt6d0 agent debug1: Will attempt key: /home/asinha/.ssh/id_herts_rsa RSA SHA256:G1eCTz32ggxWwpEmG7x3OwKCka2hklT7+t7CLAWSBgk agent debug1: Will attempt key: neuro-sig.org RSA SHA256:riETtJv+xyezXMMRqDxTs4R0f/MEbJg3hhGN6IJ0eKQ agent debug1: Will attempt key: /home/asinha/.ssh/id_package_rsa RSA SHA256:JeWeDGvTTneHF3FW8LUFCMTD8rtcq81QaVVgS6lAIac agent debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Offering public key: /home/asinha/.ssh/id_git_rsa RSA SHA256:q7cw+NxYxLnmsAcyNXHbYz9JAW2Ript+QTdm4DG7qeA explicit agent debug1: send_pubkey_test: no mutual signature algorithm debug1: No more authentication methods to try. ... git: Permission denied (publickey). If I update ssh_config to say: Host bitbucket.org Hostname bitbucket.org IdentityFile ~/.ssh/id_git_rsa PubkeyAcceptedKeyTypes=+ssh-rsa it works fine: $ ssh -Tv git OpenSSH_8.3p1, OpenSSL 1.1.1g FIPS 21 Apr 2020 debug1: Reading configuration data /home/asinha/.ssh/config debug1: /home/asinha/.ssh/config line 28: Applying options for bitbucket.org debug1: Reading configuration data /etc/ssh/ssh_config debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug1: configuration requests final Match pass debug1: re-parsing configuration debug1: Reading configuration data /home/asinha/.ssh/config debug1: /home/asinha/.ssh/config line 28: Applying options for bitbucket.org debug1: Reading configuration data /etc/ssh/ssh_config debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug1: Connecting to bitbucket.org [2406:da00:ff00::22c0:3470] port 22. debug1: Connection established. debug1: identity file /home/asinha/.ssh/id_git_rsa type 0 debug1: identity file /home/asinha/.ssh/id_git_rsa-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_8.3 debug1: Remote protocol version 2.0, remote software version conker_b9a79bcd5e-dirty conker-3000 debug1: no match: conker_b9a79bcd5e-dirty conker-3000 debug1: Authenticating to bitbucket.org:22 as 'git' debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: ssh-rsa debug1: kex: server->client cipher: chacha20-poly1305 MAC: <implicit> compression: none debug1: kex: client->server cipher: chacha20-poly1305 MAC: <implicit> compression: none debug1: kex: curve25519-sha256 need=64 dh_need=64 debug1: kex: curve25519-sha256 need=64 dh_need=64 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host key: ssh-rsa SHA256:zzXQOXSRBEiUtuE8AikJYKwbHaxvSc0ojez9YXaGp1A debug1: Host 'bitbucket.org' is known and matches the RSA host key. debug1: Found key in /home/asinha/.ssh/known_hosts:2 Warning: Permanently added the RSA host key for IP address '2406:da00:ff00::22c0:3470' to the list of known hosts. debug1: rekey out after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: rekey in after 134217728 blocks debug1: Will attempt key: /home/asinha/.ssh/id_git_rsa RSA SHA256:q7cw+NxYxLnmsAcyNXHbYz9JAW2Ript+QTdm4DG7qeA explicit agent ... debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Offering public key: /home/asinha/.ssh/id_git_rsa RSA SHA256:q7cw+NxYxLnmsAcyNXHbYz9JAW2Ript+QTdm4DG7qeA explicit agent debug1: Server accepts key: /home/asinha/.ssh/id_git_rsa RSA SHA256:q7cw+NxYxLnmsAcyNXHbYz9JAW2Ript+QTdm4DG7qeA explicit agent debug1: Authentication succeeded (publickey). Authenticated to bitbucket.org ([2406:da00:ff00::22c0:3470]:22). debug1: channel 0: new [client-session] debug1: Entering interactive session. debug1: pledge: network debug1: Sending environment. debug1: Sending env XMODIFIERS = @im=ibus debug1: Sending env LANG = en_GB.UTF-8 debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 logged in as sanjay_ankur You can use git or hg to connect to Bitbucket. Shell access is disabled debug1: channel 0: free: client-session, nchannels 1 Transferred: sent 3504, received 2048 bytes, in 0.2 seconds Bytes per second: sent 19333.9, received 11300.2 debug1: Exit status 0
> I believed the initial comment is self-explaining, sorry. I can't resist, the tone in the response to my report was inappropriate.
(In reply to Ankur Sinha (FranciscoD) from comment #6) > For completeness, this is what I'm seeing. > > I created a new key (which now I'm not so sure was necessary XD ): > > $ ssh-keygen -t rsa-sha2-512 -b 4096 -f id_git_rsa No, it is not. It generates the "same" rsa key as before. The digest is not reflected in the key at all. Can you add the logs with more verbosity? "LogLevel DEBUG3" or just "ssh -vvv bitbucket.org". > debug1: kex: host key algorithm: ssh-rsa This is bad, but it is host key so not related to the above issue. It is still accepted by our openssh as it was not disabled by openssh upstream and it is not enforced by crypto policies, but it will stop work completely with some future openssh: https://www.openssh.com/txt/release-8.3 Manually trying to connect to bitbucket, it supports only ssh-rsa and ssh-dss, which is very bad and certainly worth reporting to bitbucket if you have something useful there. > debug1: send_pubkey_test: no mutual signature algorithm This is the same issue as in original mail in the mailng list. The server supports only ssh-rsa, fedora supports only the sha2 variants here. > Host bitbucket.org > Hostname bitbucket.org > IdentityFile ~/.ssh/id_git_rsa > PubkeyAcceptedKeyTypes=+ssh-rsa This is something else than in the original report. The default does not have ssh-rsa so adding it with this configuration option for legacy servers "fixes" the issue. The side note is that the following (what was in the original report) works too: > ssh -oPubkeyAcceptedKeyTypes=+rsa-sha2-512 and it is because the "+" modifier applies the addition to the openssh defaults, not the crypto-policy defaults (see manual page for ssh_config): > The default is handled system-wide by crypto-policies(7). [...] > If the specified list begins with a ‘+’ character, then the key types after it will be appended to the > built-in default instead of replacing it. This is side-effect of the limitation of the ssh configuration, but nothing we could simply fix.
(In reply to Pavel Raiskup from comment #5) > > It would be nice to summarize the issue in the bug so I do not have to go > > through the whole thread > > I did not want you to necessarily read that, that was just FTR. I > believed the initial comment is self-explaining, sorry. The template of the bug report with steps to reproduce, actual results, expected results, package version and so forth are there not just for fun. They really help you to sort the thoughts and any future readers to follow them and reproduce the issue you see.
(In reply to Jakub Jelen from comment #8) ... > The side note is that the following (what was in the original report) works > too: > > > ssh -oPubkeyAcceptedKeyTypes=+rsa-sha2-512 > > and it is because the "+" modifier applies the addition to the openssh > defaults, not the > crypto-policy defaults (see manual page for ssh_config): > > > The default is handled system-wide by crypto-policies(7). [...] > > If the specified list begins with a ‘+’ character, then the key types after it will be appended to the > > built-in default instead of replacing it. > > This is side-effect of the limitation of the ssh configuration, but nothing > we could simply fix. So, to emphasize this point: "+..." in config not only amends the built-in default but switches to that default instead of (or in addition to?) the crypto-policy settings. I do understand this is not a bug but the way upstream defaults and crypto policies interact. It's quite a source of confusion (as proven above). Is there good place to document it?
(In reply to Jakub Jelen from comment #8) > (In reply to Ankur Sinha (FranciscoD) from comment #6) > > For completeness, this is what I'm seeing. > > > > I created a new key (which now I'm not so sure was necessary XD ): > > > > $ ssh-keygen -t rsa-sha2-512 -b 4096 -f id_git_rsa > > No, it is not. It generates the "same" rsa key as before. The digest is not > reflected in the key at all. > > Can you add the logs with more verbosity? "LogLevel DEBUG3" or just "ssh > -vvv bitbucket.org". Here, with the tweaked ssh_config file (works): $ ssh -Tvvvvvv git OpenSSH_8.3p1, OpenSSL 1.1.1g FIPS 21 Apr 2020 debug1: Reading configuration data /home/asinha/.ssh/config debug1: /home/asinha/.ssh/config line 28: Applying options for bitbucket.org debug1: Reading configuration data /etc/ssh/ssh_config debug3: /etc/ssh/ssh_config line 54: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0 debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf debug2: checking match for 'final all' host bitbucket.org originally bitbucket.org debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched 'final' debug2: match not found debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only) debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-] debug3: kex names ok: [curve25519-sha256,curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512] debug1: configuration requests final Match pass debug1: re-parsing configuration debug1: Reading configuration data /home/asinha/.ssh/config debug1: /home/asinha/.ssh/config line 28: Applying options for bitbucket.org debug2: add_identity_file: ignoring duplicate key ~/.ssh/id_git_rsa debug1: Reading configuration data /etc/ssh/ssh_config debug3: /etc/ssh/ssh_config line 54: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0 debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf debug2: checking match for 'final all' host bitbucket.org originally bitbucket.org debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched 'final' debug2: match found debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-] debug3: kex names ok: [curve25519-sha256,curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512] debug2: resolving "bitbucket.org" port 22 debug2: ssh_connect_direct debug1: Connecting to bitbucket.org [2406:da00:ff00::22c2:513] port 22. debug1: Connection established. debug1: identity file /home/asinha/.ssh/id_git_rsa type 0 debug1: identity file /home/asinha/.ssh/id_git_rsa-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_8.3 debug1: Remote protocol version 2.0, remote software version conker_c123b90d72-dirty conker-3001 debug1: no match: conker_c123b90d72-dirty conker-3001 debug2: fd 5 setting O_NONBLOCK debug1: Authenticating to bitbucket.org:22 as 'git' debug3: hostkeys_foreach: reading file "/home/asinha/.ssh/known_hosts" debug3: record_hostkey: found key type RSA in file /home/asinha/.ssh/known_hosts:2 debug3: load_hostkeys: loaded 1 keys from bitbucket.org debug3: order_hostkeyalgs: prefer hostkeyalgs: rsa-sha2-512-cert-v01,rsa-sha2-256-cert-v01,ssh-rsa-cert-v01,rsa-sha2-512,rsa-sha2-256,ssh-rsa debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: curve25519-sha256,curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,ext-info-c debug2: host key algorithms: rsa-sha2-512-cert-v01,rsa-sha2-256-cert-v01,ssh-rsa-cert-v01,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01,ecdsa-sha2-nistp384-cert-v01,ecdsa-sha2-nistp521-cert-v01,sk-ecdsa-sha2-nistp256-cert-v01,ssh-ed25519-cert-v01,sk-ssh-ed25519-cert-v01,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256,ssh-ed25519,sk-ssh-ed25519 debug2: ciphers ctos: aes256-gcm,chacha20-poly1305,aes256-ctr,aes128-gcm,aes128-ctr debug2: ciphers stoc: aes256-gcm,chacha20-poly1305,aes256-ctr,aes128-gcm,aes128-ctr debug2: MACs ctos: hmac-sha2-256-etm,hmac-sha1-etm,umac-128-etm,hmac-sha2-512-etm,hmac-sha2-256,hmac-sha1,umac-128,hmac-sha2-512 debug2: MACs stoc: hmac-sha2-256-etm,hmac-sha1-etm,umac-128-etm,hmac-sha2-512-etm,hmac-sha2-256,hmac-sha1,umac-128,hmac-sha2-512 debug2: compression ctos: none,zlib,zlib debug2: compression stoc: none,zlib,zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: host key algorithms: ssh-dss,ssh-rsa debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm,chacha20-poly1305 debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm,chacha20-poly1305 debug2: MACs ctos: hmac-sha2-256-etm,hmac-sha2-256,hmac-sha1,hmac-sha1-96 debug2: MACs stoc: hmac-sha2-256-etm,hmac-sha2-256,hmac-sha1,hmac-sha1-96 debug2: compression ctos: none debug2: compression stoc: none debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: ssh-rsa debug1: kex: server->client cipher: chacha20-poly1305 MAC: <implicit> compression: none debug1: kex: client->server cipher: chacha20-poly1305 MAC: <implicit> compression: none debug1: kex: curve25519-sha256 need=64 dh_need=64 debug1: kex: curve25519-sha256 need=64 dh_need=64 debug3: send packet: type 30 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug3: receive packet: type 31 debug1: Server host key: ssh-rsa SHA256:zzXQOXSRBEiUtuE8AikJYKwbHaxvSc0ojez9YXaGp1A debug3: hostkeys_foreach: reading file "/home/asinha/.ssh/known_hosts" debug3: record_hostkey: found key type RSA in file /home/asinha/.ssh/known_hosts:2 debug3: load_hostkeys: loaded 1 keys from bitbucket.org debug3: hostkeys_foreach: reading file "/home/asinha/.ssh/known_hosts" debug1: Host 'bitbucket.org' is known and matches the RSA host key. debug1: Found key in /home/asinha/.ssh/known_hosts:2 Warning: Permanently added the RSA host key for IP address '2406:da00:ff00::22c2:513' to the list of known hosts. debug3: send packet: type 21 debug2: set_newkeys: mode 1 debug1: rekey out after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug3: receive packet: type 21 debug1: SSH2_MSG_NEWKEYS received debug2: set_newkeys: mode 0 debug1: rekey in after 134217728 blocks debug1: Will attempt key: /home/asinha/.ssh/id_git_rsa RSA SHA256:q7cw+NxYxLnmsAcyNXHbYz9JAW2Ript+QTdm4DG7qeA explicit agent debug1: Will attempt key: /home/asinha/.ssh/id_fedora_rsa RSA SHA256:gO+IlRkrD0I72I65sjjbfRDoSq303hK3zM2WmHOt6d0 agent debug1: Will attempt key: /home/asinha/.ssh/id_herts_rsa RSA SHA256:G1eCTz32ggxWwpEmG7x3OwKCka2hklT7+t7CLAWSBgk agent debug1: Will attempt key: neuro-sig.org RSA SHA256:riETtJv+xyezXMMRqDxTs4R0f/MEbJg3hhGN6IJ0eKQ agent debug1: Will attempt key: /home/asinha/.ssh/id_package_rsa RSA SHA256:JeWeDGvTTneHF3FW8LUFCMTD8rtcq81QaVVgS6lAIac agent debug2: pubkey_prepare: done debug3: send packet: type 5 debug3: receive packet: type 6 debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug3: send packet: type 50 debug3: receive packet: type 51 debug1: Authentications that can continue: publickey debug3: start over, passed a different list publickey debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Offering public key: /home/asinha/.ssh/id_git_rsa RSA SHA256:q7cw+NxYxLnmsAcyNXHbYz9JAW2Ript+QTdm4DG7qeA explicit agent debug3: send packet: type 50 debug2: we sent a publickey packet, wait for reply debug3: receive packet: type 60 debug1: Server accepts key: /home/asinha/.ssh/id_git_rsa RSA SHA256:q7cw+NxYxLnmsAcyNXHbYz9JAW2Ript+QTdm4DG7qeA explicit agent debug3: sign_and_send_pubkey: RSA SHA256:q7cw+NxYxLnmsAcyNXHbYz9JAW2Ript+QTdm4DG7qeA debug3: sign_and_send_pubkey: signing using ssh-rsa SHA256:q7cw+NxYxLnmsAcyNXHbYz9JAW2Ript+QTdm4DG7qeA debug3: send packet: type 50 debug3: receive packet: type 52 debug1: Authentication succeeded (publickey). Authenticated to bitbucket.org ([2406:da00:ff00::22c2:513]:22). debug1: channel 0: new [client-session] debug3: ssh_session2_open: channel_new: 0 debug2: channel 0: send open debug3: send packet: type 90 debug1: Entering interactive session. debug1: pledge: network debug3: receive packet: type 91 debug2: channel_input_open_confirmation: channel 0: callback start debug2: fd 5 setting TCP_NODELAY debug3: ssh_packet_set_tos: set IPV6_TCLASS 0x20 debug2: client_session2_setup: id 0 debug1: Sending environment. debug3: Ignored env SHELL debug3: Ignored env SESSION_MANAGER debug3: Ignored env COLORTERM debug3: Ignored env _fzf_orig_completion_tee debug3: Ignored env BYOBU_PYTHON debug3: Ignored env _fzf_orig_completion_rm debug3: Ignored env _fzf_orig_completion_rmdir debug3: Ignored env HISTCONTROL debug3: Ignored env XDG_MENU_PREFIX debug3: Ignored env BYOBU_HIGHLIGHT debug3: Ignored env TMUX debug3: Ignored env HISTSIZE debug3: Ignored env HOSTNAME debug3: Ignored env _fzf_orig_completion_uniq debug3: Ignored env GIT_PS1_SHOWUPSTREAM debug3: Ignored env _fzf_orig_completion_ftp debug3: Ignored env GUESTFISH_OUTPUT debug3: Ignored env SSH_AUTH_SOCK debug3: Ignored env BYOBU_DATE debug3: Ignored env GIT_PS1_SHOWDIRTYSTATE debug3: Ignored env _fzf_orig_completion_tail debug1: Sending env XMODIFIERS = @im=ibus debug2: channel 0: request env confirm 0 debug3: send packet: type 98 debug3: Ignored env DESKTOP_SESSION debug3: Ignored env SSH_AGENT_PID debug3: Ignored env BYOBU_CONFIG_DIR debug3: Ignored env GPG_TTY debug3: Ignored env EDITOR debug3: Ignored env _fzf_orig_completion_mv debug3: Ignored env PWD debug3: Ignored env LOGNAME debug3: Ignored env XDG_SESSION_DESKTOP debug3: Ignored env XDG_SESSION_TYPE debug3: Ignored env MODULESHOME debug3: Ignored env MANPATH debug3: Ignored env CXXFLAGS debug3: Ignored env XAUTHORITY debug3: Ignored env BYOBU_EDITOR debug3: Ignored env GUESTFISH_RESTORE debug3: Ignored env GJS_DEBUG_TOPICS debug3: Ignored env GDM_LANG debug3: Ignored env _fzf_orig_completion_diff debug3: Ignored env HOME debug3: Ignored env USERNAME debug1: Sending env LANG = en_GB.UTF-8 debug2: channel 0: request env confirm 0 debug3: send packet: type 98 debug3: Ignored env _fzf_orig_completion_awk debug3: Ignored env LS_COLORS debug3: Ignored env XDG_CURRENT_DESKTOP debug3: Ignored env _fzf_orig_completion_telnet debug3: Ignored env BYOBU_TERM debug3: Ignored env BYOBU_ULIMIT debug3: Ignored env VTE_VERSION debug3: Ignored env WAYLAND_DISPLAY debug3: Ignored env BYOBU_BACKEND debug3: Ignored env BYOBU_ACCENT debug3: Ignored env _fzf_orig_completion_grep debug3: Ignored env GUESTFISH_PS1 debug3: Ignored env PROMPT_COMMAND debug3: Ignored env GNOME_TERMINAL_SCREEN debug3: Ignored env INVOCATION_ID debug3: Ignored env MANAGERPID debug3: Ignored env BYOBU_DARK debug3: Ignored env BYOBU_PAGER debug3: Ignored env _fzf_orig_completion_du debug3: Ignored env _fzf_orig_completion_sort debug3: Ignored env GJS_DEBUG_OUTPUT debug3: Ignored env GIT_PS1_SHOWSTASHSTATE debug3: Ignored env GNOME_SETUP_DISPLAY debug3: Ignored env MODULEPATH_modshare debug3: Ignored env XDG_SESSION_CLASS debug3: Ignored env _fzf_orig_completion_cat debug3: Ignored env BYOBU_SED debug3: Ignored env TERM debug3: Ignored env LESSOPEN debug3: Ignored env _fzf_orig_completion_ln debug3: Ignored env USER debug3: Ignored env _fzf_orig_completion_ld debug3: Ignored env TMUX_PANE debug3: Ignored env _fzf_orig_completion_ls debug3: Ignored env GNOME_TERMINAL_SERVICE debug3: Ignored env MODULES_RUN_QUARANTINE debug3: Ignored env LOADEDMODULES debug3: Ignored env BYOBU_PREFIX debug3: Ignored env DISPLAY debug3: Ignored env SHLVL debug3: Ignored env BYOBU_LIGHT debug3: Ignored env GUESTFISH_INIT debug3: Ignored env BYOBU_TIME debug3: Ignored env QT_IM_MODULE debug3: Ignored env CVS_RSH debug3: Ignored env BYOBU_CHARMAP debug3: Ignored env BYOBU_WINDOW_NAME debug3: Ignored env _fzf_orig_completion_pushd debug3: Ignored env BYOBU_READLINK debug3: Ignored env BYOBU_DISTRO debug3: Ignored env XDG_RUNTIME_DIR debug3: Ignored env S_COLORS debug3: Ignored env PS1 debug3: Ignored env _fzf_orig_completion_cd debug3: Ignored env JOURNAL_STREAM debug3: Ignored env _fzf_orig_completion_less debug3: Ignored env XDG_DATA_DIRS debug3: Ignored env RDIMGDIR debug3: Ignored env _fzf_orig_completion_cp debug3: Ignored env PATH debug3: Ignored env MODULEPATH debug3: Ignored env GDMSESSION debug3: Ignored env CFLAGS debug3: Ignored env DBUS_SESSION_BUS_ADDRESS debug3: Ignored env _fzf_orig_completion_sed debug3: Ignored env MAIL debug3: Ignored env GIT_PS1_SHOWUNTRACKEDFILES debug3: Ignored env CUPS_USER debug3: Ignored env _fzf_orig_completion_head debug3: Ignored env _fzf_orig_completion_wc debug3: Ignored env BYOBU_TTY debug3: Ignored env BYOBU_RUN_DIR debug3: Ignored env MODULES_CMD debug3: Ignored env BASH_FUNC_ml%% debug3: Ignored env BASH_FUNC_switchml%% debug3: Ignored env BASH_FUNC_module%% debug3: Ignored env BASH_FUNC_scl%% debug3: Ignored env BASH_FUNC__module_raw%% debug3: Ignored env _ debug3: Ignored env OLDPWD debug2: channel 0: request shell confirm 1 debug3: send packet: type 98 debug2: channel_input_open_confirmation: channel 0: callback done debug2: channel 0: open confirm rwindow 2097152 rmax 32768 debug3: receive packet: type 99 debug2: channel_input_status_confirm: type 99 id 0 debug2: shell request accepted on channel 0 logged in as sanjay_ankur You can use git or hg to connect to Bitbucket. Shell access is disabled debug3: receive packet: type 98 debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 debug3: receive packet: type 96 debug2: channel 0: rcvd eof debug2: channel 0: output open -> drain debug2: channel 0: obuf empty debug2: channel 0: chan_shutdown_write (i0 o1 sock -1 wfd 7 efd 8 [write]) debug2: channel 0: output drain -> closed debug3: receive packet: type 97 debug2: channel 0: rcvd close debug2: channel 0: chan_shutdown_read (i0 o3 sock -1 wfd 6 efd 8 [write]) debug2: channel 0: input open -> closed debug3: channel 0: will not send data after close debug2: channel 0: almost dead debug2: channel 0: gc: notify user debug2: channel 0: gc: user detached debug2: channel 0: send close debug3: send packet: type 97 debug2: channel 0: is dead debug2: channel 0: garbage collecting debug1: channel 0: free: client-session, nchannels 1 debug3: channel 0: status: The following connections are open: #0 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/8 sock -1 cc -1) debug3: send packet: type 1 debug3: fd 1 is not O_NONBLOCK Transferred: sent 3504, received 2048 bytes, in 0.2 seconds Bytes per second: sent 19103.3, received 11165.4 debug1: Exit status 0 Here, without the tweak (does not work): $ ssh -Tvvvvvv git OpenSSH_8.3p1, OpenSSL 1.1.1g FIPS 21 Apr 2020 debug1: Reading configuration data /home/asinha/.ssh/config debug1: /home/asinha/.ssh/config line 28: Applying options for bitbucket.org debug1: Reading configuration data /etc/ssh/ssh_config debug3: /etc/ssh/ssh_config line 54: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0 debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf debug2: checking match for 'final all' host bitbucket.org originally bitbucket.org debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched 'final' debug2: match not found debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only) debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-] debug3: kex names ok: [curve25519-sha256,curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512] debug1: configuration requests final Match pass debug1: re-parsing configuration debug1: Reading configuration data /home/asinha/.ssh/config debug1: /home/asinha/.ssh/config line 28: Applying options for bitbucket.org debug2: add_identity_file: ignoring duplicate key ~/.ssh/id_git_rsa debug1: Reading configuration data /etc/ssh/ssh_config debug3: /etc/ssh/ssh_config line 54: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0 debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf debug2: checking match for 'final all' host bitbucket.org originally bitbucket.org debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched 'final' debug2: match found debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-] debug3: kex names ok: [curve25519-sha256,curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512] debug2: resolving "bitbucket.org" port 22 debug2: ssh_connect_direct debug1: Connecting to bitbucket.org [2406:da00:ff00::22cd:e0db] port 22. debug1: Connection established. debug1: identity file /home/asinha/.ssh/id_git_rsa type 0 debug1: identity file /home/asinha/.ssh/id_git_rsa-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_8.3 debug1: Remote protocol version 2.0, remote software version conker_c123b90d72-dirty conker-3005 debug1: no match: conker_c123b90d72-dirty conker-3005 debug2: fd 4 setting O_NONBLOCK debug1: Authenticating to bitbucket.org:22 as 'git' debug3: hostkeys_foreach: reading file "/home/asinha/.ssh/known_hosts" debug3: record_hostkey: found key type RSA in file /home/asinha/.ssh/known_hosts:2 debug3: load_hostkeys: loaded 1 keys from bitbucket.org debug3: order_hostkeyalgs: prefer hostkeyalgs: rsa-sha2-512-cert-v01,rsa-sha2-256-cert-v01,ssh-rsa-cert-v01,rsa-sha2-512,rsa-sha2-256,ssh-rsa debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: curve25519-sha256,curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,ext-info-c debug2: host key algorithms: rsa-sha2-512-cert-v01,rsa-sha2-256-cert-v01,ssh-rsa-cert-v01,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01,ecdsa-sha2-nistp384-cert-v01,ecdsa-sha2-nistp521-cert-v01,sk-ecdsa-sha2-nistp256-cert-v01,ssh-ed25519-cert-v01,sk-ssh-ed25519-cert-v01,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256,ssh-ed25519,sk-ssh-ed25519 debug2: ciphers ctos: aes256-gcm,chacha20-poly1305,aes256-ctr,aes128-gcm,aes128-ctr debug2: ciphers stoc: aes256-gcm,chacha20-poly1305,aes256-ctr,aes128-gcm,aes128-ctr debug2: MACs ctos: hmac-sha2-256-etm,hmac-sha1-etm,umac-128-etm,hmac-sha2-512-etm,hmac-sha2-256,hmac-sha1,umac-128,hmac-sha2-512 debug2: MACs stoc: hmac-sha2-256-etm,hmac-sha1-etm,umac-128-etm,hmac-sha2-512-etm,hmac-sha2-256,hmac-sha1,umac-128,hmac-sha2-512 debug2: compression ctos: none,zlib,zlib debug2: compression stoc: none,zlib,zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: host key algorithms: ssh-dss,ssh-rsa debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm,chacha20-poly1305 debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm,chacha20-poly1305 debug2: MACs ctos: hmac-sha2-256-etm,hmac-sha2-256,hmac-sha1,hmac-sha1-96 debug2: MACs stoc: hmac-sha2-256-etm,hmac-sha2-256,hmac-sha1,hmac-sha1-96 debug2: compression ctos: none debug2: compression stoc: none debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: ssh-rsa debug1: kex: server->client cipher: chacha20-poly1305 MAC: <implicit> compression: none debug1: kex: client->server cipher: chacha20-poly1305 MAC: <implicit> compression: none debug1: kex: curve25519-sha256 need=64 dh_need=64 debug1: kex: curve25519-sha256 need=64 dh_need=64 debug3: send packet: type 30 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug3: receive packet: type 31 debug1: Server host key: ssh-rsa SHA256:zzXQOXSRBEiUtuE8AikJYKwbHaxvSc0ojez9YXaGp1A debug3: hostkeys_foreach: reading file "/home/asinha/.ssh/known_hosts" debug3: record_hostkey: found key type RSA in file /home/asinha/.ssh/known_hosts:2 debug3: load_hostkeys: loaded 1 keys from bitbucket.org debug3: hostkeys_foreach: reading file "/home/asinha/.ssh/known_hosts" debug1: Host 'bitbucket.org' is known and matches the RSA host key. debug1: Found key in /home/asinha/.ssh/known_hosts:2 Warning: Permanently added the RSA host key for IP address '2406:da00:ff00::22cd:e0db' to the list of known hosts. debug3: send packet: type 21 debug2: set_newkeys: mode 1 debug1: rekey out after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug3: receive packet: type 21 debug1: SSH2_MSG_NEWKEYS received debug2: set_newkeys: mode 0 debug1: rekey in after 134217728 blocks debug1: Will attempt key: /home/asinha/.ssh/id_git_rsa RSA SHA256:q7cw+NxYxLnmsAcyNXHbYz9JAW2Ript+QTdm4DG7qeA explicit agent debug1: Will attempt key: /home/asinha/.ssh/id_fedora_rsa RSA SHA256:gO+IlRkrD0I72I65sjjbfRDoSq303hK3zM2WmHOt6d0 agent debug1: Will attempt key: /home/asinha/.ssh/id_herts_rsa RSA SHA256:G1eCTz32ggxWwpEmG7x3OwKCka2hklT7+t7CLAWSBgk agent debug1: Will attempt key: neuro-sig.org RSA SHA256:riETtJv+xyezXMMRqDxTs4R0f/MEbJg3hhGN6IJ0eKQ agent debug1: Will attempt key: /home/asinha/.ssh/id_package_rsa RSA SHA256:JeWeDGvTTneHF3FW8LUFCMTD8rtcq81QaVVgS6lAIac agent debug2: pubkey_prepare: done debug3: send packet: type 5 debug3: receive packet: type 6 debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug3: send packet: type 50 debug3: receive packet: type 51 debug1: Authentications that can continue: publickey debug3: start over, passed a different list publickey debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Offering public key: /home/asinha/.ssh/id_git_rsa RSA SHA256:q7cw+NxYxLnmsAcyNXHbYz9JAW2Ript+QTdm4DG7qeA explicit agent debug1: send_pubkey_test: no mutual signature algorithm debug1: Offering public key: /home/asinha/.ssh/id_fedora_rsa RSA SHA256:gO+IlRkrD0I72I65sjjbfRDoSq303hK3zM2WmHOt6d0 agent debug1: send_pubkey_test: no mutual signature algorithm debug1: Offering public key: /home/asinha/.ssh/id_herts_rsa RSA SHA256:G1eCTz32ggxWwpEmG7x3OwKCka2hklT7+t7CLAWSBgk agent debug1: send_pubkey_test: no mutual signature algorithm debug1: Offering public key: neuro-sig.org RSA SHA256:riETtJv+xyezXMMRqDxTs4R0f/MEbJg3hhGN6IJ0eKQ agent debug1: send_pubkey_test: no mutual signature algorithm debug1: Offering public key: /home/asinha/.ssh/id_package_rsa RSA SHA256:JeWeDGvTTneHF3FW8LUFCMTD8rtcq81QaVVgS6lAIac agent debug1: send_pubkey_test: no mutual signature algorithm debug2: we did not send a packet, disable method debug1: No more authentication methods to try. git: Permission denied (publickey). > > > debug1: kex: host key algorithm: ssh-rsa > > This is bad, but it is host key so not related to the above issue. It is > still accepted > by our openssh as it was not disabled by openssh upstream and it is not > enforced by > crypto policies, but it will stop work completely with some future openssh: > > https://www.openssh.com/txt/release-8.3 > > Manually trying to connect to bitbucket, it supports only ssh-rsa and > ssh-dss, > which is very bad and certainly worth reporting to bitbucket if you have > something > useful there. Great. We'll do that. We just weren't sure where the issue was. > > > debug1: send_pubkey_test: no mutual signature algorithm > > This is the same issue as in original mail in the mailng list. The server > supports only ssh-rsa, > fedora supports only the sha2 variants here. > > > Host bitbucket.org > > Hostname bitbucket.org > > IdentityFile ~/.ssh/id_git_rsa > > PubkeyAcceptedKeyTypes=+ssh-rsa > > This is something else than in the original report. The default does not > have ssh-rsa so > adding it with this configuration option for legacy servers "fixes" the > issue. I expect lots of users will be working with legacy servers (mostly unknowingly), so could this issue and workaround please be noted in the change proposal? I'll see if it can also be added to the "common bugs" page just so that it's easy for F33 users to find. (In reply to Jakub Jelen from comment #9) > (In reply to Pavel Raiskup from comment #5) > > > It would be nice to summarize the issue in the bug so I do not have to go > > > through the whole thread > > > > I did not want you to necessarily read that, that was just FTR. I > > believed the initial comment is self-explaining, sorry. > > The template of the bug report with steps to reproduce, actual results, > expected results, package version and so forth are there not just for fun. > They really help you to sort the thoughts and any future readers to follow > them and reproduce the issue you see. Let's walk away from our keyboards for a bit and reply to the bug later. Bug reports do sometimes require discussion to pin-point the cause of an issue. No reason why we cannot do that while being excellent to each other. :)
> "+" modifier applies the addition to the openssh defaults, not the crypto-policy defaults (see manual page for ssh_config): > This is side-effect of the limitation of the ssh configuration, but nothing we could simply fix. Meh, it looks like you are saying that doing just ssh -oPubkeyAcceptedKeyTypes=+rsa-sha2-512 entirely drops the crypto-policies defaults, and replaces it with the openssh defaults plus 'rsa-sha2-512'? Which actualy practially brings the less secure 'ssh-rsa', instead of 'rsa-sha2-512'? > This does not do anything useful as both of these are already in defaults. This is a bit misleading then.., it does _a very bad_ thing. I'm trying to re-read all the things that were said, because I'm still missing something.. and I'll try the -vvv ssh once more from a fresh F33.
What I think is missing from this BZ is a link to the reason this happens in Fedora33. The reason is because SHA-1 is being removed along with other legacy cryptography: https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2
(In reply to Michael J Gruber from comment #10) > (In reply to Jakub Jelen from comment #8) > ... > > The side note is that the following (what was in the original report) works > > too: > > > > > ssh -oPubkeyAcceptedKeyTypes=+rsa-sha2-512 > > > > and it is because the "+" modifier applies the addition to the openssh > > defaults, not the > > crypto-policy defaults (see manual page for ssh_config): > > > > > The default is handled system-wide by crypto-policies(7). [...] > > > If the specified list begins with a ‘+’ character, then the key types after it will be appended to the > > > built-in default instead of replacing it. > > > > This is side-effect of the limitation of the ssh configuration, but nothing > > we could simply fix. > > So, to emphasize this point: "+..." in config not only amends the built-in > default but switches to that default instead of (or in addition to?) the > crypto-policy settings. instead. > I do understand this is not a bug but the way upstream defaults and crypto > policies interact. It's quite a source of confusion (as proven above). Is > there good place to document it? We attempt to do that in the manual pages of openssh and crypto policies. If you have suggestion for improvement of this documentation, we would be happy to incorporate them. > I expect lots of users will be working with legacy servers (mostly unknowingly), > so could this issue and workaround please be noted in the change proposal? > I'll see if it can also be added to the "common bugs" page just so that it's easy for F33 users to find. This is generally a thing for LEGACY policy. > Meh, it looks like you are saying that doing just > ssh -oPubkeyAcceptedKeyTypes=+rsa-sha2-512 > entirely drops the crypto-policies defaults, and replaces it with the > openssh defaults plus 'rsa-sha2-512'? Which actualy practially brings > the less secure 'ssh-rsa', instead of 'rsa-sha2-512'? Yes. This overrides crypto policies.
> I'm still missing something. The server I spoke about is claiming: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521> So it indeed is missing the rsa-sha2-512, even though I re-generated its host key with: ssh-keygen -N "" -t rsa-sha2-512 -b 4096 -f /etc/ssh/ssh_host_rsa_key So the server probably doesn't generate the 'server-sig-algs' dynamically, and F33 client is confused and thinks that it can not talk to that server securely. Reading the bug 1623929 seems to confirm this suspicion. > Yes. This overrides crypto policies. Sounds dangerous. Could we make the openssh default strict subset of crypto-policies config? Or perhaps at least warn about the _commandline_ '=+' syntax? Or disallow '=+' on commandline? Or at least fix the man page: If the specified list begins with a ‘+’ character, then the key types after it will be appended to the built-in default instead of replacing it. This doesn't reference the crypto-policies defaults at all. I'd suggest wording like: If the specified list begins with a ‘+’ character, then the key types after it will be appended to the built-in openssh defaults and will be used. Any other configuration values (e.g. defaults specified in crypto-policies) are ignored.
(In reply to Pavel Raiskup from comment #15) > > I'm still missing something. > > The server I spoke about is claiming: > > > server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2- > nistp384,ecdsa-sha2-nistp521> Yes, this is the bug in OpenSSH 7.4p1 in Debian that I talked about in comment #2 -- it is missing the SHA2 signature algorithms, even though it supports them, which confuses some clients. > So it indeed is missing the rsa-sha2-512, even though I re-generated its > host key with: > > ssh-keygen -N "" -t rsa-sha2-512 -b 4096 -f /etc/ssh/ssh_host_rsa_key That does not do anything as I already said. And does not have any impact on the authentication either. > So the server probably doesn't generate the 'server-sig-algs' dynamically, No, this is hardcoded list of algorithms that openssh support, but it is wrong in OpenSSH 7.4p1. > and F33 client is confused and thinks that it can not talk to that server > securely. It talks to your server securely already. But there is no mutual signature algorithm that could be used for authentication. > Reading the bug 1623929 seems to confirm this suspicion. > > > Yes. This overrides crypto policies. > > Sounds dangerous. Could we make the openssh default strict subset of > crypto-policies config? No, unless we would introduce significant changes to the openssh code. Additionally, crypto policies are not meant to be non-overridable so if you modify configuration file it assumes you know what you are doing. > Or perhaps at least warn about the _commandline_ > '=+' syntax? Or disallow '=+' on commandline? It does not make any difference whether it is on commandline or in configuration file with higher priority (~/.ssh/config). If it has higher priority (see man ssh_config), it will get used, if not it will not get used as any other configuration option. > Or at least fix the man page: > > If the specified list begins with a ‘+’ character, then the key types > after it will be appended to the built-in default instead of replacing > it. > > This doesn't reference the crypto-policies defaults at all. I'd suggest > wording like: Crypto policies are mentioned in previous paragraph: > The default is handled system-wide by crypto-policies(7). To see the defaults > and how to modify this default, see manual page update-crypto-policies(8). > If the specified list begins with a ‘+’ character, then the key types > after it will be appended to the built-in openssh defaults and will > be used. This paragraph starts with the following sentence: > Specifies the key types that will be used for public key authentication as a comma-separated list of patterns. Not sure using "will be used" again makes it more clear than the original wording. I can add the "openssh defaults" though, if you believe it would help. > Any other configuration values (e.g. defaults specified in crypto-policies) are ignored. This is already implied by the way how openssh configuration works as described just on the top of the manual page: > For each parameter, the first obtained value will be used. I am not sure if it needs to be repeated for every configuration option in the manual page.
Created attachment 1716835 [details] return correct key type name when connected to openssh server < 7.8 I could be wrong about all of the following, so let me just prefix everything with, "To the best of my understanding:" In OpenSSH 7.8, the semantics of PubkeyAcceptedKeyTypes was changed, and older servers were marked with SSH_BUG_SIGTYPE. For older servers, ssh->kex->server_sig_algs appears to store a list of key types the server supports, while on newer servers it stores a list of RSA signature types that are supported. sshconnect2.c:key_sig_algorithm will return a copy of the name of an SSH key's type. For everything except RSA keys, it will filter the key's type against options.pubkey_key_types, but that doesn't make much sense, because the key has already been verified to be a permitted type by sshconnect2.c:pubkey_prepare. (I haven't gone into the history, but I'm guessing that this code used to filter against ssh->kex->server_sig_algs instead.) In other words, the private key type has already been checked against the local security policy before key_sig_algorithm is called, so key_sig_algorithm shouldn't be acting as a filter. It should be finding the correct name to refer to the client's key type. I'd suggest dropping the call to match_list, and instead simply return sshkey_ssh_name(key) for all non-RSA key types, and for RSA key types when connecting to a legacy server with SSH_BUG_SIGTYPE set.
I think bugzilla did not send an email for my previous comment because the bug was closed. I didn't realize that would happen. Jakub, would you consider the patch that I attached with my previous comment? I'm also going to send it to the upstream developers. This patch restores compatibility with OpenSSH servers < 7.8 by default, while conforming to the local security policy.
Note this bug also makes it impossible to do "git remote update" for git repos hosted on gitlab.freedesktop.org with the following key-type on my side: [hans@x1 linux]$ ssh-add -l 2048 SHA256:... /home/hans/.ssh/id_rsa (RSA) Let me know if you want ssh -vv output for a connection attempt.
(In reply to Hans de Goede from comment #19) > Note this bug also makes it impossible to do "git remote update" for git > repos hosted on gitlab.freedesktop.org with the following key-type on my > side: > > [hans@x1 linux]$ ssh-add -l > 2048 SHA256:... /home/hans/.ssh/id_rsa (RSA) > > Let me know if you want ssh -vv output for a connection attempt. I am working with gitlab.freedesktop.org frequently and without any issues. Do you have also some PubkeyAcceptedKeyTypes configuraiton option in your .ssh/config?
(In reply to Gordon Messmer from comment #17) > Created attachment 1716835 [details] > return correct key type name when connected to openssh server < 7.8 > > I could be wrong about all of the following, so let me just prefix > everything with, "To the best of my understanding:" > > In OpenSSH 7.8, the semantics of PubkeyAcceptedKeyTypes was changed, and > older servers were marked with SSH_BUG_SIGTYPE. For older servers, > ssh->kex->server_sig_algs appears to store a list of key types the server > supports, while on newer servers it stores a list of RSA signature types > that are supported. Right. This was the bug that SSH_BUG_SIGTYPE refers to. Initial implementation had it correct (just the rsa-sha2 mechanisms), then the rsa-sha2 were missing for some time and now there are only these (according to RFC). But the opensh-7.8 introduced too the new sha2-based certificate signature types. Before that, in 7.7 and earlier, there was only one ssh-rsa-cert-v01. This is the reason why the condition you modified is written the way it is. https://www.openssh.com/txt/release-7.8 https://github.com/openssh/openssh-portable/blob/V_7_7/sshkey.c#L112 > sshconnect2.c:key_sig_algorithm will return a copy of the name of an SSH > key's type. For everything except RSA keys, it will filter the key's type > against options.pubkey_key_types, but that doesn't make much sense, because > the key has already been verified to be a permitted type by > sshconnect2.c:pubkey_prepare. Can you clarify where the key was verified to be permitted? The function key_type_allowed_by_config() filters out only key types (if neither ssh-rsa, rsa-sha2-256 nor rsa-sha2-512 would be in the options.pubkey_key_types, it would remove the key). It can not make any decision if particular key will be used with sha1 or sha2 signature hash at that time. > (I haven't gone into the history, but I'm > guessing that this code used to filter against ssh->kex->server_sig_algs > instead.) Well .. in this case we need to filter by both of them ssh->kex->server_sig_algs and options.pubkey_key_types. The first one is considered the server accepted list and second is the client list. Filtering by the server avoids unneeded authentication failures. > In other words, the private key type has already been checked against the > local security policy before key_sig_algorithm is called, so > key_sig_algorithm shouldn't be acting as a filter. It should be finding the > correct name to refer to the client's key type. The first is filter of the keys, the second is filter of the signature algorithm, which is not the same for the RSA signatures. > I'd suggest dropping the call to match_list, and instead simply return > sshkey_ssh_name(key) for all non-RSA key types, and for RSA key types when > connecting to a legacy server with SSH_BUG_SIGTYPE set. I don't think the patch is correct. This might work with connecting to older OpenSSH <7.8, but it will not address the other issues with third-party ssh servers not supporting sha2 at all or being broken (for example bitbucket in comment #6). Sadly enough, I did not see a debug log from Fedora 33 to Debian 9, which was said the original issue to be able to see what is the actual issue. I think the solution here is really to use the configuration to connect to legacy servers: Host legacy PubkeyAcceptedKeyTypes=ssh-rsa or change the system-wide crypto policy to legacy: update-crypto-policy --set=LEGACY
(In reply to Jakub Jelen from comment #20) > I am working with gitlab.freedesktop.org frequently and without any issues. Have you used it the last couple of days? They recently did an upgrade and since then it is broken for me when using a Fedora 33 ssh client. > Do you have also some PubkeyAcceptedKeyTypes configuraiton option in your > .ssh/config? I have one there, but I tried commenting it out before adding a "me too" comment here, commenting it out does not help.
(In reply to Hans de Goede from comment #22) > (In reply to Jakub Jelen from comment #20) > > I am working with gitlab.freedesktop.org frequently and without any issues. > > Have you used it the last couple of days? They recently did an upgrade and > since then it is broken for me when using a Fedora 33 ssh client. > > > Do you have also some PubkeyAcceptedKeyTypes configuraiton option in your > > .ssh/config? > > I have one there, but I tried commenting it out before adding a "me too" > comment here, commenting it out does not help. OK, I am still on Fedora 32. The suggestions in previous comment work you?
(In reply to Jakub Jelen from comment #23) > OK, I am still on Fedora 32. The suggestions in previous comment work you? Setting PubkeyAcceptedKeyTypes=+ssh-rsa works as a workaround for me.
Created attachment 1717547 [details] ssh -v output for "git remote update" to gitlab.freedesktop.org with Fedora 33 default PubkeyAcceptedKeyTypes setting
Note the log I just attached is without any local modifications to PubkeyAcceptedKeyTypes. So this is a failed connection log, showing the problem. I believe that the specific error causing the connection issue is the "send_pubkey_test: no mutual signature algorithm" error.
Thank you for the logs, this one is Debian 10 -- I was not aware that particular broken version is that much widely adapted (and not fixed for years by Debian maintainers). We have the same version in RHEL 7, but with the patch for this issue backported: This will probably require a bit more love than one-line patch. Not all of the version up to 7.8 are affected -- before openssh-7.4, there was explicit "rsa-sha2-256,rsa-sha2-512", which should work fine: https://github.com/openssh/openssh-portable/commit/130f5d https://bugzilla.mindrot.org/show_bug.cgi?id=2547 https://bugzilla.mindrot.org/show_bug.cgi?id=2680 This issue was fixed in 7.5: https://github.com/openssh/openssh-portable/commit/183ba55 So we will most probably need a compat for debian running this particular version here.
(In reply to Jakub Jelen from comment #27) > Thank you for the logs, this one is Debian 10 -- I was not aware that > particular broken version is that much widely adapted (and not fixed for > years by Debian maintainers). You're welcome. Let me know if you want me to test a fix for this (preferably through a koji scratch-build of a fixed pkg).
Created attachment 1719130 [details] Use RSA keys for OpenSSH 7.4 servers, if local policy permits (In reply to Jakub Jelen from comment #21) > > Can you clarify where the key was verified to be permitted? The function > key_type_allowed_by_config() filters out only key types You're right, thank you. That does undermine the premise of the previous patch. I've written a new one that targets specifically the SHA2 types missing from 7.4p1. > I don't think the patch is correct. This might work with connecting to older > OpenSSH <7.8, but it will not address the other issues with third-party ssh > servers not supporting sha2 at all or being broken (for example bitbucket > in comment #6). I'd tested the previous patch against bitbucket.org, and found that it restored client function, but as you've clarified, that's undesirable because it wasn't using a SHA2 hash permitted by local policy. The new patch does not restore RSA keys with bitbucket.org, which seems to be the right thing to do. For that service, users should generate new non-RSA keys. > Sadly enough, I did not see a debug log from Fedora 33 to Debian 9, which > was said > the original issue to be able to see what is the actual issue. Do you still need that log?
(In reply to Gordon Messmer from comment #29) > Created attachment 1719130 [details] > Use RSA keys for OpenSSH 7.4 servers, if local policy permits > > (In reply to Jakub Jelen from comment #21) > > > > Can you clarify where the key was verified to be permitted? The function > > key_type_allowed_by_config() filters out only key types > > You're right, thank you. That does undermine the premise of the previous > patch. > > I've written a new one that targets specifically the SHA2 types missing from > 7.4p1. Thank you. Please, suggest this also in the upstream bug references above as this is something other distributions will have to handle too after they will disable SHA1. > > I don't think the patch is correct. This might work with connecting to older > > OpenSSH <7.8, but it will not address the other issues with third-party ssh > > servers not supporting sha2 at all or being broken (for example bitbucket > > in comment #6). > > > I'd tested the previous patch against bitbucket.org, and found that it > restored > client function, but as you've clarified, that's undesirable because it > wasn't > using a SHA2 hash permitted by local policy. The new patch does not restore > RSA keys with bitbucket.org, which seems to be the right thing to do. For > that > service, users should generate new non-RSA keys. Either non-RSA keys or use the above workaround for legacy hosts: Host legacy PubkeyAcceptedKeyTypes=ssh-rsa > > Sadly enough, I did not see a debug log from Fedora 33 to Debian 9, which > > was said > > the original issue to be able to see what is the actual issue. > > Do you still need that log? No, the log is not needed anymore as we already figured out what is the problem.
*** Bug 1897952 has been marked as a duplicate of this bug. ***
FEDORA-2020-413ab3bca3 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-413ab3bca3
Thanks, Jakub. There haven't been any comments on the openssh bugzilla report. Is there a better way to reach those developers?
Hi, unfortunately not. They have mostly different priorities than supporting outdated versions in old distributions. At this moment, it is issue only for Fedora, but they also plan to disable sha1 support one day so it would be good to have it fixed there too for good. I will try to ping them.
FEDORA-2020-413ab3bca3 has been pushed to the Fedora 33 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-413ab3bca3` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-413ab3bca3 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2020-413ab3bca3 has been pushed to the Fedora 33 stable repository. If problem still persists, please make note of it in this bug report.
suddenly had issues logging into my openwrt router and after some debugging came across this issue. had to add the following to my ~/.ssh/config file host gateway* 192.168.0.1 PubkeyAcceptedKeyTypes=+rsa-sha2-512 and now i can once again do passwordless logins. that was a pain!
(In reply to Mohammed Arafa from comment #37) > suddenly had issues logging into my openwrt router and after some debugging > came across this issue. > > had to add the following to my ~/.ssh/config file > > host gateway* 192.168.0.1 > PubkeyAcceptedKeyTypes=+rsa-sha2-512 This is the same as host gateway* 192.168.0.1 PubkeyAcceptedKeyTypes=ssh-rsa which I recommended in the comment #30, but just obfuscated with the "+" plus sign. Your OpenWRT probably runs old SSH server not supporting SHA2 so this is a valid workaround for your use case. > and now i can once again do passwordless logins. that was a pain! Some debug logs would be helpful.