Bug 1574230
| Summary: | Keys in ssh-agent are tried before keys from AWS (format difference? Both are RSA) | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Mark R. <rhbugzilla> |
| Component: | openssh | Assignee: | Jakub Jelen <jjelen> |
| Status: | CLOSED NOTABUG | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 28 | CC: | dwalsh, jfch, jjelen, lkundrak, mattias.ellert, plautrba, tmraz |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-05-03 16:51:05 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Mark R.
2018-05-02 20:41:51 UTC
To verify that it's something about the formatting of the AWS key, here's a connection attempt where I tell it to use one of the fake test keys (k6). That key isn't first in the agent or in sort order, but you can see the ssh client does correctly try that specific key before moving on to try keys from the agent. $ ssh -v -i ~/.ssh/k6 centos.bb.50 OpenSSH_7.7p1, OpenSSL 1.1.0h-fips 27 Mar 2018 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for * debug1: Connecting to 172.aa.bb.50 [172.aa.bb.50] port 22. debug1: Connection established. debug1: identity file /home/guest/.ssh/k6 type 3 debug1: key_load_public: No such file or directory debug1: identity file /home/guest/.ssh/k6-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_7.7 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4 debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000 debug1: Authenticating to 172.aa.bb.50:22 as 'centos' debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: ecdsa-sha2-nistp256 debug1: kex: server->client cipher: aes256-gcm MAC: <implicit> compression: none debug1: kex: client->server cipher: aes256-gcm MAC: <implicit> compression: none debug1: kex: curve25519-sha256 need=32 dh_need=32 debug1: kex: curve25519-sha256 need=32 dh_need=32 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host key: ecdsa-sha2-nistp256 SHA256:DcYxjaDCSYORUp585GsysyezKy2Gnf37IZBR1W+iGWw debug1: Host '172.aa.bb.50' is known and matches the ECDSA host key. debug1: Found key in /home/guest/.ssh/known_hosts:2 debug1: rekey after 4294967296 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: rekey after 4294967296 blocks debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512> debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug1: Next authentication method: gssapi-keyex debug1: No valid Key exchange context debug1: Next authentication method: gssapi-with-mic debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available (default cache: KEYRING:persistent:1000) debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available (default cache: KEYRING:persistent:1000) debug1: Next authentication method: publickey ## Here the client correctly sends key specified before trying any others ## ## Somehow with AWS keys this doesn't happen, but does on Fedora 27 ## debug1: Offering public key: ED25519 SHA256:wMLkYc1BbcBSLq2mcf3u0X3M9iaVgBveixVjQw4nGrk /home/guest/.ssh/k6 debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug1: Offering public key: RSA SHA256:g3BZ2UTCNR+zCqLt4JMAAYIa5KhMmVLZY/tORRWR1Wc k1 debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug1: Offering public key: RSA SHA256:TdlvzFJZeToIQ+8GHXMu4PCANn+hZ+EwyrTrXO0JF6Q k4 debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug1: Offering public key: ED25519 SHA256:IEgRLeZVURUJtQTNSze7CQ/X5ni205hXc1hyHOlwYG8 key five debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug1: Offering public key: RSA SHA256:8rWovkQVrbsuztU3Hf5R+JJyOuoHXt8zbfKXhPbpBiA temptest-aws.pem debug1: Server accepts key: pkalg rsa-sha2-512 blen 279 debug1: Authentication succeeded (publickey). Authenticated to 172.aa.bb.50 ([172.aa.bb.50]:22). debug1: channel 0: new [client-session] debug1: Requesting no-more-sessions debug1: Entering interactive session. debug1: pledge: network debug1: client_input_global_request: rtype hostkeys-00 want_reply 0 debug1: Sending environment. debug1: Sending env LANG = en_US.UTF-8 I assume you are using default GNOME and therefore you are running also gnome-keyring in your environment (which is wrapping ssh-agent from Fedora 28). You can verify that by writing the SSH_AUTH_SOCK environment variable. If it will be listed under /run/user/, it is a gnome-keyring: $ echo $SSH_AUTH_SOCK /run/user/1000/keyring/ssh Can you check if you can reproduce the same issue with ssh-agent only? It should be enough to start a new agent in a new terminal: eval `ssh-agent` after you are done with it, you can kill it with ssh-agent -k If you can reproduce the same with ssh-agent only, I will try to have a look deeper in it. I did not see any change in how the logging order is handled on the first sight. Also if I create normal rsa keys it seems to work fine. This somehow seems like the agent is not providing the matching public key or filename to match the keys against each other? The tests above happened with normal ssh-agent, also happens with the gnome-keyring. I initially hit this with my own account, which would be using the gnome wrapped agent as you suspected. Then I tried by creating a test user and using ssh-agent only, which is the test cases above. $ echo $SSH_AUTH_SOCK $ eval $(ssh-agent) Agent pid 3763 $ ssh-add .ssh/k1 Identity added: .ssh/k1 (.ssh/k1) # I repeat this for the other keys $ ssh-add -l 2048 SHA256:g3BZ2UTCNR+zCqLt4JMAAYIa5KhMmVLZY/tORRWR1Wc .ssh/k1 (RSA) 2048 SHA256:0GGpBzlNgJ4w49d9LFoVqG1vkrcpkaRuiPn0oqSFsz4 .ssh/k2 (RSA) 2048 SHA256:MPC3YJYq54W6oKdCHXqqyb6rvTkOeBlNzMEiAyDGp/s .ssh/k3 (RSA) 2048 SHA256:TdlvzFJZeToIQ+8GHXMu4PCANn+hZ+EwyrTrXO0JF6Q .ssh/k4 (RSA) 256 SHA256:IEgRLeZVURUJtQTNSze7CQ/X5ni205hXc1hyHOlwYG8 key five (ED25519) 256 SHA256:wMLkYc1BbcBSLq2mcf3u0X3M9iaVgBveixVjQw4nGrk key six (ED25519) 2048 SHA256:8rWovkQVrbsuztU3Hf5R+JJyOuoHXt8zbfKXhPbpBiA .ssh/temptest-aws.pem (RSA) $ ssh -v -i ~/.ssh/temptest-aws.pem centos.bb.50 OpenSSH_7.7p1, OpenSSL 1.1.0h-fips 27 Mar 2018 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for * debug1: Connecting to 172.aa.bb.50 [172.aa.bb.50] port 22. debug1: Connection established. debug1: key_load_public: No such file or directory debug1: identity file /home/guest/.ssh/temptest-aws.pem type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/guest/.ssh/temptest-aws.pem-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_7.7 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4 debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000 debug1: Authenticating to 172.aa.bb.50:22 as 'centos' debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: ecdsa-sha2-nistp256 debug1: kex: server->client cipher: aes256-gcm MAC: <implicit> compression: none debug1: kex: client->server cipher: aes256-gcm MAC: <implicit> compression: none debug1: kex: curve25519-sha256 need=32 dh_need=32 debug1: kex: curve25519-sha256 need=32 dh_need=32 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host key: ecdsa-sha2-nistp256 SHA256:DcYxjaDCSYORUp585GsysyezKy2Gnf37IZBR1W+iGWw debug1: Host '172.aa.bb.50' is known and matches the ECDSA host key. debug1: Found key in /home/guest/.ssh/known_hosts:2 debug1: rekey after 4294967296 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: rekey after 4294967296 blocks debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512> debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug1: Next authentication method: gssapi-keyex debug1: No valid Key exchange context debug1: Next authentication method: gssapi-with-mic debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available (default cache: KEYRING:persistent:1000) debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available (default cache: KEYRING:persistent:1000) debug1: Next authentication method: publickey debug1: Offering public key: RSA SHA256:g3BZ2UTCNR+zCqLt4JMAAYIa5KhMmVLZY/tORRWR1Wc .ssh/k1 debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug1: Offering public key: RSA SHA256:0GGpBzlNgJ4w49d9LFoVqG1vkrcpkaRuiPn0oqSFsz4 .ssh/k2 debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug1: Offering public key: RSA SHA256:MPC3YJYq54W6oKdCHXqqyb6rvTkOeBlNzMEiAyDGp/s .ssh/k3 debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug1: Offering public key: RSA SHA256:TdlvzFJZeToIQ+8GHXMu4PCANn+hZ+EwyrTrXO0JF6Q .ssh/k4 debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug1: Offering public key: ED25519 SHA256:IEgRLeZVURUJtQTNSze7CQ/X5ni205hXc1hyHOlwYG8 key five debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug1: Offering public key: ED25519 SHA256:wMLkYc1BbcBSLq2mcf3u0X3M9iaVgBveixVjQw4nGrk key six Received disconnect from 172.aa.bb.50 port 22:2: Too many authentication failures Disconnected from 172.aa.bb.50 port 22 Would it help at all if I repeat these steps on a Fedora 27 host and send you a similar log where the connection succeeds? I can get that done easily enough. Well, I'm more confused now. Booted Fedora 27 from live media, pulled the .ssh directory from my above tests over to it, and I get the same issue where ssh tries all keys in the agent before trying the key specified via '-i' or as IdentityFile in .ssh/config. It does list the key as explicitly requested, but after all the agent keys:
debug2: key: .ssh/k1 (0x55d0742d0870), agent
debug2: key: .ssh/k2 (0x55d0742cfd20), agent
debug2: key: .ssh/k3 (0x55d0742cfe40), agent
debug2: key: .ssh/k4 (0x55d0742cff60), agent
debug2: key: key five (0x55d0742d00c0), agent
debug2: key: key six (0x55d0742cc5b0), agent
debug2: key: .ssh/temptest-aws.pem (0x55d0742ce4d0), agent
debug2: key: /home/guest/.ssh/temptest-aws.pem ((nil)), explicit
I'm unsure how I haven't hit this before, but I'm sshing into EC2 instances daily as part of work, and haven't hit this until I updated my laptop to 28. I haven't added any keys to my .ssh folder or agent lately... but my newer keys are ed25519 which I don't belive gnome-keyring was able to deal with, perhaps that has changed with F28 and now those are loaded, causing my agent to have more keys than it previously did?
In case it's lost in the shuffle, this does seem to only occur with a keypair as created/provided by EC2. If I have a key created via ssh-keygen loaded even further down the list in ssh-agent, I can still login with that key and what's more notable as that this behavior of walking through all the keys in the agent does _not_ happen. The "walk the agent's keys before trying explicit key" happens _only_ when trying to login with a key that was created via EC2.
So, with a "normal" rsa key listed 9th in the output of ssh-agent, and trying the connection with '-i .ssh/k9', the ssh client tries no other keys at all and just immediately authenticates with that one.
Thank you for the reproducing the issue with Fedora 27. Yes, in Fedora 27, gnome keyring was ignored these keys and therefore they were not offered and it worked fine. Your analysis is correct: If you have public-private key pair in your drive, the ssh is able to read the public part and match the private one from the agent. This is not possible if ssh it receives only encrypted private key -- it does not know which one is it from agent (or if it is there at all) and therefore it is trying the keys from agent sequentially. You should be able to workaround it by creating the public key from the private one and store it next to it: ssh-keygen -yf ~/.ssh/temptest-aws.pem > ~/.ssh/temptest-aws.pem.pub Hah, well that's embarassing, lack of a public key in the directory sent me down a rabbit hole of confusion. The combination of that plus some ed25519 keys now working as designed in F28 is what led to my failing connections. I apologize for the noise on the bug tracker, but maybe someone else will naively download their key from AWS and throw it in their .ssh directory without creating a matching .pub, and this will thread will show up in their search to save them some time. Thanks for your help Jakub. |