Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1574230 - Keys in ssh-agent are tried before keys from AWS (format difference? Both are RSA)
Summary: Keys in ssh-agent are tried before keys from AWS (format difference? Both are...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: openssh
Version: 28
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Jakub Jelen
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-05-02 20:41 UTC by Mark R.
Modified: 2018-05-03 16:51 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-05-03 16:51:05 UTC
Type: Bug


Attachments (Terms of Use)

Description Mark R. 2018-05-02 20:41:51 UTC
Description of problem:

Created a previous bug but hadn't correctly replicated the issue, now I can. I'm connecting to an EC2 instance using the key as created by AWS. I have six other test keys created to make sure I hit MaxAuthTries to trip this bug and get rejected from the server before ssh ever tries the key I have specified on the command-line via '-i' switch.

When I load the keys into ssh-agent, whether the test AWS key is included or not, I am unable to log into instances because SSH does not use the specified IdentityFile, instead always walking through keys in the agent first and therefore tripping MaxAuthTries after the sixth, killing the connection attempt. Steps are below.

Copying the same .ssh directory to a Fedora 27 machine works as expected.


Version-Release number of selected component (if applicable):

"OpenSSH_7.7p1, OpenSSL 1.1.0h-fips  27 Mar 2018" as shipped with Fedora 28

How reproducible:

I don't exactly understand yet why it's specifically AWS-created keys that are triggering this, but here are the steps that will result in you *not* being able to login via a Fedora 28 client, while a Fedora 27 client will work as expected.

# Create an AWS keypair (or try your existing). For this test
# I saved it as .ssh/temptest-aws.pem

# Create some other test keys so you'll have enough to hit MaxAuthTries before
# the client tries your AWS key, and load them into ssh-agent. You can load
# the AWS key into the agent or not, either way it is not tried before the
# others are and your connection will fail.

$ ssh-add -l
2048 SHA256:g3BZ2UTCNR+zCqLt4JMAAYIa5KhMmVLZY/tORRWR1Wc k1 (RSA)
2048 SHA256:0GGpBzlNgJ4w49d9LFoVqG1vkrcpkaRuiPn0oqSFsz4 k2 (RSA)
2048 SHA256:MPC3YJYq54W6oKdCHXqqyb6rvTkOeBlNzMEiAyDGp/s k3 (RSA)
2048 SHA256:TdlvzFJZeToIQ+8GHXMu4PCANn+hZ+EwyrTrXO0JF6Q k4 (RSA)
256 SHA256:IEgRLeZVURUJtQTNSze7CQ/X5ni205hXc1hyHOlwYG8 key five (ED25519)
256 SHA256:wMLkYc1BbcBSLq2mcf3u0X3M9iaVgBveixVjQw4nGrk key six (ED25519)
2048 SHA256:8rWovkQVrbsuztU3Hf5R+JJyOuoHXt8zbfKXhPbpBiA temptest-aws.pem (RSA)


# Try to sign in, specifying that the connection should use only your AWS key

$ ssh -v -i ~/.ssh/temptest-aws.pem centos@172.aa.bb.50
OpenSSH_7.7p1, OpenSSL 1.1.0h-fips  27 Mar 2018
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for *
debug1: Connecting to 172.aa.bb.50 [172.aa.bb.50] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/guest/.ssh/temptest-aws.pem type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/guest/.ssh/temptest-aws.pem-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 172.aa.bb.50:22 as 'centos'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:DcYxjaDCSYORUp585GsysyezKy2Gnf37IZBR1W+iGWw
debug1: Host '172.aa.bb.50' is known and matches the ECDSA host key.
debug1: Found key in /home/guest/.ssh/known_hosts:2
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:1000)

debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:1000)

debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:g3BZ2UTCNR+zCqLt4JMAAYIa5KhMmVLZY/tORRWR1Wc k1
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: RSA SHA256:0GGpBzlNgJ4w49d9LFoVqG1vkrcpkaRuiPn0oqSFsz4 k2
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: RSA SHA256:MPC3YJYq54W6oKdCHXqqyb6rvTkOeBlNzMEiAyDGp/s k3
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: RSA SHA256:TdlvzFJZeToIQ+8GHXMu4PCANn+hZ+EwyrTrXO0JF6Q k4
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: ED25519 SHA256:IEgRLeZVURUJtQTNSze7CQ/X5ni205hXc1hyHOlwYG8 key five
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: ED25519 SHA256:wMLkYc1BbcBSLq2mcf3u0X3M9iaVgBveixVjQw4nGrk key six
Received disconnect from 172.aa.bb.50 port 22:2: Too many authentication failures
Disconnected from 172.aa.bb.50 port 22


# In the above output, ssh does know that the temptest-aws.pem key was specified as the identity file, but once it comes time to actually auth it
starts walking through all keys in the agent instead of just using temptest-aws.pem

# If I remove one or two of the other keys from the agent, or indeed just kill the agent, the connection works as expected (though it still always tries the other agent keys first even though '-i' was used).

$ ssh-add -l
2048 SHA256:g3BZ2UTCNR+zCqLt4JMAAYIa5KhMmVLZY/tORRWR1Wc k1 (RSA)
2048 SHA256:TdlvzFJZeToIQ+8GHXMu4PCANn+hZ+EwyrTrXO0JF6Q k4 (RSA)
256 SHA256:IEgRLeZVURUJtQTNSze7CQ/X5ni205hXc1hyHOlwYG8 key five (ED25519)
256 SHA256:wMLkYc1BbcBSLq2mcf3u0X3M9iaVgBveixVjQw4nGrk key six (ED25519)
2048 SHA256:8rWovkQVrbsuztU3Hf5R+JJyOuoHXt8zbfKXhPbpBiA temptest-aws.pem (RSA)


$ ssh -v -i ~/.ssh/temptest-aws.pem centos@172.aa.bb.50
OpenSSH_7.7p1, OpenSSL 1.1.0h-fips  27 Mar 2018
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for *
debug1: Connecting to 172.aa.bb.50 [172.aa.bb.50] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/guest/.ssh/temptest-aws.pem type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/guest/.ssh/temptest-aws.pem-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 172.aa.bb.50:22 as 'centos'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:DcYxjaDCSYORUp585GsysyezKy2Gnf37IZBR1W+iGWw
debug1: Host '172.aa.bb.50' is known and matches the ECDSA host key.
debug1: Found key in /home/guest/.ssh/known_hosts:2
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:1000)

debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:1000)

debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:g3BZ2UTCNR+zCqLt4JMAAYIa5KhMmVLZY/tORRWR1Wc k1
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: RSA SHA256:TdlvzFJZeToIQ+8GHXMu4PCANn+hZ+EwyrTrXO0JF6Q k4
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: ED25519 SHA256:IEgRLeZVURUJtQTNSze7CQ/X5ni205hXc1hyHOlwYG8 key five
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: ED25519 SHA256:wMLkYc1BbcBSLq2mcf3u0X3M9iaVgBveixVjQw4nGrk key six
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: RSA SHA256:8rWovkQVrbsuztU3Hf5R+JJyOuoHXt8zbfKXhPbpBiA temptest-aws.pem
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
debug1: Authentication succeeded (publickey).
Authenticated to 172.aa.bb.50 ([172.aa.bb.50]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
Last login: Wed May  2 20:13:23 2018 from ip-172-xx-yy-zz.us-west-2.compute.internal
[centos@ip-172-aa-bb-50 ~]$



Actual results:

If you have > 6 keys in ssh-agent, you get rejected by the server if they key you're passing on the command-line via '-i' isn't one of the first six keys in the agent.


Expected results:

The ssh connection should only try the key specified on the command-line, as it does for Fedora 27 and all older versions.


Additional info:

Comment 1 Mark R. 2018-05-02 20:51:44 UTC
To verify that it's something about the formatting of the AWS key, here's a connection attempt where I tell it to use one of the fake test keys (k6).  That key isn't first in the agent or in sort order, but you can see the ssh client does correctly try that specific key before moving on to try keys from the agent.

$ ssh -v -i ~/.ssh/k6 centos@172.aa.bb.50
OpenSSH_7.7p1, OpenSSL 1.1.0h-fips  27 Mar 2018
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for *
debug1: Connecting to 172.aa.bb.50 [172.aa.bb.50] port 22.
debug1: Connection established.
debug1: identity file /home/guest/.ssh/k6 type 3
debug1: key_load_public: No such file or directory
debug1: identity file /home/guest/.ssh/k6-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 172.aa.bb.50:22 as 'centos'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:DcYxjaDCSYORUp585GsysyezKy2Gnf37IZBR1W+iGWw
debug1: Host '172.aa.bb.50' is known and matches the ECDSA host key.
debug1: Found key in /home/guest/.ssh/known_hosts:2
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:1000)

debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:1000)

debug1: Next authentication method: publickey

  ## Here the client correctly sends key specified before trying any others ##
  ## Somehow with AWS keys this doesn't happen, but does on Fedora 27       ##
debug1: Offering public key: ED25519 SHA256:wMLkYc1BbcBSLq2mcf3u0X3M9iaVgBveixVjQw4nGrk /home/guest/.ssh/k6
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: RSA SHA256:g3BZ2UTCNR+zCqLt4JMAAYIa5KhMmVLZY/tORRWR1Wc k1
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: RSA SHA256:TdlvzFJZeToIQ+8GHXMu4PCANn+hZ+EwyrTrXO0JF6Q k4
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: ED25519 SHA256:IEgRLeZVURUJtQTNSze7CQ/X5ni205hXc1hyHOlwYG8 key five
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: RSA SHA256:8rWovkQVrbsuztU3Hf5R+JJyOuoHXt8zbfKXhPbpBiA temptest-aws.pem
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
debug1: Authentication succeeded (publickey).
Authenticated to 172.aa.bb.50 ([172.aa.bb.50]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8

Comment 2 Jakub Jelen 2018-05-03 08:00:56 UTC
I assume you are using default GNOME and therefore you are running also gnome-keyring in your environment (which is wrapping ssh-agent from Fedora 28). You can verify that by writing the SSH_AUTH_SOCK environment variable. If it will be listed under /run/user/, it is a gnome-keyring:

$ echo $SSH_AUTH_SOCK
/run/user/1000/keyring/ssh

Can you check if you can reproduce the same issue with ssh-agent only? It should be enough to start a new agent in a new terminal:

  eval `ssh-agent`

after you are done with it, you can kill it with

  ssh-agent -k

If you can reproduce the same with ssh-agent only, I will try to have a look deeper in it. I did not see any change in how the logging order is handled on the first sight. Also if I create normal rsa keys it seems to work fine. This somehow seems like the agent is not providing the matching public key or filename to match the keys against each other?

Comment 3 Mark R. 2018-05-03 15:21:50 UTC
The tests above happened with normal ssh-agent, also happens with the gnome-keyring. I initially hit this with my own account, which would be using the gnome wrapped agent as you suspected. Then I tried by creating a test user and using ssh-agent only, which is the test cases above.

$ echo $SSH_AUTH_SOCK

$ eval $(ssh-agent)
Agent pid 3763
$ ssh-add .ssh/k1
Identity added: .ssh/k1 (.ssh/k1)   # I repeat this for the other keys
$ ssh-add -l
2048 SHA256:g3BZ2UTCNR+zCqLt4JMAAYIa5KhMmVLZY/tORRWR1Wc .ssh/k1 (RSA)
2048 SHA256:0GGpBzlNgJ4w49d9LFoVqG1vkrcpkaRuiPn0oqSFsz4 .ssh/k2 (RSA)
2048 SHA256:MPC3YJYq54W6oKdCHXqqyb6rvTkOeBlNzMEiAyDGp/s .ssh/k3 (RSA)
2048 SHA256:TdlvzFJZeToIQ+8GHXMu4PCANn+hZ+EwyrTrXO0JF6Q .ssh/k4 (RSA)
256 SHA256:IEgRLeZVURUJtQTNSze7CQ/X5ni205hXc1hyHOlwYG8 key five (ED25519)
256 SHA256:wMLkYc1BbcBSLq2mcf3u0X3M9iaVgBveixVjQw4nGrk key six (ED25519)
2048 SHA256:8rWovkQVrbsuztU3Hf5R+JJyOuoHXt8zbfKXhPbpBiA .ssh/temptest-aws.pem (RSA)
$ ssh -v -i ~/.ssh/temptest-aws.pem centos@172.aa.bb.50
OpenSSH_7.7p1, OpenSSL 1.1.0h-fips  27 Mar 2018
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for *
debug1: Connecting to 172.aa.bb.50 [172.aa.bb.50] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/guest/.ssh/temptest-aws.pem type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/guest/.ssh/temptest-aws.pem-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 172.aa.bb.50:22 as 'centos'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:DcYxjaDCSYORUp585GsysyezKy2Gnf37IZBR1W+iGWw
debug1: Host '172.aa.bb.50' is known and matches the ECDSA host key.
debug1: Found key in /home/guest/.ssh/known_hosts:2
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:1000)

debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:1000)

debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:g3BZ2UTCNR+zCqLt4JMAAYIa5KhMmVLZY/tORRWR1Wc .ssh/k1
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: RSA SHA256:0GGpBzlNgJ4w49d9LFoVqG1vkrcpkaRuiPn0oqSFsz4 .ssh/k2
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: RSA SHA256:MPC3YJYq54W6oKdCHXqqyb6rvTkOeBlNzMEiAyDGp/s .ssh/k3
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: RSA SHA256:TdlvzFJZeToIQ+8GHXMu4PCANn+hZ+EwyrTrXO0JF6Q .ssh/k4
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: ED25519 SHA256:IEgRLeZVURUJtQTNSze7CQ/X5ni205hXc1hyHOlwYG8 key five
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: ED25519 SHA256:wMLkYc1BbcBSLq2mcf3u0X3M9iaVgBveixVjQw4nGrk key six
Received disconnect from 172.aa.bb.50 port 22:2: Too many authentication failures
Disconnected from 172.aa.bb.50 port 22



Would it help at all if I repeat these steps on a Fedora 27 host and send you a similar log where the connection succeeds? I can get that done easily enough.

Comment 4 Mark R. 2018-05-03 16:20:03 UTC
Well, I'm more confused now. Booted Fedora 27 from live media, pulled the .ssh directory from my above tests over to it, and I get the same issue where ssh tries all keys in the agent before trying the key specified via '-i' or as IdentityFile in .ssh/config. It does list the key as explicitly requested, but after all the agent keys:

    debug2: key: .ssh/k1 (0x55d0742d0870), agent
    debug2: key: .ssh/k2 (0x55d0742cfd20), agent
    debug2: key: .ssh/k3 (0x55d0742cfe40), agent
    debug2: key: .ssh/k4 (0x55d0742cff60), agent
    debug2: key: key five (0x55d0742d00c0), agent
    debug2: key: key six (0x55d0742cc5b0), agent
    debug2: key: .ssh/temptest-aws.pem (0x55d0742ce4d0), agent
    debug2: key: /home/guest/.ssh/temptest-aws.pem ((nil)), explicit

I'm unsure how I haven't hit this before, but I'm sshing into EC2 instances daily as part of work, and haven't hit this until I updated my laptop to 28. I haven't added any keys to my .ssh folder or agent lately... but my newer keys are ed25519 which I don't belive gnome-keyring was able to deal with, perhaps that has changed with F28 and now those are loaded, causing my agent to have more keys than it previously did?

In case it's lost in the shuffle, this does seem to only occur with a keypair as created/provided by EC2.  If I have a key created via ssh-keygen loaded even further down the list in ssh-agent, I can still login with that key and what's more notable as that this behavior of walking through all the keys in the agent does _not_ happen.  The "walk the agent's keys before trying explicit key" happens _only_ when trying to login with a key that was created via EC2.

So, with a "normal" rsa key listed 9th in the output of ssh-agent, and trying the connection with '-i .ssh/k9', the ssh client tries no other keys at all and just immediately authenticates with that one.

Comment 5 Jakub Jelen 2018-05-03 16:30:26 UTC
Thank you for the reproducing the issue with Fedora 27.

Yes, in Fedora 27, gnome keyring was ignored these keys and therefore they were not offered and it worked fine.

Your analysis is correct: If you have public-private key pair in your drive, the ssh is able to read the public part and match the private one from the agent. This is not possible if ssh it receives only encrypted private key -- it does not know which one is it from agent (or if it is there at all) and therefore it is trying the keys from agent sequentially.

You should be able to workaround it by creating the public key from the private one and store it next to it:

  ssh-keygen -yf ~/.ssh/temptest-aws.pem > ~/.ssh/temptest-aws.pem.pub

Comment 6 Mark R. 2018-05-03 16:51:05 UTC
Hah, well that's embarassing, lack of a public key in the directory sent me down a rabbit hole of confusion. The combination of that plus some ed25519 keys now working as designed in F28 is what led to my failing connections. I apologize for the noise on the bug tracker, but maybe someone else will naively download their key from AWS and throw it in their .ssh directory without creating a matching .pub, and this will thread will show up in their search to save them some time.

Thanks for your help Jakub.


Note You need to log in before you can comment on or make changes to this bug.