Bug 2127970

Summary: Inconsistent behaviour when using -i and -J options
Product: Red Hat Enterprise Linux 9 Reporter: Dr. Stephan Wonczak <wonczak>
Component: opensshAssignee: Dmitry Belyavskiy <dbelyavs>
Status: CLOSED NOTABUG QA Contact: BaseOS QE Security Team <qe-baseos-security>
Severity: low Docs Contact:
Priority: low    
Version: CentOS StreamCC: bstinson, jjelen, jwboyer
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-30 11:22:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dr. Stephan Wonczak 2022-09-19 13:48:20 UTC
Description of problem:
Keys explicitly provided by "-i /path/to/keyfile" is ignored when -J <jumphost> is on the command line

Version-Release number of selected component (if applicable):

openssh-8.7p1-22.el9.x86_64

How reproducible:
always

Steps to Reproduce:
1. ssh -i /path/to/ssh-key -J user@jumphost user@targethost

Actual results:
ssk-key is ignored, login fails

Expected results:
ssh-key should be used, login succeeds

Additional info:
Detailed description and observations. 

In the process of setting up a jumphost I noticed what I would classify as a bug in ssh.

I set up a jumphost which knows userA, which then passes the connection though to userA@targethost. The jumphost does not allow password-login of userA, just key-login. A special key was generated for this connection chain, which is stored in /path/to/ssh-key. The public key was put into the authorized_keys-file of userA on both jumphost and targethost.

I can login into the jumphost by

ssh -i /path/to/ssh-key userA@jumphost

I can also directly login to the targethost:

ssh -i /path/to/ssh-key userA@jumphost

This will be disallowed later, hence the requirement for a jumphost. The jumphost will also disallow direct logins of users later; direct login is only allowed for debugging right now. Now for the weird part:

ssh -i /path/to/ssh-key -J userA@jumphost userA@targethost

fails with 

userA@jumphost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

checking with "ssh -v", the key provided by the -i option is not even read (keys replaced by #### for privacy):

(...)
debug1: Will attempt key: /home/localuser/.ssh/id_rsa RSA SHA256:####  agent
debug1: Will attempt key: user,11.11.2008,linux mgr user RSA SHA256:#### agent
debug1: Will attempt key: /home/localuser/.ssh/id_dsa 
debug1: Will attempt key: /home/localuser/.ssh/id_ecdsa 
(...)
debug1: Next authentication method: publickey
debug1: Offering public key: /home/localuser/.ssh/id_rsa RSA SHA256:#### agent
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: user,11.11.2008,linux mgr user RSA SHA256:#### agent
d
(...)

If I place /path/to/ssh-key into the .ssh-folder of localuser, the key -is- offered automatically (no -i option this time), and login to targethost works

(...)
debug1: Will attempt key: /home/localuser/.ssh/id_rsa RSA SHA256:####  agent
debug1: Will attempt key: user ECDSA SHA256:#### agent
debug1: Will attempt key: user,11.11.2008,linux mgr user RSA SHA256:#### agent
debug1: Will attempt key: /home/localuser/.ssh/id_dsa 
debug1: Will attempt key: /home/localuser/.ssh/id_ecdsa 
(...)
debug1: Next authentication method: publickey
debug1: Offering public key: /home/localuser/.ssh/id_rsa RSA SHA256:#### agent
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: user ECDSA SHA256:#### agent
debug1: Server accepts key: user ECDSA SHA256:#### agent
debug1: Offering public key: user,11.11.2008,linux mgr user RSA SHA256:#### agent
d
(...)

If I remove the key from localuser's .ssh-folder again, but add it to ssh-agent: (ssh-add /path/to/ssh-key), login again works (no -i option):

(...)
debug1: Will attempt key: /home/localuser/.ssh/id_rsa RSA SHA256:####  agent
debug1: Will attempt key: user ECDSA SHA256:#### agent
debug1: Will attempt key: user,11.11.2008,linux mgr user RSA SHA256:#### agent
debug1: Will attempt key: /home/localuser/.ssh/id_dsa 
debug1: Will attempt key: /home/localuser/.ssh/id_ecdsa 
(...)
debug1: Next authentication method: publickey
debug1: Offering public key: /home/localuser/.ssh/id_rsa RSA SHA256:#### agent
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: /path/to/ssh-key ECDSA SHA256:#### agent
debug1: Server accepts key: /path/to/ssh-key ECDSA SHA256:#### agent
(...)

At one point I had forgotten to remove the -i from the command line after adding the key to ssh-agent, and the last to lines changed to:

(...)
debug1: Offering public key: /path/to/ssh-kedebug1: Offering public key: /path/to/ssh-key ECDSA SHA256:#### agent
debug1: Server accepts key: /path/to/ssh-key ECDSA SHA256:#### agent
y ECDSA SHA256:#### explicit agent
debug1: Server accepts key: /path/to/ssh-key ECDSA SHA256:#### explicit agent
(...)

So it seems that the -i option -is- evaluated, but not used in all cases. Very strange, and IMHO inconsistent behaviour indeed.

Comment 1 Dr. Stephan Wonczak 2022-10-12 13:00:13 UTC
No comments? This issue is severly hampering testing/debugging when creating a new jumphost.

Comment 2 Dmitry Belyavskiy 2022-10-12 13:19:18 UTC
Sorry for the delay. I didn't have a chance to investigate it yet :(

As I'm not aware of any difference between our code and upstream here, it's probably worth raising this issue upstream.

Comment 3 Dr. Stephan Wonczak 2022-10-18 11:14:18 UTC
I think if this is brought upstream by a RedHat-Engineer, this would probably get a bit more attention than if it came from me.
In any case the behaviour is easily reproducible, as described above.

Comment 4 Dmitry Belyavskiy 2022-10-20 14:15:31 UTC
I'm sorry but at first (and second) glance I don't see any code that could cause this behavior.
 
As the questions will be definitely raised, I strongly prefer you to report this problem because only you can answer those questions.

Comment 5 Dr. Stephan Wonczak 2022-10-24 13:49:17 UTC
OK, I followed you suggestion and reported this issue in the OpenSSH-bugtracker.
Bug-ID for reference: https://bugzilla.mindrot.org/show_bug.cgi?id=3490
If there are news I will update here, too.

Comment 6 Dmitry Belyavskiy 2022-10-24 13:53:08 UTC
Thank you very much!

Comment 7 Dmitry Belyavskiy 2022-11-30 11:22:28 UTC
Upstream considers this behavior as correct, closing.