Bug 1347912 - sss_ssh_knownhostsproxy leads to silent failure for non-existent or non-co-operative hosts
Summary: sss_ssh_knownhostsproxy leads to silent failure for non-existent or non-co-op...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: sssd
Version: 32
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Pavel Březina
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-06-18 10:28 UTC by James
Modified: 2020-07-30 18:56 UTC (History)
16 users (show)

Fixed In Version: sssd-2.3.1-2.fc32
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-07-30 18:56:35 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github SSSD sssd issues 5236 0 None closed sss_ssh_knownhostsproxy leads to silent failure for non-existent or non-co-operative hosts 2020-11-17 17:16:49 UTC

Description James 2016-06-18 10:28:14 UTC
I have the following in my /etc/ssh/ssh_config, presumably from when this machine was configured as an IPA client:


GlobalKnownHostsFile /var/lib/sss/pubconf/known_hosts
ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h


If I attempt to connect to a non-existent host (i.e. a name that doesn't resolve) I get:


[james@iseult ~]$ ssh -v nonexistenthost
OpenSSH_7.2p2, OpenSSL 1.0.2h-fips  3 May 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 62: Applying options for *
debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy -p 22 nonexistenthost
debug1: permanently_drop_suid: 1311800003
debug1: identity file /home/james/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/james/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/james/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/james/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/james/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/james/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/james/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/james/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2
ssh_exchange_identification: Connection closed by remote host
[james@iseult ~]$


That is, ssh exits with code 255, but no useful message. I think it should fail more noisily -- there's a risk of me lazily assuming that login has been successful, not checking the hostname in the prompt, and carrying on thinking I'm on a remote machine.

With the two /etc/ssh/ssh_config lines commented out, I get:


[james@iseult ~]$ ssh -v nonexistenthost
OpenSSH_7.2p2, OpenSSL 1.0.2h-fips  3 May 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 62: Applying options for *
ssh: Could not resolve hostname nonexistenthost: Name or service not known


-- which is what I'd expect.

Comment 1 James 2016-06-18 10:30:23 UTC
Almost forgot: the versions are

openssh-7.2p2-3.fc23.x86_64
sssd-1.13.4-3.fc23.x86_64
freeipa-client-4.2.4-1.fc23.x86_64

Comment 2 Lukas Slebodnik 2016-10-17 21:27:39 UTC
Sorry for late reply, but I think the error message should be printed by openssh
is ProxyCommand fails

sh$ /usr/bin/sss_ssh_knownhostsproxy -p 22 nonexistenthost
sh$ echo $?
1

Comment 3 James 2016-10-17 21:58:34 UTC
(In reply to Lukas Slebodnik from comment #2)
> Sorry for late reply, but I think the error message should be printed by
> openssh
> is ProxyCommand fails
> 
> sh$ /usr/bin/sss_ssh_knownhostsproxy -p 22 nonexistenthost
> sh$ echo $?
> 1

Are you saying the component should be changed? Does ssh know the manner in which sss_ssh_knownhostsproxy has failed (i.e. does it have different exit values for different kinds of failures)?

Still present in:
openssh-7.3p1-4.fc25.x86_64
sssd-1.14.1-3.fc25.x86_64

Comment 4 Jakub Jelen 2016-10-18 09:02:41 UTC
Well ... the ssh process itself does not do any connection to the host in this case and it gets only the error from the  sss_proxy. It can not know what kind of error though (sss_proxy  returns 1 in case of general failure), but the failure is correctly propagated using exit code (255 per manual page) and error as below. 

$ ssh -oProxyCommand="/usr/bin/sss_ssh_knownhostsproxy -p 22 nonexistenthost" -oGlobalKnownHostsFile=/var/lib/sss/pubconf/known_hosts nonexistenthost
ssh_exchange_identification: Connection closed by remote host.

The proxy command failed and therefore the connection was closed. I don't think we can do anything to improve this in openssh side.

> there's a risk of me lazily assuming that login has been successful

The "Connection closed by remote host." seems like loud enough for me.


But in any case, the proxy can report errors on the stderr (or directly on tty, if attached), which would probably be a way to go if we would like more verbose messages:

$ cat exit255
#!/bin/bash
echo "Error connecting to the host $3" >&2
exit 255
$ ssh -oProxyCommand="./exit255 -p 22 f25" f25
Error connecting to the host f25
ssh_exchange_identification: Connection closed by remote host

Lukas, do you think it would be possible to modify your proxy to do that (I am not aware how does it look from inside though)?

Comment 5 Jakub Hrozek 2016-10-18 09:24:30 UTC
We can print more errors on stderr in sss_ssh_knownhostproxy if looking up the host fails. 

Also, would it help here to just return a specific error code from the sss command and let ssh print the error based on the code or does ssh only expect success/failure?

Comment 6 Jakub Jelen 2016-10-18 09:32:23 UTC
That would be nice.

Specific exit code is not going to help if I am right. SSH itself is trying to pass the exit codes of remote commands back to the caller on success so it basically distinguishes between success (0) and failure (!=0), even though the 255 is somehow reserved for a failure of SSH itself.

I would go with minimal changes to the interface and just add a error output (maybe with possibility to silent it with some switch in case some other scripts would not expect that).

Comment 7 James 2016-10-18 17:33:14 UTC
(In reply to Jakub Jelen from comment #4)
> > there's a risk of me lazily assuming that login has been successful
> 
> The "Connection closed by remote host." seems like loud enough for me.

But it's not stable - sometimes I get "Connection closed by remote host", sometimes not:


[james@iseult ~]$ ssh nonexistenthost
ssh_exchange_identification: Connection closed by remote host
[james@iseult ~]$ ssh nonexistenthost
ssh_exchange_identification: Connection closed by remote host
[james@iseult ~]$ ssh nonexistenthost
ssh_exchange_identification: Connection closed by remote host
[james@iseult ~]$ ssh nonexistenthost
ssh_exchange_identification: Connection closed by remote host
[james@iseult ~]$ ssh nonexistenthost  <--
[james@iseult ~]$ ssh nonexistenthost  <--
[james@iseult ~]$ ssh nonexistenthost
ssh_exchange_identification: Connection closed by remote host
[james@iseult ~]$

Comment 8 Lukas Slebodnik 2016-10-19 05:29:28 UTC
(In reply to Jakub Jelen from comment #4)
> Well ... the ssh process itself does not do any connection to the host in
> this case and it gets only the error from the  sss_proxy. It can not know
> what kind of error though (sss_proxy  returns 1 in case of general failure),
> but the failure is correctly propagated using exit code (255 per manual
> page) and error as below. 
> 
> $ ssh -oProxyCommand="/usr/bin/sss_ssh_knownhostsproxy -p 22
> nonexistenthost" -oGlobalKnownHostsFile=/var/lib/sss/pubconf/known_hosts
> nonexistenthost
> ssh_exchange_identification: Connection closed by remote host.
> 

The manual page (ssh_config) for the ProxyCommand says:
     ProxyCommand
             Specifies the command to use to connect to the server.  The com‐
             mand string extends to the end of the line, and is executed using
             the user's shell ‘exec’ directive to avoid a lingering shell
             process.


However, the sss_ssh_knownhostsproxy does not connect to server.
man sss_ssh_knownhostsproxy says:
       sss_ssh_knownhostsproxy acquires SSH host public keys for host HOST,
       stores them in a custom OpenSSH known_hosts file (see the
       “SSH_KNOWN_HOSTS FILE FORMAT” section of sshd(8) for more information)
       /var/lib/sss/pubconf/known_hosts and establishes the connection to the
       host.

We can consider it as a misusing of ProxyCommand for other purposes.
I do not think that sss_ssh_knownhostsproxy should fail if data cannot be fetched.
Jakub, what do you think about ignoring such situation in sss_ssh_knownhostsproxy return 0 + maybe error message.

Comment 9 Jakub Jelen 2016-10-19 06:29:21 UTC
> But it's not stable - sometimes I get "Connection closed by remote host", sometimes not:

That is weird. What is the exit code of ssh, if you don't get the message? Calling 

  /usr/bin/sss_ssh_knownhostsproxy -p 22 nonexistenthost

exits always with the same exit code?

> man sss_ssh_knownhostsproxy says:
> [...] and establishes the connection to the host.

Not when data can not be fetched, but when you can not connect to the host or

> (i.e. a name that doesn't resolve)

should be the case isn't it?

Comment 10 James 2016-10-19 07:28:00 UTC
(In reply to Jakub Jelen from comment #9)
> > But it's not stable - sometimes I get "Connection closed by remote host", sometimes not:
> 
> That is weird. What is the exit code of ssh, if you don't get the message?

Always 255, whether the message is there or not.

> Calling 
> 
>   /usr/bin/sss_ssh_knownhostsproxy -p 22 nonexistenthost
> 
> exits always with the same exit code?

Yes, always 1.

Comment 11 Jakub Jelen 2016-10-31 09:02:38 UTC
Can you post a verbose log (with -vvv) from the case when it does not show the message and probably the same log with the message? It might show up some abnormalities. But still solving the issue in sssd is probably the way to go.

Comment 12 Fedora End Of Life 2016-11-25 09:19:48 UTC
This message is a reminder that Fedora 23 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 23. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '23'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 23 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 13 Fedora End Of Life 2017-07-25 21:07:33 UTC
This message is a reminder that Fedora 24 is nearing its end of life.
Approximately 2 (two) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 24. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '24'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 24 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 14 Fedora End Of Life 2017-11-16 18:56:11 UTC
This message is a reminder that Fedora 25 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 25. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '25'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 25 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 15 James 2017-11-17 19:33:12 UTC
(In reply to Jakub Jelen from comment #11)
> Can you post a verbose log (with -vvv) from the case when it does not show
> the message and probably the same log with the message? It might show up
> some abnormalities. But still solving the issue in sssd is probably the way
> to go.

Apologies for the outrageous delay on my part in getting back to this. Where things currently stand: with openssh-7.5p1-5.fc27.x86_64 there's no longer any inconsistency in the returned error message. With a nonexistent host I always get

ssh_exchange_identification: Connection closed by remote host

Comment 16 Fabiano Fidêncio 2018-04-11 15:27:32 UTC
Jakub(s), James,

What's the status of this bug? Is this still an issue? If yes, what's the work needed from SSSD side? And from openssh side?

I'd like to have it clear in order to decide whether to close this issue or to keep it open. Also, whether an "upstream" issue would be worth it or not.

Comment 17 Jakub Hrozek 2018-04-11 15:50:50 UTC
(In reply to Fabiano Fidêncio from comment #16)
> Jakub(s), James,
> 
> What's the status of this bug? Is this still an issue? If yes, what's the
> work needed from SSSD side? And from openssh side?
> 
> I'd like to have it clear in order to decide whether to close this issue or
> to keep it open. Also, whether an "upstream" issue would be worth it or not.

I haven't worked on this issue, really, but I assume we still need to implement what comment #6 says but even more importantly test if we can reproduce the same issue as comment #7 describes.

Comment 18 James 2018-04-11 15:55:03 UTC
Despite what I said in comment 15 I do still on occasions get a silent failure, but it's rare. Mostly it gives closed-by-remote.

Comment 19 Jakub Jelen 2018-04-11 16:53:55 UTC
I have no idea. We are not able to reproduce the issue nor the original reporter is. I did not get any debug information that could help to pinpoint the problem in one of the other component.
I did not do anything here either, but some things might have changed also in OpenSSH code (I do not have a track of every change in upstream).

Comment 20 Orion Poplawski 2018-04-17 17:08:44 UTC
The problem I have with sss_ssh_knownhostsproxy is that is obscures the real issue with connecting to a remote host.

Without it, you get:

# ssh blah
ssh: Could not resolve hostname blah: Name or service not known

with it, you get:

# ssh blah
ssh_exchange_identification: Connection closed by remote host

which makes you think that blah exists but there is a problem connecting to it.

For hosts that are down, you get the same message:

# ssh vmsl6
ssh_exchange_identification: Connection closed by remote host

instead of:

# ssh vmsl6
ssh: connect to host vmsl6 port 22: No route to host

openssh-7.7p1-2.fc29.x86_64
sssd-client-1.16.1-2.fc29.x86_64

Comment 21 Ben Cotton 2018-11-27 14:42:41 UTC
This message is a reminder that Fedora 27 is nearing its end of life.
On 2018-Nov-30  Fedora will stop maintaining and issuing updates for
Fedora 27. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora  'version' of '27'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 27 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 22 Ben Cotton 2019-10-31 19:31:37 UTC
This message is a reminder that Fedora 29 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '29'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 29 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 23 Orion Poplawski 2019-11-04 15:57:42 UTC
Things seem to be better now with unknown hosts:

# ssh blah.nwra.com
sss_ssh_knownhostsproxy: Could not resolve hostname blah.nwra.com
ssh_exchange_identification: Connection closed by remote host

But the error message is still different for hosts that are down.  With proxycommnand:

kex_exchange_identification: Connection closed by remote host

Without proxycommand:

ssh: connect to host HOST port 22: No route to host

And for machines not accepting ssh connections.  With proxycommand:

kex_exchange_identification: Connection closed by remote host

Without:

ssh: connect to host HOST port 22: Connection refused

sssd-2.2.0-3.fc29.x86_64
sssd-2.2.2-1.fc31.x86_64

Comment 24 James 2019-11-05 21:05:48 UTC
Now using:
sssd-2.2.2-3.fc31.x86_64
openssh-8.1p1-1.fc31.x86_64


What I find now is with a non-existent host, I always get

$ ssh doesntexist
sss_ssh_knownhostsproxy: Could not resolve hostname doesntexist

Sometimes (but not always) there's also a

kex_exchange_identification: Connection closed by remote host

Since in either case it makes it clear that the name didn't resolve, I'd consider this an improvement. 


For an unresponsive host, the response is often BUT NOT ALWAYS

kex_exchange_identification: Connection closed by remote host

which is a reasonable reply --- but sometimes I'm just returned to the prompt with no indication of failure. There's nothing differentiating in the debug output when I run ssh with -v.

Comment 25 Ben Cotton 2019-11-27 18:13:50 UTC
Fedora 29 changed to end-of-life (EOL) status on 2019-11-26. Fedora 29 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 26 Fedora Admin user for bugzilla script actions 2020-06-18 14:59:20 UTC
This package has changed maintainer in the Fedora.
Reassigning to the new maintainer of this component.

Comment 27 Pavel Březina 2020-07-08 10:29:21 UTC
Upstream ticket:
https://github.com/SSSD/sssd/issues/5236

Comment 28 Pavel Březina 2020-07-08 10:33:07 UTC
This took ridiculously long, I'm sorry.

Upstream PR:
https://github.com/SSSD/sssd/pull/5237

Comment 29 Pavel Březina 2020-07-20 09:14:49 UTC
Pushed PR: https://github.com/SSSD/sssd/pull/5237

* `master`
    * 3be349b962b23126ebe44d6b588924ebe9a1bef8 - sss_ssh_knownhostsproxy: print error when unable to proxy data
    * 0609d0f760b36085ba0c4392d59b410fc208cdd1 - sss_ssh_knownhostsproxy: print error when unable to connect

Comment 30 Fedora Update System 2020-07-27 14:07:09 UTC
FEDORA-2020-63a418c824 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-63a418c824

Comment 31 Fedora Update System 2020-07-28 15:20:01 UTC
FEDORA-2020-63a418c824 has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-63a418c824`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-63a418c824

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 32 Fedora Update System 2020-07-30 18:56:35 UTC
FEDORA-2020-63a418c824 has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.