RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2136824 - cockpit-ssh get's a connection reset by peer
Summary: cockpit-ssh get's a connection reset by peer
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: libssh
Version: 9.2
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: rc
: ---
Assignee: Norbert Pócs
QA Contact: Stanislav Zidek
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-10-21 13:58 UTC by Jelle van der Waa
Modified: 2023-05-09 10:18 UTC (History)
2 users (show)

Fixed In Version: libssh-0.10.4-6.el9
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-05-09 08:15:49 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker CRYPTO-8627 0 None None None 2022-10-25 14:59:18 UTC
Red Hat Issue Tracker RHELPLAN-137295 0 None None None 2022-10-21 14:06:13 UTC
Red Hat Product Errata RHBA-2023:2476 0 None None None 2023-05-09 08:15:55 UTC

Description Jelle van der Waa 2022-10-21 13:58:28 UTC 
Description of problem:

Upgrading from libssh from 0.9.6-3.el9 -> 0.10.4-3.el9 breaks cockpit-ssh with a socket error.

Version-Release number of selected component (if applicable):

0.10.4-3.el9

How reproducible:

Always, and also on Fedora-37

Steps to Reproduce:

This is a bit tricky, our CI test fails 

Actual results:

[root@x0 ~]# G_MESSAGES_DEBUG=all /usr/libexec/cockpit-ssh  x0.cockpit.lan
74

{"command":"authorize","challenge":"*","cookie":"session122841666359041"}

1

(cockpit-ssh:12284): cockpit-protocol-WARNING **: 09:30:46.179: Received unparsable control message: JSON data was empty
cockpit-ssh-Message: 09:30:46.180: cockpit-ssh did not receive a valid control message
(cockpit-ssh:12284): cockpit-ssh-DEBUG: 09:30:46.182: cockpit-ssh x0.cockpit.lan: host argument 'x0.cockpit.lan', host 'x0.cockpit.lan', username 'root', port '0'
[2022/10/21 09:30:46.184159, 3] ssh_config_parse_file:  Reading configuration data from /etc/libssh/libssh_client.config
[2022/10/21 09:30:46.185336, 3] local_parse_file:  Reading additional configuration data from /etc/crypto-policies/back-ends/libssh.config
[2022/10/21 09:30:46.189563, 3] local_parse_file:  Reading additional configuration data from /etc/ssh/ssh_config
[2022/10/21 09:30:46.191045, 3] local_parse_file:  Reading additional configuration data from /etc/ssh/ssh_config.d/04-ipa.conf
[2022/10/21 09:30:46.191836, 4] ssh_config_parse_line:  line 9: Processing Match keyword 'exec'
[2022/10/21 09:30:46.192814, 3] ssh_exec_shell:  Running command 'true'
[2022/10/21 09:30:46.199157, 4] ssh_exec_shell:  Command 'true' returned 0
[2022/10/21 09:30:46.199429, 4] ssh_match_exec:  Matched 'exec' command 'true' (rv=0)
[2022/10/21 09:30:46.199943, 3] local_parse_file:  Reading additional configuration data from /etc/ssh/ssh_config.d/50-redhat.conf
[2022/10/21 09:30:46.200417, 4] ssh_config_parse_line:  line 3: Processing Match keyword 'final'
[2022/10/21 09:30:46.200855, 2] ssh_config_parse_line:  line 3: Unsupported Match keyword 'final', skipping
[2022/10/21 09:30:46.201058, 4] ssh_config_parse_line:  line 3: Processing Match keyword 'all'
[2022/10/21 09:30:46.201256, 3] local_parse_file:  Reading additional configuration data from /etc/crypto-policies/back-ends/openssh.config
[2022/10/21 09:30:46.201555, 2] ssh_config_parse_line:  Unknown option: GSSAPIKexAlgorithms, line: 3
[2022/10/21 09:30:46.201995, 2] ssh_config_parse_line:  Unknown option: CASignatureAlgorithms, line: 6
[2022/10/21 09:30:46.202228, 2] ssh_config_parse_line:  Unknown option: RSAMinSize, line: 7
[2022/10/21 09:30:46.202498, 2] ssh_config_parse_line:  Unapplicable option: ForwardX11Trusted, line: 12
[2022/10/21 09:30:46.202757, 1] ssh_session_has_known_hosts_entry:  Cannot access file /root/.ssh/known_hosts
[2022/10/21 09:30:46.204838, 1] ssh_key_cmp:  key types don't match!
[2022/10/21 09:30:46.205105, 1] ssh_key_cmp:  key types don't match!
[2022/10/21 09:30:46.205434, 1] ssh_key_cmp:  key types don't match!
(cockpit-ssh:12284): cockpit-ssh-DEBUG: 09:30:46.206: cockpit-ssh x0.cockpit.lan: using known hosts file (null); host known: 1; connect to unknown hosts: 0          [0/581]
[2022/10/21 09:30:46.206315, 2] ssh_connect:  libssh 0.10.4 (c) 2003-2022 Aris Adamantiadis, Andreas Schneider and libssh contributors. Distributed under the LGPL, please r
efer to COPYING file for information about your rights, using threading threads_pthread
[2022/10/21 09:30:46.206743, 2] ssh_socket_connect_proxycommand:  Executing proxycommand 'exec exec /usr/bin/sss_ssh_knownhostsproxy -p 22 x0.cockpit.lan'
[2022/10/21 09:30:46.207504, 2] ssh_socket_connect_proxycommand:  ProxyCommand connection pipe: [4,5]
[2022/10/21 09:30:46.207842, 1] socket_callback_connected:  Socket connection callback: 1 (0)
[2022/10/21 09:30:46.208256, 3] ssh_socket_unbuffered_write:  Enabling POLLOUT for socket
[2022/10/21 09:30:46.208462, 2] ssh_connect:  Socket connecting, now waiting for the callbacks to work
[2022/10/21 09:30:46.208767, 3] ssh_connect:  Actual timeout : 10000
[2022/10/21 09:30:46.209101, 4] ssh_socket_pollcallback:  Poll callback on socket 5 (POLLOUT ), out buffer 0
[2022/10/21 09:30:46.211931, 4] ssh_socket_pollcallback:  Poll callback on socket 5 (POLLIN POLLERR), out buffer 0
[2022/10/21 09:30:46.212129, 1] ssh_socket_exception_callback:  Socket exception callback: 2 (104)
[2022/10/21 09:30:46.212361, 1] ssh_socket_exception_callback:  Socket error: Connection reset by peer
[2022/10/21 09:30:46.212769, 4] ssh_socket_close:  Proxy command returned 127
[2022/10/21 09:30:46.212931, 3] ssh_connect:  current state : 9
cockpit-ssh-Message: 09:30:46.213: cockpit-ssh x0.cockpit.lan: -1 couldn't connect: Socket error: Connection reset by peer 'x0.cockpit.lan' '22'

Additional info:

This test server runs with it's own realm and DNS server on a different machine:

[root@x0 ~]# cat /etc/resolv.conf
domain cockpit.lan
search cockpit.lan
nameserver 10.111.112.100

x0.cockpit.lan is the test machine itself:

[root@x0 ~]# ping x0.cockpit.lan
PING x0.cockpit.lan (10.111.113.1) 56(84) bytes of data.

ssh works:

[root@x0 ~]# ssh admin.lan
(admin.lan) Password:
Web console: https://x0.cockpit.lan:9090/


Test:

https://cockpit-logs.us-east-1.linodeobjects.com/pull-3980-20221020-101845-bef9cefc-rhel-9-2-cockpit-project-cockpit/log.html#280-2
Comment 1 Martin Pitt 2022-10-24 07:11:32 UTC 
While this runs, there are *zero* messages in `tail -f /var/log/sssd/*`. I'm not sure what `Proxy command returned 127` refers to exactly, supposedly the ssh_knownhostsproxy? cockpit-ssh does call this [1], but with different options than what the log says:

    ssh_socket_connect_proxycommand:  Executing proxycommand 'exec exec /usr/bin/sss_ssh_knownhostsproxy -p 22 x0.cockpit.lan'

plus, cockpit-ssh's own call succeeds (log says "host known: 1"). The above command seems to happen in ssh_connect(), which fails with that connection reset. I think it's from /etc/ssh/ssh_config.d/04-ipa.conf , which does


    Match exec true   
	ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h

I can call

    $ /usr/bin/sss_ssh_knownhostsproxy -p 22 x0.cockpit.lan
    SSH-2.0-OpenSSH_8.8

but I don't know how to drive this from the CLI. When I press enter, it just fails with "Invalid SSH identification string."

Running `ssh -vv -K x0.cockpit.lan echo hello` works fine, and it runs the same proxy command:

    debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy -p 22 x0.cockpit.lan



[1] https://github.com/cockpit-project/cockpit/blob/a180ea3a134bde4adc40d6713ad8b6cbde594768/src/ssh/cockpitsshrelay.c#L539
Comment 2 Marius Vollmer 2022-10-24 07:49:49 UTC 
"exec exec" looks suspicous.  If I run this in the shell:

$ exec exec /usr/bin/sss_ssh_knownhostsproxy -p 22 x0.cockpit.lan
bash: exec: exec: not found
$ echo $?
127
Comment 3 Marius Vollmer 2022-10-24 09:41:41 UTC 
This is a bug in libssh (or maybe a bug in how we use it, but that would make me sad).

libssh has the concept of "applying the options", and you can only do that once per ssh_session.  In our use it happens twice, which leads to two "exec" prefixes of the ProxyCommand option.  ssh_session_connect "applies the options" unconditionally, but several other ssh_session functions can do it as well on-demand, such as ssh_session_ssh_session_has_known_hosts_entry, which cockpit-ssh might call a couple of times.  I'll file a bug.

We can work around this by calling ssh_session_has_known_hosts_entry from a copy of the main ssh_session structure, made with ssh_options_copy.
Comment 4 Marius Vollmer 2022-10-24 10:06:59 UTC 
Proposed workaround: https://github.com/cockpit-project/cockpit/pull/17834
libssh bug report: https://gitlab.com/libssh/libssh-mirror/-/issues/156
Comment 9 errata-xmlrpc 2023-05-09 08:15:49 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (libssh bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2476
Note You need to log in before you can comment on or make changes to this bug.