Bug 2136824
| Summary: | cockpit-ssh get's a connection reset by peer | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Jelle van der Waa <jvanderwaa> |
| Component: | libssh | Assignee: | Norbert Pócs <npocs> |
| Status: | CLOSED ERRATA | QA Contact: | Stanislav Zidek <szidek> |
| Severity: | high | Docs Contact: | |
| Priority: | medium | ||
| Version: | 9.2 | CC: | mpitt, mvollmer |
| Target Milestone: | rc | Keywords: | Triaged |
| Target Release: | --- | Flags: | pm-rhel:
mirror+
|
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | libssh-0.10.4-6.el9 | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-05-09 08:15:49 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
While this runs, there are *zero* messages in `tail -f /var/log/sssd/*`. I'm not sure what `Proxy command returned 127` refers to exactly, supposedly the ssh_knownhostsproxy? cockpit-ssh does call this [1], but with different options than what the log says:
ssh_socket_connect_proxycommand: Executing proxycommand 'exec exec /usr/bin/sss_ssh_knownhostsproxy -p 22 x0.cockpit.lan'
plus, cockpit-ssh's own call succeeds (log says "host known: 1"). The above command seems to happen in ssh_connect(), which fails with that connection reset. I think it's from /etc/ssh/ssh_config.d/04-ipa.conf , which does
Match exec true
ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h
I can call
$ /usr/bin/sss_ssh_knownhostsproxy -p 22 x0.cockpit.lan
SSH-2.0-OpenSSH_8.8
but I don't know how to drive this from the CLI. When I press enter, it just fails with "Invalid SSH identification string."
Running `ssh -vv -K x0.cockpit.lan echo hello` works fine, and it runs the same proxy command:
debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy -p 22 x0.cockpit.lan
[1] https://github.com/cockpit-project/cockpit/blob/a180ea3a134bde4adc40d6713ad8b6cbde594768/src/ssh/cockpitsshrelay.c#L539
"exec exec" looks suspicous. If I run this in the shell: $ exec exec /usr/bin/sss_ssh_knownhostsproxy -p 22 x0.cockpit.lan bash: exec: exec: not found $ echo $? 127 This is a bug in libssh (or maybe a bug in how we use it, but that would make me sad). libssh has the concept of "applying the options", and you can only do that once per ssh_session. In our use it happens twice, which leads to two "exec" prefixes of the ProxyCommand option. ssh_session_connect "applies the options" unconditionally, but several other ssh_session functions can do it as well on-demand, such as ssh_session_ssh_session_has_known_hosts_entry, which cockpit-ssh might call a couple of times. I'll file a bug. We can work around this by calling ssh_session_has_known_hosts_entry from a copy of the main ssh_session structure, made with ssh_options_copy. Proposed workaround: https://github.com/cockpit-project/cockpit/pull/17834 libssh bug report: https://gitlab.com/libssh/libssh-mirror/-/issues/156 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (libssh bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:2476 |
Description of problem: Upgrading from libssh from 0.9.6-3.el9 -> 0.10.4-3.el9 breaks cockpit-ssh with a socket error. Version-Release number of selected component (if applicable): 0.10.4-3.el9 How reproducible: Always, and also on Fedora-37 Steps to Reproduce: This is a bit tricky, our CI test fails Actual results: [root@x0 ~]# G_MESSAGES_DEBUG=all /usr/libexec/cockpit-ssh x0.cockpit.lan 74 {"command":"authorize","challenge":"*","cookie":"session122841666359041"} 1 (cockpit-ssh:12284): cockpit-protocol-WARNING **: 09:30:46.179: Received unparsable control message: JSON data was empty cockpit-ssh-Message: 09:30:46.180: cockpit-ssh did not receive a valid control message (cockpit-ssh:12284): cockpit-ssh-DEBUG: 09:30:46.182: cockpit-ssh x0.cockpit.lan: host argument 'x0.cockpit.lan', host 'x0.cockpit.lan', username 'root', port '0' [2022/10/21 09:30:46.184159, 3] ssh_config_parse_file: Reading configuration data from /etc/libssh/libssh_client.config [2022/10/21 09:30:46.185336, 3] local_parse_file: Reading additional configuration data from /etc/crypto-policies/back-ends/libssh.config [2022/10/21 09:30:46.189563, 3] local_parse_file: Reading additional configuration data from /etc/ssh/ssh_config [2022/10/21 09:30:46.191045, 3] local_parse_file: Reading additional configuration data from /etc/ssh/ssh_config.d/04-ipa.conf [2022/10/21 09:30:46.191836, 4] ssh_config_parse_line: line 9: Processing Match keyword 'exec' [2022/10/21 09:30:46.192814, 3] ssh_exec_shell: Running command 'true' [2022/10/21 09:30:46.199157, 4] ssh_exec_shell: Command 'true' returned 0 [2022/10/21 09:30:46.199429, 4] ssh_match_exec: Matched 'exec' command 'true' (rv=0) [2022/10/21 09:30:46.199943, 3] local_parse_file: Reading additional configuration data from /etc/ssh/ssh_config.d/50-redhat.conf [2022/10/21 09:30:46.200417, 4] ssh_config_parse_line: line 3: Processing Match keyword 'final' [2022/10/21 09:30:46.200855, 2] ssh_config_parse_line: line 3: Unsupported Match keyword 'final', skipping [2022/10/21 09:30:46.201058, 4] ssh_config_parse_line: line 3: Processing Match keyword 'all' [2022/10/21 09:30:46.201256, 3] local_parse_file: Reading additional configuration data from /etc/crypto-policies/back-ends/openssh.config [2022/10/21 09:30:46.201555, 2] ssh_config_parse_line: Unknown option: GSSAPIKexAlgorithms, line: 3 [2022/10/21 09:30:46.201995, 2] ssh_config_parse_line: Unknown option: CASignatureAlgorithms, line: 6 [2022/10/21 09:30:46.202228, 2] ssh_config_parse_line: Unknown option: RSAMinSize, line: 7 [2022/10/21 09:30:46.202498, 2] ssh_config_parse_line: Unapplicable option: ForwardX11Trusted, line: 12 [2022/10/21 09:30:46.202757, 1] ssh_session_has_known_hosts_entry: Cannot access file /root/.ssh/known_hosts [2022/10/21 09:30:46.204838, 1] ssh_key_cmp: key types don't match! [2022/10/21 09:30:46.205105, 1] ssh_key_cmp: key types don't match! [2022/10/21 09:30:46.205434, 1] ssh_key_cmp: key types don't match! (cockpit-ssh:12284): cockpit-ssh-DEBUG: 09:30:46.206: cockpit-ssh x0.cockpit.lan: using known hosts file (null); host known: 1; connect to unknown hosts: 0 [0/581] [2022/10/21 09:30:46.206315, 2] ssh_connect: libssh 0.10.4 (c) 2003-2022 Aris Adamantiadis, Andreas Schneider and libssh contributors. Distributed under the LGPL, please r efer to COPYING file for information about your rights, using threading threads_pthread [2022/10/21 09:30:46.206743, 2] ssh_socket_connect_proxycommand: Executing proxycommand 'exec exec /usr/bin/sss_ssh_knownhostsproxy -p 22 x0.cockpit.lan' [2022/10/21 09:30:46.207504, 2] ssh_socket_connect_proxycommand: ProxyCommand connection pipe: [4,5] [2022/10/21 09:30:46.207842, 1] socket_callback_connected: Socket connection callback: 1 (0) [2022/10/21 09:30:46.208256, 3] ssh_socket_unbuffered_write: Enabling POLLOUT for socket [2022/10/21 09:30:46.208462, 2] ssh_connect: Socket connecting, now waiting for the callbacks to work [2022/10/21 09:30:46.208767, 3] ssh_connect: Actual timeout : 10000 [2022/10/21 09:30:46.209101, 4] ssh_socket_pollcallback: Poll callback on socket 5 (POLLOUT ), out buffer 0 [2022/10/21 09:30:46.211931, 4] ssh_socket_pollcallback: Poll callback on socket 5 (POLLIN POLLERR), out buffer 0 [2022/10/21 09:30:46.212129, 1] ssh_socket_exception_callback: Socket exception callback: 2 (104) [2022/10/21 09:30:46.212361, 1] ssh_socket_exception_callback: Socket error: Connection reset by peer [2022/10/21 09:30:46.212769, 4] ssh_socket_close: Proxy command returned 127 [2022/10/21 09:30:46.212931, 3] ssh_connect: current state : 9 cockpit-ssh-Message: 09:30:46.213: cockpit-ssh x0.cockpit.lan: -1 couldn't connect: Socket error: Connection reset by peer 'x0.cockpit.lan' '22' Additional info: This test server runs with it's own realm and DNS server on a different machine: [root@x0 ~]# cat /etc/resolv.conf domain cockpit.lan search cockpit.lan nameserver 10.111.112.100 x0.cockpit.lan is the test machine itself: [root@x0 ~]# ping x0.cockpit.lan PING x0.cockpit.lan (10.111.113.1) 56(84) bytes of data. ssh works: [root@x0 ~]# ssh admin.lan (admin.lan) Password: Web console: https://x0.cockpit.lan:9090/ Test: https://cockpit-logs.us-east-1.linodeobjects.com/pull-3980-20221020-101845-bef9cefc-rhel-9-2-cockpit-project-cockpit/log.html#280-2