Bug 2137839
Summary: | libssh 0.10 can no longer parse "+" character in .ssh/config lines | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 9 | Reporter: | Xiaodai Wang <xiaodwan> |
Component: | libssh | Assignee: | Norbert Pócs <npocs> |
Status: | CLOSED ERRATA | QA Contact: | Stanislav Zidek <szidek> |
Severity: | high | Docs Contact: | |
Priority: | high | ||
Version: | 9.2 | CC: | jjelen, jpazdziora, juzhou, lersek, mxie, rjones, tyan, tzheng, vwu |
Target Milestone: | rc | Keywords: | Regression, Triaged |
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | libssh-0.10.4-6.el9 | Doc Type: | No Doc Update |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2023-05-09 08:15:49 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Xiaodai Wang
2022-10-26 11:23:00 UTC
Thanks Laszlo Ersek's help. ------------------------------ Hi Xiaodai, this seems to be a regression in RHEL-9 libssh; more closely in those parts that parse the ~/.ssh/config file. Namely, on your system, libssh-0.10.4-3.el9 was originally installed; that's when the ssh connection from nbdkit's ssh plugin fails (confirmed by passing -v -x to virt-v2v). After I had downgraded libssh to libssh-0.9.6-3.el9, which is what the latest *released* RHEL-9 variant ships, virt-v2v is back to functional state. For more details... it was not just the libssh package that I needed to downgrade; I needed to keep the libssh-devel and libssh-config subpackages in sync too. These are the package builds in Brew: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1791187 https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=2198242 Note that the %changelog in the latter highlights: Rebase to version 0.10.4 so that's most likely where this regression comes from. Note that the system-wide libssh client config (/etc/libssh/libssh_client.config) had not changed between both builds, so it's very likely a regression in libssh's config parser -- not just in RHEL-9.2, but upstream. (I see a bunch of changes upstream, for "src/options.c", in the libssh-0.9.6..libssh-0.10.4 commit range.) Yet more details... if you change the ~/.ssh/config file as follows: --- .ssh/config.bak 2022-10-26 11:25:25.015777474 +0200 +++ .ssh/config 2022-10-26 11:23:57.127559875 +0200 @@ -1,5 +1,5 @@ Host 10.73.224.33 - KexAlgorithms +diffie-hellman-group14-sha1 + KexAlgorithms diffie-hellman-group14-sha1 MACs +hmac-sha1 HostKeyAlgorithms +ssh-rsa PubkeyAcceptedKeyTypes +ssh-rsa That is, if you remove the plus sign, then libssh-0.10.4-3.el9 successfully parses the "KexAlgorithms" key, and the error message *changes* to: nbdkit: ssh[1]: error: failed to connect to remote host: 10.73.224.33: kex error : no match for method server host key algo: server [ssh-rsa,ssh-dss], client [rsa-sha2-512,rsa-sha2-256,ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256] In other words, now it complains about the *host key* algorithm, and not the key exchange algorithm. Obviously, this complaint is just as bogus: we do have HostKeyAlgorithms +ssh-rsa in our config file; it's just that libssh has lost the ability to parse the plus sign (+) at the front of the value. I suggest reporting a bug (a regression) for libssh in RHEL-9.2, and adding Jakub Jelen <jjelen> to the CC list of the ticket. Reproducing this is fairly easy *if* you have access to a RHEL 6 system. I have a RHEL 6 system called 'onuma'. The other commands are run on Fedora or RHEL 9. Prepare RHEL 6 system (onuma) by creating a remote disk image: onuma# truncate -s 1M /var/tmp/disk.img Everything else is run on Fedora or RHEL 9: (1) Edit $HOME/.ssh/config and add: Host onuma # replace with name of your RHEL 6 system HostKeyAlgorithms +ssh-rsa PubkeyAcceptedAlgorithms +ssh-rsa (2) Install nbdkit-ssh-plugin & nbdinfo: # dnf install nbdkit nbdkit-ssh-plugin libnbd (3) Run this command (replacing the host= with your RHEL 6 host): $ nbdkit ssh host=onuma /var/tmp/disk.img config=$HOME/.ssh/config --run 'nbdinfo $uri' It will fail with several errors, starting with: nbdkit: ssh[1]: error: failed to connect to remote host: onuma: kex error : no match for method server host key algo: server [ssh-rsa,ssh-dss], client [rsa-sha2-512,rsa-sha2-256,ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256] (4) Edit ~/.ssh/config and remove the "+" signs. (5) Repeat the command in (3) and it will work. Note that the "+" signs appear to be valid, in as much as openssh can parse them fine. Git bisect says this, but I'm pretty sure it's not correct. I think what is happening is my test works before this commit because RSA + SHA1 is enabled by default and it never has to read the configuration file. (This is just a guess, I didn't look at the code yet.) Possible therefore that it never parsed the "+" sign correctly, but the removal of SHA1 has just exposed the error because it now needs to parse the config file? fecdc3cc0e6d051ebbe06414a15c6634a4126a8b is the first bad commit commit fecdc3cc0e6d051ebbe06414a15c6634a4126a8b Author: Jakub Jelen <jjelen> Date: Tue Apr 14 12:26:50 2020 +0200 Disable RSA and DSA keys with sha1 by default Fixes: T218 Signed-off-by: Jakub Jelen <jjelen> Reviewed-by: Anderson Toshiyuki Sasaki <ansasaki> src/kex.c | 32 ++++++++++++++++++---------- tests/unittests/torture_knownhosts_parsing.c | 16 ++++---------- 2 files changed, 25 insertions(+), 23 deletions(-) The config parser eventually calls: ssh_options_set(session, SSH_OPTIONS_HOSTKEYS, "+ssh-rsa"); which is documented (https://api.libssh.org/stable/group__libssh__session.html) as accepting “comma-separated list ex: "ssh-rsa,ssh-dss,ecdh-sha2-nistp256” so not the "+xxx" format. Also verified by examining the code for this function - it just copies the string verbatim into the session->opts.wanted_methods[SSH_HOSTKEYS] field without any attempt to parse '+' characters. And the kex code and the ssh_find_matching functions just split the string on ',' and do not attempt anything with '+' characters. I'm not sure the best place to solve this. I suppose we might modify src/config.c so that if the first character is "+" then we read the current SSH_OPTIONS_HOSTKEYS and append the new keys instead of replacing? I defer to someone who is more expert in the code to tell me if this is a good idea or not. I've tried to look up the background on "T218". Some unfortunate project management decisions make that impossible however: - the original bug supposedly used to exist at <https://bugs.libssh.org/T218>, but the link no longer works, and the redirect to gitlab also doesn't work - searching the ticket list at <https://gitlab.com/libssh/libssh-mirror/-/issues/?sort=created_date&state=all&first_page_size=20> for T218 returns no results, IOW the bug has not been migrated from the original trac system to gitlab - The libssh mailing lists described at <https://www.libssh.org/communication/> include "libssh-tickets", which looks promising; however, this list has not been publicly archived -- only the main discussion list is archived at <https://archive.libssh.org/>. I've not been able to find a different (independent) archive either. Wiping your historical bug DB off the face of the earth is not the best move for the community. IA has it although there's not a lot of content in the bug: https://web.archive.org/web/20210127004644/https://bugs.libssh.org/T218 Disabling of SHA1 by default was system-wide change in RHEL to tighten security and it is along the lines of what was done in both OpenSSH and other crypto libraries and crypto-policies. The + sign was never supported in libssh, but it looks like something we will look into and try to implement for libssh as it looks like there is many people using this OpenSSH configuration in libssh. Can you comment on whether my suggested fix in comment 4 would be the right approach? (In reply to Richard W.M. Jones from comment #8) > Can you comment on whether my suggested fix in comment 4 would be the right > approach? Yes, that sounds like a good approach. Norbert is already looking into this issue. OK let me know if you need any help. Is there an upstream patch or package we can try out? Here is the upstream pull request which is under review atm. https://gitlab.com/libssh/libssh-mirror/-/merge_requests/319 I also tested it by virt-v2v, the function works well. # rpm -q libssh virt-v2v libssh-0.10.4-6.el9.x86_64 virt-v2v-2.0.7-6.el9.x86_64 # cat ~/.ssh/config Host 10.73.224.33 KexAlgorithms +diffie-hellman-group14-sha1 MACs +hmac-sha1 HostKeyAlgorithms +ssh-rsa PubkeyAcceptedKeyTypes +ssh-rsa PubkeyAcceptedAlgorithms +ssh-rsa # cat ~/openssl-sha1.cnf .include /etc/ssl/openssl.cnf [openssl_init] alg_section = evp_properties [evp_properties] rh-allow-sha1-signatures = yes # avocado run --vt-type v2v function_test_xen.positive_test.rhev.multiconsole.output_mode.rhev.rhv_upload JOB ID : a32d10aa4957bc5d58e484f80f0f5b04837c77c0 JOB LOG : /root/avocado/job-results/job-2022-12-04T08.40-a32d10a/job.log (1/1) type_specific.io-github-autotest-libvirt.function_test_xen.positive_test.rhev.multiconsole.output_mode.rhev.rhv_upload: STARTED █ 100% [****************************************] (1/1) type_specific.io-github-autotest-libvirt.function_test_xen.positive_test.rhev.multiconsole.output_mode.rhev.rhv_upload: PASS (393.76 s) RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 397.45 s Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (libssh bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:2476 |