Thanks Laszlo Ersek's help.

------------------------------
Hi Xiaodai,
this seems to be a regression in RHEL-9 libssh; more closely in those
parts that parse the ~/.ssh/config file.

Namely, on your system, libssh-0.10.4-3.el9 was originally installed;
that's when the ssh connection from nbdkit's ssh plugin fails (confirmed
by passing -v -x to virt-v2v). After I had downgraded libssh to
libssh-0.9.6-3.el9, which is what the latest *released* RHEL-9 variant
ships, virt-v2v is back to functional state.

For more details... it was not just the libssh package that I needed to
downgrade; I needed to keep the libssh-devel and libssh-config
subpackages in sync too.

These are the package builds in Brew:

https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1791187
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=2198242

Note that the %changelog in the latter highlights:

  Rebase to version 0.10.4

so that's most likely where this regression comes from.

Note that the system-wide libssh client config
(/etc/libssh/libssh_client.config) had not changed between both builds,
so it's very likely a regression in libssh's config parser -- not just
in RHEL-9.2, but upstream. (I see a bunch of changes upstream, for
"src/options.c", in the libssh-0.9.6..libssh-0.10.4 commit range.)

Yet more details... if you change the ~/.ssh/config file as follows:

--- .ssh/config.bak     2022-10-26 11:25:25.015777474 +0200
+++ .ssh/config 2022-10-26 11:23:57.127559875 +0200
@@ -1,5 +1,5 @@
 Host 10.73.224.33
-  KexAlgorithms            +diffie-hellman-group14-sha1
+  KexAlgorithms            diffie-hellman-group14-sha1
   MACs                     +hmac-sha1
   HostKeyAlgorithms        +ssh-rsa
   PubkeyAcceptedKeyTypes   +ssh-rsa

That is, if you remove the plus sign, then libssh-0.10.4-3.el9
successfully parses the "KexAlgorithms" key, and the error message
*changes* to:

nbdkit: ssh[1]: error: failed to connect to remote host: 10.73.224.33:
kex error : no match for method server host key algo: server
[ssh-rsa,ssh-dss], client
[rsa-sha2-512,rsa-sha2-256,ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256]

In other words, now it complains about the *host key* algorithm, and not
the key exchange algorithm. Obviously, this complaint is just as bogus:
we do have

   HostKeyAlgorithms        +ssh-rsa

in our config file; it's just that libssh has lost the ability to parse
the plus sign (+) at the front of the value.

I suggest reporting a bug (a regression) for libssh in RHEL-9.2, and
adding Jakub Jelen <jjelen> to the CC list of the ticket.

Reproducing this is fairly easy *if* you have access to a RHEL 6 system.
I have a RHEL 6 system called 'onuma'.  The other commands are run on
Fedora or RHEL 9.

Prepare RHEL 6 system (onuma) by creating a remote disk image:

onuma# truncate -s 1M /var/tmp/disk.img

Everything else is run on Fedora or RHEL 9:

(1) Edit $HOME/.ssh/config and add:

Host onuma   # replace with name of your RHEL 6 system                       
    HostKeyAlgorithms +ssh-rsa                                                    
    PubkeyAcceptedAlgorithms +ssh-rsa                                             

(2) Install nbdkit-ssh-plugin & nbdinfo:

# dnf install nbdkit nbdkit-ssh-plugin libnbd

(3) Run this command (replacing the host= with your RHEL 6 host):

$ nbdkit ssh host=onuma /var/tmp/disk.img config=$HOME/.ssh/config --run 'nbdinfo $uri'

It will fail with several errors, starting with:

nbdkit: ssh[1]: error: failed to connect to remote host: onuma: kex error : no match for method server host key algo: server [ssh-rsa,ssh-dss], client [rsa-sha2-512,rsa-sha2-256,ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256]

(4) Edit ~/.ssh/config and remove the "+" signs.

(5) Repeat the command in (3) and it will work.

Note that the "+" signs appear to be valid, in as much as openssh can parse them fine.

Git bisect says this, but I'm pretty sure it's not correct.  I think what
is happening is my test works before this commit because RSA + SHA1 is
enabled by default and it never has to read the configuration file.
(This is just a guess, I didn't look at the code yet.)  Possible therefore
that it never parsed the "+" sign correctly, but the removal of SHA1 has
just exposed the error because it now needs to parse the config file?

fecdc3cc0e6d051ebbe06414a15c6634a4126a8b is the first bad commit
commit fecdc3cc0e6d051ebbe06414a15c6634a4126a8b
Author: Jakub Jelen <jjelen>
Date:   Tue Apr 14 12:26:50 2020 +0200

    Disable RSA and DSA keys with sha1 by default
    
    Fixes: T218
    
    Signed-off-by: Jakub Jelen <jjelen>
    Reviewed-by: Anderson Toshiyuki Sasaki <ansasaki>

 src/kex.c                                    | 32 ++++++++++++++++++----------
 tests/unittests/torture_knownhosts_parsing.c | 16 ++++----------
 2 files changed, 25 insertions(+), 23 deletions(-)

The config parser eventually calls:

            ssh_options_set(session, SSH_OPTIONS_HOSTKEYS, "+ssh-rsa");

which is documented (https://api.libssh.org/stable/group__libssh__session.html)
as accepting “comma-separated list ex: "ssh-rsa,ssh-dss,ecdh-sha2-nistp256”
so not the "+xxx" format.  Also verified by examining the code for this function -
it just copies the string verbatim into the session->opts.wanted_methods[SSH_HOSTKEYS]
field without any attempt to parse '+' characters.  And the kex code and the
ssh_find_matching functions just split the string on ',' and do not attempt anything
with '+' characters.

I'm not sure the best place to solve this.  I suppose we might modify src/config.c
so that if the first character is "+" then we read the current SSH_OPTIONS_HOSTKEYS
and append the new keys instead of replacing?

I defer to someone who is more expert in the code to tell me if this is a good idea or not.

I've tried to look up the background on "T218". Some unfortunate project management decisions make that impossible however:

- the original bug supposedly used to exist at <https://bugs.libssh.org/T218>, but the link no longer works, and the redirect to gitlab also doesn't work

- searching the ticket list at <https://gitlab.com/libssh/libssh-mirror/-/issues/?sort=created_date&state=all&first_page_size=20> for T218 returns no results, IOW the bug has not been migrated from the original trac system to gitlab

- The libssh mailing lists described at <https://www.libssh.org/communication/> include "libssh-tickets", which looks promising; however, this list has not been publicly archived -- only the main discussion list is archived at <https://archive.libssh.org/>. I've not been able to find a different (independent) archive either.

Wiping your historical bug DB off the face of the earth is not the best move for the community.

IA has it although there's not a lot of content in the bug:
https://web.archive.org/web/20210127004644/https://bugs.libssh.org/T218

Disabling of SHA1 by default was system-wide change in RHEL to tighten security and it is along the lines of what was done in both OpenSSH and other crypto libraries and crypto-policies. The + sign was never supported in libssh, but it looks like something we will look into and try to implement for libssh as it looks like there is many people using this OpenSSH configuration in libssh.

Can you comment on whether my suggested fix in comment 4 would be the right approach?

(In reply to Richard W.M. Jones from comment #8)
> Can you comment on whether my suggested fix in comment 4 would be the right
> approach?

Yes, that sounds like a good approach. Norbert is already looking into this issue.

OK let me know if you need any help.

Is there an upstream patch or package we can try out?

Here is the upstream pull request which is under review atm.

https://gitlab.com/libssh/libssh-mirror/-/merge_requests/319

I also tested it by virt-v2v, the function works well.

# rpm -q libssh virt-v2v 
libssh-0.10.4-6.el9.x86_64
virt-v2v-2.0.7-6.el9.x86_64

# cat ~/.ssh/config 
Host 10.73.224.33
  KexAlgorithms            +diffie-hellman-group14-sha1
  MACs                     +hmac-sha1
  HostKeyAlgorithms        +ssh-rsa
  PubkeyAcceptedKeyTypes   +ssh-rsa
  PubkeyAcceptedAlgorithms +ssh-rsa
# cat ~/openssl-sha1.cnf 
.include /etc/ssl/openssl.cnf
[openssl_init]
alg_section = evp_properties
[evp_properties]
rh-allow-sha1-signatures = yes

# avocado run --vt-type v2v function_test_xen.positive_test.rhev.multiconsole.output_mode.rhev.rhv_upload
JOB ID     : a32d10aa4957bc5d58e484f80f0f5b04837c77c0
JOB LOG    : /root/avocado/job-results/job-2022-12-04T08.40-a32d10a/job.log
 (1/1) type_specific.io-github-autotest-libvirt.function_test_xen.positive_test.rhev.multiconsole.output_mode.rhev.rhv_upload: STARTED
█ 100% [****************************************]
 (1/1) type_specific.io-github-autotest-libvirt.function_test_xen.positive_test.rhev.multiconsole.output_mode.rhev.rhv_upload: PASS (393.76 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 397.45 s

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (libssh bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2476