Bug 2062360

(In reply to Richard W.M. Jones from comment #0)
> Description of problem:
>
> Currently virt-v2v advertises this advice in the manual:
>
>   With some modern ssh implementations, legacy crypto policies
>   required to interoperate with RHEL 5 sshd are disabled. To
>   enable them you may need to run this command on the conversion
>   server (ie. ssh client), but read update-crypto-policies(8) first:
>
>    # update-crypto-policies --set LEGACY
>
> (from
> https://libguestfs.org/virt-v2v-input-xen.1.html#ssh-authentication)
>
> We should replace this with some more targeted mechanism to
> be defined.  Several suggestions have been made:
>
>  * Replace the documentation with host-specific configuration
>    in .ssh/config so that at least the whole system doesn't
>    use legacy crypto.  Note that we need to check that all of
>    the ssh techniques we use will actually consume this
>    configuration - does nbdkit-ssh-plugin?
>
>  * Modify virt-v2v invocation of ssh, scp, nbdkit-ssh-plugin in
>    some way to weaken crypto requirements when connecting to
>    hosts.  (This is not as secure as the previous method, but
>    considerably easier for end users.)

The second option is much-much worse IMO. First, we'd be taking
responsibility for weakening crypto, which is never a good position to
be in. Second, we'd either have to bake those settings into the source
code (making it unmodifiable for users -- in either direction
(weakening/strengthening)), or invent a separate config format for them.
I consider the Host stanzas in ~/.ssh/config the canonical way to steer
these settings.

nbdkit-ssh-plugin does consume both ~/.ssh/config and
/etc/ssh/ssh_config:

https://libguestfs.org/nbdkit-ssh-plugin.1.html#PARAMETERS

> config=CONFIG_FILE
>
>     Read local SSH configuration from an alternate configuration file.
>     Libssh expands some %-sequences in CONFIG_FILE, see "Path
>     expansion" below. CONFIG_FILE must expand to an absolute path.
>
> config=
>
>     Do not read any local SSH configuration.
>
>     The config parameter is optional. If it is not specified at all
>     then ~/.ssh/config and /etc/ssh/ssh_config are both read. Missing
>     or unreadable files are ignored.

(In reply to Laszlo Ersek from comment #3)
> nbdkit-ssh-plugin does consume both ~/.ssh/config and
> /etc/ssh/ssh_config:

This was intended as a note that we should be careful to check
any recommendations we make are really parsed by libssh.  libssh uses
its own code (separate from openssh) to parse these config files so
might not parse exactly the same language and/or features.

I think your argument is on the right lines here.  We shouldn't
weaken all uses of ssh.  Documentation for modifying the config
file sounds better.

Some Fedora mailing list discussion starting here (unfortunately I
cannot link to the whole subthread because hyperkitty is annoying):

https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/BZ66E2DDSKYXRYDCZ3JIQAKEVM4YQDPY/

See also bug 2064740

*** Bug 1778090 has been marked as a duplicate of this bug. ***

I believe virt-v2v uses ssh for the following purposes:

(1) -i vmx -it ssh:

  setup                            [input/input_vmx.ml]
    parse_domain_from_vmx          [input/parse_domain_from_vmx.ml]
      scp_from_remote_to_temporary [input/parse_domain_from_vmx.ml]
        calls "scp"
    remote_file_exists             [input/parse_domain_from_vmx.ml]
      calls "ssh"
    Nbdkit_ssh.create_ssh          [input/nbdkit_ssh.ml]
      starts nbdkit with the ssh
        plugin

(2) -ic 'xen+ssh://root.com'

  setup                            [input/input_xen_ssh.ml]
    Libvirt.Connect.connect_auth
    Nbdkit_ssh.create_ssh          [input/nbdkit_ssh.ml]
      starts nbdkit with the ssh
        plugin

(3) -ic vpx://... -it vddk

  setup                          [input/input_vddk.ml]
    Libvirt.Connect.connect_auth

(4) -ic vpx://...

  setup                          [input/input_vcenter_https.ml]
    Libvirt.Connect.connect_auth

From these, (1), (3) and (4) connect to various vmware components using
SSL / SSH , and I don't think we've seen complaints there -- I think
those vmware server components all support the restricted crypto
settings of modern Fedora / RHEL clients, out of the box.

So what seems to remain is (2): connecting to a RHEL 5 Xen server over
SSH. (Here I'm basically just confirming / asking if there is no wider
scope to this bug than stated in comment#0.)

Under (2), two methods are used: one where the libvirt client starts
"ssh" (Libvirt.Connect.connect_auth), and another where nbdkit's ssh
plugin (via Nbdkit_ssh.create_ssh) uses libssh.

Both of these methods should be testable if I manage to set up a RHEL 5
VM, running Xen in L1, dom0 in L2, and one domU in L2. (I might not even
need nested virt: purely PV domUs work on CPUs without virtualization
extensions, so I can hide "vmx" from L1.) With this configured, I should
attempt the conversion, and keep tweaking the local ssh config file
surgically until the conversion succeeds.

Is there anything else we should cover?

I agree with the analysis.  Is libvirt virConnectAuth* using
ssh at all?  There is no mention of ssh under libvirt.git/src/esx,
and I assumed it uses some kind of HTTPS-based API.

Anyway nbdkit uses libssh, and should by default read
the ssh config file, although it's a bit unclear if it reads
$HOME/.ssh/config or only the global /etc/ssh/ssh_config.

(In reply to Richard W.M. Jones from comment #10)
> Is libvirt virConnectAuth* using
> ssh at all?  There is no mention of ssh under libvirt.git/src/esx,
> and I assumed it uses some kind of HTTPS-based API.

I've had my doubts about that as well. The libvirt URIs look deceptively similar <https://libvirt.org/uri.html>:

> qemu+ssh://root.example.com/system
> xen+ssh://rjones@towada/system

Underneath, the "driver" parts of the URI schemes work differently: the "qemu" driver connects to libvirtd, but the "xen" driver connects to... something else Xen specific I guess?

However, the "transport" part of the scheme, "ssh", should work identically in both cases: <https://libvirt.org/uri.html#ssh-transport> -- "The ssh transport uses the standard SSH protocol via the system installed binary".

In the libvirt tree, there seems to be a "remote" driver that handles URI schemes that no other driver handles.

For example, there is a local-only Xen driver that handles only the "xen" URI scheme: "libxlConnectDriver" in "src/libxl/libxl_driver.c". The internal name of this driver is "Xen" (libxlHypervisorDriver.name = LIBXL_DRIVER_EXTERNAL_NAME).

But the driver that I think matters here is the "remote" driver: "connect_driver" in "src/remote/remote_driver.c", which references "hypervisor_driver" in the same file (.name = "remote"). The connectOpen member of this structure is remoteConnectOpen(), and then we have:

remoteConnectOpen()                        [src/remote/remote_driver.c]
  remoteSplitURIScheme()                   [src/remote/remote_sockets.c]
    -> outputs REMOTE_DRIVER_TRANSPORT_SSH
  doRemoteOpen()                           [src/remote/remote_driver.c]
    -> starts "ssh"

The comment on "doRemoteOpen" is:

/*
 * URIs that this driver needs to handle:
 *
 * The easy answer:
 *   - Everything that no one else has yet claimed, but nothing if
 *     we're inside the libvirtd daemon
 *
 * The hard answer:
 *   - Plain paths (///var/lib/xen/xend-socket)  -> UNIX domain socket
 *   - xxx://servername/      -> TLS connection
 *   - xxx+tls://servername/  -> TLS connection
 *   - xxx+tls:///            -> TLS connection to localhost
 *   - xxx+tcp://servername/  -> TCP connection
 *   - xxx+tcp:///            -> TCP connection to localhost
 *   - xxx+unix:///           -> UNIX domain socket
 *   - xxx:///                -> UNIX domain socket
 *   - xxx+ssh:///            -> SSH connection (legacy)
 *   - xxx+libssh2:///        -> SSH connection (using libssh2)
 *   - xxx+libssh:///         -> SSH connection (using libssh)
 */

So I think "xen+ssh" would not be matched directly by the "xen" driver; the "remote" driver would first launch "ssh" for setting up the transport. And that would make virConnectAuth* indirectly depend on the ssh config.

> Anyway nbdkit uses libssh, and should by default read
> the ssh config file, although it's a bit unclear if it reads
> $HOME/.ssh/config or only the global /etc/ssh/ssh_config.

That should become clear from the testing (if it doesn't work, I'll look at the libssh source too).

OK, so the troubles start as soon as I try to log in to dom0 via ssh
from Fedora 35:

> Unable to negotiate with 192.168.122.108 port 22: no matching key
> exchange method found. Their offer: diffie-hellman-group-exchange-sha1,
> diffie-hellman-group14-sha1, diffie-hellman-group1-sha1

A relevant RFC was published in January 2022:

  RFC 9142 -- Key Exchange (KEX) Method Updates and Recommendations for
  Secure Shell (SSH)

  https://www.rfc-editor.org/rfc/rfc9142.html

This RFC contains a summary table of recommendations:

  https://www.rfc-editor.org/rfc/rfc9142.html#name-summary-guidance-for-implem

We can build the intersection of the RHEL5 offer and this table; the result
is:

- diffie-hellman-group-exchange-sha1: SHOULD NOT
- diffie-hellman-group14-sha1:        MAY
- diffie-hellman-group1-sha1:         SHOULD NOT

The "least deprecated" KEX method is therefore
"diffie-hellman-group14-sha1"; that's what we should recommend too.

The ssh_config(5) manual explains the "+" format for the KexAlgorithms
keyword:

> [...] if the specified value begins with a ‘+’ character, then the
> specified methods will be appended to the default set instead of
> replacing them

Therefore our documentation should recommend, for $HOME/.ssh/config:

> Host xen.example.com
>   KexAlgorithms +diffie-hellman-group14-sha1

With this, no further tweaks were necessary, I could log in.


... However, I don't really find that a comprehensive answer. Therefore,
I reconfigured my F35 system with:

> # update-crypto-policies --set FUTURE

Which is the strongest pre-packaged policy, based on CRYPTO-POLICIES(7).

Indeed, this roots out further deprecated algorithms:

> Unable to negotiate with 192.168.122.108 port 22: no matching MAC found.
> Their offer: hmac-md5, hmac-sha1, hmac-ripemd160,
> hmac-ripemd160, hmac-sha1-96, hmac-md5-96

From these, we can immediately remove everything MD5, and the truncated
length SHA1 as well, leaving us with:

  hmac-sha1, hmac-ripemd160, hmac-ripemd160

SHA1 is considered broken, while (to my understanding) RIPEMD-160 is not.
(In other words, there is no public cryptanalysis of RIPEMD-160 that breaks
it to the extent of SHA1's public breakage.) For example, BitCoin uses
RIPEMD-160. Therefore I thought our suggestion should be "hmac-ripemd160",
*but* it turns out that the RIPEMD MAC algos are not supported by the F35
ssh *client*! The only option we're left with is "hmac-sha1":

> Host xen.example.com
>   MACs +hmac-sha1

I didn't have to relax any other crypto settings (for example, I didn't have
to enable "weak" cipher algos); with these two (KexAlgorithms and MACs)
relaxed, I managed to log in. In summary:

> Host xen.example.com
>   KexAlgorithms +diffie-hellman-group14-sha1
>   MACs          +hmac-sha1

I'll try a full conversion later, with these.

Rich reminded me of pubkey auth, and he pointed out (and I verified) that "ssh-rsa" is both necessary and sufficient for "PubkeyAcceptedAlgorithms". So thus far we have:

> Host xen.example.com
>   KexAlgorithms            +diffie-hellman-group14-sha1
>   MACs                     +hmac-sha1
>   PubkeyAcceptedAlgorithms +ssh-rsa

After quite some struggle, I managed to install a RHEL6 Xen PV guest inside the RHEL5 Xen (Server) domain.

Then, with the settings from comment 15 in effect:

$ ./run virt-v2v -ic 'xen+ssh://root.122.108' -ip $HOME/src/v2v/xen.passwd -o null rhel6PV
[   0.0] Setting up the source: -i libvirt -ic xen+ssh://root.122.108 rhel6PV
[   1.6] Opening the source
nbdkit: ssh[1]: error: failed to connect to remote host: 192.168.122.108: kex error : no match for method server host key algo: server [ssh-rsa,ssh-dss], client [rsa-sha2-256,rsa-sha2-512,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519]
virt-v2v: error: libguestfs error: guestfs_launch failed.


This confirms the other knob pointed out by Rich: "HostKeyAlgorithms".

Let's update the recommendation:

> Host xen.example.com
>   KexAlgorithms            +diffie-hellman-group14-sha1
>   MACs                     +hmac-sha1
>   PubkeyAcceptedAlgorithms +ssh-rsa
>   HostKeyAlgorithms        +ssh-rsa

With these in place, I get a successful (-o null) conversion:

$ ./run virt-v2v -ic 'xen+ssh://root.122.108' -ip $HOME/src/v2v/xen.passwd -o null rhel6PV
[   0.0] Setting up the source: -i libvirt -ic xen+ssh://root.122.108 rhel6PV
[   1.6] Opening the source
[   7.0] Inspecting the source
[  13.8] Checking for sufficient free disk space in the guest
[  13.8] Converting Red Hat Enterprise Linux Server release 6.10 (Santiago) to run on KVM
virt-v2v: The QEMU Guest Agent will be installed for this guest at first 
boot.
virt-v2v: This guest has virtio drivers installed.
[ 113.7] Mapping filesystem data to avoid copying unused and blank areas
[ 116.1] Closing the overlay
[ 116.2] Assigning disks to buses
[ 116.2] Checking if the guest needs BIOS or UEFI to boot
[ 116.2] Setting up the destination: -o null
[ 117.5] Copying disk 1/1
█ 100% [****************************************]
[ 205.8] Creating output metadata
[ 205.8] Finishing off

Hmmm, something is still wrong. Without setting the crypto policy to LEGACY (i.e., when using DEFAULT or FUTURE), *passwordless* login does no work (regardless of whether I run the SSH agent or not). For some reason, DEFAULT and FUTURE prevent nbdkit-ssh-plugin from using pubkey auth, even though ssh *itself* is perfectly happy.

This is amazing.

In a really bad way.

OpenSSH renamed the config option "PubkeyAcceptedKeyTypes" to "PubkeyAcceptedAlgorithms" at some point.

libssh picked up the new option name (and preserved the old one for compatibility) in libssh commit af9018ce13b4 ("Solve issue #113 "Remove unneeded configuration options not supported by OpenSSH anymore"", 2022-04-14). This commit has not been included in *any* upstream release yet, while Fedora provides the latest upstream release.

Therefore, on Fedora, ssh (the utility) recognizes PubkeyAcceptedAlgorithms *only*, while libssh (the library) recognizes PubkeyAcceptedKeyTypes *only*.

Well, almost. ssh (the utility) *continues* to recognize PubkeyAcceptedKeyTypes for compatibility, except its *manual* (ssh_config(5)) is silent about it. So you only *think* PubkeyAcceptedAlgorithms is not recognized anymore, if you read and TRUST the documentation.

Honestly this has been executed amazingly badly. Both libssh and ssh have been doing the right thing by introducing the new name for the option and preserving compatibility with the old name, BUT the documentation writers must have figured, "hey how can we best convince people to migrate to the new option name -- I know, let's pretend the old one does not exist!!!" And then people that work off of the documentation as their PRIMARY INFORMATION SOURCE are screwed because they are led to believe there is no functional intersection between new ssh and old libssh.

Well, there is, you just have to actively DISTRUST the official ssh documentation. Well done!

FML.

Unfortunately, my commit 46298c651471, for bug 1854275, was incorrect. Namely, from comment 9:

(2) -ic 'xen+ssh://root.com'

  setup                            [input/input_xen_ssh.ml]
    Libvirt.Connect.connect_auth
    Nbdkit_ssh.create_ssh          [input/nbdkit_ssh.ml]
      starts nbdkit with the ssh
        plugin

and "Libvirt.Connect.connect_auth" starts the "ssh" binary, which requires a password (in the absence of pubkey auth), and does not adhere to the "-ip" option.

Therefore I need to port commit 3172a509bb43 to virt-v2v-input-xen as well.

OK, I'm going to post two patches tomorrow (hopefully).

#1 -- port commit 3172a509bb43 to virt-v2v-input-xen; "Libvirt.Connect.connect_auth" will start "ssh" internally, and "ssh" ignores our "-ip" option. The only way to get around this is to use pubkey auth.

#2 -- recommend the following client settings specifically, for connecting to RHEL 5 Xen hosts (beyond the existent hint to enable PasswordAuthentication on the server):

Host xen.example.com
  KexAlgorithms            +diffie-hellman-group14-sha1
  MACs                     +hmac-sha1
  HostKeyAlgorithms        +ssh-rsa

  # old option name:
  PubkeyAcceptedKeyTypes   +ssh-rsa

  # new option name:
  PubkeyAcceptedAlgorithms +ssh-rsa

These are required and sufficient even when the FUTURE crypto-policy is set.

Justification for each line:

- KexAlgorithms, MACs: without these, not even interactive "ssh" will log in to the server.

- HostKeyAlgorithms: without this one, nbdkit-ssh-plugin won't even get as far as attempting authentication.

- PubkeyAcceptedKeyTypes: this is the old name of the option. Recognized by both libssh-0.9.6-1.fc35 and openssh-8.7p1-3.fc35, but not documented in the openssh-8.7p1-3.fc35 manual. Required by all three of: (a) interactive "ssh", (b) nbdkit-ssh-plugin, and (c) the "ssh" process launched by "Libvirt.Connect.connect_auth". If this option is absent, then (a) interactive "ssh" and (c) "Libvirt.Connect.connect_auth" will ask for a password; the "-ip" option is only honored by (b) nbdkit-ssh-plugin.

- PubkeyAcceptedAlgorithms: the new name of the option. Recognized only by openssh-8.7p1-3.fc35; ignored by libssh-0.9.6-1.fc35. Included in the recommendation for future-proofing only.

I didn't justify the *values* for the knobs. Here goes:

- KexAlgorithms: diffie-hellman-group14-sha1 is the least deprecated one, according to RFC 9142, from the three possible diffie-hellman-group-exchange-sha1, diffie-hellman-group14-sha1, diffie-hellman-group1-sha1.

- MACs: after removing everything MD5-related (MD5 is broken) and the truncated SHA1 based MAC(s), only RIPEMD ones and hmac-sha1 remains. RIPEMD is not supported (as in, it is compiled out) on the client side, on Fedora. Thus, only hmac-sha1 remains.

- HostKeyAlgorithms, PubkeyAcceptedKeyTypes/PubkeyAcceptedAlgorithms: the RHEL5 server supports only ssh-rsa and ssh-dss, and the former is more commonly used.

(e.g. ssh-keygen defaults to creating ssh-rsa keys)

[v2v PATCH 0/2] input-xen: SSH authentication documentation improvements
Message-Id: <20220711070157.5399-1-lersek>
https://listman.redhat.com/archives/libguestfs/2022-July/029430.html

(In reply to Laszlo Ersek from comment #23)
> [v2v PATCH 0/2] input-xen: SSH authentication documentation improvements
> Message-Id: <20220711070157.5399-1-lersek>
> https://listman.redhat.com/archives/libguestfs/2022-July/029430.html

Upstream commit range 4368b94ee172..af4a0454cdd2.

Test the bug with below builds:
virt-v2v-2.0.7-2.el9.x86_64
libguestfs-1.48.4-1.el9.x86_64
guestfs-tools-1.48.2-5.el9.x86_64
libvirt-libs-8.5.0-2.el9.x86_64
qemu-img-7.0.0-9.el9.x86_64
nbdkit-server-1.30.8-1.el9.x86_64
libnbd-1.12.5-1.el9.x86_64
openssh-8.7p1-19.el9.x86_64
libssh-0.9.6-3.el9.x86_64
crypto-policies-20220427-1.gitb2323a1.el9.noarch
crypto-policies-scripts-20220427-1.gitb2323a1.el9.noarch
libblockdev-crypto-2.25-12.el9.x86_64


Steps:
1. Check info about legacy crypto in virt-v2v-input-xen man page
# man virt-v2v-input-xen |grep 'legacy' -A 15
       With some modern ssh implementations, legacy crypto algorithms required to interoperate with RHEL 5
       sshd are disabled.  To enable them, you may need to add the following "Host" stanza to your
       ~/.ssh/config:

        Host xen.example.com
          KexAlgorithms            +diffie-hellman-group14-sha1
          MACs                     +hmac-sha1
          HostKeyAlgorithms        +ssh-rsa
          PubkeyAcceptedKeyTypes   +ssh-rsa
          PubkeyAcceptedAlgorithms +ssh-rsa

       ("PubkeyAcceptedKeyTypes" and "PubkeyAcceptedAlgorithms" have identical meaning; the former is the
       old option name, the latter is the new one. Virt-v2v uses both "libssh" and "ssh" when converting a
       guest from Xen, and on some operating systems, "libssh" and "ssh" may not both accept the same
       option variant.)

2. Make sure test server has default crypto policie
# update-crypto-policies --show
DEFAULT

3. Convert a guest from Xen to local by v2v without setting '~/.ssh/config'
# virt-v2v -ic xen+ssh://root.224.33 xen-hvm-rhel7.8-x86_64 -ip /home/esxpwd
[   0.0] Setting up the source: -i libvirt -ic xen+ssh://root.224.33 xen-hvm-rhel7.8-x86_64
virt-v2v: error: exception: libvirt: VIR_ERR_SYSTEM_ERROR: VIR_FROM_RPC: 
Cannot recv data: Unable to negotiate with 10.73.224.33 port 22: no 
matching key exchange method found. Their offer: 
diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1: 
Connection reset by peer

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

4. Convert a guest from Xen to local via -ip option by v2v with setting '~/.ssh/config'
4.1 # cat ~/.ssh/config
Host 10.73.224.33
  KexAlgorithms            +diffie-hellman-group14-sha1
  MACs                     +hmac-sha1
  HostKeyAlgorithms        +ssh-rsa
  PubkeyAcceptedKeyTypes   +ssh-rsa
  PubkeyAcceptedAlgorithms +ssh-rsa

4.2 # virt-v2v -ic xen+ssh://root.224.33 xen-hvm-rhel7.8-x86_64 -ip /home/esxpwd
[   0.0] Setting up the source: -i libvirt -ic xen+ssh://root.224.33 xen-hvm-rhel7.8-x86_64
virt-v2v: error: exception: libvirt: VIR_ERR_SYSTEM_ERROR: VIR_FROM_RPC: 
Cannot recv data: ssh_dispatch_run_fatal: Connection to 10.73.224.33 port 
22: error in libcrypto: Connection reset by peer

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

5.Switch to regular user, then convert a guest from Xen to local via -ip option by v2v with setting '~/.ssh/config'
5.1 $ cat ~/.ssh/config
Host 10.73.224.33
  KexAlgorithms            +diffie-hellman-group14-sha1
  MACs                     +hmac-sha1
  HostKeyAlgorithms        +ssh-rsa
  PubkeyAcceptedKeyTypes   +ssh-rsa
  PubkeyAcceptedAlgorithms +ssh-rsa


5.2 $ virt-v2v -ic xen+ssh://root.224.33 xen-hvm-rhel7.8-x86_64 -ip /home/esxpwd
[   0.0] Setting up the source: -i libvirt -ic xen+ssh://root.224.33 xen-hvm-rhel7.8-x86_64
virt-v2v: error: exception: libvirt: VIR_ERR_SYSTEM_ERROR: VIR_FROM_RPC: 
Cannot recv data: ssh_dispatch_run_fatal: Connection to 10.73.224.33 port 
22: error in libcrypto: Connection reset by peer

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]


Hi Laszlo,

   Please check the results of step4 and step5, is there anything missing in my steps?

What's the version of:
* openssl
* compat-openssl11 (if installed) ?

(In reply to Richard W.M. Jones from comment #26)
> What's the version of:
> * openssl
> * compat-openssl11 (if installed) ?

# rpm -qa |grep openssl
openssl-pkcs11-0.4.11-7.el9.x86_64
xmlsec1-openssl-1.2.29-9.el9.x86_64
openssl-libs-3.0.1-37.el9_0.x86_64
openssl-3.0.1-37.el9_0.x86_64

Thanks - is there any error message associated with the libcrypto failure.
I suppose it might be in the journal or dmesg:
# journalctl -S today

(In reply to Richard W.M. Jones from comment #28)
> Thanks - is there any error message associated with the libcrypto failure.
> I suppose it might be in the journal or dmesg:
> # journalctl -S today

#  journalctl -S today |grep libcrypto
nothing

# dmesg |grep libcrypto
nothing

BTW, have same error when ssh xen host

# ssh root.224.33
ssh_dispatch_run_fatal: Connection to 10.73.224.33 port 22: error in libcrypto

I think this is a bug in RHEL9 crypto. I've messaged the rhel-crypto list, and CC'd Rich and Ming Xie.

https://listman.redhat.com/archives/libvir-list/2022-July/233149.html

[v2v PATCH] input-xen: cover RHEL9 OpenSSL crypto settings
Message-Id: <20220729105703.10150-1-lersek>
https://listman.redhat.com/archives/libguestfs/2022-July/029568.html

(In reply to Laszlo Ersek from comment #51)
> [v2v PATCH] input-xen: cover RHEL9 OpenSSL crypto settings
> Message-Id: <20220729105703.10150-1-lersek>
> https://listman.redhat.com/archives/libguestfs/2022-July/029568.html

Commit ddab06d5eb99 on the master branch.

Cherry-picked to the rhel-9.1 branch as commit 9e1c78a4dda8, with commit hash references updated / annotated in the commit message.

Test the bug with below builds:
virt-v2v-2.0.7-4.el9.x86_64
libguestfs-1.48.4-1.el9.x86_64
libvirt-libs-8.5.0-3.el9.x86_64
qemu-img-7.0.0-9.el9.x86_64


Steps:
1. Check info about legacy crypto in virt-v2v-input-xen man page
# man virt-v2v-input-xen |grep 'legacy' -A 31
       With some modern ssh implementations, legacy crypto algorithms required to interoperate with RHEL 5
       sshd are disabled.  To enable them, you may need to add the following "Host" stanza to your
       ~/.ssh/config:

        Host xen.example.com
          KexAlgorithms            +diffie-hellman-group14-sha1
          MACs                     +hmac-sha1
          HostKeyAlgorithms        +ssh-rsa
          PubkeyAcceptedKeyTypes   +ssh-rsa
          PubkeyAcceptedAlgorithms +ssh-rsa

       ("PubkeyAcceptedKeyTypes" and "PubkeyAcceptedAlgorithms" have identical meaning; the former is the old
       option name, the latter is the new one. Virt-v2v uses both "libssh" and "ssh" when converting a guest
       from Xen, and on some operating systems, "libssh" and "ssh" may not both accept the same option
       variant.)

       When connecting to RHEL 5 sshd from RHEL 9, the SHA1 algorithm's use in signatures has to be re-enabled
       at the OpenSSL level, in addition to the above SSH configuration.  Create a file called
       $HOME/openssl-sha1.cnf with the following contents:

        .include /etc/ssl/openssl.cnf
        [openssl_init]
        alg_section = evp_properties
        [evp_properties]
        rh-allow-sha1-signatures = yes

       and export the following variable into the environment of the "virt-v2v" process:

        OPENSSL_CONF=$HOME/openssl-sha1.cnf

       Note that the "OPENSSL_CONF" environment variable will only take effect if the libvirt client library
       used by virt-v2v is at least version 8.6.0.


2. Make sure test server has default crypto policie
# update-crypto-policies --show
DEFAULT

3.Set ~/.ssh/config and $HOME/openssl-sha1.cnf according to virt-v2v-input-xen man page, then export env variable 
3.1 # cat ~/.ssh/config
Host 10.73.224.33
  KexAlgorithms            +diffie-hellman-group14-sha1
  MACs                     +hmac-sha1
  HostKeyAlgorithms        +ssh-rsa
  PubkeyAcceptedKeyTypes   +ssh-rsa
  PubkeyAcceptedAlgorithms +ssh-rsa

3.2 # cat $HOME/openssl-sha1.cnf
.include /etc/ssl/openssl.cnf
[openssl_init]
alg_section = evp_properties
[evp_properties]
rh-allow-sha1-signatures = yes

3.3 # export  OPENSSL_CONF=$HOME/openssl-sha1.cnf

4. Convert a guest from Xen by v2v
# virt-v2v -ic xen+ssh://root.224.33 xen-hvm-rhel7.8-x86_64 -ip /home/esxpwd
[   0.2] Setting up the source: -i libvirt -ic xen+ssh://root.224.33 xen-hvm-rhel7.8-x86_64
virt-v2v: error: exception: libvirt: VIR_ERR_SYSTEM_ERROR: VIR_FROM_RPC: 
Cannot recv data: ssh_dispatch_run_fatal: Connection to 10.73.224.33 port 
22: error in libcrypto: Connection reset by peer

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]


Result:

   Can't get expected result because the version of latest rhel9 libvirt is still 8.5, will test the bug when bug2112348 is fixed

Test the bug again because bug2112348 has been fixed

Package versions:
virt-v2v-2.0.7-4.el9.x86_64
libguestfs-1.48.4-1.el9.x86_64
guestfs-tools-1.48.2-5.el9.x86_64
libvirt-libs-8.5.0-5.el9.x86_64
qemu-img-7.0.0-9.el9.x86_64
nbdkit-server-1.30.8-1.el9.x86_64
libnbd-1.12.6-1.el9.x86_64

Steps:
1. Make sure test server has default crypto policie
# update-crypto-policies --show
DEFAULT

2.Set ~/.ssh/config and $HOME/openssl-sha1.cnf according to virt-v2v-input-xen man page, then export env variable 
2.1 # cat ~/.ssh/config
Host 10.73.224.33
  KexAlgorithms            +diffie-hellman-group14-sha1
  MACs                     +hmac-sha1
  HostKeyAlgorithms        +ssh-rsa
  PubkeyAcceptedKeyTypes   +ssh-rsa
  PubkeyAcceptedAlgorithms +ssh-rsa

2.2 # cat $HOME/openssl-sha1.cnf
.include /etc/ssl/openssl.cnf
[openssl_init]
alg_section = evp_properties
[evp_properties]
rh-allow-sha1-signatures = yes

2.3 # export  OPENSSL_CONF=$HOME/openssl-sha1.cnf

3. Convert a guest from Xen by v2v with root user
# virt-v2v -ic xen+ssh://root.224.33 xen-hvm-rhel7.8-x86_64 -ip /home/esxpwd
[   1.4] Setting up the source: -i libvirt -ic xen+ssh://root.224.33 xen-hvm-rhel7.8-x86_64
root.224.33's password: 
[   6.9] Opening the source
[  41.5] Inspecting the source
[  53.3] Checking for sufficient free disk space in the guest
[  53.3] Converting Red Hat Enterprise Linux Server 7.8 Beta (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 208.6] Mapping filesystem data to avoid copying unused and blank areas
[ 209.7] Closing the overlay
[ 210.0] Assigning disks to buses
[ 210.0] Checking if the guest needs BIOS or UEFI to boot
[ 210.0] Setting up the destination: -o libvirt
[ 212.3] Copying disk 1/1
█ 100% [****************************************]
[ 335.6] Creating output metadata
[ 335.7] Finishing off

4.Switch account to regular user and convert a guest from Xen by v2v again
4.1

$ cat $HOME/.ssh/config
Host 10.73.224.33
  KexAlgorithms            +diffie-hellman-group14-sha1
  MACs                     +hmac-sha1
  HostKeyAlgorithms        +ssh-rsa
  PubkeyAcceptedKeyTypes   +ssh-rsa
  PubkeyAcceptedAlgorithms +ssh-rsa

$ cat $HOME/openssl-sha1.cnf 
.include /etc/ssl/openssl.cnf
[openssl_init]
alg_section = evp_properties
[evp_properties]
rh-allow-sha1-signatures = yes

$ export  OPENSSL_CONF=$HOME/openssl-sha1.cnf

4.2 $ virt-v2v -ic xen+ssh://root.224.33 xen-hvm-rhel7.8-x86_64 -ip /home/esxpwd -o null
[   0.0] Setting up the source: -i libvirt -ic xen+ssh://root.224.33 xen-hvm-rhel7.8-x86_64
root.224.33's password: 
[   4.2] Opening the source
[   9.1] Inspecting the source
[  20.6] Checking for sufficient free disk space in the guest
[  20.6] Converting Red Hat Enterprise Linux Server 7.8 Beta (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 158.6] Mapping filesystem data to avoid copying unused and blank areas
[ 159.6] Closing the overlay
[ 159.9] Assigning disks to buses
[ 159.9] Checking if the guest needs BIOS or UEFI to boot
[ 159.9] Setting up the destination: -o null
[ 161.2] Copying disk 1/1
█ 100% [****************************************]
[ 255.4] Creating output metadata
[ 255.4] Finishing off


Hi Laszlo,

   Just one question, virt-v2v-input-xen man page shows the version of libvirt for the bug is at least version 8.6.0, but bug2112348 is fixed in libvirt-8.5.0-5, does the man page need to be modified?

Hi mxie,

good catch. I did consider this question for the docs update patch (see comments 48 through 51). I specifically considered the case that the (very simple) libvirtd patch could be backported to RHEL-9 without a rebase, and so our base version would remain 8.5.0. That's one of the reasons why I asked Michal on virt-devel about potentially rebasing libvirt to v8.6.0 in RHEL-9.1.

We could consider two changes here:

- Replace "will" with "may", in the sentence.

> Note that the OPENSSL_CONF environment variable [will] only take effect if the libvirt client library used by virt-v2v is at least version 8.6.0.

Again I considered this originally, and rejected it, because it leaves the user with no usable information. What does it depend on? And then we need to get into explaining backports, etc. I didn't (and don't) think that was appropriate for the virt-v2v manual.

- Add a downstream-only patch where we remove the entire sentence (given that in RHEL-9.1, the components should be in sync), *and* file a reminder bug for RHEL-9.2, to drop the downstream only patch.

This is doable, but IMO way too much work. If the current wording ever causes confusion, we can always put up a small kbase article explaining this.

This kind of problem tends to solve itself over time; next time libvirtd is rebased in RHEL-9 (which is inevitably coming), the sentence will kind of "fix itself".

So I'd suggest leaving the language as-is.

Thanks Laszlo, move the bug from ON_QA to VERIFIED status according to comment57 and comment58

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: virt-v2v security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7968