Bug 2175703

According to the conversion log in comment#0:

==========
kernels offered by the bootloader in this guest (first in list is default):
* kernel-core 5.14.0-182.el9.x86_64 (x86_64)
	/boot/vmlinuz-5.14.0-182.el9.x86_64
	/boot/initramfs-5.14.0-182.el9.x86_64.img
	/boot/config-5.14.0-182.el9.x86_64
	/lib/modules/5.14.0-182.el9.x86_64
	2154 modules found
	virtio: blk=true net=true rng=true balloon=true
	pvpanic=true vsock=true xen=false debug=false
* kernel-core 5.14.0-185.el9.x86_64 (x86_64)
	/boot/vmlinuz-5.14.0-185.el9.x86_64
	/boot/initramfs-5.14.0-185.el9.x86_64.img
	/boot/config-5.14.0-185.el9.x86_64
	/lib/modules/5.14.0-185.el9.x86_64
	2155 modules found
	virtio: blk=true net=true rng=true balloon=true
	pvpanic=true vsock=true xen=false debug=false
==========

However, the kmod-kvdo package you have installed in the guest (kmod-kvdo-8.2.1.6-74.el9.x86_64) provides the following files:

==========
/etc/depmod.d/kvdo.conf
/lib/modules/5.14.0-283.el9.x86_64
/lib/modules/5.14.0-283.el9.x86_64/extra
/lib/modules/5.14.0-283.el9.x86_64/extra/kmod-kvdo
/lib/modules/5.14.0-283.el9.x86_64/extra/kmod-kvdo/vdo
/lib/modules/5.14.0-283.el9.x86_64/extra/kmod-kvdo/vdo/kvdo.ko
/usr/lib/.build-id
/usr/lib/.build-id/f9
/usr/lib/.build-id/f9/dcf9e96207d48704190591d89ac79457550144
/usr/share/doc/kmod-kvdo/greylist.txt
==========

This means that you have a misconfigured guest. The VDO kernel module does not match your installed kernel(s).

If you check the older kmod-kvdo package (kmod-kvdo-8.2.0.18-55.el9_2.x86_64):

==========
...
/lib/modules/5.14.0-185.el9.x86_64/extra/kmod-kvdo/vdo/kvdo.ko
...
==========

That one does match your installed -185 kernel.

So, I think virt-v2v is right to fail the conversion; the guest is misconfigured on the source side.

I think this can happen at any time when the kmod-kvdo package gets out of sync with the installed kernel package(s). If you check the build history for kmod-kvdo in Brew, you'll see that its %changelog almost entirely consists of "Rebuilt for latest kernel" (bug 2119820). This symptom can therefore result from the kmod-kvdo package having a bug, *or* from your distribution channel -- rhsm or otherwise -- offering such kernel and kmod-kvdo packages in the BaseOS repository that are *mismatched*.

I think you've simply hit bug 2119820 / bug 2172911, and that there's nothing for us to do here, for virt-v2v.

It fails when we try to run the command:

/usr/bin/dracut --verbose --add-drivers "virtio_blk virtio_scsi virtio_net bochs" /boot/initramfs-5.14.0-185.el9.x86_64.img 5.14.0-185.el9.x86_64

We do this to update the bootloader with the virtio drivers.

Regretfully it does appear that this error is not possible to recover
from.  If we cannot update the bootloader then we cannot be assured
that the guest will boot on KVM with virtio devices, so the only
good thing here is to fail.

So yes I agree with Laszlo.

I wonder why VDO is supplied as this out of tree module?  It seems
like it is free software, assuming it is this:
https://github.com/dm-vdo/kvdo
Perhaps it'd be better to work to get this included upstream.

s/bootloader/initramfs/ !

I *think* VDO is provided as an out-of-tree module because it is out of tree even upstream. I don't know what the reason for that is. IIRC, VDO was originally introduced in RHEL7, and some documentation I read at the time (maybe the RHEL7 documentation, or VDO's own description at their github page) stated VDO "was being upstreamed" or something like that. I don't know if that effort has failed permanently, and/or abandoned. I agree it's super annoying, in particular the version numbering in the RPM "nevra"s is totally opaque. It's impossible to know from the kmod-kvdo RPM's version-release what kernel RPM it matches!

Right, their github page still states, "This software and technology has been acquired by Red Hat, has been relicensed under the GPL (v2 or later), and this repository begins the process of preparing for integration with the upstream kernel".

Please retest with package(s) fixing bug 2119820 installed in the guest before conversion. Thanks.

Tried with the following rpms in the guest.

kmod-kvdo-8.2.1.6-75.el9.x86_64
kernel-modules-core-5.14.0-285.el9.x86_64
kernel-core-5.14.0-285.el9.x86_64
kernel-modules-5.14.0-285.el9.x86_64

Convert the guest to local libvirt via virt-v2v-2.2.0-5.el9.x86_64.

# virt-v2v -i libvirt -ic vpx://root.212.149/data/10.73.212.36/?no_verify=1 -o libvirt -of raw --mac 00:50:56:83:62:d2:bridge:virbr0  -ip /v2v-ops/esxpw esx8.0-rhel9.2-x86_64-bz2175703
[   0.0] Setting up the source: -i libvirt -ic vpx://root.212.149/data/10.73.212.36/?no_verify=1 esx8.0-rhel9.2-x86_64-bz2175703
[   2.5] Opening the source
[  39.3] Inspecting the source
[ 958.2] Checking for sufficient free disk space in the guest
[ 958.2] Converting Red Hat Enterprise Linux 9.2 Beta (Plow) to run on KVM
virt-v2v: error: no installed kernel packages were found.

This probably indicates that virt-v2v was unable to inspect this guest 
properly.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]


Please check the attachment on debug log.

Oh what a friggin mess this is.

When inspecting a Linux guest, the "detect_kernels" funciton in
"libguestfs-common/mldrivers/linux_kernels.ml" works like this (I'm
omitting some details):

- collects packages that are named "kernel-*"

- for each such package, lists the files in the package

- if the file list does not contain a file called "/boot/vmlinuz-*", or
  that file does not actually exist in the filesystem, then the package
  in question is thrown away (it is not considered to provide a kernel),
  and we advance to the next package

- we check the file list for the FIRST pathname that matches
  "/lib/modules/*". If this pathname exists, then (a) we ASSUME the
  pathname to be of the form "/lib/modules/5.14.0-182.el9.x86_64", and
  (b) we EXPECT the pathname to reference a DIRECTORY in the filesystem.
  If our expectation (b) fails, then the package in question is thrown
  away (it is not considered to provide a kernel).

Up to and including RHEL-9.1:

- the "kernel-core" package would:

  - both contain the kernel file "/boot/vmlinuz-*"

  - and a directory named like "/lib/modules/5.14.0-182.el9.x86_64" as
    the FIRST pathname in the file list matching the pattern
    "/lib/modules/*"

- therefore the logic outlined above would identify the "kernel-core"
  package as a "kernel package".

Starting with RHEL-9.2:

- a new kernel subpackage has been introduced apparently, called
  "kernel-modules-core".


- the "kernel-core" package still contains the kernel file
  "/boot/vmlinuz-*"

- the "kernel-core" package no longer contains a directory like
  "/lib/modules/5.14.0-182.el9.x86_64"

- therefore the "kernel-core" package is ruled out as a "kernel package"


- the new subpackage "kernel-modules-core" now contains the directory
  "/lib/modules/5.14.0-283.el9.x86_64"

- however the "/boot/vmlinuz-*" file still belongs to the "kernel-core"
  package!

- therefore the "kernel-modules-core" package is ruled out as a "kernel
  package" *as well*.

In short: Linux guest inspection currently REQUIRES that any one kernel
package contain *BOTH* the "/boot/vmlinuz-*" file *AND* the
"/lib/modules/<version>" DIRECTORY. This was broken in RHEL-9.2, because
the "/lib/modules/<version>" directory was moved to the
"kernel-modules-core" package, BUT the "/boot/vmlinuz-*" file continues
to belong to the "kernel-core" package. In other words, there is NO
package now in RHEL-9.2 that satisfies *both* conditions of the
inspection.

Here's the kernel commit that broke the inspection logic:

commit 95440c0fbdb59078c2ed779494789d33a1ab6106
Author: Gerd Hoffmann <kraxel>
Date:   Wed Dec 7 06:42:48 2022 +0100

    redhat: split sub-rpm kernel-modules-core from kernel-core
    
    Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2142102
    Upstream Status: RHEL only
    
    All kernel modules plus support files (such as the files generated
    by depmod) are moved to the new kernel-modules-core sub-rpm.
    
    The kernel binary plus support files stay in the kernel-core sub-rpm.
    This essentially includes the files which are copied over to /boot by
    the kernel-install utility (vmlinuz, System.map, ...).
    
    With this in place we have a strict separation between sub-rpms carrying
    a kernel image and sub-rpms carrying kernel modules.  This should make it
    easier to use alternative kernel image packages, for example an unified
    kernel.
    
    Signed-off-by: Gerd Hoffmann <kraxel>
    Signed-off-by: Vitaly Kuznetsov <vkuznets>

This first appeared in kernel-5.14.0-269.el9.

In retrospect, the original kmod-kvdo version mismatch may be a red herring. It's entirely possible that the proper kmod-kvdo version was installed in the source guest, it's just that due to the subpackage split, virt-v2v never considered the *right* kernel *at all*, and so all the *other* kernels were found to mismatch kmod-kvdo.

The issue doesn't only affect virt-v2v, it also affects the new virt-drivers tool in guestfs-tools (virt-drivers also calls "detect_kernels").

virt-drivers was introduced in guestfs-tools commit 11fee3f83fb0 ("New 'virt-drivers' tool", 2023-01-19), part of v1.49.9. We don't have that version downstream yet (the latest build is guestfs-tools-1.48.2-8.el9, as of this writing); however, we'll get it when we rebase guestfs-tools to 1.50 (under bug 2168626).

So this issue blocks bug 2168626.

This comes at the absolute worst possible time. The fix I have in mind for upstream libguestfs-common is not easily backportable to RHEL-9.2.

At the time of this writing, the latest downstream build of virt-v2v is "virt-v2v-2.2.0-5.el9". This corresponds to dist-git commit f1abc5da694c, which is based on the rhel-9.2 branch of the upstream repository being at commit 86517b17be98 ("RHEL 9: tests: Remove btrfs test", 2023-02-06).

In turn, in the upstream repository, the rhel-9.2 branch (currently at 86517b17be98, see above) consumes the libguestfs-common submodule at commit 2c8e81f10e45 ("options: Also free ks->keys", 2022-10-11). In other words, the latest downstream build embeds (flattens) the submodule as of upstream commit 2c8e81f10e45.

The difficulty is that the fix affects the "detect_kernels" function. As of libguestfs-common commit 2c8e81f10e45 -- that is, as of "virt-v2v-2.2.0-5.el9" --, this function lives in the virt-v2v source code proper, and *not* in libguestfs-common. However, upstream has advanced meanwhile, and since libguestfs-common commit d34502ab586c ("mldrivers: New directory containing driver detection code", 2023-01-19), and since virt-v2v commit 4b9c8e1560d6 ("convert: Move Linux driver detection code to common/", 2023-01-19), the code has been moved from virt-v2v to libguestfs-common.

Which is to say that we need to fix it once for upstream, in the libguestfs-common submodule, and another time, for RHEL-9.2, in virt-v2v, on the rhel-9.2 branch. Worst of all, it's not even a "usual backport", because I can't even test my upstream fix. I need to do the upstream fix, then port it to rhel-9.2, build a downstream package for QE to test, and *then* upstream the upstream variant of the patch.

[libguestfs-common PATCH 0/2] detect_kernels: deal with RHEL's kernel-core / kernel-modules-core split
Message-Id: <20230320115301.43051-1-lersek>
https://listman.redhat.com/archives/libguestfs/2023-March/031111.html

(In reply to Laszlo Ersek from comment #19)
> [libguestfs-common PATCH 0/2] detect_kernels: deal with RHEL's kernel-core / kernel-modules-core split
> Message-Id: <20230320115301.43051-1-lersek>
> https://listman.redhat.com/archives/libguestfs/2023-March/031111.html

merged upstream as commit range 402f7600d707..70c10a079a30 (on the master branch)

RHEL-9.2.z port posted:

[rhel-9.2 v2v PATCH 0/2] detect_kernels: deal with RHEL's kernel-core / kernel-modules-core split
Message-Id: <20230405150829.171720-1-lersek>
https://listman.redhat.com/archives/libguestfs/2023-April/031214.html

(In reply to Laszlo Ersek from comment #40)
> RHEL-9.2.z port posted:
> 
> [rhel-9.2 v2v PATCH 0/2] detect_kernels: deal with RHEL's kernel-core / kernel-modules-core split
> Message-Id: <20230405150829.171720-1-lersek>
> https://listman.redhat.com/archives/libguestfs/2023-April/031214.html

Merged as commits e7aa456e26e5 and 9449d21eeae8 on the rhel-9.2 branch.

I've rebased (non-fast-forwardably) the same branch, in order to keep the "RHEL*" patches clustered near HEAD, as follows:

 1:  e8c61da73b62 =  1:  e8c61da73b62 Translated using Weblate (Ukrainian)
 2:  fd9694a3a2ea =  2:  fd9694a3a2ea convert: windows: Remove extraneous blank lines in source
 3:  9a5c900fdb53 =  3:  9a5c900fdb53 convert: windows: Document what copy_qemu_ga function returns
 4:  48f15935cff5 =  4:  48f15935cff5 convert: windows: Remove unused 'open Utils'
 5:  4623b2aab891 =  5:  4623b2aab891 -o kubevirt: Fix mistake in error message
 6:  6b7ef3efe748 =  6:  6b7ef3efe748 -o kubevirt: Move "cpu" element under "domain"
 7:  8009825c3963 =  7:  8009825c3963 -o kubevirt: Error on invalid output guest names
 8:  bcd60820de12 =  8:  bcd60820de12 Split long lines in messages
 9:  64a86bb9ef02 =  9:  64a86bb9ef02 -o kubevirt: Implement -oo compressed for qcow2 files
10:  8802e8b4135c = 10:  8802e8b4135c v2v: Remove use of ~anchored
11:  628ee708464e = 11:  628ee708464e -o kubevirt: Replace PCRE ~anchored with ^...$
12:  a9630d3981cb = 12:  a9630d3981cb -o libvirt: Add correct xmlns:libosinfo for Rocky Linux
13:  227313ca73d2 = 13:  227313ca73d2 convert: linux: Require host cpu for all RHEL-alike >= 9
 -:  ------------ > 14:  e7aa456e26e5 detect_kernels: tighten "try" scope
 -:  ------------ > 15:  9449d21eeae8 detect_kernels: deal with RHEL's kernel-core / kernel-modules-core split
14:  6633c5ca0fe4 = 16:  da3ae25bdd18 RHEL: v2v: Select correct qemu binary for -o qemu mode (RHBZ#1147313).
15:  cf636bf489fd = 17:  b638c2083ee9 RHEL: v2v: Disable the --qemu-boot / -oo qemu-boot option (RHBZ#1147313).
16:  a4ed97d92b38 = 18:  d8a6a0577f0a RHEL: Fix list of supported sound cards to match RHEL qemu (RHBZ#1176493).
17:  ef9a020874b8 = 19:  94f4aaeea12a RHEL: Fixes for libguestfs-winsupport.
18:  5703bf73a706 = 20:  ffcd5358b8f6 RHEL: v2v: -i disk: force VNC as display (RHBZ#1372671)
19:  4cc16f9bdf58 = 21:  907b8c0f9f0a RHEL: v2v: do not mention SUSE Xen hosts (RHBZ#1430203)
20:  f00e06aecd8e = 22:  ee2bf7286e96 RHEL: point to KB for supported v2v hypervisors/guests
21:  5ef909eb4d65 = 23:  2d85522fe7b0 RHEL: Disable -o glance
22:  ade0fd71c59b = 24:  8fbfb57de81e RHEL: Remove the --in-place option
23:  d6fc11c3f99f = 25:  b04577d80b89 RHEL 9: -oo compressed: Remove nbdcopy version check and test
24:  86517b17be98 = 26:  462f9b9eb810 RHEL 9: tests: Remove btrfs test

The test suite passes at 462f9b9eb810.

Verified with the versions:
virt-v2v-2.3.4-1.el9.x86_64
libnbd-1.16.0-1.el9.x86_64
qemu-kvm-8.0.0-1.el9.x86_64
libvirt-9.2.0-1.el9.x86_64
nbdkit-1.34.1-1.el9.x86_64

The versions inside the guest:
kmod-28-8.el9.x86_64
kmod-libs-28-8.el9.x86_64
kmod-kvdo-8.2.1.6-82.el9.x86_64

Steps:
1. Prepare RHEL9.2 guest with the latest kernel rpms on ESXi
2. Convert the guest to local libvirt/rhv via virt-v2v

#virt-v2v -ic vpx://root.212.149/data/10.73.212.36/?no_verify=1 -o libvirt -of raw -os v2v_dir --mac 00:50:56:af:a9:ac:network:v2v_net_0 esx8.0-rhel9.2-x86_64 -it vddk -io vddk-libdir=/root/vddk_libdir/latest -io vddk-thumbprint=D1:03:96:7E:11:3D:7C:4C:B6:50:28:1B:63:74:B5:40:5F:9D:9F:94 -ip /tmp/v2v_vpx_passwd

#virt-v2v -ic vpx://root.212.149/data/10.73.212.36/?no_verify=1 -o rhv-upload -of raw -os nfs_data -oc https://dell-per740-48.lab.eng.pek2.redhat.com/ovirt-engine/api -op /tmp/rhv_upload_passwd_file -oo rhv-cafile=/tmp/rhv_upload_ca.pem -oo rhv-cluster=NFS -oa preallocated -oo rhv-verifypeer=true --mac 00:50:56:af:a9:ac:bridge:ovirtmgmt esx8.0-rhel9.2-x86_64 -it vddk -io vddk-libdir=/home/v2v_auto/vddk_libdir/latest -io vddk-thumbprint=D1:03:96:7E:11:3D:7C:4C:B6:50:28:1B:63:74:B5:40:5F:9D:9F:94 -on esx8.0-rhel9.2-x86_64-bz2175703j4a -ip /tmp/v2v_vpx_passwd

Both conversions are successful.

3. Start the guests after conversion and do the checkpoints.

Marking as Verified:Tested.

Verified with the versions:
virt-v2v-2.3.4-2.el9.x86_64
kmod-28-8.el9.x86_64
kmod-libs-28-8.el9.x86_64
libnbd-1.16.0-1.el9.x86_64
nbdkit-1.34.1-1.el9.x86_64
qemu-kvm-8.0.0-3.el9.x86_64
kmod-kvdo-8.2.1.6-84.el9.x86_64
libvirt-9.3.0-2.el9.x86_64

The versions inside the guest:
kmod-28-8.el9.x86_64
kmod-libs-28-8.el9.x86_64
kmod-kvdo-8.2.1.6-82.el9.x86_64

Steps:
1. Prepare RHEL9.2 guest with the latest kernel rpms on ESXi
2. Convert the guest to local libvirt/rhv via virt-v2v
3. Start the guests after conversion and do the checkpoints.

Closed the bug with CURRENTRELEASE.