Can reproduce the bug when convert the VMware guest which has rhel7 and rhel8 installed in one disk and select rhel8 OS during v2v conversion

# virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.2 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78 -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -ip /home/passwd -op /home/rhvpasswd  -os nfs_data -b ovirtmgmt  esx7.0-rhel7.9-rhel8.4-multiple_linux_os
[   1.0] Opening the source -i libvirt -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 esx7.0-rhel7.9-rhel8.4-multiple_linux_os -it vddk  -io vddk-libdir=/home/vddk7.0.2 -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78
[   2.8] Creating an overlay to protect the source from being modified
[   3.7] Opening the overlay
[  10.6] Inspecting the overlay

***
Dual- or multi-boot operating system detected.  Choose the root filesystem
that contains the main operating system from the list below:

 [1] /dev/rhel_bootp-73-199-22/root (Red Hat Enterprise Linux Server 7.9 (Maipo))
 [2] /dev/rhel_bootp-73-199-6/root (Red Hat Enterprise Linux 8.4 (Ootpa))

Enter a number between 1 and 2, or ‘exit’: 2
[ 788.3] Checking for sufficient free disk space in the guest
[ 788.3] Converting Red Hat Enterprise Linux 8.4 (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 872.4] Mapping filesystem data to avoid copying unused and blank areas
virt-v2v: warning: fstrim on guest filesystem 
/dev/mapper/osprober-linux-sda1 failed.  Usually you can ignore this 
message.  To find out more read "Trimming" in virt-v2v(1).

Original message: fstrim: fstrim: /sysroot/: FITRIM ioctl failed: Operation 
not permitted
virt-v2v: warning: fstrim on guest filesystem 
/dev/mapper/osprober-linux-rhel_bootp--73--199--22-root failed.  Usually 
you can ignore this message.  To find out more read "Trimming" in 
virt-v2v(1).

Original message: fstrim: fstrim: /sysroot/: FITRIM ioctl failed: Operation 
not permitted
[ 876.3] Closing the overlay
[ 876.6] Assigning disks to buses
[ 876.6] Checking if the guest needs BIOS or UEFI to boot
[ 876.6] Initializing the target -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data
[ 896.3] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/tmp/v2vnbdkit.lAVk2C/nbdkit5.sock", "file.export": "/" } (qcow2)
    (100.00/100%)
[1268.3] Creating output metadata
[1331.0] Finishing off

Why is the warning a problem?

The warning correctly informs the user that we cannot minimize the number of blocks that need to be copied. The effort to minimize the number of blocks that need to be copied, by trimming, applies equally to the system (root) disk and to any data disks. The conversion does not fail. The warning provides a pointer to <https://libguestfs.org/virt-v2v.1.html#trimming>, which correctly explains the idea in guest filesystem terms -- independently of "OS vs. non-OS disks".

I do not agree that we should to hide this warning *specifically* for non-system disks. Either the warning is entirely worthless, or it is equally useful for all disks. For now I'm closing this as NOTABUG.

(In reply to Laszlo Ersek from comment #3)
> Why is the warning a problem?
> 
> The warning correctly informs the user that we cannot minimize the number of
> blocks that need to be copied. The effort to minimize the number of blocks
> that need to be copied, by trimming, applies equally to the system (root)
> disk and to any data disks. The conversion does not fail. The warning
> provides a pointer to <https://libguestfs.org/virt-v2v.1.html#trimming>,
> which correctly explains the idea in guest filesystem terms -- independently
> of "OS vs. non-OS disks".
> 
> I do not agree that we should to hide this warning *specifically* for
> non-system disks. Either the warning is entirely worthless, or it is equally
> useful for all disks. For now I'm closing this as NOTABUG.

Hi Leszlo,

   Please check comment2, this problem can be reproduced when converting the VMware guest which has rhel7 and rhel8 installed in one disk and select rhel8 OS during v2v conversion, I reproduced this scenario with latest versions as below, you can see that there is no warning if select rhel7 OS, so I think the warning ' FITRIM ioctl failed: Operation not permitted' should be a problem

Package versions:
virt-v2v-2.0.4-1.el9.x86_64
libguestfs-1.48.1-1.el9.x86_64
guestfs-tools-1.48.0-1.el9.x86_64
nbdkit-server-1.30.4-1.el9.x86_64
libnbd-1.12.2-1.el9.x86_64


Steps:
1.Converting the VMware guest which has rhel7 and rhel8 installed in one disk and select rhel8 OS during v2v conversion
 #virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.3 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78   -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt  ^C
[root@dell-per740-53 ~]# virt-v2v -ic vpx://root.227.27/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.3 -io  vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api  -op /home/rhvpasswd  -os nfs_data -b ovirtmgmt -ip /home/passwd esx7.0-rhel7.9-rhel8.4-multiple_linux_os
[   0.0] Setting up the source: -i libvirt -ic vpx://root.227.27/data/10.73.199.217/?no_verify=1 -it vddk esx7.0-rhel7.9-rhel8.4-multiple_linux_os
[   1.9] Opening the source
[   9.0] Inspecting the source

***
Dual- or multi-boot operating system detected.  Choose the root filesystem
that contains the main operating system from the list below:

 [1] /dev/rhel_bootp-73-199-22/root (Red Hat Enterprise Linux Server 7.9 (Maipo))
 [2] /dev/rhel_bootp-73-199-6/root (Red Hat Enterprise Linux 8.4 (Ootpa))

Enter a number between 1 and 2, or ‘exit’: 2
[ 136.3] Checking for sufficient free disk space in the guest
[ 136.3] Converting Red Hat Enterprise Linux 8.4 (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 210.3] Mapping filesystem data to avoid copying unused and blank areas
virt-v2v: warning: fstrim on guest filesystem 
/dev/mapper/osprober-linux-sda1 failed.  Usually you can ignore this 
message.  To find out more read "Trimming" in virt-v2v(1).

Original message: fstrim: fstrim: /sysroot/: FITRIM ioctl failed: Operation 
not permitted
virt-v2v: warning: fstrim on guest filesystem 
/dev/mapper/osprober-linux-rhel_bootp--73--199--22-root failed.  Usually 
you can ignore this message.  To find out more read "Trimming" in 
virt-v2v(1).

Original message: fstrim: fstrim: /sysroot/: FITRIM ioctl failed: Operation 
not permitted
[ 212.8] Closing the overlay
[ 213.0] Assigning disks to buses
[ 213.0] Checking if the guest needs BIOS or UEFI to boot
[ 213.0] Setting up the destination: -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -os nfs_data
[ 233.5] Copying disk 1/1
....


2.Converting same guest with step1 and select rhel7 OS during v2v conversion
# virt-v2v -ic vpx://root.227.27/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.3 -io  vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api  -op /home/rhvpasswd  -os nfs_data -b ovirtmgmt -ip /home/passwd esx7.0-rhel7.9-rhel8.4-multiple_linux_os
[   0.0] Setting up the source: -i libvirt -ic vpx://root.227.27/data/10.73.199.217/?no_verify=1 -it vddk esx7.0-rhel7.9-rhel8.4-multiple_linux_os
[   1.8] Opening the source
[   7.2] Inspecting the source

***
Dual- or multi-boot operating system detected.  Choose the root filesystem
that contains the main operating system from the list below:

 [1] /dev/rhel_bootp-73-199-22/root (Red Hat Enterprise Linux Server 7.9 (Maipo))
 [2] /dev/rhel_bootp-73-199-6/root (Red Hat Enterprise Linux 8.4 (Ootpa))

Enter a number between 1 and 2, or ‘exit’: 1
[ 904.4] Checking for sufficient free disk space in the guest
[ 904.4] Converting Red Hat Enterprise Linux Server 7.9 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[1072.6] Mapping filesystem data to avoid copying unused and blank areas
[1075.0] Closing the overlay
[1075.3] Assigning disks to buses
[1075.3] Checking if the guest needs BIOS or UEFI to boot
[1075.3] Setting up the destination: -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -os nfs_data
[1088.7] Copying disk 1/1
█ 100% [****************************************]
[1563.0] Creating output metadata
[1585.7] Finishing off

Hi mxie,

this is also described by the documentation <https://libguestfs.org/virt-v2v.1.html#trimming>:

"Unfortunately support for fstrim is not universal, and it also depends on specific details of the filesystem, partition alignment, and backing storage."

Most probably, the storage stack underlying the RHEL7 OS root, and the storage stack underlying the RHEL8 OS root, differ in this regard. One supports trimming, the other doesn't.

You could perhaps verify by booting either guest and running "fstrim /" as root (both before conversion and after conversion).

It's usually pretty difficult to figure out why a storage stack does not effectively support trimming, even on a bare metal machine. Especially if you have LVM and/or LUKS between the actual block device and the filesystem. Each level must support and enable trimming (enablement usually requires editing different config files). If you can make the guest disk available (with both LVs on it), we could look into it, but the reason almost certainly lies much more deeply than virt-v2v. Virt-v2v only attempts to mount each filesystem in turn with the "discard" option enabled, and then to run fstrim.

https://libguestfs.org/guestfs.3.html#guestfs_mount_options
https://libguestfs.org/guestfs.3.html#guestfs_fstrim

"This operation requires support in libguestfs, the mounted filesystem, the host filesystem, qemu and the host kernel. If this support isn't present it may give an error or even appear to run but do nothing."

https://listman.redhat.com/archives/libguestfs/2022-May/028863.html

(Still testing it locally)

Laszlo asked if we could, in virt-v2v, find and "dmsetup remove" the
/dev/mapper/osprober-* devices after running grub2.

Technically yes this works.  Following the same instructions as comment 12:

...
><rescue> /sbin/grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
done
><rescue> ls -l /dev/mapper 
total 0
crw------- 1 root root 10, 236 May 17 18:35 control
lrwxrwxrwx 1 root root       7 May 17 18:45 osprober-linux-rhel_bootp--73--199--22-root -> ../dm-5
lrwxrwxrwx 1 root root       7 May 17 18:45 osprober-linux-sda3 -> ../dm-4
lrwxrwxrwx 1 root root       7 May 17 18:36 rhel_bootp--73--199--22-root -> ../dm-1
lrwxrwxrwx 1 root root       7 May 17 18:36 rhel_bootp--73--199--22-swap -> ../dm-0
lrwxrwxrwx 1 root root       7 May 17 18:36 rhel_bootp--73--199--6-root -> ../dm-2
lrwxrwxrwx 1 root root       7 May 17 18:36 rhel_bootp--73--199--6-swap -> ../dm-3
><rescue> dmsetup remove /dev/mapper/osprober-linux-rhel_bootp--73--199--22-root
><rescue> dmsetup remove /dev/mapper/osprober-linux-sda3 
><rescue> ls -l /dev/mapper
total 0
crw------- 1 root root 10, 236 May 17 18:35 control
lrwxrwxrwx 1 root root       7 May 17 18:36 rhel_bootp--73--199--22-root -> ../dm-1
lrwxrwxrwx 1 root root       7 May 17 18:36 rhel_bootp--73--199--22-swap -> ../dm-0
lrwxrwxrwx 1 root root       7 May 17 18:36 rhel_bootp--73--199--6-root -> ../dm-2
lrwxrwxrwx 1 root root       7 May 17 18:36 rhel_bootp--73--199--6-swap -> ../dm-3

However from the virt-v2v point of view it would involve either
adding a new libguestfs API for this purpose, of uncertain
generality ("guestfs_remove_broken_osprober_devices").  Or use the
debug sh "backdoor" into the appliance to run the shell command
(https://libguestfs.org/guestfs-faq.1.html#whats-the-difference-between-guestfish-and-virt-rescue)

So it's sort of possible but also a bit of a hack.

> Or use the debug sh "backdoor" into the appliance to run the shell command
(https://libguestfs.org/guestfs-faq.1.html#whats-the-difference-between-guestfish-and-virt-rescue)

Actually is this bit true?  Maybe because /dev is bind-mounted into
the chroot when running commands, we could run the command using g#command.
I will try a patch ...

I think g#command is not a hack here (for running "dmsetup remove"), but business as usual!

Here's why I'm saying that: the problem is created by grub2-mkconfig (that's what we're trying to "clean up after"), and we *already* call grub2-mkconfig in virt-v2v's "convert/linux_bootloaders.ml" with g#command, namely in the grub2_update_console() and update() methods:

  method private grub2_update_console ~remove () =
...
          ignore (g#command [| grub2_mkconfig_cmd; "-o"; grub_config |])

  method update () =
    ignore (g#command [| grub2_mkconfig_cmd; "-o"; grub_config |])

The idea is to factor out this g#command call to a common helper function, then add *another* g#command call to that helper function, for removing any runaway /dev/mapper/osprober-* block devices.

Yup, that's pretty much the conclusion I arrived at too:

https://listman.redhat.com/archives/libguestfs/2022-May/028883.html
https://listman.redhat.com/archives/libguestfs/2022-May/028884.html

Note I'm still testing this one as well as the libguestfs one.

Fix upstream in:

https://github.com/libguestfs/virt-v2v/commit/4ad661b545f476f8dfa00f3cdc28be01f6cc0510
https://github.com/libguestfs/virt-v2v/commit/e06cf5b5dcbfd3f0a798eaf75c8778502c9e3bc6

Verify the bug with below builds:
virt-v2v-2.0.6-1.el9.x86_64
libguestfs-1.48.3-1.el9.x86_64
guestfs-tools-1.48.2-1.el9.x86_64
nbdkit-server-1.30.5-1.el9.x86_64
libnbd-1.12.2-1.el9.x86_64
libvirt-libs-8.3.0-1.el9.x86_64
qemu-img-7.0.0-4.el9.x86_64

Steps:
Scenario1:
1.1 Convert a VMware guest which has rhel7 and rhel8 installed in one disk by v2v and select rhel8 OS during v2v conversion
#  virt-v2v -ic vpx://root.227.27/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.3 -io  vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api  -op /home/rhvpasswd  -os nfs_data -b ovirtmgmt -ip /home/passwd esx7.0-rhel7.9-rhel8.4-multiple_linux_os
[   0.2] Setting up the source: -i libvirt -ic vpx://root.227.27/data/10.73.199.217/?no_verify=1 -it vddk esx7.0-rhel7.9-rhel8.4-multiple_linux_os
[   2.0] Opening the source
[  10.3] Inspecting the source

***
Dual- or multi-boot operating system detected.  Choose the root filesystem
that contains the main operating system from the list below:

 [1] /dev/rhel_bootp-73-199-22/root (Red Hat Enterprise Linux Server 7.9 (Maipo))
 [2] /dev/rhel_bootp-73-199-6/root (Red Hat Enterprise Linux 8.4 (Ootpa))

Enter a number between 1 and 2, or ‘exit’: 2
[  40.6] Checking for sufficient free disk space in the guest
[  40.6] Converting Red Hat Enterprise Linux 8.4 (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[  99.1] Mapping filesystem data to avoid copying unused and blank areas
[ 101.0] Closing the overlay
[ 101.3] Assigning disks to buses
[ 101.3] Checking if the guest needs BIOS or UEFI to boot
[ 101.3] Setting up the destination: -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -os nfs_data
[ 114.4] Copying disk 1/1
█ 100% [****************************************]
[ 675.8] Creating output metadata
[ 692.6] Finishing off

1.2 Check the guest after v2v conversion, the checkpoints of guest are passed


Scenario2:

2.1 Prepare a linux guest and its non-os disk has few/no inodes left on VMware, then convert it by v2v

# df -i
Filesystem                          Inodes  IUsed   IFree IUse% Mounted on
devtmpfs                            227695    341  227354    1% /dev
tmpfs                               232604      1  232603    1% /dev/shm
tmpfs                               232604    556  232048    1% /run
tmpfs                               232604     17  232587    1% /sys/fs/cgroup
/dev/mapper/rhel_vm--197--147-root 5134336  43461 5090875    1% /
/dev/sda1                           524288    310  523978    1% /boot
tmpfs                               232604      5  232599    1% /run/user/0
/dev/sdb1                           327680 327680       0  100% /home/sdb

#  virt-v2v -ic vpx://root.227.27/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.3 -io  vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api  -op /home/rhvpasswd  -os nfs_data -b ovirtmgmt -ip /home/passwd esx7.0-rhel8.5-second-disk-has-no-inodes
[   0.2] Setting up the source: -i libvirt -ic vpx://root.227.27/data/10.73.199.217/?no_verify=1 -it vddk esx7.0-rhel8.5-second-disk-has-no-inodes
[   3.1] Opening the source
[   9.8] Inspecting the source
[  21.5] Checking for sufficient free disk space in the guest
[  21.5] Converting Red Hat Enterprise Linux 8.5 Beta (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[  84.7] Mapping filesystem data to avoid copying unused and blank areas
[  85.9] Closing the overlay
[  86.2] Assigning disks to buses
[  86.2] Checking if the guest needs BIOS or UEFI to boot
[  86.2] Setting up the destination: -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -os nfs_data
[ 111.7] Copying disk 1/2
█ 100% [****************************************]
[ 307.9] Copying disk 2/2
█ 100% [****************************************]
[ 314.2] Creating output metadata
[ 349.1] Finishing off

2.2 Check the guest after v2v conversion, the checkpoints of guest are passed


Result: 
   The bug has been fixed, move the bug from ON_QA to VERIFIED

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: virt-v2v security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7968