Bug 2003503
Summary: | There is virt-v2v warning: fstrim on guest filesystem /dev/mapper/osprober-linux-sdb1 failed if non-os disk of source guest has few/no inodes lef | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 9 | Reporter: | mxie <mxie> |
Component: | virt-v2v | Assignee: | Richard W.M. Jones <rjones> |
Status: | CLOSED ERRATA | QA Contact: | mxie <mxie> |
Severity: | low | Docs Contact: | |
Priority: | low | ||
Version: | 9.0 | CC: | chhu, juzhou, kkiwi, lersek, mzhan, rjones, tyan, tzheng, vwu, xiaodwan |
Target Milestone: | rc | Keywords: | Reopened, Triaged |
Target Release: | --- | ||
Hardware: | x86_64 | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | virt-v2v-2.0.7-1.el9 | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2022-11-15 09:55:51 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Deadline: | 2022-07-11 |
Description
mxie@redhat.com
2021-09-13 03:29:40 UTC
Can reproduce the bug when convert the VMware guest which has rhel7 and rhel8 installed in one disk and select rhel8 OS during v2v conversion # virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.2 -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78 -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -ip /home/passwd -op /home/rhvpasswd -os nfs_data -b ovirtmgmt esx7.0-rhel7.9-rhel8.4-multiple_linux_os [ 1.0] Opening the source -i libvirt -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 esx7.0-rhel7.9-rhel8.4-multiple_linux_os -it vddk -io vddk-libdir=/home/vddk7.0.2 -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78 [ 2.8] Creating an overlay to protect the source from being modified [ 3.7] Opening the overlay [ 10.6] Inspecting the overlay *** Dual- or multi-boot operating system detected. Choose the root filesystem that contains the main operating system from the list below: [1] /dev/rhel_bootp-73-199-22/root (Red Hat Enterprise Linux Server 7.9 (Maipo)) [2] /dev/rhel_bootp-73-199-6/root (Red Hat Enterprise Linux 8.4 (Ootpa)) Enter a number between 1 and 2, or ‘exit’: 2 [ 788.3] Checking for sufficient free disk space in the guest [ 788.3] Converting Red Hat Enterprise Linux 8.4 (Ootpa) to run on KVM virt-v2v: This guest has virtio drivers installed. [ 872.4] Mapping filesystem data to avoid copying unused and blank areas virt-v2v: warning: fstrim on guest filesystem /dev/mapper/osprober-linux-sda1 failed. Usually you can ignore this message. To find out more read "Trimming" in virt-v2v(1). Original message: fstrim: fstrim: /sysroot/: FITRIM ioctl failed: Operation not permitted virt-v2v: warning: fstrim on guest filesystem /dev/mapper/osprober-linux-rhel_bootp--73--199--22-root failed. Usually you can ignore this message. To find out more read "Trimming" in virt-v2v(1). Original message: fstrim: fstrim: /sysroot/: FITRIM ioctl failed: Operation not permitted [ 876.3] Closing the overlay [ 876.6] Assigning disks to buses [ 876.6] Checking if the guest needs BIOS or UEFI to boot [ 876.6] Initializing the target -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data [ 896.3] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/tmp/v2vnbdkit.lAVk2C/nbdkit5.sock", "file.export": "/" } (qcow2) (100.00/100%) [1268.3] Creating output metadata [1331.0] Finishing off Why is the warning a problem? The warning correctly informs the user that we cannot minimize the number of blocks that need to be copied. The effort to minimize the number of blocks that need to be copied, by trimming, applies equally to the system (root) disk and to any data disks. The conversion does not fail. The warning provides a pointer to <https://libguestfs.org/virt-v2v.1.html#trimming>, which correctly explains the idea in guest filesystem terms -- independently of "OS vs. non-OS disks". I do not agree that we should to hide this warning *specifically* for non-system disks. Either the warning is entirely worthless, or it is equally useful for all disks. For now I'm closing this as NOTABUG. (In reply to Laszlo Ersek from comment #3) > Why is the warning a problem? > > The warning correctly informs the user that we cannot minimize the number of > blocks that need to be copied. The effort to minimize the number of blocks > that need to be copied, by trimming, applies equally to the system (root) > disk and to any data disks. The conversion does not fail. The warning > provides a pointer to <https://libguestfs.org/virt-v2v.1.html#trimming>, > which correctly explains the idea in guest filesystem terms -- independently > of "OS vs. non-OS disks". > > I do not agree that we should to hide this warning *specifically* for > non-system disks. Either the warning is entirely worthless, or it is equally > useful for all disks. For now I'm closing this as NOTABUG. Hi Leszlo, Please check comment2, this problem can be reproduced when converting the VMware guest which has rhel7 and rhel8 installed in one disk and select rhel8 OS during v2v conversion, I reproduced this scenario with latest versions as below, you can see that there is no warning if select rhel7 OS, so I think the warning ' FITRIM ioctl failed: Operation not permitted' should be a problem Package versions: virt-v2v-2.0.4-1.el9.x86_64 libguestfs-1.48.1-1.el9.x86_64 guestfs-tools-1.48.0-1.el9.x86_64 nbdkit-server-1.30.4-1.el9.x86_64 libnbd-1.12.2-1.el9.x86_64 Steps: 1.Converting the VMware guest which has rhel7 and rhel8 installed in one disk and select rhel8 OS during v2v conversion #virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.3 -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78 -ip /home/passwd -o rhv-upload -oo rhv-direct -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt ^C [root@dell-per740-53 ~]# virt-v2v -ic vpx://root.227.27/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.3 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data -b ovirtmgmt -ip /home/passwd esx7.0-rhel7.9-rhel8.4-multiple_linux_os [ 0.0] Setting up the source: -i libvirt -ic vpx://root.227.27/data/10.73.199.217/?no_verify=1 -it vddk esx7.0-rhel7.9-rhel8.4-multiple_linux_os [ 1.9] Opening the source [ 9.0] Inspecting the source *** Dual- or multi-boot operating system detected. Choose the root filesystem that contains the main operating system from the list below: [1] /dev/rhel_bootp-73-199-22/root (Red Hat Enterprise Linux Server 7.9 (Maipo)) [2] /dev/rhel_bootp-73-199-6/root (Red Hat Enterprise Linux 8.4 (Ootpa)) Enter a number between 1 and 2, or ‘exit’: 2 [ 136.3] Checking for sufficient free disk space in the guest [ 136.3] Converting Red Hat Enterprise Linux 8.4 (Ootpa) to run on KVM virt-v2v: This guest has virtio drivers installed. [ 210.3] Mapping filesystem data to avoid copying unused and blank areas virt-v2v: warning: fstrim on guest filesystem /dev/mapper/osprober-linux-sda1 failed. Usually you can ignore this message. To find out more read "Trimming" in virt-v2v(1). Original message: fstrim: fstrim: /sysroot/: FITRIM ioctl failed: Operation not permitted virt-v2v: warning: fstrim on guest filesystem /dev/mapper/osprober-linux-rhel_bootp--73--199--22-root failed. Usually you can ignore this message. To find out more read "Trimming" in virt-v2v(1). Original message: fstrim: fstrim: /sysroot/: FITRIM ioctl failed: Operation not permitted [ 212.8] Closing the overlay [ 213.0] Assigning disks to buses [ 213.0] Checking if the guest needs BIOS or UEFI to boot [ 213.0] Setting up the destination: -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -os nfs_data [ 233.5] Copying disk 1/1 .... 2.Converting same guest with step1 and select rhel7 OS during v2v conversion # virt-v2v -ic vpx://root.227.27/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.3 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data -b ovirtmgmt -ip /home/passwd esx7.0-rhel7.9-rhel8.4-multiple_linux_os [ 0.0] Setting up the source: -i libvirt -ic vpx://root.227.27/data/10.73.199.217/?no_verify=1 -it vddk esx7.0-rhel7.9-rhel8.4-multiple_linux_os [ 1.8] Opening the source [ 7.2] Inspecting the source *** Dual- or multi-boot operating system detected. Choose the root filesystem that contains the main operating system from the list below: [1] /dev/rhel_bootp-73-199-22/root (Red Hat Enterprise Linux Server 7.9 (Maipo)) [2] /dev/rhel_bootp-73-199-6/root (Red Hat Enterprise Linux 8.4 (Ootpa)) Enter a number between 1 and 2, or ‘exit’: 1 [ 904.4] Checking for sufficient free disk space in the guest [ 904.4] Converting Red Hat Enterprise Linux Server 7.9 (Maipo) to run on KVM virt-v2v: This guest has virtio drivers installed. [1072.6] Mapping filesystem data to avoid copying unused and blank areas [1075.0] Closing the overlay [1075.3] Assigning disks to buses [1075.3] Checking if the guest needs BIOS or UEFI to boot [1075.3] Setting up the destination: -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -os nfs_data [1088.7] Copying disk 1/1 █ 100% [****************************************] [1563.0] Creating output metadata [1585.7] Finishing off Hi mxie, this is also described by the documentation <https://libguestfs.org/virt-v2v.1.html#trimming>: "Unfortunately support for fstrim is not universal, and it also depends on specific details of the filesystem, partition alignment, and backing storage." Most probably, the storage stack underlying the RHEL7 OS root, and the storage stack underlying the RHEL8 OS root, differ in this regard. One supports trimming, the other doesn't. You could perhaps verify by booting either guest and running "fstrim /" as root (both before conversion and after conversion). It's usually pretty difficult to figure out why a storage stack does not effectively support trimming, even on a bare metal machine. Especially if you have LVM and/or LUKS between the actual block device and the filesystem. Each level must support and enable trimming (enablement usually requires editing different config files). If you can make the guest disk available (with both LVs on it), we could look into it, but the reason almost certainly lies much more deeply than virt-v2v. Virt-v2v only attempts to mount each filesystem in turn with the "discard" option enabled, and then to run fstrim. https://libguestfs.org/guestfs.3.html#guestfs_mount_options https://libguestfs.org/guestfs.3.html#guestfs_fstrim "This operation requires support in libguestfs, the mounted filesystem, the host filesystem, qemu and the host kernel. If this support isn't present it may give an error or even appear to run but do nothing." https://listman.redhat.com/archives/libguestfs/2022-May/028863.html (Still testing it locally) Laszlo asked if we could, in virt-v2v, find and "dmsetup remove" the /dev/mapper/osprober-* devices after running grub2. Technically yes this works. Following the same instructions as comment 12: ... ><rescue> /sbin/grub2-mkconfig -o /boot/grub2/grub.cfg Generating grub configuration file ... done ><rescue> ls -l /dev/mapper total 0 crw------- 1 root root 10, 236 May 17 18:35 control lrwxrwxrwx 1 root root 7 May 17 18:45 osprober-linux-rhel_bootp--73--199--22-root -> ../dm-5 lrwxrwxrwx 1 root root 7 May 17 18:45 osprober-linux-sda3 -> ../dm-4 lrwxrwxrwx 1 root root 7 May 17 18:36 rhel_bootp--73--199--22-root -> ../dm-1 lrwxrwxrwx 1 root root 7 May 17 18:36 rhel_bootp--73--199--22-swap -> ../dm-0 lrwxrwxrwx 1 root root 7 May 17 18:36 rhel_bootp--73--199--6-root -> ../dm-2 lrwxrwxrwx 1 root root 7 May 17 18:36 rhel_bootp--73--199--6-swap -> ../dm-3 ><rescue> dmsetup remove /dev/mapper/osprober-linux-rhel_bootp--73--199--22-root ><rescue> dmsetup remove /dev/mapper/osprober-linux-sda3 ><rescue> ls -l /dev/mapper total 0 crw------- 1 root root 10, 236 May 17 18:35 control lrwxrwxrwx 1 root root 7 May 17 18:36 rhel_bootp--73--199--22-root -> ../dm-1 lrwxrwxrwx 1 root root 7 May 17 18:36 rhel_bootp--73--199--22-swap -> ../dm-0 lrwxrwxrwx 1 root root 7 May 17 18:36 rhel_bootp--73--199--6-root -> ../dm-2 lrwxrwxrwx 1 root root 7 May 17 18:36 rhel_bootp--73--199--6-swap -> ../dm-3 However from the virt-v2v point of view it would involve either adding a new libguestfs API for this purpose, of uncertain generality ("guestfs_remove_broken_osprober_devices"). Or use the debug sh "backdoor" into the appliance to run the shell command (https://libguestfs.org/guestfs-faq.1.html#whats-the-difference-between-guestfish-and-virt-rescue) So it's sort of possible but also a bit of a hack. > Or use the debug sh "backdoor" into the appliance to run the shell command (https://libguestfs.org/guestfs-faq.1.html#whats-the-difference-between-guestfish-and-virt-rescue) Actually is this bit true? Maybe because /dev is bind-mounted into the chroot when running commands, we could run the command using g#command. I will try a patch ... I think g#command is not a hack here (for running "dmsetup remove"), but business as usual! Here's why I'm saying that: the problem is created by grub2-mkconfig (that's what we're trying to "clean up after"), and we *already* call grub2-mkconfig in virt-v2v's "convert/linux_bootloaders.ml" with g#command, namely in the grub2_update_console() and update() methods: method private grub2_update_console ~remove () = ... ignore (g#command [| grub2_mkconfig_cmd; "-o"; grub_config |]) method update () = ignore (g#command [| grub2_mkconfig_cmd; "-o"; grub_config |]) The idea is to factor out this g#command call to a common helper function, then add *another* g#command call to that helper function, for removing any runaway /dev/mapper/osprober-* block devices. Yup, that's pretty much the conclusion I arrived at too: https://listman.redhat.com/archives/libguestfs/2022-May/028883.html https://listman.redhat.com/archives/libguestfs/2022-May/028884.html Note I'm still testing this one as well as the libguestfs one. Fix upstream in: https://github.com/libguestfs/virt-v2v/commit/4ad661b545f476f8dfa00f3cdc28be01f6cc0510 https://github.com/libguestfs/virt-v2v/commit/e06cf5b5dcbfd3f0a798eaf75c8778502c9e3bc6 Verify the bug with below builds: virt-v2v-2.0.6-1.el9.x86_64 libguestfs-1.48.3-1.el9.x86_64 guestfs-tools-1.48.2-1.el9.x86_64 nbdkit-server-1.30.5-1.el9.x86_64 libnbd-1.12.2-1.el9.x86_64 libvirt-libs-8.3.0-1.el9.x86_64 qemu-img-7.0.0-4.el9.x86_64 Steps: Scenario1: 1.1 Convert a VMware guest which has rhel7 and rhel8 installed in one disk by v2v and select rhel8 OS during v2v conversion # virt-v2v -ic vpx://root.227.27/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.3 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data -b ovirtmgmt -ip /home/passwd esx7.0-rhel7.9-rhel8.4-multiple_linux_os [ 0.2] Setting up the source: -i libvirt -ic vpx://root.227.27/data/10.73.199.217/?no_verify=1 -it vddk esx7.0-rhel7.9-rhel8.4-multiple_linux_os [ 2.0] Opening the source [ 10.3] Inspecting the source *** Dual- or multi-boot operating system detected. Choose the root filesystem that contains the main operating system from the list below: [1] /dev/rhel_bootp-73-199-22/root (Red Hat Enterprise Linux Server 7.9 (Maipo)) [2] /dev/rhel_bootp-73-199-6/root (Red Hat Enterprise Linux 8.4 (Ootpa)) Enter a number between 1 and 2, or ‘exit’: 2 [ 40.6] Checking for sufficient free disk space in the guest [ 40.6] Converting Red Hat Enterprise Linux 8.4 (Ootpa) to run on KVM virt-v2v: This guest has virtio drivers installed. [ 99.1] Mapping filesystem data to avoid copying unused and blank areas [ 101.0] Closing the overlay [ 101.3] Assigning disks to buses [ 101.3] Checking if the guest needs BIOS or UEFI to boot [ 101.3] Setting up the destination: -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -os nfs_data [ 114.4] Copying disk 1/1 █ 100% [****************************************] [ 675.8] Creating output metadata [ 692.6] Finishing off 1.2 Check the guest after v2v conversion, the checkpoints of guest are passed Scenario2: 2.1 Prepare a linux guest and its non-os disk has few/no inodes left on VMware, then convert it by v2v # df -i Filesystem Inodes IUsed IFree IUse% Mounted on devtmpfs 227695 341 227354 1% /dev tmpfs 232604 1 232603 1% /dev/shm tmpfs 232604 556 232048 1% /run tmpfs 232604 17 232587 1% /sys/fs/cgroup /dev/mapper/rhel_vm--197--147-root 5134336 43461 5090875 1% / /dev/sda1 524288 310 523978 1% /boot tmpfs 232604 5 232599 1% /run/user/0 /dev/sdb1 327680 327680 0 100% /home/sdb # virt-v2v -ic vpx://root.227.27/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.3 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data -b ovirtmgmt -ip /home/passwd esx7.0-rhel8.5-second-disk-has-no-inodes [ 0.2] Setting up the source: -i libvirt -ic vpx://root.227.27/data/10.73.199.217/?no_verify=1 -it vddk esx7.0-rhel8.5-second-disk-has-no-inodes [ 3.1] Opening the source [ 9.8] Inspecting the source [ 21.5] Checking for sufficient free disk space in the guest [ 21.5] Converting Red Hat Enterprise Linux 8.5 Beta (Ootpa) to run on KVM virt-v2v: This guest has virtio drivers installed. [ 84.7] Mapping filesystem data to avoid copying unused and blank areas [ 85.9] Closing the overlay [ 86.2] Assigning disks to buses [ 86.2] Checking if the guest needs BIOS or UEFI to boot [ 86.2] Setting up the destination: -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -os nfs_data [ 111.7] Copying disk 1/2 █ 100% [****************************************] [ 307.9] Copying disk 2/2 █ 100% [****************************************] [ 314.2] Creating output metadata [ 349.1] Finishing off 2.2 Check the guest after v2v conversion, the checkpoints of guest are passed Result: The bug has been fixed, move the bug from ON_QA to VERIFIED Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Low: virt-v2v security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:7968 |