Description of problem: Unable to migrate a Windows Server 2016 Datacenter VM from VMWare to RHV 4.4.3. The RHV Host has all errata applied. The virtio-win-1.9.14.iso is used and in the ISO storage domain. The following errors occur and the VM is not migrated. ERROR: virt-v2v error: failure: Unexpected file type msg.dictionary.load.openFailed] Cannot open file "/etc/vmware/vsphereFeatures/vsphereFeatures.cfg": No such file or directory. Version-Release number of selected component (if applicable): - Red Hat Virtualization 4.4.3 Host - CloudForms 5 How reproducible: Steps to Reproduce: 1. Install fresh RHV 4.4.3 Host with VDDK and ISO domain 2. Install fresh CloudForms. 3. Create the migration and successfully added Host. 4. Start migration of single Windows Server 2016 Datacenter VM Actual results: Migration fails with only Windows VMs. Expected results: Successful migration with errors. Additional info: The actual logs are available on the linked case. Assuming this is related to the virtio-win-1.9.14-4 package. But recreating Host with this did not help.
Created attachment 1733295 [details] 20201125-143256 - virt-v2v log
The error comes from libosinfo code, here: https://gitlab.com/libosinfo/libosinfo/-/blob/master/osinfo/osinfo_loader.c#L2368 To understand how we get the error message, see these pieces of code: https://github.com/libguestfs/virt-v2v/blob/20180abc544358f0856245b9e0758f8f51a11a58/v2v/libosinfo-c.c#L132 https://github.com/libguestfs/libguestfs-common/blob/c184cb9a1c838e34035cb5c37261570366e00358/mltools/tools_utils.ml#L191 I pushed a trivial patch this morning which improves this error message so at least it won't be so obscure in future.
To figure out what's wrong with osinfo on customer's host we will need more information. Could we get the output of `osinfo-query` on the host? As root and as vdsm user, like this: # osinfo-query os # sudo -u vdsm osinfo-query os It should produce similar error to what we see in virt-v2v output. Idealy, if it is possible to install strace on the host capturing the oututput of following commands would be even more helpful # strace osinfo-query os # sudo -u vdsm strace osinfo-query os
Error shown with the modified libosinfo: virt-v2v error: failure: Unexpected file type 'G_FILE_TYPE_UNKNOWN' for file 'file:///root/.config/osinfo'.
One line reproducer (thanks to Tomas for finding the root cause): $ HOME=/root osinfo-query os Error loading OS data: Unexpected file type libosinfo-1.8.0-3.fc33.x86_64
The fix for the issue is here: https://github.com/ManageIQ/manageiq-v2v-conversion_host/pull/97
(In reply to Tomáš Golembiovský from comment #11) > The fix for the issue is here: > https://github.com/ManageIQ/manageiq-v2v-conversion_host/pull/97 This bug should then probably move to CFME?
Yes it probably needs to be moved there. However I will also clone it now for libosinfo, to fix the (lower priority) issue described in comment 10.
(In reply to Michal Skrivanek from comment #12) > (In reply to Tomáš Golembiovský from comment #11) > > The fix for the issue is here: > > https://github.com/ManageIQ/manageiq-v2v-conversion_host/pull/97 > > This bug should then probably move to CFME? No, this requires fix in RPM in RHV.
It is not entirely obvious from the bug description and comments. The situation here is like this: 1) There is a bug in libosinfo or in glib where problems with file permissions are not handled properly which results in the bogus error 'unexpected file type'. A separate bug 1902720 has been opened to track that. 2) While investigating the error message it was discovered that wrapper in v2v-conversion-host is leaking USER and HOME environment variables from root when running virt-v2v via systemd-run as vdsm user. This is the reason why libosinfo later runs into permission related problems (when trying to access /root/.config/osinfo which is unreadable from vdsm user). The fix for the environment issue is tracked by this bug.
Try to reproduce the bug with below builds: virt-v2v-1.42.0-6.module+el8.4.0+8855+a9e237a9.x86_64 libguestfs-1.42.0-2.module+el8.4.0+8855+a9e237a9.x86_64 libvirt-6.10.0-1.module+el8.4.0+8898+a84e86e1.x86_64 qemu-kvm-5.2.0-0.module+el8.4.0+8855+a9e237a9.x86_64 libosinfo-1.8.0-1.el8.x86_64 Steps to reproduce: 1. Log into v2v conversion server with regular server 2. Set $ HOME=/root 3. Execute command "osinfo-query os", bug1902720 can be reproduced $ osinfo-query os Error loading OS data: Unexpected file type 4.Use v2v to convert a guest from VMware, but met permission error info during conversion $ virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib/ -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA esx6.7-rhel6.10-x86_64 -ip /home/passwd [ 0.0] Opening the source -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 esx6.7-rhel6.10-x86_64 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib/ -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA [ 1.7] Creating an overlay to protect the source from being modified [ 2.6] Opening the overlay virt-v2v: error: libguestfs error: could not connect to libvirt (URI = qemu:///session): Cannot create user runtime directory '/root/.cache/libvirt': Permission denied [code=38 int1=13] If reporting bugs, run virt-v2v with debugging enabled and include the complete output: virt-v2v -v -x [...] 5. Have to add root permission to regular user and convert the guest from VMware again 5.1 $ sudo virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib/ -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA esx6.7-rhel6.10-x86_64 -ip /home/passwd We trust you have received the usual lecture from the local System Administrator. It usually boils down to these three things: #1) Respect the privacy of others. #2) Think before you type. #3) With great power comes great responsibility. [sudo] password for mxie: mxie is not in the sudoers file. This incident will be reported. 5.2 # vi /etc/sudoers .... ## Allow root to run any commands anywhere root ALL=(ALL) ALL mxie ALL=(ALL) NOPASSWD:ALL .... 5.3 $ virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib/ -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA esx6.7-rhel6.10-x86_64 -ip /home/passwd^C [mxie@lz-8 mxie]$ sudo virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib/ -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA esx6.7-rhel6.10-x86_64 -ip /home/passwd [ 0.0] Opening the source -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 esx6.7-rhel6.10-x86_64 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib/ -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA [ 1.7] Creating an overlay to protect the source from being modified [ 2.5] Opening the overlay [ 6.9] Inspecting the overlay [ 15.8] Checking for sufficient free disk space in the guest [ 15.8] Estimating space required on target for each disk [ 15.8] Converting Red Hat Enterprise Linux Server release 6.10 (Santiago) to run on KVM virt-v2v: This guest has virtio drivers installed. [ 77.8] Mapping filesystem data to avoid copying unused and blank areas [ 78.3] Closing the overlay [ 78.5] Assigning disks to buses [ 78.5] Checking if the guest needs BIOS or UEFI to boot [ 78.5] Initializing the target -o libvirt -os default [ 78.5] Copying disk 1/1 to /var/lib/libvirt/images/esx6.7-rhel6.10-x86_64-sda (raw) (100.00/100%) [ 532.5] Creating output metadata [ 532.6] Finishing off Hi Tomas, I can reproduce bug1902720 in step3 , but v2v reports permission error if set $HOME environment variable as root (step4). So I have to add permission to regular user, then can't reproduce the bug from v2v side, could you please give me some suggestions how to reproduce the bug from v2v side? I don't know how to set $HOME environment variable for vdsm user on rhv node before importing guest from VMware
(In reply to mxie from comment #29) > Hi Tomas, > > I can reproduce bug1902720 in step3 , but v2v reports permission error if > set $HOME environment variable as root (step4). So I have to add permission > to regular user, then can't reproduce the bug from v2v side, could you > please give me some suggestions how to reproduce the bug from v2v side? I > don't know how to set $HOME environment variable for vdsm user on rhv node > before importing guest from VMware Don't try to add a new user as that may get you into all sorts of permission problems. But if you really want to then it may possibly work better with LIBGUESTFS_BACKEND="direct". I would rather suggest logging in as root and running virt-v2v as vdsm user and just replace the HOME env. on start. Either of these two options should work: 1) sudo -u vdsm -E HOME=/root virt-v2v -ic ... 2) systemd-run --pipe --uid=36 --gid=36 -E HOME=/root virt-v2v -ic ...
Thanks Tomas and Richard, can reproduce the bug from v2v side with below steps: Package version: virt-v2v-1.42.0-6.module+el8.4.0+8855+a9e237a9.x86_64 libguestfs-1.42.0-2.module+el8.4.0+8855+a9e237a9.x86_64 libvirt-6.10.0-1.module+el8.4.0+8898+a84e86e1.x86_64 qemu-kvm-5.2.0-0.module+el8.4.0+8855+a9e237a9.x86_64 libosinfo-1.8.0-1.el8.x86_64 Steps: 1. Log into v2v conversion server with regular server 'mxie' 2. Set $ HOME=/root and $ export LIBGUESTFS_BACKEND=direct 2.1 $ HOME=/root 2.2 $ export LIBGUESTFS_BACKEND=direct 3. Use v2v to convert a windows guest from VMware $ virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib/ -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA esx6.7-win2019-x86_64 -ip /home/passwd [ 0.0] Opening the source -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 esx6.7-win2019-x86_64 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib/ -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA [ 1.7] Creating an overlay to protect the source from being modified [ 2.7] Opening the overlay [ 6.9] Inspecting the overlay [ 11.5] Checking for sufficient free disk space in the guest [ 11.5] Estimating space required on target for each disk [ 11.5] Converting Windows Server 2019 Standard to run on KVM virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing. Firstboot scripts may conflict with PnP. virt-v2v: error: failure: Unexpected file type If reporting bugs, run virt-v2v with debugging enabled and include the complete output: virt-v2v -v -x [...]
Verified the BZ on v2v-conversion-host-1.16.2-8.el8ev My test environment: CFME 5.11.10.1 RHV 4.4.3
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat Virtualization security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5611