Description of problem: - Trying to import VM from Xen or VMware using RHEL7.3 host failed with the next vdsm.log error: Thread-1287::ERROR::2016-08-17 17:48:12,679::v2v::670::root::(_run) Job u'4235cea0-e706-af4f-c296-0346c347741e' failed Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/v2v.py", line 665, in _run self._import() File "/usr/lib/python2.7/site-packages/vdsm/v2v.py", line 692, in _import self._proc.stderr.read(1024))) V2VProcessError: Job u'4235cea0-e706-af4f-c296-0346c347741e' process failed exit-code: 1, stderr: virt-v2v: error: internal error: Not_found exception was thrown - Manually importing the same VM using virt-v2v command succeeds: root@**** nisim]# LIBGUESTFS_BACKEND=direct virt-v2v -ic vpx://administrator@****/Folder1/Folder2/Compute/Folder3/Cluster/*****?no_verify=1 tg-centos -o null [ 0.0] Opening the source -i libvirt -ic vpx://administrator@*****/Folder1/Folder2/Compute/Folder3/Cluster/*****?no_verify=1 tg-centos Enter administrator's password for *****: Enter host password for user 'administrator': [ 8.5] Creating an overlay to protect the source from being modified [ 9.3] Initializing the target -o null [ 9.3] Opening the overlay [ 37.8] Inspecting the overlay [ 80.8] Checking for sufficient free disk space in the guest [ 80.8] Estimating space required on target for each disk [ 80.8] Converting Red Hat Enterprise Linux Server 7.0 (Maipo) to run on KVM virt-v2v: This guest has virtio drivers installed. [ 807.7] Mapping filesystem data to avoid copying unused and blank areas [ 810.3] Closing the overlay [ 810.3] Checking if the guest needs BIOS or UEFI to boot [ 810.3] Assigning disks to buses [ 810.3] Copying disk 1/1 to /var/tmp/null.Sy8WQa/sda (raw) (100.00/100%) [ 950.0] Creating output metadata [ 950.0] Finishing off Version-Release number of selected component (if applicable): rhevm-4.0.2.6-0.1.el7ev.noarch libvirt-client-2.0.0-5.el7.x86_64 vdsm-4.18.11-1.el7ev.x86_64 qemu-kvm-rhev-2.6.0-21.el7.x86_64 sanlock-3.4.0-1.el7.x86_64 3.10.0-492.el7.x86_64 How reproducible: Consistently Steps to Reproduce: 1. Try to import VM from Xen or VMware. 2. 3. Actual results: import failed. Expected results: VM should be imported successfully. Additional info: Bug is relevant only for virt-v2v. importing from RHEL KVM environment is possible. vdsm and engine logs attached.
Created attachment 1191664 [details] engine.log
Created attachment 1191665 [details] vdsm.log
Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.
This bug is also relevant for rhevm-3.6.8-0.1.el6 with RHEL 7.3 host.
The problem is that we start virt-v2v with empty environment (besides LIBGUESTFS_BACKEND and possibly VIRTIO_WIN). As a result virt-v2v cannot find qemu-img. However this worked in RHEL 7.2 with default virt-v2v version and also with virt-v2v from preview repo. It can be verified like this: *** works *** env -i PATH=/sbin:/bin:/usr/sbin:/usr/bin LIBGUESTFS_BACKEND=direct /usr/bin/virt-v2v -v -x -ic 'vpx://administrator@****/Folder1/Folder2/Compute/Folder3/Cluster/****?no_verify=1' -o null --password-file /var/run/vdsm/v2v/894967a0-bed4-48e5-a209-1a6efa71ce4a.tmp --machine-readable tg-centos *** doesn't work *** env -i LIBGUESTFS_BACKEND=direct /usr/bin/virt-v2v -v -x -ic 'vpx://administrator@****/Folder1/Folder2/Compute/Folder3/Cluster/****?no_verify=1' -o null --password-file /var/run/vdsm/v2v/894967a0-bed4-48e5-a209-1a6efa71ce4a.tmp --machine-readable tg-centos
The cause of this is this upstream commit: https://github.com/libguestfs/libguestfs/commit/c605765c95fa5f8a0994e6676bad967b0448767b Basically it changes the code from (in 7.2) using the C function system(3), to (in 7.3) directly execvp(3)-ing qemu-img. The specific difference is that because system(3) runs /bin/sh, and because the shell starts with a default PATH if PATH is not set at all, qemu-img can still be found even though virt-v2v itself has no PATH in its environment. When, in RHEL 7.3, we exec qemu-img directly, we use the PATH in the virt-v2v environment, which is apparently non-existent. However having said all that, I don't believe this is a bug in virt-v2v. I think you should run virt-v2v with a PATH. and virt-v2v should use the PATH to find utilities like qemu-img. So just pass in a simple PATH that includes all the tools needed. PATH=/sbin:/bin:/usr/sbin:/usr/bin should work fine for this purpose. --- There is also a bug in virt-v2v here! virt-v2v shouldn't be throwing Not_found exception, ever. Pino, any idea why it's doing that? I couldn't see where that's coming from.
(In reply to Tomáš Golembiovský from comment #5) > [...] and also with virt-v2v from preview repo. I have rechecked this and the truth is that it worked with version 1.32.5-10.el7.x86_64 but doesn't work with 1.32.7-1.el7.x86_64. I assume the mentioned commit got there between those versions. (In reply to Richard W.M. Jones from comment #6) > The cause of this is this upstream commit: > > https://github.com/libguestfs/libguestfs/commit/ > c605765c95fa5f8a0994e6676bad967b0448767b > > Basically it changes the code from (in 7.2) using the C > function system(3), to (in 7.3) directly execvp(3)-ing qemu-img. > > The specific difference is that because system(3) runs /bin/sh, > and because the shell starts with a default PATH if PATH is not > set at all, qemu-img can still be found even though virt-v2v > itself has no PATH in its environment. Thanks for the clarification. > When, in RHEL 7.3, we exec qemu-img directly, we use the PATH in > the virt-v2v environment, which is apparently non-existent. > > However having said all that, I don't believe this is a bug > in virt-v2v. I think you should run virt-v2v with a PATH. and > virt-v2v should use the PATH to find utilities like qemu-img. Yes, this makes actually sense.
(In reply to Richard W.M. Jones from comment #6) > There is also a bug in virt-v2v here! virt-v2v shouldn't > be throwing Not_found exception, ever. Pino, any idea why it's > doing that? I couldn't see where that's coming from. I guess it's the: let paths = String.nsplit ":" (Sys.getenv "PATH") in line of Common_utils.ml, "which" function. I'm going to send a patch to handle its lack as empty list of paths. (In reply to Richard W.M. Jones from comment #6) > However having said all that, I don't believe this is a bug > in virt-v2v. I think you should run virt-v2v with a PATH. and > virt-v2v should use the PATH to find utilities like qemu-img. > > So just pass in a simple PATH that includes all the tools needed. I agree -- even if we used absolute paths for the tools we call directly, an empty PATH might still break the other tools. IMHO using "sh -c" instead of "env -i" should be better, as provides a reset environment which contains the base stuff.
(In reply to Pino Toscano from comment #8) > (In reply to Richard W.M. Jones from comment #6) > > There is also a bug in virt-v2v here! virt-v2v shouldn't > > be throwing Not_found exception, ever. Pino, any idea why it's > > doing that? I couldn't see where that's coming from. > > I guess it's the: > > let paths = String.nsplit ":" (Sys.getenv "PATH") in > > line of Common_utils.ml, "which" function. I'm going to send a patch to > handle its lack as empty list of paths. Which is https://www.redhat.com/archives/libguestfs/2016-August/msg00120.html Note that this avoid the Not_found exception thrown, but still virt-v2v will not work if it cannot find the tools it needs (e.g. qemu-img), and will print a better error message in that case. The real fix, as hinted above already, is to run virt-v2v with a proper $PATH with system paths.
(In reply to Pino Toscano from comment #8) > IMHO using "sh -c" instead of "env -i" should be better, as provides a reset > environment which contains the base stuff. We don't use "env -i" to run virt-v2v. It was just an example how to reproduce the problem. But I agree with you, I will fix that on our side.
Why do you set LIBGUESTFS_BACKEND=direct? That is a bug in oVirt.
(In reply to Richard W.M. Jones from comment #11) > Why do you set LIBGUESTFS_BACKEND=direct? That is a bug in oVirt. OK because you're reading from ESX, so this is in fact NOT a bug in oVirt. We should still check if virt-v2v can now use the libvirt backend, because bug 1134878 is supposedly fixed.
(In reply to Richard W.M. Jones from comment #12) > (In reply to Richard W.M. Jones from comment #11) > > Why do you set LIBGUESTFS_BACKEND=direct? That is a bug in oVirt. > > OK because you're reading from ESX, so this is in fact NOT a bug > in oVirt. Yes, we are importing from VMware and Xen.
I sent this patch upstream: https://www.redhat.com/archives/libguestfs/2016-August/msg00125.html which should mean in the ~ RHEL 7.4 timeframe we can remove this environment variable and use libvirt.
Reassigned, same error logged again in vdsm.log (attached): Thread-253::ERROR::2016-09-01 16:17:12,589::v2v::670::root::(_run) Job u'42354403-ed50-de34-6a8f-34e3cad5e81c' failed Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/v2v.py", line 665, in _run self._import() File "/usr/lib/python2.7/site-packages/vdsm/v2v.py", line 692, in _import self._proc.stderr.read(1024))) V2VProcessError: Job u'42354403-ed50-de34-6a8f-34e3cad5e81c' process failed exit-code: 1, stderr: virt-v2v: error: internal error: Not_found exception was thrown If reporting bugs, run virt-v2v with debugging enabled and include the complete output: virt-v2v -v -x [...] Version: ovirt-engine-4.0.4-0.1.el7ev vdsm-4.18.11-1.el7ev.x86_64 libvirt-client-2.0.0-6.el7.x86_64 qemu-kvm-rhev-2.6.0-22.el7.x86_64 virt-v2v-1.32.7-3.el7.x86_64
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
Created attachment 1196836 [details] reassigned, vdsm.log of build 4.0.4.0-1
the fix is in 4.18.12, not 4.18.11, putting back to on_qa
Verified: ovirt-engine-4.0.4-0.1.el7ev vdsm-4.18.12-1.el7ev.x86_64 qemu-kvm-rhev-2.6.0-22.el7.x86_64 libvirt-client-2.0.0-6.el7.x86_64 virt-v2v-1.32.7-3.el7.x86_64
*** Bug 1400412 has been marked as a duplicate of this bug. ***
*** Bug 1416375 has been marked as a duplicate of this bug. ***