Bug 1367839 - Cannot import VMs from Xen and VMware when using RHEL7.3 host.
Summary: Cannot import VMs from Xen and VMware when using RHEL7.3 host.
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: vdsm
Classification: oVirt
Component: Core
Version: ---
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ovirt-4.0.4
: 4.18.13
Assignee: Tomáš Golembiovský
QA Contact: Nisim Simsolo
URL:
Whiteboard:
: 1416375 (view as bug list)
Depends On:
Blocks: 1368258
TreeView+ depends on / blocked
 
Reported: 2016-08-17 15:26 UTC by Nisim Simsolo
Modified: 2017-02-01 11:59 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1368258 1370055 (view as bug list)
Environment:
Last Closed: 2016-09-26 12:33:08 UTC
oVirt Team: Virt
Embargoed:
rule-engine: ovirt-4.0.z+
rule-engine: blocker+
mgoldboi: planning_ack+
michal.skrivanek: devel_ack+
mavital: testing_ack+


Attachments (Terms of Use)
engine.log (128.42 KB, application/x-xz)
2016-08-17 15:26 UTC, Nisim Simsolo
no flags Details
vdsm.log (582.55 KB, application/x-xz)
2016-08-17 15:27 UTC, Nisim Simsolo
no flags Details
reassigned, vdsm.log of build 4.0.4.0-1 (302.23 KB, application/x-xz)
2016-09-01 13:40 UTC, Nisim Simsolo
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 62765 0 master MERGED v2v: Running virt-v2v with some sane environment 2021-02-09 10:02:20 UTC
oVirt gerrit 62919 0 ovirt-4.0 MERGED v2v: Running virt-v2v with some sane environment 2021-02-09 10:02:20 UTC

Description Nisim Simsolo 2016-08-17 15:26:27 UTC
Description of problem:
- Trying to import VM from Xen or VMware using RHEL7.3 host failed with the next vdsm.log error: 
Thread-1287::ERROR::2016-08-17 17:48:12,679::v2v::670::root::(_run) Job u'4235cea0-e706-af4f-c296-0346c347741e' failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/v2v.py", line 665, in _run
    self._import()
  File "/usr/lib/python2.7/site-packages/vdsm/v2v.py", line 692, in _import
    self._proc.stderr.read(1024)))
V2VProcessError: Job u'4235cea0-e706-af4f-c296-0346c347741e' process failed exit-code: 1, stderr: virt-v2v: error: internal error: Not_found exception was thrown

- Manually importing the same VM using virt-v2v command succeeds:
root@**** nisim]# LIBGUESTFS_BACKEND=direct virt-v2v -ic vpx://administrator@****/Folder1/Folder2/Compute/Folder3/Cluster/*****?no_verify=1 tg-centos -o null
[   0.0] Opening the source -i libvirt -ic vpx://administrator@*****/Folder1/Folder2/Compute/Folder3/Cluster/*****?no_verify=1 tg-centos
Enter administrator's password for *****:
Enter host password for user 'administrator':
[   8.5] Creating an overlay to protect the source from being modified
[   9.3] Initializing the target -o null
[   9.3] Opening the overlay
[  37.8] Inspecting the overlay
[  80.8] Checking for sufficient free disk space in the guest
[  80.8] Estimating space required on target for each disk
[  80.8] Converting Red Hat Enterprise Linux Server 7.0 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 807.7] Mapping filesystem data to avoid copying unused and blank areas
[ 810.3] Closing the overlay
[ 810.3] Checking if the guest needs BIOS or UEFI to boot
[ 810.3] Assigning disks to buses
[ 810.3] Copying disk 1/1 to /var/tmp/null.Sy8WQa/sda (raw)
    (100.00/100%)
[ 950.0] Creating output metadata
[ 950.0] Finishing off


Version-Release number of selected component (if applicable):
rhevm-4.0.2.6-0.1.el7ev.noarch
libvirt-client-2.0.0-5.el7.x86_64
vdsm-4.18.11-1.el7ev.x86_64
qemu-kvm-rhev-2.6.0-21.el7.x86_64
sanlock-3.4.0-1.el7.x86_64
3.10.0-492.el7.x86_64

How reproducible:
Consistently

Steps to Reproduce:
1. Try to import VM  from Xen or VMware.
2.
3.

Actual results:
import failed.

Expected results:
VM should be imported successfully.

Additional info:
Bug is relevant only for virt-v2v. importing from RHEL KVM environment is possible.
vdsm and engine logs attached.

Comment 1 Nisim Simsolo 2016-08-17 15:26:52 UTC
Created attachment 1191664 [details]
engine.log

Comment 2 Nisim Simsolo 2016-08-17 15:27:27 UTC
Created attachment 1191665 [details]
vdsm.log

Comment 3 Red Hat Bugzilla Rules Engine 2016-08-18 08:02:30 UTC
Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.

Comment 4 Nisim Simsolo 2016-08-18 14:34:28 UTC
This bug is also relevant for rhevm-3.6.8-0.1.el6 with RHEL 7.3 host.

Comment 5 Tomáš Golembiovský 2016-08-23 12:57:58 UTC
The problem is that we start virt-v2v with empty environment (besides LIBGUESTFS_BACKEND and possibly VIRTIO_WIN). As a result virt-v2v cannot find qemu-img. However this worked in RHEL 7.2 with default virt-v2v version and also with virt-v2v from preview repo.

It can be verified like this: 

*** works ***

        env -i PATH=/sbin:/bin:/usr/sbin:/usr/bin LIBGUESTFS_BACKEND=direct  /usr/bin/virt-v2v -v -x -ic 'vpx://administrator@****/Folder1/Folder2/Compute/Folder3/Cluster/****?no_verify=1' -o null --password-file /var/run/vdsm/v2v/894967a0-bed4-48e5-a209-1a6efa71ce4a.tmp  --machine-readable tg-centos

*** doesn't work ***

        env -i LIBGUESTFS_BACKEND=direct  /usr/bin/virt-v2v -v -x -ic 'vpx://administrator@****/Folder1/Folder2/Compute/Folder3/Cluster/****?no_verify=1' -o null --password-file /var/run/vdsm/v2v/894967a0-bed4-48e5-a209-1a6efa71ce4a.tmp  --machine-readable tg-centos

Comment 6 Richard W.M. Jones 2016-08-23 13:18:21 UTC
The cause of this is this upstream commit:

https://github.com/libguestfs/libguestfs/commit/c605765c95fa5f8a0994e6676bad967b0448767b

Basically it changes the code from (in 7.2) using the C
function system(3), to (in 7.3) directly execvp(3)-ing qemu-img.

The specific difference is that because system(3) runs /bin/sh,
and because the shell starts with a default PATH if PATH is not
set at all, qemu-img can still be found even though virt-v2v
itself has no PATH in its environment.

When, in RHEL 7.3, we exec qemu-img directly, we use the PATH in
the virt-v2v environment, which is apparently non-existent.

However having said all that, I don't believe this is a bug
in virt-v2v.  I think you should run virt-v2v with a PATH. and
virt-v2v should use the PATH to find utilities like qemu-img.

So just pass in a simple PATH that includes all the tools needed.

PATH=/sbin:/bin:/usr/sbin:/usr/bin

should work fine for this purpose.

---

There is also a bug in virt-v2v here!  virt-v2v shouldn't
be throwing Not_found exception, ever.  Pino, any idea why it's
doing that?  I couldn't see where that's coming from.

Comment 7 Tomáš Golembiovský 2016-08-23 13:33:24 UTC
(In reply to Tomáš Golembiovský from comment #5)
> [...] and also with virt-v2v from preview repo.

I have rechecked this and the truth is that it worked with version 1.32.5-10.el7.x86_64 but doesn't work with 1.32.7-1.el7.x86_64. I assume the mentioned commit got there between those versions.


(In reply to Richard W.M. Jones from comment #6)
> The cause of this is this upstream commit:
> 
> https://github.com/libguestfs/libguestfs/commit/
> c605765c95fa5f8a0994e6676bad967b0448767b
> 
> Basically it changes the code from (in 7.2) using the C
> function system(3), to (in 7.3) directly execvp(3)-ing qemu-img.
> 
> The specific difference is that because system(3) runs /bin/sh,
> and because the shell starts with a default PATH if PATH is not
> set at all, qemu-img can still be found even though virt-v2v
> itself has no PATH in its environment.

Thanks for the clarification.


> When, in RHEL 7.3, we exec qemu-img directly, we use the PATH in
> the virt-v2v environment, which is apparently non-existent.
> 
> However having said all that, I don't believe this is a bug
> in virt-v2v.  I think you should run virt-v2v with a PATH. and
> virt-v2v should use the PATH to find utilities like qemu-img.

Yes, this makes actually sense.

Comment 8 Pino Toscano 2016-08-23 13:49:41 UTC
(In reply to Richard W.M. Jones from comment #6)
> There is also a bug in virt-v2v here!  virt-v2v shouldn't
> be throwing Not_found exception, ever.  Pino, any idea why it's
> doing that?  I couldn't see where that's coming from.

I guess it's the:

  let paths = String.nsplit ":" (Sys.getenv "PATH") in

line of Common_utils.ml, "which" function. I'm going to send a patch to handle its lack as empty list of paths.

(In reply to Richard W.M. Jones from comment #6)
> However having said all that, I don't believe this is a bug
> in virt-v2v.  I think you should run virt-v2v with a PATH. and
> virt-v2v should use the PATH to find utilities like qemu-img.
> 
> So just pass in a simple PATH that includes all the tools needed.

I agree -- even if we used absolute paths for the tools we call directly, an empty PATH might still break the other tools.

IMHO using "sh -c" instead of "env -i" should be better, as provides a reset environment which contains the base stuff.

Comment 9 Pino Toscano 2016-08-23 14:02:51 UTC
(In reply to Pino Toscano from comment #8)
> (In reply to Richard W.M. Jones from comment #6)
> > There is also a bug in virt-v2v here!  virt-v2v shouldn't
> > be throwing Not_found exception, ever.  Pino, any idea why it's
> > doing that?  I couldn't see where that's coming from.
> 
> I guess it's the:
> 
>   let paths = String.nsplit ":" (Sys.getenv "PATH") in
> 
> line of Common_utils.ml, "which" function. I'm going to send a patch to
> handle its lack as empty list of paths.

Which is
  https://www.redhat.com/archives/libguestfs/2016-August/msg00120.html

Note that this avoid the Not_found exception thrown, but still virt-v2v will not work if it cannot find the tools it needs (e.g. qemu-img), and will print a better error message in that case.

The real fix, as hinted above already, is to run virt-v2v with a proper $PATH with system paths.

Comment 10 Tomáš Golembiovský 2016-08-23 19:11:42 UTC
(In reply to Pino Toscano from comment #8)

> IMHO using "sh -c" instead of "env -i" should be better, as provides a reset
> environment which contains the base stuff.

We don't use "env -i" to run virt-v2v. It was just an example how to reproduce the problem.

But I agree with you, I will fix that on our side.

Comment 11 Richard W.M. Jones 2016-08-24 13:11:27 UTC
Why do you set LIBGUESTFS_BACKEND=direct?  That is a bug in oVirt.

Comment 12 Richard W.M. Jones 2016-08-24 13:21:18 UTC
(In reply to Richard W.M. Jones from comment #11)
> Why do you set LIBGUESTFS_BACKEND=direct?  That is a bug in oVirt.

OK because you're reading from ESX, so this is in fact NOT a bug
in oVirt.

We should still check if virt-v2v can now use the libvirt backend,
because bug 1134878 is supposedly fixed.

Comment 13 Tomáš Golembiovský 2016-08-24 13:34:04 UTC
(In reply to Richard W.M. Jones from comment #12)
> (In reply to Richard W.M. Jones from comment #11)
> > Why do you set LIBGUESTFS_BACKEND=direct?  That is a bug in oVirt.
> 
> OK because you're reading from ESX, so this is in fact NOT a bug
> in oVirt.

Yes, we are importing from VMware and Xen.

Comment 14 Richard W.M. Jones 2016-08-24 14:02:56 UTC
I sent this patch upstream:

https://www.redhat.com/archives/libguestfs/2016-August/msg00125.html

which should mean in the ~ RHEL 7.4 timeframe we can remove
this environment variable and use libvirt.

Comment 15 Nisim Simsolo 2016-09-01 13:37:10 UTC
Reassigned, same error logged again in vdsm.log (attached): 
Thread-253::ERROR::2016-09-01 16:17:12,589::v2v::670::root::(_run) Job u'42354403-ed50-de34-6a8f-34e3cad5e81c' failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/v2v.py", line 665, in _run
    self._import()
  File "/usr/lib/python2.7/site-packages/vdsm/v2v.py", line 692, in _import
    self._proc.stderr.read(1024)))
V2VProcessError: Job u'42354403-ed50-de34-6a8f-34e3cad5e81c' process failed exit-code: 1, stderr: virt-v2v: error: internal error: Not_found exception was thrown

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Version:
ovirt-engine-4.0.4-0.1.el7ev
vdsm-4.18.11-1.el7ev.x86_64
libvirt-client-2.0.0-6.el7.x86_64
qemu-kvm-rhev-2.6.0-22.el7.x86_64
virt-v2v-1.32.7-3.el7.x86_64

Comment 16 Red Hat Bugzilla Rules Engine 2016-09-01 13:37:17 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 17 Nisim Simsolo 2016-09-01 13:40:12 UTC
Created attachment 1196836 [details]
reassigned, vdsm.log of build 4.0.4.0-1

Comment 18 Tomas Jelinek 2016-09-01 14:29:14 UTC
the fix is in 4.18.12, not 4.18.11, putting back to on_qa

Comment 19 Nisim Simsolo 2016-09-04 08:48:43 UTC
Verified:
ovirt-engine-4.0.4-0.1.el7ev
vdsm-4.18.12-1.el7ev.x86_64
qemu-kvm-rhev-2.6.0-22.el7.x86_64
libvirt-client-2.0.0-6.el7.x86_64
virt-v2v-1.32.7-3.el7.x86_64

Comment 20 Tomáš Golembiovský 2016-12-02 14:13:56 UTC
*** Bug 1400412 has been marked as a duplicate of this bug. ***

Comment 21 Tomáš Golembiovský 2017-02-01 11:59:29 UTC
*** Bug 1416375 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.