Bug 1138504

Summary: Support conversion of guests from remote kvm/xen/esx connections by virt-v2v
Product: Red Hat Enterprise Linux 7 Reporter: tingting zheng <tzheng>
Component: libguestfsAssignee: Richard W.M. Jones <rjones>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.1CC: codong, juzhou, mbooth, mzhan, ptoscano, rjones, tzheng
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: V2V
Fixed In Version: libguestfs-1.27.38-1.1.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-03-05 13:44:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1132569, 1138359    
Bug Blocks:    

Description tingting zheng 2014-09-05 03:09:48 UTC
Description
Support conversion of guests from remote kvm/xen/esx connections by virt-v2v

Version:
libguestfs-tools-c-1.27.32-1.1.el7.x86_64
libguestfs-1.27.32-1.1.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Old version of virt-v2v support convert guests from remote libvirt/xen/esx connections.
eg: # virt-v2v -ic xen+ssh://$xenserver/ -os default $guest
    # virt-v2v -ic qemu+ssh://$kvmserver/ -o rhev -os $export $guest
    # virt-v2v -ic esx://$esxserver -o rhev -os $export $guest

2.New version of virt-v2v only support conversion fo guest from remote vmware vcenter connections.
2.1 Try remote xen connections:
# virt-v2v -ic xen+ssh://10.66.106.64 -os default rhel6.6-pv-x64-test
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.66.106.64 rhel6.6-pv-x64-test
virt-v2v: warning: no support for remote libvirt connections to '-ic
xen+ssh://10.66.106.64'.  The conversion may fail when it tries to read
the source disks.
The authenticity of host '10.66.106.64 (10.66.106.64)' can't be established.
RSA key fingerprint is c7:91:05:44:14:57:8f:82:02:81:11:3a:63:2b:cb:ca.
Are you sure you want to continue connecting (yes/no)? yes
root.106.64's password:
[  28.0] Creating an overlay to protect the source from being modified
qemu-img: /tmp/v2vovl5174b3.qcow2: Could not open '/var/lib/xen/images/rhel6.6-pv-x64-test.img': Could not open file: No such file or directory: No such file or directory
virt-v2v: error: qemu-img command failed, see earlier errors

virt-v2v: If reporting bugs, run virt-v2v with debugging enabled (-v
-x) and include the complete output.

2.2 Try remote kvm connections:
# virt-v2v -ic qemu+ssh://10.66.106.65/system -o rhev -os 10.66.90.115:/vol/v2v_auto/auto_export kvm-rhel6.4-x86_64
[   0.0] Opening the source -i libvirt -ic qemu+ssh://10.66.106.65/system kvm-rhel6.4-x86_64
virt-v2v: warning: no support for remote libvirt connections to '-ic
qemu+ssh://10.66.106.65/system'.  The conversion may fail when it tries
to read the source disks.
root.106.65's password:
[  19.0] Creating an overlay to protect the source from being modified
qemu-img: /tmp/v2vovl5c3e87.qcow2: Could not open '/var/lib/libvirt/images/kvm/kvm-rhel/kvm-rhel6.4-x86_64.img': Could not open file: No such file or directory: No such file or directory
virt-v2v: error: qemu-img command failed, see earlier errors

virt-v2v: If reporting bugs, run virt-v2v with debugging enabled (-v
-x) and include the complete output.


2.3 Try remote esx connections:
# virt-v2v -ic esx://10.66.71.84/?no_verify=1 esx5.1-clone-test
[   0.0] Opening the source -i libvirt -ic esx://10.66.71.84/?no_verify=1 esx5.1-clone-test
Enter username for 10.66.71.84 [root]:
Enter root's password for 10.66.71.84:
curl -s --insecure -I 'https://10.66.71.84/folder/esx5.0-clone-test/esx5.0-clone-test-flat.vmdk?dcPath=ha-datacenter&dsName=esx5.1' ||:
HTTP/1.1 401 Unauthorized
Date: Wed, 3 Sep 2014 09:21:15 GMT
Set-Cookie: vmware_soap_session="52eb8815-d2e8-86d4-cdfc-642879ba2b84"; Path=/; HttpOnly;
Connection: close
WWW-Authenticate: Basic realm="VMware HTTP server"
Content-Length: 0

virt-v2v: error: esx: incorrect username or password

virt-v2v: If reporting bugs, run virt-v2v with debugging enabled (-v
-x) and include the complete output.


Actual results:
As description.

Expected results:
As old version of virt-v2v support conversion guests from remote kvm/xen/esx connections from virt-v2v,new version miss these functionalities.

Additional info:

Comment 2 Richard W.M. Jones 2014-09-05 08:29:49 UTC
This is now implemented (in 1.27.37).  However it requires
also an updated qemu-kvm-rhev containing the ssh driver (bug 1138359).

Comment 3 Richard W.M. Jones 2014-09-05 08:31:27 UTC
A further note:

    # virt-v2v -ic qemu+ssh://$kvmserver/ -o rhev -os $export $guest

We're not going to support this one.  There is no point
using virt-v2v if the guest already runs on KVM.

    # virt-v2v -ic esx://$esxserver -o rhev -os $export $guest

You'll have to use the vpx:// URL + vCenter.  Importing directly
from an ESXi hypervisor cannot be supported because the webserver
on ESXi hypervisor is broken.

Comment 4 Richard W.M. Jones 2014-09-09 20:09:37 UTC
This bug is fixed, but needs QA ack before I can add it to the
erratum.

Also the description covers a lot of unrelated issues.  I believe
that I have fixed all the issues covered, at least, all the issues
that need to be fixed (qemu+ssh:// URLs are NOT going to work).
However if there are other things still not working then it's
best to file separate bugs for them.

Comment 5 tingting zheng 2014-09-10 03:17:14 UTC
(In reply to Richard W.M. Jones from comment #3)
> A further note:
> 
>     # virt-v2v -ic qemu+ssh://$kvmserver/ -o rhev -os $export $guest
> 
> We're not going to support this one.  There is no point
> using virt-v2v if the guest already runs on KVM.
> 
>     # virt-v2v -ic esx://$esxserver -o rhev -os $export $guest
> 
> You'll have to use the vpx:// URL + vCenter.  Importing directly
> from an ESXi hypervisor cannot be supported because the webserver
> on ESXi hypervisor is broken.

I will try later,unfortunatly there is some problem with my vcenter environment,I will try to fix it and test this bug again.

Comment 7 tingting zheng 2014-09-12 07:11:40 UTC
Hi,Richard
   I think there are still some parts we need to confirm:
   1.Support remote kvm connections:
   # virt-v2v -ic qemu+ssh://$kvmserver/ -o rhev -os $export $guest
   As old version of virt-v2v can support this function,it's better to include this part in new version of virt-v2v.Users may have a stable kvm server with lots of guests installed but use a latest host as v2v server for conversion.
  
   2. Support remote esx connection:
    # virt-v2v -ic esx://$esxserver -o rhev -os $export $guest
    I noticed in comment 3:You'll have to use the vpx:// URL + vCenter. Importing directly from an ESXi hypervisor cannot be supported because the webserver on ESXi hypervisor is broken.

    I don't understand the exact meaning of the last sentence,webserver is a managment tool for esxi hypervisor,right? It seems that the old way -ic esx://$esxserver has nothing to do with webserver.
    BTW,users can use esx vclient to manage guests running on esxi,I don't think everyone use vcenter,so I think if possible it's better to support the old conversion way:-ic esx://$esxserver.

Comment 8 Richard W.M. Jones 2014-09-12 16:00:39 UTC
(In reply to tingting zheng from comment #7)
> Hi,Richard
>    I think there are still some parts we need to confirm:
>    1.Support remote kvm connections:
>    # virt-v2v -ic qemu+ssh://$kvmserver/ -o rhev -os $export $guest
>    As old version of virt-v2v can support this function,it's better to
> include this part in new version of virt-v2v.Users may have a stable kvm
> server with lots of guests installed but use a latest host as v2v server for
> conversion.

Apparently people used this as a back door for importing
KVM guests into RHEV, because amazingly RHEV doesn't have a
way to import disk images.

The problem is that virt-v2v is quite a brutal operation, deleting
and rewriting config files and so on.  It may happen not to break
a KVM guest, but that would only be by luck.

virt-v2v is a tool for importing from foreign hypervisors to KVM,
not a back door for a missing feature of RHEV.

So I don't think we should support this.  RHEV should add an import
disk image feature.

>    2. Support remote esx connection:
>     # virt-v2v -ic esx://$esxserver -o rhev -os $export $guest
>     I noticed in comment 3:You'll have to use the vpx:// URL + vCenter.
> Importing directly from an ESXi hypervisor cannot be supported because the
> webserver on ESXi hypervisor is broken.
> 
>     I don't understand the exact meaning of the last sentence,webserver is a
> managment tool for esxi hypervisor,right? It seems that the old way -ic
> esx://$esxserver has nothing to do with webserver.
>     BTW,users can use esx vclient to manage guests running on esxi,I don't
> think everyone use vcenter,so I think if possible it's better to support the
> old conversion way:-ic esx://$esxserver.

So firstly the web server included in ESXi doesn't advertise 'Accept: ranges'
(https://en.wikipedia.org/wiki/Byte_serving).

The old virt-v2v used to copy the entire disk image off the server in one
huge operation.  (It was slow, right?)  The new virt-v2v uses byte ranges
to just download parts of the disk image it needs.  So it simply doesn't
work against ESXi.  Users could use curl/wget if they want to do things
the old way ...

Another reason is that accessing ESXi directly isn't a supported method
for accessing disk images (as in, supported by VMware), but using vCenter is.

So that's why we can't access ESXi directly any longer.

The reason why the URL has changed from esx:// to vpx:// is because of
libvirt which uses different URI schemes for each
(http://libvirt.org/drvesx.html#uri)

Comment 9 zhoujunqin 2014-09-16 09:55:18 UTC
I can reproduce this bug issue.
And try to verify with new build:
libguestfs-1.27.46-1.1.el7.x86_64
virt-v2v-1.27.46-1.1.el7.x86_64

1. Test support remote xen connection:
# virt-v2v -ic xen+ssh://10.66.106.64 -os default   rhel6.6-pv-x64-test
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.66.106.64 rhel6.6-pv-x64-test
[  16.0] Creating an overlay to protect the source from being modified
[  30.0] Opening the overlay
[  49.0] Initializing the target -o libvirt -os default
[  49.0] Inspecting the overlay
[  53.0] Checking for sufficient free disk space in the guest
[  53.0] Estimating space required on target for each disk
[  53.0] Converting Red Hat Enterprise Linux Server release 6.6 Beta (Santiago) to run on KVM
[  88.0] Mapping filesystem data to avoid copying unused and blank areas
[  91.0] Closing the overlay
[  91.0] Copying disk 1/1 to /var/lib/libvirt/images/rhel6.6-pv-x64-test-sda (raw)
    (100.00/100%)
[ 164.0] Creating output metadata
Pool default refreshed

Domain rhel6.6-pv-x64-test defined from /tmp/v2vlibvirt2f5a26.xml

[ 164.0] Finishing off

2. Test support remote esx connection:
# virt-v2v -ic vpx://administrator.7.125/tzheng-test/10.66.71.84/?no_verify=1  esx-rhel6
[   0.0] Opening the source -i libvirt -ic vpx://administrator.7.125/tzheng-test/10.66.71.84/?no_verify=1 esx-rhel6
Enter administrator's password for 10.66.7.125: 
Enter host password for user 'administrator':
[  14.0] Creating an overlay to protect the source from being modified
[  14.0] Opening the overlay
[  33.0] Initializing the target -o libvirt -os default
[  33.0] Inspecting the overlay
[ 141.0] Checking for sufficient free disk space in the guest
[ 141.0] Estimating space required on target for each disk
[ 141.0] Converting Red Hat Enterprise Linux Server release 6.5 (Santiago) to run on KVM
[ 816.0] Mapping filesystem data to avoid copying unused and blank areas
[ 828.0] Closing the overlay
[ 828.0] Copying disk 1/1 to /var/lib/libvirt/images/esx-rhel6-sda (raw)
    (100.00/100%)
[2717.0] Creating output metadata
Pool default refreshed

Domain esx-rhel6 defined from /tmp/v2vlibvirt16217c.xml

[2717.0] Finishing off


And also seen Comment 8, move this bug from ON_QA to VERIFIED.

Comment 11 errata-xmlrpc 2015-03-05 13:44:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0303.html