Bug 1733168 - virt-v2v: Use scp -T in -i vmx -it ssh mode
Summary: virt-v2v: Use scp -T in -i vmx -it ssh mode
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: libguestfs
Version: 8.1
Hardware: x86_64
OS: Unspecified
urgent
urgent
Target Milestone: rc
: 8.0
Assignee: Libvirt Maintainers
QA Contact: Virtualization Bugs
URL:
Whiteboard: V2V
Depends On: 1666581
Blocks: 1738886
TreeView+ depends on / blocked
 
Reported: 2019-07-25 10:03 UTC by mxie@redhat.com
Modified: 2019-11-06 07:18 UTC (History)
9 users (show)

Fixed In Version: libguestfs-1.40.2-12.module+el8.1.0+3908+8a8c5ed4
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1738886 (view as bug list)
Environment:
Last Closed: 2019-11-06 07:17:49 UTC
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:3723 None None None 2019-11-06 07:18:16 UTC

Description mxie@redhat.com 2019-07-25 10:03:50 UTC
Description of problem:
Can't convert guest from VMware via vmx+ssh by virt-v2v due to openssh protocol  problem

Version-Release number of selected component (if applicable):
openssh-8.0p1-2.el8.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Convert a windows guest from VMware to KVM managed by libvirt via vmx+ssh using virt-v2v but the conversion is failed

# virt-v2v -i vmx -it ssh ssh://root@10.73.75.219/vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/esx6.7-win2019-x86_64-efi/esx6.7-win2019-x86_64-efi.vmx -of raw -o local -os /home
[   0.0] Opening the source -i vmx ssh://root@10.73.75.219/vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/esx6.7-win2019-x86_64-efi/esx6.7-win2019-x86_64-efi.vmx
protocol error: filename does not match request
virt-v2v: error: could not copy the VMX file from the remote server, see 
earlier error messages

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

2.Downgrade openssh from openssh-8.0p1-2.el8.x86_64 to openssh-7.8p1-4.el8.x86_64

#yum downgrade http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/openssh/7.8p1/4.el8/x86_64/openssh-7.8p1-4.el8.x86_64.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/openssh/7.8p1/4.el8/x86_64/openssh-server-7.8p1-4.el8.x86_64.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/openssh/7.8p1/4.el8/x86_64/openssh-clients-7.8p1-4.el8.x86_64.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/openssh/7.8p1/4.el8/x86_64/openssh-askpass-7.8p1-4.el8.x86_64.rpm -y

# rpm -q openssh
openssh-7.8p1-4.el8.x86_64

3.Convert a windows guest from VMware to KVM managed by libvirt via vmx+ssh using virt-v2v again, the conversion can be finished without error

# virt-v2v -i vmx -it ssh ssh://root@10.73.75.219/vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/esx6.7-win2019-x86_64-efi/esx6.7-win2019-x86_64-efi.vmx -of raw -o local -os /home
[   0.0] Opening the source -i vmx ssh://root@10.73.75.219/vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/esx6.7-win2019-x86_64-efi/esx6.7-win2019-x86_64-efi.vmx
[   0.6] Creating an overlay to protect the source from being modified
[   0.7] Initializing the target -o local -os /home
[   0.7] Opening the overlay
[   5.5] Inspecting the overlay
[  13.0] Checking for sufficient free disk space in the guest
[  13.0] Estimating space required on target for each disk
[  13.0] Converting Windows Server 2019 Standard to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.  
Firstboot scripts may conflict with PnP.
virt-v2v: warning: there is no QXL driver for this version of Windows (10.0 
x86_64).  virt-v2v looks for this driver in 
/usr/share/virtio-win/virtio-win.iso

The guest will be configured to use a basic VGA display driver.
virt-v2v: This guest has virtio drivers installed.
[  19.9] Mapping filesystem data to avoid copying unused and blank areas
virt-v2v: warning: fstrim on guest filesystem /dev/sda2 failed.  Usually 
you can ignore this message.  To find out more read "Trimming" in 
virt-v2v(1).

Original message: fstrim: fstrim: /sysroot/: the discard operation is not 
supported
[  20.4] Closing the overlay
[  20.5] Checking if the guest needs BIOS or UEFI to boot
virt-v2v: This guest requires UEFI on the target to boot.
[  20.5] Assigning disks to buses
[  20.5] Copying disk 1/1 to /home/esx6.7-win2019-x86_64-efi-sda (raw)
    (100.00/100%)
[ 448.6] Creating output metadata
[ 448.6] Finishing off


Actual results:


Expected results:
Can convert guest from VMware via vmx+ssh by virt-v2v with latest openssh


Additional info:

Comment 1 Jakub Jelen 2019-07-25 10:17:57 UTC
Please, provide the debug logs. This sounds like consequence of fix for CVE-2019-6111. Try to pass the -T switch to the scp, if that is using scp.

Comment 2 Jakub Jelen 2019-07-25 10:19:13 UTC
See related bug #1722229 which broke oracle for possible workarounds.

Comment 3 Richard W.M. Jones 2019-07-25 10:25:36 UTC
Here are my comments from IRC:

10:50 < rjones> mxie: it's likely to be a quoting problem, so you could try copying a remote file with a space in the name
10:51 < rjones> and see if quoting it with "file' 'name" works or whatever
10:51 < rjones> or backslashes or quotes
10:52 < rjones> mxie: see also:
10:52 < rjones> https://github.com/libguestfs/libguestfs/blob/6d251e3828ff94deac7589b5892df174430e01f9/v2v/input_vmx.ml#L82
10:53 < rjones> and also https://stackoverflow.com/questions/19858176/how-to-escape-spaces-in-path-during-scp-copy-in-linux
10:53 < rjones> notice all the backslashes in the error message

Comment 4 Jakub Jelen 2019-07-25 11:12:36 UTC
As already mentioned in the previous comments, there are two possible workarounds:

 * Use sftp for sensible file transfers.
 * Use -T switch to avoid these checks if you are using trusted servers

Comment 5 Richard W.M. Jones 2019-07-25 11:19:18 UTC
sftp (command line tool) isn't going to work for us because it's interactive.
There's a batch mode (-b option) which may be worth looking into.  Using sftp
from libssh is also possible, but a rather huge change.  I need to make
sure that VMware's ssh actually supports sftp because it could be disabled.

We can easily add the -T option.  The server is a VMware server so I guess we
trust it.  It's at least always owned by the same people running virt-v2v.

Comment 6 Jakub Jelen 2019-07-25 13:45:46 UTC
Richard, do you want me to change this bug to your component or can I close it as duplicate of #1722229?

Comment 7 Richard W.M. Jones 2019-07-25 13:47:51 UTC
I don't really agree with breaking scp for everyone, but here we are.

Comment 8 Richard W.M. Jones 2019-07-25 14:02:37 UTC
Patch posted:
https://www.redhat.com/archives/libguestfs/2019-July/msg00268.html

Comment 9 Jakub Jelen 2019-07-25 15:10:14 UTC
Thank you. Technically, the upstream fix was trying to accommodate as much as possible use cases, but because of the flexibility of this "API" and many possible various bash expansions or quotes ignorance that could be used in the scp, it is really not possible to catch everything.

Comment 10 Richard W.M. Jones 2019-08-08 11:09:16 UTC
Upstream in:
https://github.com/libguestfs/libguestfs/commit/7692c31494f7b1d37e380eed9eb99c5952940dbf

(Sorry, I kept the "lazy", I quite like it)

Comment 12 mxie@redhat.com 2019-08-09 13:16:01 UTC
Verify the bug with builds:
virt-v2v-1.40.2-12.module+el8.1.0+3908+8a8c5ed4.x86_64
libguestfs-1.40.2-12.module+el8.1.0+3908+8a8c5ed4.x86_64
libvirt-5.6.0-1.module+el8.1.0+3890+4d3d259c.x86_64
qemu-kvm-4.0.0-6.module+el8.1.0+3736+a2aefea3.x86_64
nbdkit-1.12.5-1.module+el8.1.0+3868+35f94834.x86_64
libssh-0.9.0-4.el8.x86_64
openssh-8.0p1-3.el8.x86_64
kernel-4.18.0-128.el8.x86_64

Steps:
Scenario1:
1.1 Convert guests from VMware to RHV via vmx+ssh using virt-v2v 
# virt-v2v -i vmx -it ssh ssh://root@10.73.75.219//vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0//esx6.7-rhel7.6-uefi-raid//esx6.7-rhel7.6-uefi-raid.vmx -o rhv-upload -oo rhv-cafile=/home/ca.pem -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /tmp/rhvpasswd --password-file /tmp/passwd -os nfs_data -b ovirtmgmt  -oo rhv-cluster=nfs
# virt-v2v -i vmx -it ssh ssh://root@10.73.75.219//vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0//esx6.7-rhel7.6-uefi-raid//esx6.7-rhel7.6-uefi-raid.vmx -o rhv-upload -oo rhv-cafile=/home/ca.pem -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /tmp/rhvpasswd --password-file /tmp/passwd -os nfs_data -b ovirtmgmt  -oo rhv-cluster=nfs
[   0.5] Opening the source -i vmx ssh://root@10.73.75.219//vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0//esx6.7-rhel7.6-uefi-raid//esx6.7-rhel7.6-uefi-raid.vmx
[   1.5] Creating an overlay to protect the source from being modified
[   2.6] Opening the overlay
[  23.3] Inspecting the overlay
[  50.5] Checking for sufficient free disk space in the guest
[  50.5] Estimating space required on target for each disk
[  50.5] Converting Red Hat Enterprise Linux Server 7.6 (Maipo) to run on KVM
virt-v2v: warning: guest tools directory ‘linux/el7’ is missing from 
the virtio-win directory or ISO.

Guest tools are only provided in the RHV Guest Tools ISO, so this can 
happen if you are using the version of virtio-win which contains just the 
virtio drivers.  In this case only virtio drivers can be installed in the 
guest, and installation of Guest Tools will be skipped.

virt-v2v: This guest has virtio drivers installed.
[ 240.8] Mapping filesystem data to avoid copying unused and blank areas
virt-v2v: warning: fstrim on guest filesystem /dev/md127 failed.  Usually 
you can ignore this message.  To find out more read "Trimming" in 
virt-v2v(1).

Original message: fstrim: fstrim: /sysroot/: the discard operation is not 
supported
[ 242.7] Closing the overlay
[ 242.8] Assigning disks to buses
[ 242.8] Checking if the guest needs BIOS or UEFI to boot
virt-v2v: This guest requires UEFI on the target to boot.
[ 242.8] Initializing the target -o rhv-upload -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /tmp/rhvpasswd -os nfs_data
[ 246.4] Copying disk 1/3 to qemu URI json:{ "file.driver": "nbd", "file.path": "/var/tmp/rhvupload.LTgO8Q/nbdkit0.sock", "file.export": "/" } (raw)
    (100.00/100%)
[ 744.1] Copying disk 2/3 to qemu URI json:{ "file.driver": "nbd", "file.path": "/var/tmp/rhvupload.LTgO8Q/nbdkit1.sock", "file.export": "/" } (raw)
    (100.00/100%)
[ 868.2] Copying disk 3/3 to qemu URI json:{ "file.driver": "nbd", "file.path": "/var/tmp/rhvupload.LTgO8Q/nbdkit2.sock", "file.export": "/" } (raw)
    (100.00/100%)
[1055.7] Creating output metadata
[1074.4] Finishing off

1.2 Power on guest and checkpoint of guest are passed

1.3 Check if "scp -T " is used in v2v vmx+ssh conversion
# cat vmx-ssh-fast.log |grep "scp -T"
LANG=C scp -T |& grep "unknown option"
scp -T 'root'@'10.73.75.219':''\''//vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0//esx6.7-rhel7.6-uefi-raid//esx6.7-rhel7.6-uefi-raid.vmx'\''' '/var/tmp/vmx.GxVglt/source.vmx'


Scenario2:
2.1 Convert guests from VMware to RHV via vmx using virt-v2v 
# virt-v2v -i vmx esx6.5-win2019-x86_64/esx6.5-win2019-x86_64.vmx  -o rhv-upload -oo rhv-cafile=/home/ca.pem -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /tmp/rhvpasswd --password-file /tmp/passwd -os nfs_data -b ovirtmgmt  -oo rhv-cluster=nfs -of raw
[   0.5] Opening the source -i vmx esx6.5-win2019-x86_64/esx6.5-win2019-x86_64.vmx
[   0.5] Creating an overlay to protect the source from being modified
[   0.6] Opening the overlay
[   4.8] Inspecting the overlay
[   7.8] Checking for sufficient free disk space in the guest
[   7.8] Estimating space required on target for each disk
[   7.8] Converting Windows Server 2019 Standard to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.  
Firstboot scripts may conflict with PnP.
virt-v2v: warning: there is no QXL driver for this version of Windows (10.0 
x86_64).  virt-v2v looks for this driver in 
/usr/share/virtio-win/virtio-win.iso

The guest will be configured to use a basic VGA display driver.
virt-v2v: This guest has virtio drivers installed.
[  15.2] Mapping filesystem data to avoid copying unused and blank areas
[  16.1] Closing the overlay
[  16.2] Assigning disks to buses
[  16.2] Checking if the guest needs BIOS or UEFI to boot
[  16.2] Initializing the target -o rhv-upload -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /tmp/rhvpasswd -os nfs_data
[  17.5] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/var/tmp/rhvupload.NwNXbH/nbdkit0.sock", "file.export": "/" } (raw)
    (100.00/100%)
[ 796.1] Creating output metadata
[ 819.8] Finishing off

2.2 Power on guest and checkpoint of guest are passed

2.3 Check if "scp -T " is used in v2v vmx conversion
# cat vmx.log |grep "scp -T"
nothing

Scenario3:
3.1 Convert a guest from Xen to rhv by virt-v2v
# virt-v2v -ic xen+ssh://root@10.73.3.21 xen-hvm-sles12sp1-x86_64 -of raw -o rhv-upload -oo rhv-cafile=/home/ca.pem -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /tmp/rhvpasswd --password-file /tmp/passwd -os nfs_data -b ovirtmgmt  -oo rhv-cluster=nfs
[   0.5] Opening the source -i libvirt -ic xen+ssh://root@10.73.3.21 xen-hvm-sles12sp1-x86_64
[   1.2] Creating an overlay to protect the source from being modified
[   1.7] Opening the overlay
[   6.7] Inspecting the overlay
[  24.7] Checking for sufficient free disk space in the guest
[  24.7] Estimating space required on target for each disk
[  24.7] Converting SUSE Linux Enterprise Server 12 SP1 to run on KVM
virt-v2v: warning: guest tools directory ‘linux/lp151’ is missing from 
the virtio-win directory or ISO.

Guest tools are only provided in the RHV Guest Tools ISO, so this can 
happen if you are using the version of virtio-win which contains just the 
virtio drivers.  In this case only virtio drivers can be installed in the 
guest, and installation of Guest Tools will be skipped.
virt-v2v: This guest has virtio drivers installed.
[  69.5] Mapping filesystem data to avoid copying unused and blank areas
[  70.3] Closing the overlay
[  70.3] Assigning disks to buses
[  70.3] Checking if the guest needs BIOS or UEFI to boot
[  70.3] Initializing the target -o rhv-upload -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /tmp/rhvpasswd -os nfs_data
[  71.7] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/var/tmp/rhvupload.MAEmTd/nbdkit0.sock", "file.export": "/" } (raw)
    (100.00/100%)
[ 509.1] Creating output metadata
[ 532.7] Finishing off

3.2 Power on guest and checkpoint of guest are passed

3.3 Check if "scp -T " is used in v2v xen conversion
# cat xen-fast.log  |grep "scp -T"
nothing

Result:
  Virt-v2v only use ‘scp -T’ to convert guest via vmx+ssh, so move the bug from ON_QA to VERIFIED

Comment 14 errata-xmlrpc 2019-11-06 07:17:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3723


Note You need to log in before you can comment on or make changes to this bug.