Bug 1738886

Summary: virt-v2v: Use scp -T in -i vmx -it ssh mode
Product: Red Hat Enterprise Linux 8 Reporter: Richard W.M. Jones <rjones>
Component: libguestfsAssignee: Pino Toscano <ptoscano>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 8.1CC: jjelen, juzhou, knoel, libvirt-maint, mxie, mzhan, ptoscano, rjones, tmraz, tzheng, virt-bugs, xiaodwan, zili
Target Milestone: rcKeywords: Regression
Target Release: 8.1   
Hardware: x86_64   
OS: Unspecified   
Whiteboard: V2V
Fixed In Version: libguestfs-1.38.4-14.module+el8.1.0+3910+3a68cd83 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1733168 Environment:
Last Closed: 2019-11-05 20:51:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1666581, 1733168    
Bug Blocks:    

Description Richard W.M. Jones 2019-08-08 11:12:29 UTC
+++ This bug was initially created as a clone of Bug #1733168 +++

Description of problem:
Can't convert guest from VMware via vmx+ssh by virt-v2v due to openssh protocol  problem

Version-Release number of selected component (if applicable):
openssh-8.0p1-2.el8.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Convert a windows guest from VMware to KVM managed by libvirt via vmx+ssh using virt-v2v but the conversion is failed

# virt-v2v -i vmx -it ssh ssh://root.75.219/vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/esx6.7-win2019-x86_64-efi/esx6.7-win2019-x86_64-efi.vmx -of raw -o local -os /home
[   0.0] Opening the source -i vmx ssh://root.75.219/vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/esx6.7-win2019-x86_64-efi/esx6.7-win2019-x86_64-efi.vmx
protocol error: filename does not match request
virt-v2v: error: could not copy the VMX file from the remote server, see 
earlier error messages

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

2.Downgrade openssh from openssh-8.0p1-2.el8.x86_64 to openssh-7.8p1-4.el8.x86_64

#yum downgrade http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/openssh/7.8p1/4.el8/x86_64/openssh-7.8p1-4.el8.x86_64.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/openssh/7.8p1/4.el8/x86_64/openssh-server-7.8p1-4.el8.x86_64.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/openssh/7.8p1/4.el8/x86_64/openssh-clients-7.8p1-4.el8.x86_64.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/openssh/7.8p1/4.el8/x86_64/openssh-askpass-7.8p1-4.el8.x86_64.rpm -y

# rpm -q openssh
openssh-7.8p1-4.el8.x86_64

3.Convert a windows guest from VMware to KVM managed by libvirt via vmx+ssh using virt-v2v again, the conversion can be finished without error

# virt-v2v -i vmx -it ssh ssh://root.75.219/vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/esx6.7-win2019-x86_64-efi/esx6.7-win2019-x86_64-efi.vmx -of raw -o local -os /home
[   0.0] Opening the source -i vmx ssh://root.75.219/vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/esx6.7-win2019-x86_64-efi/esx6.7-win2019-x86_64-efi.vmx
[   0.6] Creating an overlay to protect the source from being modified
[   0.7] Initializing the target -o local -os /home
[   0.7] Opening the overlay
[   5.5] Inspecting the overlay
[  13.0] Checking for sufficient free disk space in the guest
[  13.0] Estimating space required on target for each disk
[  13.0] Converting Windows Server 2019 Standard to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.  
Firstboot scripts may conflict with PnP.
virt-v2v: warning: there is no QXL driver for this version of Windows (10.0 
x86_64).  virt-v2v looks for this driver in 
/usr/share/virtio-win/virtio-win.iso

The guest will be configured to use a basic VGA display driver.
virt-v2v: This guest has virtio drivers installed.
[  19.9] Mapping filesystem data to avoid copying unused and blank areas
virt-v2v: warning: fstrim on guest filesystem /dev/sda2 failed.  Usually 
you can ignore this message.  To find out more read "Trimming" in 
virt-v2v(1).

Original message: fstrim: fstrim: /sysroot/: the discard operation is not 
supported
[  20.4] Closing the overlay
[  20.5] Checking if the guest needs BIOS or UEFI to boot
virt-v2v: This guest requires UEFI on the target to boot.
[  20.5] Assigning disks to buses
[  20.5] Copying disk 1/1 to /home/esx6.7-win2019-x86_64-efi-sda (raw)
    (100.00/100%)
[ 448.6] Creating output metadata
[ 448.6] Finishing off


Actual results:


Expected results:
Can convert guest from VMware via vmx+ssh by virt-v2v with latest openssh


Additional info:

--- Additional comment from Jakub Jelen on 2019-07-25 10:17:57 UTC ---

Please, provide the debug logs. This sounds like consequence of fix for CVE-2019-6111. Try to pass the -T switch to the scp, if that is using scp.

--- Additional comment from Jakub Jelen on 2019-07-25 10:19:13 UTC ---

See related bug #1722229 which broke oracle for possible workarounds.

--- Additional comment from Richard W.M. Jones on 2019-07-25 10:25:36 UTC ---

Here are my comments from IRC:

10:50 < rjones> mxie: it's likely to be a quoting problem, so you could try copying a remote file with a space in the name
10:51 < rjones> and see if quoting it with "file' 'name" works or whatever
10:51 < rjones> or backslashes or quotes
10:52 < rjones> mxie: see also:
10:52 < rjones> https://github.com/libguestfs/libguestfs/blob/6d251e3828ff94deac7589b5892df174430e01f9/v2v/input_vmx.ml#L82
10:53 < rjones> and also https://stackoverflow.com/questions/19858176/how-to-escape-spaces-in-path-during-scp-copy-in-linux
10:53 < rjones> notice all the backslashes in the error message

--- Additional comment from Jakub Jelen on 2019-07-25 11:12:36 UTC ---

As already mentioned in the previous comments, there are two possible workarounds:

 * Use sftp for sensible file transfers.
 * Use -T switch to avoid these checks if you are using trusted servers

--- Additional comment from Richard W.M. Jones on 2019-07-25 11:19:18 UTC ---

sftp (command line tool) isn't going to work for us because it's interactive.
There's a batch mode (-b option) which may be worth looking into.  Using sftp
from libssh is also possible, but a rather huge change.  I need to make
sure that VMware's ssh actually supports sftp because it could be disabled.

We can easily add the -T option.  The server is a VMware server so I guess we
trust it.  It's at least always owned by the same people running virt-v2v.

--- Additional comment from Jakub Jelen on 2019-07-25 13:45:46 UTC ---

Richard, do you want me to change this bug to your component or can I close it as duplicate of #1722229?

--- Additional comment from Richard W.M. Jones on 2019-07-25 13:47:51 UTC ---

I don't really agree with breaking scp for everyone, but here we are.

--- Additional comment from Richard W.M. Jones on 2019-07-25 14:02:37 UTC ---

Patch posted:
https://www.redhat.com/archives/libguestfs/2019-July/msg00268.html

--- Additional comment from Jakub Jelen on 2019-07-25 15:10:14 UTC ---

Thank you. Technically, the upstream fix was trying to accommodate as much as possible use cases, but because of the flexibility of this "API" and many possible various bash expansions or quotes ignorance that could be used in the scp, it is really not possible to catch everything.

--- Additional comment from Richard W.M. Jones on 2019-08-08 11:09:16 UTC ---

Upstream in:
https://github.com/libguestfs/libguestfs/commit/7692c31494f7b1d37e380eed9eb99c5952940dbf

(Sorry, I kept the "lazy", I quite like it)

Comment 1 Richard W.M. Jones 2019-08-08 11:13:40 UTC
Unfortunately I think we're going to need a z-stream here since the
version of openssh which broke this is going into RHEL 8.1.0.

Comment 3 mxie@redhat.com 2019-08-09 09:06:21 UTC
Verify the bug with builds:
virt-v2v-1.38.4-14.module+el8.1.0+3910+3a68cd83.x86_64
libguestfs-1.38.4-14.module+el8.1.0+3910+3a68cd83.x86_64
libvirt-4.5.0-31.module+el8.1.0+3808+3325c1a3.x86_64
qemu-kvm-2.12.0-83.module+el8.1.0+3852+0ba8aef0.x86_64
libssh-0.9.0-4.el8.x86_64
openssh-8.0p1-3.el8.x86_64
kernel-4.18.0-128.el8.x86_64

Steps:
Scenario1:
1.1 Convert guests from VMware to kvm managed by libvirt via vmx+ssh using virt-v2v 
# virt-v2v -i vmx -it ssh ssh://root.75.219/vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/esx6.7-win2019-x86_64-efi/esx6.7-win2019-x86_64-efi.vmx 
[   0.0] Opening the source -i vmx ssh://root.75.219/vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/esx6.7-win2019-x86_64-efi/esx6.7-win2019-x86_64-efi.vmx
[   0.6] Creating an overlay to protect the source from being modified
[   0.8] Initializing the target -o libvirt -os default
[   0.9] Opening the overlay
[   5.9] Inspecting the overlay
[  13.4] Checking for sufficient free disk space in the guest
[  13.4] Estimating space required on target for each disk
[  13.4] Converting Windows Server 2019 Standard to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.  
Firstboot scripts may conflict with PnP.
virt-v2v: warning: there is no QXL driver for this version of Windows (10.0 
x86_64).  virt-v2v looks for this driver in 
/usr/share/virtio-win/virtio-win.iso

The guest will be configured to use a basic VGA display driver.
virt-v2v: This guest has virtio drivers installed.
[  20.8] Mapping filesystem data to avoid copying unused and blank areas
virt-v2v: warning: fstrim on guest filesystem /dev/sda2 failed.  Usually 
you can ignore this message.  To find out more read "Trimming" in 
virt-v2v(1).

Original message: fstrim: fstrim: /sysroot/: the discard operation is not 
supported
[  21.2] Closing the overlay
[  21.3] Checking if the guest needs BIOS or UEFI to boot
virt-v2v: This guest requires UEFI on the target to boot.
[  21.3] Assigning disks to buses
[  21.3] Copying disk 1/1 to /var/lib/libvirt/images/esx6.7-win2019-x86_64-efi-sda (raw)
    (100.00/100%)
[ 440.1] Creating output metadata
Pool default refreshed

error: Failed to define domain from /tmp/v2vlibvirta8118d.xml
error: unsupported configuration: IDE controllers are unsupported for this QEMU binary or machine type

virt-v2v: warning: could not define libvirt domain.  The libvirt XML is 
still available in ‘/tmp/v2vlibvirta8118d.xml’.  Try running ‘virsh 
define /tmp/v2vlibvirta8118d.xml’ yourself instead.
[ 440.3] Finishing off

1.2 Modify /tmp/v2vlibvirta8118d.xml to change the IDE device to SATA,then define the guest by virsh
# vi /tmp/v2vlibvirta8118d.xml
# virsh define /tmp/v2vlibvirta8118d.xml
Domain esx6.7-win2019-x86_64-efi defined from /tmp/v2vlibvirta8118d.xml

1.3 Start guest and checkpoints of guest are passed

Scenario2:
2.1 Convert guests from VMware to RHV via vmx+ssh using virt-v2v 
# virt-v2v -i vmx -it ssh ssh://root.75.219//vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/rhel8-mxie-none-efi/rhel8-mxie-none-efi.vmx -o rhv -os 10.73.194.236:/home/nfs_export  -b ovirtmgmt
[   0.0] Opening the source -i vmx ssh://root.75.219//vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/rhel8-mxie-none-efi/rhel8-mxie-none-efi.vmx
[   0.3] Creating an overlay to protect the source from being modified
[   0.6] Initializing the target -o rhv -os 10.73.194.236:/home/nfs_export
[   1.0] Opening the overlay
[   5.8] Inspecting the overlay
[  14.8] Checking for sufficient free disk space in the guest
[  14.8] Estimating space required on target for each disk
[  14.8] Converting Red Hat Enterprise Linux 8.0 Beta (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[  82.3] Mapping filesystem data to avoid copying unused and blank areas
[  82.7] Closing the overlay
[  82.8] Checking if the guest needs BIOS or UEFI to boot
[  82.8] Assigning disks to buses
[  82.8] Copying disk 1/1 to /tmp/v2v.NARZx6/d46d664d-c030-42de-a8ee-ef109f8c3766/images/57d4f19e-b839-46c1-9e52-7a0d20e3fb97/c930c6e6-4294-496d-8652-e25313015d04 (raw)
    (100.00/100%)
[ 233.6] Creating output metadata
[ 233.9] Finishing off

2.2 Check if "scp -T " is used in v2v vmx+ssh conversion
# cat virt-v2v-vmx-ssh-debug.log |grep "scp -T"
LANG=C scp -T |& grep "unknown option"
scp -T 'root'@'10.73.75.219':''\''//vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/rhel8-mxie-none-efi/rhel8-mxie-none-efi.vmx'\''' '/var/tmp/vmx.ydUBBh/source.vmx'

2.3 Try to import the guest from export domain to data domain and power on guest,checkpoints of guest are passed


Scenario3:
3.1 Convert a guest from Xen to rhv by virt-v2v
 # virt-v2v -ic xen+ssh://root.3.21 xen-hvm-rhel6.9-x86_64 -of raw -o rhv -os 10.73.194.236:/home/nfs_export  -b ovirtmgmt
[   0.0] Opening the source -i libvirt -ic xen+ssh://root.3.21 xen-hvm-rhel6.9-x86_64
[   0.7] Creating an overlay to protect the source from being modified
[   1.2] Initializing the target -o rhv -os 10.73.194.236:/home/nfs_export
[   2.0] Opening the overlay
[   7.2] Inspecting the overlay
[  30.0] Checking for sufficient free disk space in the guest
[  30.0] Estimating space required on target for each disk
[  30.0] Converting Red Hat Enterprise Linux Server release 6.9 (Santiago) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 153.1] Mapping filesystem data to avoid copying unused and blank areas
[ 153.5] Closing the overlay
[ 153.6] Checking if the guest needs BIOS or UEFI to boot
[ 153.6] Assigning disks to buses
[ 153.6] Copying disk 1/1 to /tmp/v2v.bWRVHo/d46d664d-c030-42de-a8ee-ef109f8c3766/images/4ea0602b-fbd9-46da-94b1-43aeb1ac5457/71f150a7-bc06-418a-ac63-ad3b9c6a2d42 (raw)
    (100.00/100%)
[ 547.5] Creating output metadata
[ 547.6] Finishing off

3.2 Check if "scp -T " is used in v2v xen conversion
# cat v2v-xen-debug.log |grep "scp -T"
nothing

3.3 Try to import the guest from export domain to data domain and power on guest,checkpoints of guest are passed

Result:
  Virt-v2v only use ‘scp -T’ to convert guest via vmx+ssh now, so move the bug from ON_QA to VERIFIED

Comment 4 Karen Noel 2019-08-13 01:46:39 UTC
(In reply to Richard W.M. Jones from comment #1)
> Unfortunately I think we're going to need a z-stream here since the
> version of openssh which broke this is going into RHEL 8.1.0.

Are we all set now? This is 8.1.0. Thanks.

Comment 5 Pino Toscano 2019-08-13 05:10:40 UTC
(In reply to Karen Noel from comment #4)
> (In reply to Richard W.M. Jones from comment #1)
> > Unfortunately I think we're going to need a z-stream here since the
> > version of openssh which broke this is going into RHEL 8.1.0.
> 
> Are we all set now?

Yes.

Comment 6 Richard W.M. Jones 2019-08-13 07:40:58 UTC
Yes it's all good.  When I wrote that comment I thought (wrongly) that
there wasn't enough time to still add the patch to 8.1, but Pino has
done that now.

Comment 8 errata-xmlrpc 2019-11-05 20:51:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3345