RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1738886 - virt-v2v: Use scp -T in -i vmx -it ssh mode
Summary: virt-v2v: Use scp -T in -i vmx -it ssh mode
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: libguestfs
Version: 8.1
Hardware: x86_64
OS: Unspecified
urgent
urgent
Target Milestone: rc
: 8.1
Assignee: Pino Toscano
QA Contact: Virtualization Bugs
URL:
Whiteboard: V2V
Depends On: 1666581 1733168
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-08 11:12 UTC by Richard W.M. Jones
Modified: 2020-11-14 06:30 UTC (History)
13 users (show)

Fixed In Version: libguestfs-1.38.4-14.module+el8.1.0+3910+3a68cd83
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1733168
Environment:
Last Closed: 2019-11-05 20:51:11 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:3345 0 None None None 2019-11-05 20:51:39 UTC

Description Richard W.M. Jones 2019-08-08 11:12:29 UTC 
+++ This bug was initially created as a clone of Bug #1733168 +++

Description of problem:
Can't convert guest from VMware via vmx+ssh by virt-v2v due to openssh protocol  problem

Version-Release number of selected component (if applicable):
openssh-8.0p1-2.el8.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Convert a windows guest from VMware to KVM managed by libvirt via vmx+ssh using virt-v2v but the conversion is failed

# virt-v2v -i vmx -it ssh ssh://root.75.219/vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/esx6.7-win2019-x86_64-efi/esx6.7-win2019-x86_64-efi.vmx -of raw -o local -os /home
[   0.0] Opening the source -i vmx ssh://root.75.219/vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/esx6.7-win2019-x86_64-efi/esx6.7-win2019-x86_64-efi.vmx
protocol error: filename does not match request
virt-v2v: error: could not copy the VMX file from the remote server, see 
earlier error messages

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

2.Downgrade openssh from openssh-8.0p1-2.el8.x86_64 to openssh-7.8p1-4.el8.x86_64

#yum downgrade http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/openssh/7.8p1/4.el8/x86_64/openssh-7.8p1-4.el8.x86_64.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/openssh/7.8p1/4.el8/x86_64/openssh-server-7.8p1-4.el8.x86_64.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/openssh/7.8p1/4.el8/x86_64/openssh-clients-7.8p1-4.el8.x86_64.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/openssh/7.8p1/4.el8/x86_64/openssh-askpass-7.8p1-4.el8.x86_64.rpm -y

# rpm -q openssh
openssh-7.8p1-4.el8.x86_64

3.Convert a windows guest from VMware to KVM managed by libvirt via vmx+ssh using virt-v2v again, the conversion can be finished without error

# virt-v2v -i vmx -it ssh ssh://root.75.219/vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/esx6.7-win2019-x86_64-efi/esx6.7-win2019-x86_64-efi.vmx -of raw -o local -os /home
[   0.0] Opening the source -i vmx ssh://root.75.219/vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/esx6.7-win2019-x86_64-efi/esx6.7-win2019-x86_64-efi.vmx
[   0.6] Creating an overlay to protect the source from being modified
[   0.7] Initializing the target -o local -os /home
[   0.7] Opening the overlay
[   5.5] Inspecting the overlay
[  13.0] Checking for sufficient free disk space in the guest
[  13.0] Estimating space required on target for each disk
[  13.0] Converting Windows Server 2019 Standard to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.  
Firstboot scripts may conflict with PnP.
virt-v2v: warning: there is no QXL driver for this version of Windows (10.0 
x86_64).  virt-v2v looks for this driver in 
/usr/share/virtio-win/virtio-win.iso

The guest will be configured to use a basic VGA display driver.
virt-v2v: This guest has virtio drivers installed.
[  19.9] Mapping filesystem data to avoid copying unused and blank areas
virt-v2v: warning: fstrim on guest filesystem /dev/sda2 failed.  Usually 
you can ignore this message.  To find out more read "Trimming" in 
virt-v2v(1).

Original message: fstrim: fstrim: /sysroot/: the discard operation is not 
supported
[  20.4] Closing the overlay
[  20.5] Checking if the guest needs BIOS or UEFI to boot
virt-v2v: This guest requires UEFI on the target to boot.
[  20.5] Assigning disks to buses
[  20.5] Copying disk 1/1 to /home/esx6.7-win2019-x86_64-efi-sda (raw)
    (100.00/100%)
[ 448.6] Creating output metadata
[ 448.6] Finishing off


Actual results:


Expected results:
Can convert guest from VMware via vmx+ssh by virt-v2v with latest openssh


Additional info:

--- Additional comment from Jakub Jelen on 2019-07-25 10:17:57 UTC ---

Please, provide the debug logs. This sounds like consequence of fix for CVE-2019-6111. Try to pass the -T switch to the scp, if that is using scp.

--- Additional comment from Jakub Jelen on 2019-07-25 10:19:13 UTC ---

See related bug #1722229 which broke oracle for possible workarounds.

--- Additional comment from Richard W.M. Jones on 2019-07-25 10:25:36 UTC ---

Here are my comments from IRC:

10:50 < rjones> mxie: it's likely to be a quoting problem, so you could try copying a remote file with a space in the name
10:51 < rjones> and see if quoting it with "file' 'name" works or whatever
10:51 < rjones> or backslashes or quotes
10:52 < rjones> mxie: see also:
10:52 < rjones> https://github.com/libguestfs/libguestfs/blob/6d251e3828ff94deac7589b5892df174430e01f9/v2v/input_vmx.ml#L82
10:53 < rjones> and also https://stackoverflow.com/questions/19858176/how-to-escape-spaces-in-path-during-scp-copy-in-linux
10:53 < rjones> notice all the backslashes in the error message

--- Additional comment from Jakub Jelen on 2019-07-25 11:12:36 UTC ---

As already mentioned in the previous comments, there are two possible workarounds:

 * Use sftp for sensible file transfers.
 * Use -T switch to avoid these checks if you are using trusted servers

--- Additional comment from Richard W.M. Jones on 2019-07-25 11:19:18 UTC ---

sftp (command line tool) isn't going to work for us because it's interactive.
There's a batch mode (-b option) which may be worth looking into.  Using sftp
from libssh is also possible, but a rather huge change.  I need to make
sure that VMware's ssh actually supports sftp because it could be disabled.

We can easily add the -T option.  The server is a VMware server so I guess we
trust it.  It's at least always owned by the same people running virt-v2v.

--- Additional comment from Jakub Jelen on 2019-07-25 13:45:46 UTC ---

Richard, do you want me to change this bug to your component or can I close it as duplicate of #1722229?

--- Additional comment from Richard W.M. Jones on 2019-07-25 13:47:51 UTC ---

I don't really agree with breaking scp for everyone, but here we are.

--- Additional comment from Richard W.M. Jones on 2019-07-25 14:02:37 UTC ---

Patch posted:
https://www.redhat.com/archives/libguestfs/2019-July/msg00268.html

--- Additional comment from Jakub Jelen on 2019-07-25 15:10:14 UTC ---

Thank you. Technically, the upstream fix was trying to accommodate as much as possible use cases, but because of the flexibility of this "API" and many possible various bash expansions or quotes ignorance that could be used in the scp, it is really not possible to catch everything.

--- Additional comment from Richard W.M. Jones on 2019-08-08 11:09:16 UTC ---

Upstream in:
https://github.com/libguestfs/libguestfs/commit/7692c31494f7b1d37e380eed9eb99c5952940dbf

(Sorry, I kept the "lazy", I quite like it)
Comment 1 Richard W.M. Jones 2019-08-08 11:13:40 UTC 
Unfortunately I think we're going to need a z-stream here since the
version of openssh which broke this is going into RHEL 8.1.0.
Comment 3 mxie@redhat.com 2019-08-09 09:06:21 UTC 
Verify the bug with builds:
virt-v2v-1.38.4-14.module+el8.1.0+3910+3a68cd83.x86_64
libguestfs-1.38.4-14.module+el8.1.0+3910+3a68cd83.x86_64
libvirt-4.5.0-31.module+el8.1.0+3808+3325c1a3.x86_64
qemu-kvm-2.12.0-83.module+el8.1.0+3852+0ba8aef0.x86_64
libssh-0.9.0-4.el8.x86_64
openssh-8.0p1-3.el8.x86_64
kernel-4.18.0-128.el8.x86_64

Steps:
Scenario1:
1.1 Convert guests from VMware to kvm managed by libvirt via vmx+ssh using virt-v2v 
# virt-v2v -i vmx -it ssh ssh://root.75.219/vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/esx6.7-win2019-x86_64-efi/esx6.7-win2019-x86_64-efi.vmx 
[   0.0] Opening the source -i vmx ssh://root.75.219/vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/esx6.7-win2019-x86_64-efi/esx6.7-win2019-x86_64-efi.vmx
[   0.6] Creating an overlay to protect the source from being modified
[   0.8] Initializing the target -o libvirt -os default
[   0.9] Opening the overlay
[   5.9] Inspecting the overlay
[  13.4] Checking for sufficient free disk space in the guest
[  13.4] Estimating space required on target for each disk
[  13.4] Converting Windows Server 2019 Standard to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.  
Firstboot scripts may conflict with PnP.
virt-v2v: warning: there is no QXL driver for this version of Windows (10.0 
x86_64).  virt-v2v looks for this driver in 
/usr/share/virtio-win/virtio-win.iso

The guest will be configured to use a basic VGA display driver.
virt-v2v: This guest has virtio drivers installed.
[  20.8] Mapping filesystem data to avoid copying unused and blank areas
virt-v2v: warning: fstrim on guest filesystem /dev/sda2 failed.  Usually 
you can ignore this message.  To find out more read "Trimming" in 
virt-v2v(1).

Original message: fstrim: fstrim: /sysroot/: the discard operation is not 
supported
[  21.2] Closing the overlay
[  21.3] Checking if the guest needs BIOS or UEFI to boot
virt-v2v: This guest requires UEFI on the target to boot.
[  21.3] Assigning disks to buses
[  21.3] Copying disk 1/1 to /var/lib/libvirt/images/esx6.7-win2019-x86_64-efi-sda (raw)
    (100.00/100%)
[ 440.1] Creating output metadata
Pool default refreshed

error: Failed to define domain from /tmp/v2vlibvirta8118d.xml
error: unsupported configuration: IDE controllers are unsupported for this QEMU binary or machine type

virt-v2v: warning: could not define libvirt domain.  The libvirt XML is 
still available in ‘/tmp/v2vlibvirta8118d.xml’.  Try running ‘virsh 
define /tmp/v2vlibvirta8118d.xml’ yourself instead.
[ 440.3] Finishing off

1.2 Modify /tmp/v2vlibvirta8118d.xml to change the IDE device to SATA,then define the guest by virsh
# vi /tmp/v2vlibvirta8118d.xml
# virsh define /tmp/v2vlibvirta8118d.xml
Domain esx6.7-win2019-x86_64-efi defined from /tmp/v2vlibvirta8118d.xml

1.3 Start guest and checkpoints of guest are passed

Scenario2:
2.1 Convert guests from VMware to RHV via vmx+ssh using virt-v2v 
# virt-v2v -i vmx -it ssh ssh://root.75.219//vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/rhel8-mxie-none-efi/rhel8-mxie-none-efi.vmx -o rhv -os 10.73.194.236:/home/nfs_export  -b ovirtmgmt
[   0.0] Opening the source -i vmx ssh://root.75.219//vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/rhel8-mxie-none-efi/rhel8-mxie-none-efi.vmx
[   0.3] Creating an overlay to protect the source from being modified
[   0.6] Initializing the target -o rhv -os 10.73.194.236:/home/nfs_export
[   1.0] Opening the overlay
[   5.8] Inspecting the overlay
[  14.8] Checking for sufficient free disk space in the guest
[  14.8] Estimating space required on target for each disk
[  14.8] Converting Red Hat Enterprise Linux 8.0 Beta (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[  82.3] Mapping filesystem data to avoid copying unused and blank areas
[  82.7] Closing the overlay
[  82.8] Checking if the guest needs BIOS or UEFI to boot
[  82.8] Assigning disks to buses
[  82.8] Copying disk 1/1 to /tmp/v2v.NARZx6/d46d664d-c030-42de-a8ee-ef109f8c3766/images/57d4f19e-b839-46c1-9e52-7a0d20e3fb97/c930c6e6-4294-496d-8652-e25313015d04 (raw)
    (100.00/100%)
[ 233.6] Creating output metadata
[ 233.9] Finishing off

2.2 Check if "scp -T " is used in v2v vmx+ssh conversion
# cat virt-v2v-vmx-ssh-debug.log |grep "scp -T"
LANG=C scp -T |& grep "unknown option"
scp -T 'root'@'10.73.75.219':''\''//vmfs/volumes/5aefd41e-1d448cf8-0b1f-001018d0c8e0/rhel8-mxie-none-efi/rhel8-mxie-none-efi.vmx'\''' '/var/tmp/vmx.ydUBBh/source.vmx'

2.3 Try to import the guest from export domain to data domain and power on guest,checkpoints of guest are passed


Scenario3:
3.1 Convert a guest from Xen to rhv by virt-v2v
 # virt-v2v -ic xen+ssh://root.3.21 xen-hvm-rhel6.9-x86_64 -of raw -o rhv -os 10.73.194.236:/home/nfs_export  -b ovirtmgmt
[   0.0] Opening the source -i libvirt -ic xen+ssh://root.3.21 xen-hvm-rhel6.9-x86_64
[   0.7] Creating an overlay to protect the source from being modified
[   1.2] Initializing the target -o rhv -os 10.73.194.236:/home/nfs_export
[   2.0] Opening the overlay
[   7.2] Inspecting the overlay
[  30.0] Checking for sufficient free disk space in the guest
[  30.0] Estimating space required on target for each disk
[  30.0] Converting Red Hat Enterprise Linux Server release 6.9 (Santiago) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 153.1] Mapping filesystem data to avoid copying unused and blank areas
[ 153.5] Closing the overlay
[ 153.6] Checking if the guest needs BIOS or UEFI to boot
[ 153.6] Assigning disks to buses
[ 153.6] Copying disk 1/1 to /tmp/v2v.bWRVHo/d46d664d-c030-42de-a8ee-ef109f8c3766/images/4ea0602b-fbd9-46da-94b1-43aeb1ac5457/71f150a7-bc06-418a-ac63-ad3b9c6a2d42 (raw)
    (100.00/100%)
[ 547.5] Creating output metadata
[ 547.6] Finishing off

3.2 Check if "scp -T " is used in v2v xen conversion
# cat v2v-xen-debug.log |grep "scp -T"
nothing

3.3 Try to import the guest from export domain to data domain and power on guest,checkpoints of guest are passed

Result:
  Virt-v2v only use ‘scp -T’ to convert guest via vmx+ssh now, so move the bug from ON_QA to VERIFIED
Comment 4 Karen Noel 2019-08-13 01:46:39 UTC 
(In reply to Richard W.M. Jones from comment #1)
> Unfortunately I think we're going to need a z-stream here since the
> version of openssh which broke this is going into RHEL 8.1.0.

Are we all set now? This is 8.1.0. Thanks.
Comment 5 Pino Toscano 2019-08-13 05:10:40 UTC 
(In reply to Karen Noel from comment #4)
> (In reply to Richard W.M. Jones from comment #1)
> > Unfortunately I think we're going to need a z-stream here since the
> > version of openssh which broke this is going into RHEL 8.1.0.
> 
> Are we all set now?

Yes.
Comment 6 Richard W.M. Jones 2019-08-13 07:40:58 UTC 
Yes it's all good.  When I wrote that comment I thought (wrongly) that
there wasn't enough time to still add the patch to 8.1, but Pino has
done that now.
Comment 8 errata-xmlrpc 2019-11-05 20:51:11 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3345
Note You need to log in before you can comment on or make changes to this bug.