RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2158300 - Improve error message when PasswordAuthentication is set to 'no' on VMware server
Summary: Improve error message when PasswordAuthentication is set to 'no' on VMware se...
Keywords:
Status: CLOSED ERRATA
Alias: None
Deadline: 2023-01-16
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: nbdkit
Version: 9.2
Hardware: x86_64
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Richard W.M. Jones
QA Contact: mxie@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-01-05 02:35 UTC by mxie@redhat.com
Modified: 2023-05-09 09:06 UTC (History)
12 users (show)

Fixed In Version: nbdkit-1.32.5-4.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-05-09 07:47:50 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-143983 0 None None None 2023-01-05 02:52:08 UTC
Red Hat Product Errata RHBA-2023:2347 0 None None None 2023-05-09 07:48:11 UTC

Description mxie@redhat.com 2023-01-05 02:35:42 UTC
Description of problem:
Virt-v2v can't convert guest from ESXi8.0 host via vmx+ssh

Version-Release number of selected component (if applicable):
virt-v2v-2.0.7-7.el9.x86_64
libguestfs-1.48.4-4.el9.x86_64
guestfs-tools-1.48.2-8.el9.x86_64
libvirt-libs-8.10.0-2.el9.x86_64
qemu-img-7.2.0-3.el9.x86_64
nbdkit-server-1.32.5-1.el9.x86_64
libnbd-1.14.2-1.el9.x86_64


How reproducible:
100%

Steps to Reproduce:
1.Convert a guest from ESXi8.0 host via vmx+ssh by v2v
# virt-v2v -i vmx -it ssh ssh://root.212.36/vmfs/volumes/esx8.0-matrix/esx8.0-rhel9.2-x86_64/esx8.0-rhel9.2-x86_64.vmx -ip /home/esxpwd
[   0.0] Setting up the source: -i vmx ssh://root.212.36/vmfs/volumes/esx8.0-matrix/esx8.0-rhel9.2-x86_64/esx8.0-rhel9.2-x86_64.vmx
(root.212.36) Password: 
(root.212.36) Password: 
[   6.3] Opening the source
nbdkit: ssh[1]: error: all possible authentication methods failed
virt-v2v: error: libguestfs error: could not create appliance through 
libvirt.

Try running qemu directly without libvirt using this environment variable:
export LIBGUESTFS_BACKEND=direct

Original error from libvirt: internal error: process exited while 
connecting to monitor: 2023-01-04T15:56:55.558596Z qemu-kvm: -blockdev 
{"driver":"nbd","server":{"type":"unix","path":"/tmp/v2v.rewGKS/in0"},"node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":true},"auto-read-only":true,"discard":"unmap"}: 
Requested export not available [code=1 int1=-1]

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]


Actual results:
As above description

Expected results:
v2v can convert guest from ESXi8.0 host via vmx+ssh

Additional info:
v2v can convert guest from ESXi7.0 host via vmx+ssh
# virt-v2v -i vmx -it ssh ssh://root.212.38/vmfs/volumes/esx7.0-matrix/esx7.0-rhel9.2-x86_64/esx7.0-rhel9.2-x86_64.vmx -ip /home/esxpwd -o null
[   0.0] Setting up the source: -i vmx ssh://root.212.38/vmfs/volumes/esx7.0-matrix/esx7.0-rhel9.2-x86_64/esx7.0-rhel9.2-x86_64.vmx
(root.212.38) Password: 
(root.212.38) Password: 
[   7.0] Opening the source
[  11.6] Inspecting the source
[  18.5] Checking for sufficient free disk space in the guest
[  18.5] Converting Red Hat Enterprise Linux 9.2 Beta (Plow) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 121.4] Mapping filesystem data to avoid copying unused and blank areas
[ 122.5] Closing the overlay
[ 122.8] Assigning disks to buses
[ 122.8] Checking if the guest needs BIOS or UEFI to boot
[ 122.8] Setting up the destination: -o null
[ 124.0] Copying disk 1/1
█ 100% [****************************************]
[ 217.5] Creating output metadata
[ 217.5] Finishing off

Comment 2 mxie@redhat.com 2023-01-05 10:00:39 UTC
Can't reproduce the bug on rhel8.8 

virt-v2v-1.42.0-21.module+el8.8.0+16781+9f4724c2.x86_64
libguestfs-1.44.0-9.module+el8.8.0+16781+9f4724c2.x86_64
nbdkit-server-1.24.0-5.module+el8.8.0+17308+05924798.x86_64
libnbd-1.6.0-5.module+el8.8.0+16781+9f4724c2.x86_64
libvirt-libs-8.0.0-13.module+el8.8.0+17719+f18c2d1b.x86_64
qemu-img-6.2.0-28.module+el8.8.0+17721+63bacee8.x86_64

#  virt-v2v -i vmx -it ssh ssh://root.212.36/vmfs/volumes/esx8.0-matrix/esx8.0-rhel9.2-x86_64/esx8.0-rhel9.2-x86_64.vmx
[   0.0] Opening the source -i vmx ssh://root.212.36/vmfs/volumes/esx8.0-matrix/esx8.0-rhel9.2-x86_64/esx8.0-rhel9.2-x86_64.vmx
[   1.6] Creating an overlay to protect the source from being modified
[   1.9] Opening the overlay
[  65.2] Inspecting the overlay
[  72.4] Checking for sufficient free disk space in the guest
[  72.4] Estimating space required on target for each disk
[  72.4] Converting Red Hat Enterprise Linux 9.2 Beta (Plow) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 174.6] Mapping filesystem data to avoid copying unused and blank areas
[ 175.2] Closing the overlay
[ 175.5] Assigning disks to buses
[ 175.5] Checking if the guest needs BIOS or UEFI to boot
[ 175.5] Initializing the target -o libvirt -os default
[ 175.6] Copying disk 1/1 to /var/lib/libvirt/images/esx8.0-rhel9.2-x86_64-sda (raw)
    (100.00/100%)
[ 350.5] Creating output metadata
[ 350.6] Finishing off

Comment 4 Richard W.M. Jones 2023-01-05 10:55:47 UTC
The reason is shown in the debug output:

  nbdkit: ssh[1]: debug: authentication methods offered by the server [0x14]:  publickey keyboard-interactive

Because we're trying password authentication, not SSH agent ("publickey")
we would expect the server to offer something more like this:

  nbdkit: ssh[1]: debug: authentication methods offered by the server [0x26]:  password publickey gssapi-with-mic

where we could use "password" to do password authentication.

One way to fix this would be to edit /etc/ssh/sshd_config (on the
server) and change:

  PasswordAuthentication yes  # originally set to "no"

Note this is already mentioned in the documentation:
https://libguestfs.org/virt-v2v-input-vmware.1.html#vmx:-access-to-the-storage-containing-the-vmx-and-vmdk-files

We might make this a bit clearer, for example by including the
actual error and debug messages in the docs.

The reason it works for RHEL 8.8 is a bit mysterious.  However I think
we were using the qemu ssh client (instead of nbdkit) so maybe it behaves
differently, or maybe you have SSH agent set up on that client.

Comment 5 Richard W.M. Jones 2023-01-05 11:00:56 UTC
> We might make this a bit clearer, for example by including the
actual error and debug messages in the docs.

Or the error from nbdkit could be better too ...

Comment 6 Richard W.M. Jones 2023-01-05 11:35:21 UTC
Potential improvement to nbdkit error message:

https://listman.redhat.com/archives/libguestfs/2023-January/030448.html

Comment 7 Richard W.M. Jones 2023-01-05 15:09:44 UTC
Changing the component because the proposed fix is in nbdkit.

Comment 10 mxie@redhat.com 2023-01-09 08:14:17 UTC
Test the bug with below builds:
virt-v2v-2.0.7-7.el9.x86_64
nbdkit-server-1.32.5-2.el9.x86_64

Steps:
1.Log into the VMware server to set "PasswordAuthentication no".
#cat /etc/ssh/sshd_config |grep PasswordAuthentication
PasswordAuthentication no

2.Convert a guest from VMware server by v2v with SSH password authentication
# virt-v2v -i vmx -it ssh ssh://root.212.36/vmfs/volumes/esx8.0-matrix/esx8.0-rhel9.2-x86_64/esx8.0-rhel9.2-x86_64.vmx -ip /home/passwd  -o null 
[   0.0] Setting up the source: -i vmx ssh://root.212.36/vmfs/volumes/esx8.0-matrix/esx8.0-rhel9.2-x86_64/esx8.0-rhel9.2-x86_64.vmx
(root.212.36) Password: 
(root.212.36) Password: 
[   6.3] Opening the source
nbdkit: ssh[1]: error: the server does not offer password authentication but you tried to use a password; if you have root access to the server, try editing 'sshd_config' and setting 'PasswordAuthentication yes'; otherwise try setting up public key authentication
virt-v2v: error: libguestfs error: could not create appliance through 
libvirt.

Try running qemu directly without libvirt using this environment variable:
export LIBGUESTFS_BACKEND=direct

Original error from libvirt: internal error: process exited while 
connecting to monitor: 2023-01-09T08:06:17.007683Z qemu-kvm: -blockdev 
{"driver":"nbd","server":{"type":"unix","path":"/tmp/v2v.dQxsPi/in0"},"node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":true},"auto-read-only":true,"discard":"unmap"}: 
Requested export not available [code=1 int1=-1]

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

3.Setup ssh-agent with VMware server
#cat ~/.ssh/id_rsa.pub | ssh root.212.36 "cat >> /etc/ssh/keys-root/authorized_keys"
#eval `ssh-agent`
#ssh-add

4.Convert a guest from VMware server by v2v with ssh-agent authentication
# virt-v2v -i vmx -it ssh ssh://root.212.36/vmfs/volumes/esx8.0-matrix/esx8.0-rhel9.2-x86_64/esx8.0-rhel9.2-x86_64.vmx  -o null 
[   0.0] Setting up the source: -i vmx ssh://root.212.36/vmfs/volumes/esx8.0-matrix/esx8.0-rhel9.2-x86_64/esx8.0-rhel9.2-x86_64.vmx
[   1.7] Opening the source
[   6.3] Inspecting the source
[  13.2] Checking for sufficient free disk space in the guest
[  13.2] Converting Red Hat Enterprise Linux 9.2 Beta (Plow) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 114.7] Mapping filesystem data to avoid copying unused and blank areas
[ 115.9] Closing the overlay
[ 116.2] Assigning disks to buses
[ 116.2] Checking if the guest needs BIOS or UEFI to boot
[ 116.2] Setting up the destination: -o null
[ 117.4] Copying disk 1/1
█ 100% [****************************************]
[ 216.7] Creating output metadata
[ 216.7] Finishing off

Result:
   The error message is clearer when ssh password authentication method fails

Comment 13 mxie@redhat.com 2023-01-11 06:47:58 UTC
Verify the bug with below builds, test steps are same with comment10, no new problem was found, move the bug from ON_QA to VERIFIED
virt-v2v-2.2.0-1.el9.x86_64
nbdkit-1.32.5-3.el9.x86_64

Comment 15 errata-xmlrpc 2023-05-09 07:47:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (nbdkit bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2347


Note You need to log in before you can comment on or make changes to this bug.