RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1841820 - RFE: nbdkit-retry-filter doesn't work in vmx+ssh and esx7.0+vddk v2v conversion, should support retrying open
Summary: RFE: nbdkit-retry-filter doesn't work in vmx+ssh and esx7.0+vddk v2v conversi...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: nbdkit
Version: unspecified
Hardware: x86_64
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Eric Blake
QA Contact: mxie@redhat.com
URL:
Whiteboard:
Depends On: 2168629
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-29 14:47 UTC by mxie@redhat.com
Modified: 2023-11-07 09:26 UTC (History)
10 users (show)

Fixed In Version: nbdkit-1.33.11-1.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-11-07 08:28:48 UTC
Type: Feature Request
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
vmx+ssh-nbdkit-retry-filter.log (2.08 MB, text/plain)
2020-05-29 14:47 UTC, mxie@redhat.com 		no flags Details
disable-ssh-esxi7.0.png (18.56 KB, image/png)
2020-05-29 14:47 UTC, mxie@redhat.com 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2023:6374 0 None None None 2023-11-07 08:29:02 UTC

Description mxie@redhat.com 2020-05-29 14:47:18 UTC 
Created attachment 1693400 [details]
vmx+ssh-nbdkit-retry-filter.log

Description of problem:
nbdkit-retry-filter doesn't work in vmx+ssh v2v conversion

Version-Release number of selected component (if applicable):
virt-v2v-1.42.0-4.module+el8.3.0+6798+ad6e66be.x86_64
libguestfs-1.42.0-2.module+el8.3.0+6798+ad6e66be.x86_64
libvirt-6.3.0-1.module+el8.3.0+6478+69f490bb.x86_64
qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420.x86_64
nbdkit-1.20.2-1.module+el8.3.0+6764+cc503f20.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Convert a guest from VMX via SSH by v2v
# virt-v2v -i vmx -it ssh ssh://root.199.217/vmfs/volumes/esx7.0-matrix/esx7.0-ubuntu18.04-x86_64/esx7.0-ubuntu18.04-x86_64.vmx -v -x 

2. Disable SSH function on ESXi host before v2v copying disk, pls refer to screenshot"disable-ssh-esxi7.0.png"

3. Found v2v conversion will fail during copying disk without any retrying, details pls refer to 'vmx+ssh-nbdkit-retry-filter.log'
......
[  62.2] Copying disk 1/1 to /var/lib/libvirt/images/esx7.0-ubuntu18.04-x86_64-sda (raw)
......
qemu-img 'convert' '-p' '-n' '-f' 'qcow2' '-O' 'raw' '-S' '64k' '/var/tmp/v2vovlf07f8e.qcow2' '/var/lib/libvirt/images/esx7.0-ubuntu18.04-x86_64-sda'
nbdkit: debug: accepted connection
......
nbdkit: ssh[3]: debug: cacheextents: open readonly=1
nbdkit: ssh[3]: debug: readahead: open readonly=1
nbdkit: ssh[3]: debug: retry: open readonly=1
nbdkit: ssh[3]: debug: ssh: open readonly=1
nbdkit: ssh[3]: error: failed to connect to remote host: 10.73.199.217: Connection refused
nbdkit: ssh[3]: debug: ssh: open returned handle (nil)
nbdkit: ssh[3]: debug: retry: open returned handle (nil)
nbdkit: ssh[3]: debug: readahead: open returned handle (nil)
nbdkit: ssh[3]: debug: cacheextents: open returned handle (nil)
nbdkit: ssh[3]: debug: write: NBD_OPT_ABORT: Broken pipe
qemu-img: Could not open '/var/tmp/v2vovlf07f8e.qcow2': Could not open backing file: Requested export not available

virt-v2v: error: qemu-img command failed, see earlier errors


Actual results:
As above description

Expected results:
nbdkit-retry-filter works well in vmx+ssh v2v conversion

Additional info:
Comment 1 mxie@redhat.com 2020-05-29 14:47:57 UTC 
Created attachment 1693401 [details]
disable-ssh-esxi7.0.png
Comment 2 Richard W.M. Jones 2020-05-29 15:29:02 UTC 
> 2. Disable SSH function on ESXi host before v2v copying disk, pls refer to screenshot"disable-ssh-esxi7.0.png"

I'm fairly certain what happens here is the following:

(a) The (earlier) conversion step has opened an SSH connection.  Disabling
the SSH server does not stop existing SSH connections, so the conversion
step keeps running to completion.

(b) mxie disables the SSH server, step 2. above.

(c) After conversion completes, the copying step starts.  This tries to open
a new SSH connection to the server and this is what fails.

Note that the retry filter doesn't retry failed opens (only
failed data operations like read/write), as you can see from
the following simple test:

  nbdkit -v -U - --filter=retry sh - --run 'qemu-img info $nbd' <<'EOF'
  case "$1" in
       open) echo EIO >&2; exit 1 ;;
       get_size) echo 1M ;;
       pread) dd if=/dev/zero count=$3 iflag=count_bytes ;;
       *) exit 2 ;;
  esac
  EOF

(note the open method returns an error).  It prints a log very similar to yours,
ending in:

nbdkit: sh[1]: error: /tmp/nbdkitbhYyrx/inline-script.sh: 
nbdkit: sh[1]: debug: sh: open returned handle (nil)
nbdkit: sh[1]: debug: retry: open returned handle (nil)
nbdkit: sh[1]: debug: write: NBD_OPT_ABORT: Broken pipe
qemu-img: Could not open 'nbd:unix:/tmp/nbdkitpDSwwy/socket': Requested export not available

Since this is essentially something which nbdkit-retry-filter does not
currently implement, I'm going to add "RFE" to the summary line.
Comment 5 Eric Hadley 2021-09-08 16:45:07 UTC 
Bulk update: Move RHEL-AV bugs to RHEL9. If necessary to resolve in RHEL8, then clone to the current RHEL8 release.
Comment 7 RHEL Program Management 2021-11-29 07:27:03 UTC 
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.
Comment 8 Richard W.M. Jones 2021-11-29 10:22:42 UTC 
This bug was closed in error by a process we have no control over.
Comment 9 mxie@redhat.com 2022-02-22 14:39:26 UTC 
I think retry filter also doesn't work when convert guest from ESXi7.0 host via vddk and copying disk has not started in v2v conversion

Package versions:
virt-v2v-1.45.99-1.el9.x86_64
libguestfs-1.46.1-2.el9.x86_64
guestfs-tools-1.46.1-6.el9.x86_64
nbdkit-1.28.5-1.el9.x86_64
libnbd-1.10.5-1.el9.x86_64
libvirt-libs-8.0.0-4.el9.x86_64
qemu-img-6.2.0-9.el9.x86_64

Steps:
1.Convert a guest from ESXi7.0 via vddk by v2v
# virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.2 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78  -ip /home/passwd esx7.0-win2022-x86_64 -v -x 

2.Disconnect the ESXi7.0 host on vsphere client before copying disk has not started in v2v conversion

3.Then v2v conversion will be failed with below error, there is no any info about retrying reconnecting in v2v debug log
.....
nbdkit: vddk[2]: debug: vddk: open returned handle (nil)
nbdkit: vddk[2]: debug: retry: open returned handle (nil)
nbdkit: vddk[2]: debug: cacheextents: open returned handle (nil)
nbdkit: vddk[2]: debug: blocksize: open returned handle (nil)
nbdkit: vddk[2]: debug: cow: open returned handle (nil)
nbdkit: vddk[2]: debug: multi-conn: open returned handle (nil)
libnbd: debug: nbd1: nbd_connect_unix: poll end: r=1 revents=1
libnbd: debug: nbd1: nbd_connect_unix: transition: NEWSTYLE.OPT_GO.RECV_REPLY -> NEWSTYLE.OPT_GO.RECV_REPLY_PAYLOAD
libnbd: debug: nbd1: nbd_connect_unix: transition: NEWSTYLE.OPT_GO.RECV_REPLY_PAYLOAD -> NEWSTYLE.OPT_GO.CHECK_REPLY
libnbd: debug: nbd1: nbd_connect_unix: transition: NEWSTYLE.OPT_GO.CHECK_REPLY -> NEWSTYLE.PREPARE_OPT_ABORT
libnbd: debug: nbd1: nbd_connect_unix: transition: NEWSTYLE.PREPARE_OPT_ABORT -> NEWSTYLE.SEND_OPT_ABORT
nbdkit: vddk[2]: debug: client sent NBD_OPT_ABORT to abort the connection
libnbd: debug: nbd1: nbd_connect_unix: transition: NEWSTYLE.SEND_OPT_ABORT -> NEWSTYLE.SEND_OPTION_SHUTDOWN
libnbd: debug: nbd1: nbd_connect_unix: transition: NEWSTYLE.SEND_OPTION_SHUTDOWN -> DEAD
libnbd: debug: nbd1: nbd_connect_unix: leave: error="nbd_connect_unix: handshake: server has no export named '': No such file or directory"
libnbd: debug: nbd1: nbd_close: closing handle
virt-v2v: error: exception: NBD.Error("nbd_connect_unix: handshake: server 
has no export named '': No such file or directory", 2)


Additional info：
retry filter works if convert guest from ESXi6.7 host via vddk
Comment 12 John Ferlan 2022-09-02 19:42:53 UTC 
Eric - assigning directly to you for handling. This bug has been open for a while and we need to make a decision whether or not to add the feature.  I've placed 9.2.0 in the ITR as a way to try and force that decision. If the decision is to not do this, we should just close.
Comment 14 John Ferlan 2022-12-14 12:31:27 UTC 
Updating the stale date to the end of ITM 26 for RHEL 9.2.0
Comment 15 Eric Blake 2023-01-27 20:42:26 UTC 
Upstream patch proposed:
https://listman.redhat.com/archives/libguestfs/2023-January/030531.html
Now the question is whether this is important enough to backport.
Comment 16 Richard W.M. Jones 2023-02-06 09:48:02 UTC 
Moving to RHEL 9.3
Comment 18 mxie@redhat.com 2023-03-29 12:51:26 UTC 
Test the bug with below builds:
nbdkit-server-1.33.11-1.el9.x86_64
libnbd-1.15.12-1.el9.x86_64
virt-v2v-2.2.0-5.el9.x86_64
libguestfs-1.50.1-3.el9.x86_64
guestfs-tools-1.48.2-8.el9.x86_64


Steps:
Scenario1:
1.1 Convert a guest from VMX via SSH by v2v
# virt-v2v -i vmx -it ssh ssh://root.225.34/vmfs/volumes/datastore2/esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmx -ip /home/esxpwd -v -x

1.2 Disable SSH function on ESXi host before v2v copying disk

1.3 Found v2v conversion will fail after retrying 5 times
# virt-v2v -i vmx -it ssh ssh://root.225.34/vmfs/volumes/datastore2/esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmx -ip /home/esxpwd -v -x
.....
nbdkit: ssh[2]: error: failed to connect to remote host: 10.73.225.34: Connection refused
nbdkit: ssh[2]: debug: ssh: open returned handle (nil)
nbdkit: ssh[2]: debug: open failed: original errno = 108
nbdkit: ssh[2]: debug: retry 1: waiting 2 seconds before retrying
.....
.....
nbdkit: ssh[2]: error: failed to connect to remote host: 10.73.225.34: Connection refused
nbdkit: ssh[2]: debug: ssh: open returned handle (nil)
nbdkit: ssh[2]: debug: open failed: original errno = 108
nbdkit: ssh[2]: debug: retry 5: waiting 32 seconds before retrying
nbdkit: ssh[2]: debug: ssh: open readonly=1 exportname="" tls=0
nbdkit: ssh[2]: debug: ssh: default_export readonly=1 tls=0
nbdkit: ssh[2]: error: failed to connect to remote host: 10.73.225.34: Connection refused
nbdkit: ssh[2]: debug: ssh: open returned handle (nil)
nbdkit: ssh[2]: debug: could not recover after 5 retries
.....
libnbd: debug: nbd1: nbd_close: closing handle
virt-v2v: error: exception: NBD.Error("nbd_connect_unix: handshake: server 
has no export named '': No such file or directory", _)
.....
.....


Scenario2:
2.1 Convert a guest from VMX via SSH by v2v
# virt-v2v -i vmx -it ssh ssh://root.225.34/vmfs/volumes/datastore2/esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmx -ip /home/esxpwd -v -x

2.2 Disable SSH function on ESXi host before copying disk, v2v conversion can be continued if recover ssh connection before retrying 5th times
# virt-v2v -i vmx -it ssh ssh://root.225.34/vmfs/volumes/datastore2/esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmx -ip /home/esxpwd -v -x
....
....
nbdkit: ssh[2]: error: failed to connect to remote host: 10.73.225.34: Connection refused
nbdkit: ssh[2]: debug: ssh: open returned handle (nil)
nbdkit: ssh[2]: debug: open failed: original errno = 108
nbdkit: ssh[2]: debug: retry 1: waiting 2 seconds before retrying
nbdkit: ssh[2]: debug: ssh: open readonly=1 exportname="" tls=0
nbdkit: ssh[2]: debug: ssh: default_export readonly=1 tls=0
nbdkit: ssh[2]: error: failed to connect to remote host: 10.73.225.34: Connection refused
nbdkit: ssh[2]: debug: ssh: open returned handle (nil)
nbdkit: ssh[2]: debug: open failed: original errno = 108
nbdkit: ssh[2]: debug: retry 2: waiting 4 seconds before retrying
nbdkit: ssh[2]: debug: ssh: open readonly=1 exportname="" tls=0
nbdkit: ssh[2]: debug: ssh: default_export readonly=1 tls=0
nbdkit: ssh[2]: error: failed to connect to remote host: 10.73.225.34: Connection refused
nbdkit: ssh[2]: debug: ssh: open returned handle (nil)
nbdkit: ssh[2]: debug: open failed: original errno = 108
nbdkit: ssh[2]: debug: retry 3: waiting 8 seconds before retrying
nbdkit: ssh[2]: debug: ssh: open readonly=1 exportname="" tls=0
nbdkit: ssh[2]: debug: ssh: default_export readonly=1 tls=0
nbdkit: ssh[2]: debug: authentication methods offered by the server [0x16]:  password publickey keyboard-interactive
nbdkit: ssh[2]: debug: opened libssh handle
nbdkit: ssh[2]: debug: ssh: open returned handle 0x7f5a14048e00
nbdkit: ssh[2]: debug: ssh: prepare readonly=1
nbdkit: ssh[2]: debug: retry: open returned handle 0x7f5a14022c10
nbdkit: ssh[2]: debug: cacheextents: open returned handle 0x7f5a1a4cd4b8
nbdkit: ssh[2]: debug: cow: open returned handle 0x7f5a1a4cd4b8
nbdkit: ssh[2]: debug: retry: prepare readonly=1
.....

2.3 Disable SSH function on ESXi host during copying disk

2.4 v2v conversion can finish successfully, there is no any error about 'ssh[2]: error: failed to connect to remote host' during conversion
# virt-v2v -i vmx -it ssh ssh://root.225.34/vmfs/volumes/datastore2/esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmx -ip /home/esxpwd -v -x
.....
.....
[ 250.2] Finishing off
.....


Scenario3: 
3.1 Convert a guest from ESXi7.0 via vddk by v2v
# virt-v2v  -ic vpx://vsphere.local%5cAdministrator.227.27/data/10.73.225.34/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 esx7.0-win11-x86_64 -ip /home/passwd -v -x

3.2 Disconnect the ESXi7.0 host on vsphere client before copying disk 

3.3 Found v2v conversion will fail after retrying 5 times
virt-v2v  -ic vpx://vsphere.local%5cAdministrator.227.27/data/10.73.225.34/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 esx7.0-win11-x86_64 -ip /home/passwd -v -x
.....
.....
nbdkit: vddk[2]: error: VixDiskLib_Open: [datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk: You do not have access rights to this file
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_FreeErrorText ()
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_Disconnect (connection)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_Disconnect: Disconnect.
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_FreeConnectParams (params)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
nbdkit: vddk[2]: debug: vddk: open returned handle (nil)
nbdkit: vddk[2]: debug: open failed: original errno = 108
nbdkit: vddk[2]: debug: retry 1: waiting 2 seconds before retrying
.....
nbdkit: vddk[2]: debug: retry 5: waiting 32 seconds before retrying
.....
nbdkit: vddk[2]: error: VixDiskLib_Open: [datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk: Unknown error
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_FreeErrorText ()
nbdkit: vddk[2]: error: Please verify whether the "thumbprint" parameter (76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21) matches the SHA1 fingerprint of the remote VMware server.  Refer to nbdkit-vddk-plugin(1) section "THUMBPRINTS" for details.
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_Disconnect (connection)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_Disconnect: Disconnect.
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_FreeConnectParams (params)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
nbdkit: vddk[2]: debug: vddk: open returned handle (nil)
nbdkit: vddk[2]: debug: could not recover after 5 retries
.....
.....


Scenario4:
4.1 Convert a guest from ESXi7.0 via vddk by v2v
# virt-v2v  -ic vpx://vsphere.local%5cAdministrator.227.27/data/10.73.225.34/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 esx7.0-win11-x86_64 -ip /home/passwd -v -x

4.2 Disconnect ESXi host before copying disk, v2v conversion can be continued if recover connection before retrying 5th times
# virt-v2v  -ic vpx://vsphere.local%5cAdministrator.227.27/data/10.73.225.34/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 esx7.0-win11-x86_64 -ip /home/passwd -v -x
....
....
nbdkit: vddk[2]: error: VixDiskLib_Open: [datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk: Unknown error
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_FreeErrorText ()
nbdkit: vddk[2]: error: Please verify whether the "thumbprint" parameter (76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21) matches the SHA1 fingerprint of the remote VMware server.  Refer to nbdkit-vddk-plugin(1) section "THUMBPRINTS" for details.
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_Disconnect (connection)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_Disconnect: Disconnect.
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_FreeConnectParams (params)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
nbdkit: vddk[2]: debug: vddk: open returned handle (nil)
nbdkit: vddk[2]: debug: open failed: original errno = 108
nbdkit: vddk[2]: debug: retry 3: waiting 8 seconds before retrying
nbdkit: vddk[2]: debug: vddk: open readonly=1 exportname="" tls=0
nbdkit: vddk[2]: debug: vddk: default_export readonly=1 tls=0
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_AllocateConnectParams ()
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_ConnectEx (h->params, 1, NULL, NULL, &connection)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_ConnectEx: Establish connection using (null).
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLibConnectInt: Establish connection.
nbdkit: vddk[2]: debug: VixDiskLib: Resolve host.
....
nbdkit: vddk[2]: debug: vddk: open returned handle 0x7f36fc0553a0
nbdkit: vddk[2]: debug: vddk: prepare readonly=1
.....

4.3 Disconnect the ESXi7.0 host on vsphere client during copying disk 

4.4 Virt-v2v conversion can finish successfully
#virt-v2v  -ic vpx://vsphere.local%5cAdministrator.227.27/data/10.73.225.34/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 esx7.0-win11-x86_64 -ip /home/passwd -v -x
....
....
[ 246.2] Finishing off


Result:
  The bug has been fixed
Comment 21 mxie@redhat.com 2023-05-05 06:56:17 UTC 
Verify the bug with below builds:
nbdkit-server-1.34.1-1.el9.x86_64
libnbd-1.16.0-1.el9.x86_64
virt-v2v-2.3.4-1.el9.x86_64
libguestfs-1.50.1-4.el9.x86_64
guestfs-tools-1.50.1-2.el9.x86_64


Steps:
Scenario1:
1.1 Convert a guest from VMX via SSH by v2v
# virt-v2v -i vmx -it ssh ssh://root.225.34/vmfs/volumes/datastore2/esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmx -ip /home/esxpwd -v -x

1.2 Disable SSH service of ESXi host on vsphere client before v2v copying disk

1.3 Found v2v conversion will fail after retrying 5 times
# virt-v2v -i vmx -it ssh ssh://root.225.34/vmfs/volumes/datastore2/esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmx -ip /home/esxpwd -v -x
.....
nbdkit: ssh[2]: error: failed to connect to remote host: 10.73.225.34: Connection refused
nbdkit: ssh[2]: debug: ssh: open returned handle (nil)
nbdkit: ssh[2]: debug: open failed: original errno = 108
nbdkit: ssh[2]: debug: retry 1: waiting 2 seconds before retrying
.....
.....
nbdkit: ssh[2]: error: failed to connect to remote host: 10.73.225.34: Connection refused
nbdkit: ssh[2]: debug: ssh: open returned handle (nil)
nbdkit: ssh[2]: debug: open failed: original errno = 108
nbdkit: ssh[2]: debug: retry 5: waiting 32 seconds before retrying
nbdkit: ssh[2]: debug: ssh: open readonly=1 exportname="" tls=0
nbdkit: ssh[2]: debug: ssh: default_export readonly=1 tls=0
nbdkit: ssh[2]: error: failed to connect to remote host: 10.73.225.34: Connection refused
nbdkit: ssh[2]: debug: ssh: open returned handle (nil)
nbdkit: ssh[2]: debug: could not recover after 5 retries
nbdkit: ssh[2]: debug: retry: open returned handle (nil)
.....
libnbd: debug: nbd1: nbd_close: closing handle
virt-v2v: error: exception: NBD.Error("nbd_connect_unix: handshake: server 
has no export named '': No such file or directory", _)
.....
.....


Scenario2:
2.1 Convert a guest from VMX via SSH by v2v
# virt-v2v -i vmx -it ssh ssh://root.225.34/vmfs/volumes/datastore2/esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmx -ip /home/esxpwd -v -x

2.2 Disable SSH service of ESXi host on vsphere client before copying disk, v2v conversion can be continued if recover ssh connection before retrying 5th times
# virt-v2v -i vmx -it ssh ssh://root.225.34/vmfs/volumes/datastore2/esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmx -ip /home/esxpwd -v -x
....
nbdkit: ssh[2]: error: failed to connect to remote host: 10.73.225.34: Connection refused
nbdkit: ssh[2]: debug: ssh: open returned handle (nil)
nbdkit: ssh[2]: debug: open failed: original errno = 108
nbdkit: ssh[2]: debug: retry 4: waiting 16 seconds before retrying
nbdkit: ssh[2]: debug: ssh: open readonly=1 exportname="" tls=0
nbdkit: ssh[2]: debug: ssh: default_export readonly=1 tls=0
nbdkit: ssh[2]: debug: authentication methods offered by the server [0x16]:  password publickey keyboard-interactive
nbdkit: ssh[2]: debug: opened libssh handle
nbdkit: ssh[2]: debug: ssh: open returned handle 0x7fbf0401ca70
nbdkit: ssh[2]: debug: ssh: prepare readonly=1
nbdkit: ssh[2]: debug: retry: open returned handle 0x7fbf04022c10
nbdkit: ssh[2]: debug: cacheextents: open returned handle 0x7fbf0ad244b8
nbdkit: ssh[2]: debug: cow: open returned handle 0x7fbf0ad244b8
nbdkit: ssh[2]: debug: retry: prepare readonly=1
nbdkit: ssh[2]: debug: cacheextents: prepare readonly=1
nbdkit: ssh[2]: debug: cow: prepare readonly=0
.....

2.3 Disable SSH service of ESXi host on vsphere client during copying disk

2.4 v2v conversion can finish successfully, there is no any error about 'ssh[2]: error: failed to connect to remote host' during conversion
# virt-v2v -i vmx -it ssh ssh://root.225.34/vmfs/volumes/datastore2/esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmx -ip /home/esxpwd -v -x
.....
.....
[ 250.2] Finishing off
.....


Scenario3: 
3.1 Convert a guest from ESXi7.0 via vddk by v2v
# virt-v2v  -ic vpx://vsphere.local%5cAdministrator.eng.pek2.redhat.com/data/cluster/dell-per740xd-20.lab.eng.pek2.redhat.com/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 esx7.0-win11-x86_64 -ip /home/passwd -v -x

3.2 Disconnect the ESXi7.0 host on vsphere client before v2v copying disk 

3.3 Found v2v conversion will fail after retrying 5 times
# virt-v2v  -ic vpx://vsphere.local%5cAdministrator.eng.pek2.redhat.com/data/cluster/dell-per740xd-20.lab.eng.pek2.redhat.com/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 esx7.0-win11-x86_64 -ip /home/passwd -v -x
.....
nbdkit: vddk[2]: error: VixDiskLib_Open: [datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk: Unknown error
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_FreeErrorText ()
nbdkit: vddk[2]: error: Please verify whether the "thumbprint" parameter (76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21) matches the SHA1 fingerprint of the remote VMware server.  Refer to nbdkit-vddk-plugin(1) section "THUMBPRINTS" for details.
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_Disconnect (connection)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_Disconnect: Disconnect.
nbdkit: vddk[2]: debug: VixDiskLib: Backup Failure Function: VixDiskLib_Open 
nbdkit: vddk[2]: debug: VixDiskLib: Backup Failure Error: 1 
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_FreeConnectParams (params)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
nbdkit: vddk[2]: debug: vddk: open returned handle (nil)
nbdkit: vddk[2]: debug: could not recover after 5 retries
.....
virt-v2v: error: exception: NBD.Error("nbd_connect_unix: handshake: server 
has no export named '': No such file or directory", _)
.....


Scenario4:
4.1 Convert a guest from ESXi7.0 via vddk by v2v
# virt-v2v  -ic vpx://vsphere.local%5cAdministrator.eng.pek2.redhat.com/data/cluster/dell-per740xd-20.lab.eng.pek2.redhat.com/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 esx7.0-win11-x86_64 -ip /home/passwd -v -x

4.2 Disconnect ESXi host before copying disk, v2v conversion can be continued if recover connection before retrying 5th times
#  virt-v2v  -ic vpx://vsphere.local%5cAdministrator.eng.pek2.redhat.com/data/cluster/dell-per740xd-20.lab.eng.pek2.redhat.com/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 esx7.0-win11-x86_64 -ip /home/passwd -v -x
....
nbdkit: vddk[2]: error: VixDiskLib_Open: [datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk: Unknown error
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_FreeErrorText ()
nbdkit: vddk[2]: error: Please verify whether the "thumbprint" parameter (76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21) matches the SHA1 fingerprint of the remote VMware server.  Refer to nbdkit-vddk-plugin(1) section "THUMBPRINTS" for details.
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_Disconnect (connection)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_Disconnect: Disconnect.
nbdkit: vddk[2]: debug: VixDiskLib: Backup Failure Function: VixDiskLib_Open 
nbdkit: vddk[2]: debug: VixDiskLib: Backup Failure Error: 1 
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_FreeConnectParams (params)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
nbdkit: vddk[2]: debug: vddk: open returned handle (nil)
nbdkit: vddk[2]: debug: open failed: original errno = 108
nbdkit: vddk[2]: debug: retry 3: waiting 8 seconds before retrying
nbdkit: vddk[2]: debug: vddk: open readonly=1 exportname="" tls=0
nbdkit: vddk[2]: debug: vddk: default_export readonly=1 tls=0
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_AllocateConnectParams ()
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_ConnectEx (h->params, 1, NULL, NULL, &connection)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_ConnectEx: Establish connection using (null).
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLibConnectInt: Establish connection.
nbdkit: vddk[2]: debug: VixDiskLib: Resolve host.
nbdkit: vddk[2]: debug: 2023-05-05T14:48:32.795+08:00 error -[45917] [Originator@6876 sub=Default] Cannot use advanced transport modes for 10.73.227.27/moref=vm-4120/: Cannot create directory /tmp/vmware-root/42348194-815e-3440-b69f-d916a521d166-vm-4120.
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_ConnectEx: Advanced transport modes not available for opening moref=vm-4120. (Cannot use advanced transport modes for 10.73.227.27/moref=vm-4120/: Cannot create directory /tmp/vmware-root/42348194-815e-3440-b69f-d916a521d166-vm-4120.)
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_Open (connection, [datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk, 4, &handle)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_OpenEx: Open a disk.
nbdkit: vddk[2]: debug: NBD_ClientOpen: attempting to create connection to vpxa-nfc://[datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk.eng.pek2.redhat.com:902
nbdkit: vddk[2]: debug: Setting NFC log level to 1
nbdkit: vddk[2]: debug: Setting NFC log level to 1
nbdkit: vddk[2]: debug: NFC Async IO session is established for '[datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk' with log level 1.
nbdkit: vddk[2]: debug: NfcAioOpenSession: the socket options client snd buffer size 87040,           rcv buffer size 131072.
nbdkit: vddk[2]: debug: NfcAioOpenSession: the socket options server snd buffer size 293058,           rcv buffer size 293058.
nbdkit: vddk[2]: debug: Opening file [datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk (vpxa-nfc://[datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk.eng.pek2.redhat.com:902)
nbdkit: vddk[2]: debug: Nbd_ClientSetCallback: Set callback on '[datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk' with type 0.
nbdkit: vddk[2]: debug: VDDK_PhoneHome: Record transport mode "nbd" used...
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_GetTransportMode (handle)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_GetTransportMode: Retrieve transport mode.
nbdkit: vddk[2]: debug: transport mode: nbd
nbdkit: vddk[2]: debug: vddk: open returned handle 0x7ffa3c1a8ae0
nbdkit: vddk[2]: debug: vddk: prepare readonly=1
nbdkit: vddk[2]: debug: retry: open returned handle 0x7ffa3c1622c0
.....

4.3 Disconnect the ESXi7.0 host on vsphere client during copying disk 

4.4 Virt-v2v conversion can finish successfully
#virt-v2v  -ic vpx://vsphere.local%5cAdministrator.227.27/data/10.73.225.34/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 esx7.0-win11-x86_64 -ip /home/passwd -v -x
....
....
[ 246.2] Finishing off


Result:
  No problems found, move the bug from ON_QA to VERIFIED
Comment 23 errata-xmlrpc 2023-11-07 08:28:48 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (nbdkit bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6374
Note You need to log in before you can comment on or make changes to this bug.