Bug 1841820

Summary:

RFE: nbdkit-retry-filter doesn't work in vmx+ssh and esx7.0+vddk v2v conversion, should support retrying open

Product:

Red Hat Enterprise Linux 9

Reporter:

mxie <mxie>

Component:

nbdkit

Assignee:

Eric Blake <eblake>

Status:

CLOSED ERRATA

QA Contact:

mxie <mxie>

Severity:

medium

Docs Contact:

Priority:

medium

Version:

unspecified

CC:

eblake, jsuchane, juzhou, lersek, mzhan, rjones, tyan, tzheng, virt-maint, xiaodwan

Target Milestone:

Keywords:

FutureFeature, Reopened, Triaged

Target Release:

---

Hardware:

x86_64

OS:

Unspecified

Whiteboard:

Fixed In Version:

nbdkit-1.33.11-1.el9

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2023-11-07 08:28:48 UTC

Type:

Feature Request

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

2168629

Bug Blocks:

Attachments:

Description	Flags
vmx+ssh-nbdkit-retry-filter.log	none
disable-ssh-esxi7.0.png	none

Description mxie@redhat.com 2020-05-29 14:47:18 UTC

Created attachment 1693400 [details]
vmx+ssh-nbdkit-retry-filter.log

Description of problem:
nbdkit-retry-filter doesn't work in vmx+ssh v2v conversion

Version-Release number of selected component (if applicable):
virt-v2v-1.42.0-4.module+el8.3.0+6798+ad6e66be.x86_64
libguestfs-1.42.0-2.module+el8.3.0+6798+ad6e66be.x86_64
libvirt-6.3.0-1.module+el8.3.0+6478+69f490bb.x86_64
qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420.x86_64
nbdkit-1.20.2-1.module+el8.3.0+6764+cc503f20.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Convert a guest from VMX via SSH by v2v
# virt-v2v -i vmx -it ssh ssh://root.199.217/vmfs/volumes/esx7.0-matrix/esx7.0-ubuntu18.04-x86_64/esx7.0-ubuntu18.04-x86_64.vmx -v -x 

2. Disable SSH function on ESXi host before v2v copying disk, pls refer to screenshot"disable-ssh-esxi7.0.png"

3. Found v2v conversion will fail during copying disk without any retrying, details pls refer to 'vmx+ssh-nbdkit-retry-filter.log'
......
[  62.2] Copying disk 1/1 to /var/lib/libvirt/images/esx7.0-ubuntu18.04-x86_64-sda (raw)
......
qemu-img 'convert' '-p' '-n' '-f' 'qcow2' '-O' 'raw' '-S' '64k' '/var/tmp/v2vovlf07f8e.qcow2' '/var/lib/libvirt/images/esx7.0-ubuntu18.04-x86_64-sda'
nbdkit: debug: accepted connection
......
nbdkit: ssh[3]: debug: cacheextents: open readonly=1
nbdkit: ssh[3]: debug: readahead: open readonly=1
nbdkit: ssh[3]: debug: retry: open readonly=1
nbdkit: ssh[3]: debug: ssh: open readonly=1
nbdkit: ssh[3]: error: failed to connect to remote host: 10.73.199.217: Connection refused
nbdkit: ssh[3]: debug: ssh: open returned handle (nil)
nbdkit: ssh[3]: debug: retry: open returned handle (nil)
nbdkit: ssh[3]: debug: readahead: open returned handle (nil)
nbdkit: ssh[3]: debug: cacheextents: open returned handle (nil)
nbdkit: ssh[3]: debug: write: NBD_OPT_ABORT: Broken pipe
qemu-img: Could not open '/var/tmp/v2vovlf07f8e.qcow2': Could not open backing file: Requested export not available

virt-v2v: error: qemu-img command failed, see earlier errors


Actual results:
As above description

Expected results:
nbdkit-retry-filter works well in vmx+ssh v2v conversion

Additional info:

Comment 1 mxie@redhat.com 2020-05-29 14:47:57 UTC

Created attachment 1693401 [details]
disable-ssh-esxi7.0.png

Comment 2 Richard W.M. Jones 2020-05-29 15:29:02 UTC

> 2. Disable SSH function on ESXi host before v2v copying disk, pls refer to screenshot"disable-ssh-esxi7.0.png"

I'm fairly certain what happens here is the following:

(a) The (earlier) conversion step has opened an SSH connection.  Disabling
the SSH server does not stop existing SSH connections, so the conversion
step keeps running to completion.

(b) mxie disables the SSH server, step 2. above.

(c) After conversion completes, the copying step starts.  This tries to open
a new SSH connection to the server and this is what fails.

Note that the retry filter doesn't retry failed opens (only
failed data operations like read/write), as you can see from
the following simple test:

  nbdkit -v -U - --filter=retry sh - --run 'qemu-img info $nbd' <<'EOF'
  case "$1" in
       open) echo EIO >&2; exit 1 ;;
       get_size) echo 1M ;;
       pread) dd if=/dev/zero count=$3 iflag=count_bytes ;;
       *) exit 2 ;;
  esac
  EOF

(note the open method returns an error).  It prints a log very similar to yours,
ending in:

nbdkit: sh[1]: error: /tmp/nbdkitbhYyrx/inline-script.sh: 
nbdkit: sh[1]: debug: sh: open returned handle (nil)
nbdkit: sh[1]: debug: retry: open returned handle (nil)
nbdkit: sh[1]: debug: write: NBD_OPT_ABORT: Broken pipe
qemu-img: Could not open 'nbd:unix:/tmp/nbdkitpDSwwy/socket': Requested export not available

Since this is essentially something which nbdkit-retry-filter does not
currently implement, I'm going to add "RFE" to the summary line.

Comment 5 Eric Hadley 2021-09-08 16:45:07 UTC

Bulk update: Move RHEL-AV bugs to RHEL9. If necessary to resolve in RHEL8, then clone to the current RHEL8 release.

Comment 7 RHEL Program Management 2021-11-29 07:27:03 UTC

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Comment 8 Richard W.M. Jones 2021-11-29 10:22:42 UTC

This bug was closed in error by a process we have no control over.

Comment 9 mxie@redhat.com 2022-02-22 14:39:26 UTC

I think retry filter also doesn't work when convert guest from ESXi7.0 host via vddk and copying disk has not started in v2v conversion

Package versions:
virt-v2v-1.45.99-1.el9.x86_64
libguestfs-1.46.1-2.el9.x86_64
guestfs-tools-1.46.1-6.el9.x86_64
nbdkit-1.28.5-1.el9.x86_64
libnbd-1.10.5-1.el9.x86_64
libvirt-libs-8.0.0-4.el9.x86_64
qemu-img-6.2.0-9.el9.x86_64

Steps:
1.Convert a guest from ESXi7.0 via vddk by v2v
# virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.2 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78  -ip /home/passwd esx7.0-win2022-x86_64 -v -x 

2.Disconnect the ESXi7.0 host on vsphere client before copying disk has not started in v2v conversion

3.Then v2v conversion will be failed with below error, there is no any info about retrying reconnecting in v2v debug log
.....
nbdkit: vddk[2]: debug: vddk: open returned handle (nil)
nbdkit: vddk[2]: debug: retry: open returned handle (nil)
nbdkit: vddk[2]: debug: cacheextents: open returned handle (nil)
nbdkit: vddk[2]: debug: blocksize: open returned handle (nil)
nbdkit: vddk[2]: debug: cow: open returned handle (nil)
nbdkit: vddk[2]: debug: multi-conn: open returned handle (nil)
libnbd: debug: nbd1: nbd_connect_unix: poll end: r=1 revents=1
libnbd: debug: nbd1: nbd_connect_unix: transition: NEWSTYLE.OPT_GO.RECV_REPLY -> NEWSTYLE.OPT_GO.RECV_REPLY_PAYLOAD
libnbd: debug: nbd1: nbd_connect_unix: transition: NEWSTYLE.OPT_GO.RECV_REPLY_PAYLOAD -> NEWSTYLE.OPT_GO.CHECK_REPLY
libnbd: debug: nbd1: nbd_connect_unix: transition: NEWSTYLE.OPT_GO.CHECK_REPLY -> NEWSTYLE.PREPARE_OPT_ABORT
libnbd: debug: nbd1: nbd_connect_unix: transition: NEWSTYLE.PREPARE_OPT_ABORT -> NEWSTYLE.SEND_OPT_ABORT
nbdkit: vddk[2]: debug: client sent NBD_OPT_ABORT to abort the connection
libnbd: debug: nbd1: nbd_connect_unix: transition: NEWSTYLE.SEND_OPT_ABORT -> NEWSTYLE.SEND_OPTION_SHUTDOWN
libnbd: debug: nbd1: nbd_connect_unix: transition: NEWSTYLE.SEND_OPTION_SHUTDOWN -> DEAD
libnbd: debug: nbd1: nbd_connect_unix: leave: error="nbd_connect_unix: handshake: server has no export named '': No such file or directory"
libnbd: debug: nbd1: nbd_close: closing handle
virt-v2v: error: exception: NBD.Error("nbd_connect_unix: handshake: server 
has no export named '': No such file or directory", 2)


Additional info：
retry filter works if convert guest from ESXi6.7 host via vddk

Comment 12 John Ferlan 2022-09-02 19:42:53 UTC

Eric - assigning directly to you for handling. This bug has been open for a while and we need to make a decision whether or not to add the feature.  I've placed 9.2.0 in the ITR as a way to try and force that decision. If the decision is to not do this, we should just close.

Comment 14 John Ferlan 2022-12-14 12:31:27 UTC

Updating the stale date to the end of ITM 26 for RHEL 9.2.0

Comment 15 Eric Blake 2023-01-27 20:42:26 UTC

Upstream patch proposed:
https://listman.redhat.com/archives/libguestfs/2023-January/030531.html
Now the question is whether this is important enough to backport.

Comment 16 Richard W.M. Jones 2023-02-06 09:48:02 UTC

Moving to RHEL 9.3

Comment 18 mxie@redhat.com 2023-03-29 12:51:26 UTC

Test the bug with below builds:
nbdkit-server-1.33.11-1.el9.x86_64
libnbd-1.15.12-1.el9.x86_64
virt-v2v-2.2.0-5.el9.x86_64
libguestfs-1.50.1-3.el9.x86_64
guestfs-tools-1.48.2-8.el9.x86_64


Steps:
Scenario1:
1.1 Convert a guest from VMX via SSH by v2v
# virt-v2v -i vmx -it ssh ssh://root.225.34/vmfs/volumes/datastore2/esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmx -ip /home/esxpwd -v -x

1.2 Disable SSH function on ESXi host before v2v copying disk

1.3 Found v2v conversion will fail after retrying 5 times
# virt-v2v -i vmx -it ssh ssh://root.225.34/vmfs/volumes/datastore2/esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmx -ip /home/esxpwd -v -x
.....
nbdkit: ssh[2]: error: failed to connect to remote host: 10.73.225.34: Connection refused
nbdkit: ssh[2]: debug: ssh: open returned handle (nil)
nbdkit: ssh[2]: debug: open failed: original errno = 108
nbdkit: ssh[2]: debug: retry 1: waiting 2 seconds before retrying
.....
.....
nbdkit: ssh[2]: error: failed to connect to remote host: 10.73.225.34: Connection refused
nbdkit: ssh[2]: debug: ssh: open returned handle (nil)
nbdkit: ssh[2]: debug: open failed: original errno = 108
nbdkit: ssh[2]: debug: retry 5: waiting 32 seconds before retrying
nbdkit: ssh[2]: debug: ssh: open readonly=1 exportname="" tls=0
nbdkit: ssh[2]: debug: ssh: default_export readonly=1 tls=0
nbdkit: ssh[2]: error: failed to connect to remote host: 10.73.225.34: Connection refused
nbdkit: ssh[2]: debug: ssh: open returned handle (nil)
nbdkit: ssh[2]: debug: could not recover after 5 retries
.....
libnbd: debug: nbd1: nbd_close: closing handle
virt-v2v: error: exception: NBD.Error("nbd_connect_unix: handshake: server 
has no export named '': No such file or directory", _)
.....
.....


Scenario2:
2.1 Convert a guest from VMX via SSH by v2v
# virt-v2v -i vmx -it ssh ssh://root.225.34/vmfs/volumes/datastore2/esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmx -ip /home/esxpwd -v -x

2.2 Disable SSH function on ESXi host before copying disk, v2v conversion can be continued if recover ssh connection before retrying 5th times
# virt-v2v -i vmx -it ssh ssh://root.225.34/vmfs/volumes/datastore2/esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmx -ip /home/esxpwd -v -x
....
....
nbdkit: ssh[2]: error: failed to connect to remote host: 10.73.225.34: Connection refused
nbdkit: ssh[2]: debug: ssh: open returned handle (nil)
nbdkit: ssh[2]: debug: open failed: original errno = 108
nbdkit: ssh[2]: debug: retry 1: waiting 2 seconds before retrying
nbdkit: ssh[2]: debug: ssh: open readonly=1 exportname="" tls=0
nbdkit: ssh[2]: debug: ssh: default_export readonly=1 tls=0
nbdkit: ssh[2]: error: failed to connect to remote host: 10.73.225.34: Connection refused
nbdkit: ssh[2]: debug: ssh: open returned handle (nil)
nbdkit: ssh[2]: debug: open failed: original errno = 108
nbdkit: ssh[2]: debug: retry 2: waiting 4 seconds before retrying
nbdkit: ssh[2]: debug: ssh: open readonly=1 exportname="" tls=0
nbdkit: ssh[2]: debug: ssh: default_export readonly=1 tls=0
nbdkit: ssh[2]: error: failed to connect to remote host: 10.73.225.34: Connection refused
nbdkit: ssh[2]: debug: ssh: open returned handle (nil)
nbdkit: ssh[2]: debug: open failed: original errno = 108
nbdkit: ssh[2]: debug: retry 3: waiting 8 seconds before retrying
nbdkit: ssh[2]: debug: ssh: open readonly=1 exportname="" tls=0
nbdkit: ssh[2]: debug: ssh: default_export readonly=1 tls=0
nbdkit: ssh[2]: debug: authentication methods offered by the server [0x16]:  password publickey keyboard-interactive
nbdkit: ssh[2]: debug: opened libssh handle
nbdkit: ssh[2]: debug: ssh: open returned handle 0x7f5a14048e00
nbdkit: ssh[2]: debug: ssh: prepare readonly=1
nbdkit: ssh[2]: debug: retry: open returned handle 0x7f5a14022c10
nbdkit: ssh[2]: debug: cacheextents: open returned handle 0x7f5a1a4cd4b8
nbdkit: ssh[2]: debug: cow: open returned handle 0x7f5a1a4cd4b8
nbdkit: ssh[2]: debug: retry: prepare readonly=1
.....

2.3 Disable SSH function on ESXi host during copying disk

2.4 v2v conversion can finish successfully, there is no any error about 'ssh[2]: error: failed to connect to remote host' during conversion
# virt-v2v -i vmx -it ssh ssh://root.225.34/vmfs/volumes/datastore2/esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmx -ip /home/esxpwd -v -x
.....
.....
[ 250.2] Finishing off
.....


Scenario3: 
3.1 Convert a guest from ESXi7.0 via vddk by v2v
# virt-v2v  -ic vpx://vsphere.local%5cAdministrator.227.27/data/10.73.225.34/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 esx7.0-win11-x86_64 -ip /home/passwd -v -x

3.2 Disconnect the ESXi7.0 host on vsphere client before copying disk 

3.3 Found v2v conversion will fail after retrying 5 times
virt-v2v  -ic vpx://vsphere.local%5cAdministrator.227.27/data/10.73.225.34/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 esx7.0-win11-x86_64 -ip /home/passwd -v -x
.....
.....
nbdkit: vddk[2]: error: VixDiskLib_Open: [datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk: You do not have access rights to this file
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_FreeErrorText ()
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_Disconnect (connection)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_Disconnect: Disconnect.
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_FreeConnectParams (params)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
nbdkit: vddk[2]: debug: vddk: open returned handle (nil)
nbdkit: vddk[2]: debug: open failed: original errno = 108
nbdkit: vddk[2]: debug: retry 1: waiting 2 seconds before retrying
.....
nbdkit: vddk[2]: debug: retry 5: waiting 32 seconds before retrying
.....
nbdkit: vddk[2]: error: VixDiskLib_Open: [datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk: Unknown error
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_FreeErrorText ()
nbdkit: vddk[2]: error: Please verify whether the "thumbprint" parameter (76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21) matches the SHA1 fingerprint of the remote VMware server.  Refer to nbdkit-vddk-plugin(1) section "THUMBPRINTS" for details.
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_Disconnect (connection)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_Disconnect: Disconnect.
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_FreeConnectParams (params)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
nbdkit: vddk[2]: debug: vddk: open returned handle (nil)
nbdkit: vddk[2]: debug: could not recover after 5 retries
.....
.....


Scenario4:
4.1 Convert a guest from ESXi7.0 via vddk by v2v
# virt-v2v  -ic vpx://vsphere.local%5cAdministrator.227.27/data/10.73.225.34/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 esx7.0-win11-x86_64 -ip /home/passwd -v -x

4.2 Disconnect ESXi host before copying disk, v2v conversion can be continued if recover connection before retrying 5th times
# virt-v2v  -ic vpx://vsphere.local%5cAdministrator.227.27/data/10.73.225.34/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 esx7.0-win11-x86_64 -ip /home/passwd -v -x
....
....
nbdkit: vddk[2]: error: VixDiskLib_Open: [datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk: Unknown error
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_FreeErrorText ()
nbdkit: vddk[2]: error: Please verify whether the "thumbprint" parameter (76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21) matches the SHA1 fingerprint of the remote VMware server.  Refer to nbdkit-vddk-plugin(1) section "THUMBPRINTS" for details.
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_Disconnect (connection)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_Disconnect: Disconnect.
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_FreeConnectParams (params)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
nbdkit: vddk[2]: debug: vddk: open returned handle (nil)
nbdkit: vddk[2]: debug: open failed: original errno = 108
nbdkit: vddk[2]: debug: retry 3: waiting 8 seconds before retrying
nbdkit: vddk[2]: debug: vddk: open readonly=1 exportname="" tls=0
nbdkit: vddk[2]: debug: vddk: default_export readonly=1 tls=0
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_AllocateConnectParams ()
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_ConnectEx (h->params, 1, NULL, NULL, &connection)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_ConnectEx: Establish connection using (null).
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLibConnectInt: Establish connection.
nbdkit: vddk[2]: debug: VixDiskLib: Resolve host.
....
nbdkit: vddk[2]: debug: vddk: open returned handle 0x7f36fc0553a0
nbdkit: vddk[2]: debug: vddk: prepare readonly=1
.....

4.3 Disconnect the ESXi7.0 host on vsphere client during copying disk 

4.4 Virt-v2v conversion can finish successfully
#virt-v2v  -ic vpx://vsphere.local%5cAdministrator.227.27/data/10.73.225.34/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 esx7.0-win11-x86_64 -ip /home/passwd -v -x
....
....
[ 246.2] Finishing off


Result:
  The bug has been fixed

Comment 21 mxie@redhat.com 2023-05-05 06:56:17 UTC

Verify the bug with below builds:
nbdkit-server-1.34.1-1.el9.x86_64
libnbd-1.16.0-1.el9.x86_64
virt-v2v-2.3.4-1.el9.x86_64
libguestfs-1.50.1-4.el9.x86_64
guestfs-tools-1.50.1-2.el9.x86_64


Steps:
Scenario1:
1.1 Convert a guest from VMX via SSH by v2v
# virt-v2v -i vmx -it ssh ssh://root.225.34/vmfs/volumes/datastore2/esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmx -ip /home/esxpwd -v -x

1.2 Disable SSH service of ESXi host on vsphere client before v2v copying disk

1.3 Found v2v conversion will fail after retrying 5 times
# virt-v2v -i vmx -it ssh ssh://root.225.34/vmfs/volumes/datastore2/esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmx -ip /home/esxpwd -v -x
.....
nbdkit: ssh[2]: error: failed to connect to remote host: 10.73.225.34: Connection refused
nbdkit: ssh[2]: debug: ssh: open returned handle (nil)
nbdkit: ssh[2]: debug: open failed: original errno = 108
nbdkit: ssh[2]: debug: retry 1: waiting 2 seconds before retrying
.....
.....
nbdkit: ssh[2]: error: failed to connect to remote host: 10.73.225.34: Connection refused
nbdkit: ssh[2]: debug: ssh: open returned handle (nil)
nbdkit: ssh[2]: debug: open failed: original errno = 108
nbdkit: ssh[2]: debug: retry 5: waiting 32 seconds before retrying
nbdkit: ssh[2]: debug: ssh: open readonly=1 exportname="" tls=0
nbdkit: ssh[2]: debug: ssh: default_export readonly=1 tls=0
nbdkit: ssh[2]: error: failed to connect to remote host: 10.73.225.34: Connection refused
nbdkit: ssh[2]: debug: ssh: open returned handle (nil)
nbdkit: ssh[2]: debug: could not recover after 5 retries
nbdkit: ssh[2]: debug: retry: open returned handle (nil)
.....
libnbd: debug: nbd1: nbd_close: closing handle
virt-v2v: error: exception: NBD.Error("nbd_connect_unix: handshake: server 
has no export named '': No such file or directory", _)
.....
.....


Scenario2:
2.1 Convert a guest from VMX via SSH by v2v
# virt-v2v -i vmx -it ssh ssh://root.225.34/vmfs/volumes/datastore2/esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmx -ip /home/esxpwd -v -x

2.2 Disable SSH service of ESXi host on vsphere client before copying disk, v2v conversion can be continued if recover ssh connection before retrying 5th times
# virt-v2v -i vmx -it ssh ssh://root.225.34/vmfs/volumes/datastore2/esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmx -ip /home/esxpwd -v -x
....
nbdkit: ssh[2]: error: failed to connect to remote host: 10.73.225.34: Connection refused
nbdkit: ssh[2]: debug: ssh: open returned handle (nil)
nbdkit: ssh[2]: debug: open failed: original errno = 108
nbdkit: ssh[2]: debug: retry 4: waiting 16 seconds before retrying
nbdkit: ssh[2]: debug: ssh: open readonly=1 exportname="" tls=0
nbdkit: ssh[2]: debug: ssh: default_export readonly=1 tls=0
nbdkit: ssh[2]: debug: authentication methods offered by the server [0x16]:  password publickey keyboard-interactive
nbdkit: ssh[2]: debug: opened libssh handle
nbdkit: ssh[2]: debug: ssh: open returned handle 0x7fbf0401ca70
nbdkit: ssh[2]: debug: ssh: prepare readonly=1
nbdkit: ssh[2]: debug: retry: open returned handle 0x7fbf04022c10
nbdkit: ssh[2]: debug: cacheextents: open returned handle 0x7fbf0ad244b8
nbdkit: ssh[2]: debug: cow: open returned handle 0x7fbf0ad244b8
nbdkit: ssh[2]: debug: retry: prepare readonly=1
nbdkit: ssh[2]: debug: cacheextents: prepare readonly=1
nbdkit: ssh[2]: debug: cow: prepare readonly=0
.....

2.3 Disable SSH service of ESXi host on vsphere client during copying disk

2.4 v2v conversion can finish successfully, there is no any error about 'ssh[2]: error: failed to connect to remote host' during conversion
# virt-v2v -i vmx -it ssh ssh://root.225.34/vmfs/volumes/datastore2/esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmx -ip /home/esxpwd -v -x
.....
.....
[ 250.2] Finishing off
.....


Scenario3: 
3.1 Convert a guest from ESXi7.0 via vddk by v2v
# virt-v2v  -ic vpx://vsphere.local%5cAdministrator.eng.pek2.redhat.com/data/cluster/dell-per740xd-20.lab.eng.pek2.redhat.com/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 esx7.0-win11-x86_64 -ip /home/passwd -v -x

3.2 Disconnect the ESXi7.0 host on vsphere client before v2v copying disk 

3.3 Found v2v conversion will fail after retrying 5 times
# virt-v2v  -ic vpx://vsphere.local%5cAdministrator.eng.pek2.redhat.com/data/cluster/dell-per740xd-20.lab.eng.pek2.redhat.com/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 esx7.0-win11-x86_64 -ip /home/passwd -v -x
.....
nbdkit: vddk[2]: error: VixDiskLib_Open: [datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk: Unknown error
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_FreeErrorText ()
nbdkit: vddk[2]: error: Please verify whether the "thumbprint" parameter (76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21) matches the SHA1 fingerprint of the remote VMware server.  Refer to nbdkit-vddk-plugin(1) section "THUMBPRINTS" for details.
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_Disconnect (connection)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_Disconnect: Disconnect.
nbdkit: vddk[2]: debug: VixDiskLib: Backup Failure Function: VixDiskLib_Open 
nbdkit: vddk[2]: debug: VixDiskLib: Backup Failure Error: 1 
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_FreeConnectParams (params)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
nbdkit: vddk[2]: debug: vddk: open returned handle (nil)
nbdkit: vddk[2]: debug: could not recover after 5 retries
.....
virt-v2v: error: exception: NBD.Error("nbd_connect_unix: handshake: server 
has no export named '': No such file or directory", _)
.....


Scenario4:
4.1 Convert a guest from ESXi7.0 via vddk by v2v
# virt-v2v  -ic vpx://vsphere.local%5cAdministrator.eng.pek2.redhat.com/data/cluster/dell-per740xd-20.lab.eng.pek2.redhat.com/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 esx7.0-win11-x86_64 -ip /home/passwd -v -x

4.2 Disconnect ESXi host before copying disk, v2v conversion can be continued if recover connection before retrying 5th times
#  virt-v2v  -ic vpx://vsphere.local%5cAdministrator.eng.pek2.redhat.com/data/cluster/dell-per740xd-20.lab.eng.pek2.redhat.com/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 esx7.0-win11-x86_64 -ip /home/passwd -v -x
....
nbdkit: vddk[2]: error: VixDiskLib_Open: [datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk: Unknown error
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_FreeErrorText ()
nbdkit: vddk[2]: error: Please verify whether the "thumbprint" parameter (76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21) matches the SHA1 fingerprint of the remote VMware server.  Refer to nbdkit-vddk-plugin(1) section "THUMBPRINTS" for details.
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_Disconnect (connection)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_Disconnect: Disconnect.
nbdkit: vddk[2]: debug: VixDiskLib: Backup Failure Function: VixDiskLib_Open 
nbdkit: vddk[2]: debug: VixDiskLib: Backup Failure Error: 1 
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_FreeConnectParams (params)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.
nbdkit: vddk[2]: debug: vddk: open returned handle (nil)
nbdkit: vddk[2]: debug: open failed: original errno = 108
nbdkit: vddk[2]: debug: retry 3: waiting 8 seconds before retrying
nbdkit: vddk[2]: debug: vddk: open readonly=1 exportname="" tls=0
nbdkit: vddk[2]: debug: vddk: default_export readonly=1 tls=0
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_AllocateConnectParams ()
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_ConnectEx (h->params, 1, NULL, NULL, &connection)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_ConnectEx: Establish connection using (null).
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLibConnectInt: Establish connection.
nbdkit: vddk[2]: debug: VixDiskLib: Resolve host.
nbdkit: vddk[2]: debug: 2023-05-05T14:48:32.795+08:00 error -[45917] [Originator@6876 sub=Default] Cannot use advanced transport modes for 10.73.227.27/moref=vm-4120/: Cannot create directory /tmp/vmware-root/42348194-815e-3440-b69f-d916a521d166-vm-4120.
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_ConnectEx: Advanced transport modes not available for opening moref=vm-4120. (Cannot use advanced transport modes for 10.73.227.27/moref=vm-4120/: Cannot create directory /tmp/vmware-root/42348194-815e-3440-b69f-d916a521d166-vm-4120.)
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_Open (connection, [datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk, 4, &handle)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_OpenEx: Open a disk.
nbdkit: vddk[2]: debug: NBD_ClientOpen: attempting to create connection to vpxa-nfc://[datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk.eng.pek2.redhat.com:902
nbdkit: vddk[2]: debug: Setting NFC log level to 1
nbdkit: vddk[2]: debug: Setting NFC log level to 1
nbdkit: vddk[2]: debug: NFC Async IO session is established for '[datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk' with log level 1.
nbdkit: vddk[2]: debug: NfcAioOpenSession: the socket options client snd buffer size 87040,           rcv buffer size 131072.
nbdkit: vddk[2]: debug: NfcAioOpenSession: the socket options server snd buffer size 293058,           rcv buffer size 293058.
nbdkit: vddk[2]: debug: Opening file [datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk (vpxa-nfc://[datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk.eng.pek2.redhat.com:902)
nbdkit: vddk[2]: debug: Nbd_ClientSetCallback: Set callback on '[datastore2] esx7.0-win11-x86_64/esx7.0-win11-x86_64.vmdk' with type 0.
nbdkit: vddk[2]: debug: VDDK_PhoneHome: Record transport mode "nbd" used...
nbdkit: vddk[2]: debug: VDDK call: VixDiskLib_GetTransportMode (handle)
nbdkit: vddk[2]: debug: VixDiskLib: VixDiskLib_GetTransportMode: Retrieve transport mode.
nbdkit: vddk[2]: debug: transport mode: nbd
nbdkit: vddk[2]: debug: vddk: open returned handle 0x7ffa3c1a8ae0
nbdkit: vddk[2]: debug: vddk: prepare readonly=1
nbdkit: vddk[2]: debug: retry: open returned handle 0x7ffa3c1622c0
.....

4.3 Disconnect the ESXi7.0 host on vsphere client during copying disk 

4.4 Virt-v2v conversion can finish successfully
#virt-v2v  -ic vpx://vsphere.local%5cAdministrator.227.27/data/10.73.225.34/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 esx7.0-win11-x86_64 -ip /home/passwd -v -x
....
....
[ 246.2] Finishing off


Result:
  No problems found, move the bug from ON_QA to VERIFIED

Comment 23 errata-xmlrpc 2023-11-07 08:28:48 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (nbdkit bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6374