Bug 1709211 - Suppress error message printed by nbdkit-vddk-plugin when VDDK extents support is broken
Summary: Suppress error message printed by nbdkit-vddk-plugin when VDDK extents suppor...
Keywords:
Status: NEW
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: nbdkit
Version: 8.1
Hardware: x86_64
OS: Unspecified
medium
medium
Target Milestone: rc
: 8.0
Assignee: Richard W.M. Jones
QA Contact: Virtualization Bugs
URL:
Whiteboard: V2V
Depends On:
Blocks: TRACKER-bugs-affecting-libguestfs
TreeView+ depends on / blocked
 
Reported: 2019-05-13 08:07 UTC by mxie@redhat.com
Modified: 2020-07-09 08:09 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)
vddk6.7-nfc-error-rhel8.1.log (2.36 MB, text/plain)
2019-05-13 08:07 UTC, mxie@redhat.com
no flags Details
esx5.5-vddk6.7.3.log (347.01 KB, text/plain)
2019-11-19 09:23 UTC, mxie@redhat.com
no flags Details

Description mxie@redhat.com 2019-05-13 08:07:14 UTC
Created attachment 1567813 [details]
vddk6.7-nfc-error-rhel8.1.log

Description of problem:
There is NFC error when convert a guest from ESXi5.5 host via vddk6.7 on rhel8.1 fast train

Version-Release number of selected component (if applicable):
virt-v2v-1.40.2-3.module+el8.1.0+3164+94495c71.x86_64
libguestfs-1.40.2-3.module+el8.1.0+3164+94495c71.x86_64
libvirt-5.3.0-1.module+el8.1.0+3164+94495c71.x86_64
qemu-kvm-4.0.0-0.module+el8.1.0+3169+3c501422.x86_64
nbdkit-1.12.1-1.module+el8.1.0+3164+94495c71.x86_64
kernel-4.18.0-83.el8.x86_64
VMware-vix-disklib-6.7.0-8173251.x86_64.tar.gz


How reproducible:
100%

Steps to Reproduce:
1.Convert a guest from ESXi5.5 host via vddk6.7 on rhel8.1 v2v conversion server
# virt-v2v -ic vpx://root@10.73.75.182/data/10.73.3.19?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io  vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF -o rhv --password-file /tmp/passwd -os 10.66.144.40:/home/nfs_export -b ovirtmgmt esx5.5-win8.1-x86_64 
[   0.1] Opening the source -i libvirt -ic vpx://root@10.73.75.182/data/10.73.3.19?no_verify=1 esx5.5-win8.1-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF
[  16.2] Creating an overlay to protect the source from being modified
nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49
nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server
[  40.8] Opening the overlay
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server
[  70.1] Inspecting the overlay
[  77.8] Checking for sufficient free disk space in the guest
[  77.8] Estimating space required on target for each disk
[  77.8] Converting Windows 8.1 Enterprise to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.  
Firstboot scripts may conflict with PnP.
virt-v2v: warning: there is no QXL driver for this version of Windows (6.3 
x86_64).  virt-v2v looks for this driver in 
/usr/share/virtio-win/virtio-win.iso

The guest will be configured to use a basic VGA display driver.
virt-v2v: This guest has virtio drivers installed.
[  87.0] Mapping filesystem data to avoid copying unused and blank areas
[  87.7] Closing the overlay
[  87.9] Assigning disks to buses
[  87.9] Checking if the guest needs BIOS or UEFI to boot
[  87.9] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export
[  88.2] Copying disk 1/1 to /tmp/v2v.j8tJoW/c980b640-cb59-47f7-a0fe-90312d33deae/images/c8eecdd8-8906-4cfd-8cd2-f018515e868b/6de49971-6544-43d4-8acf-3dd13063f9d5 (raw)
nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49
nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server
^C  (73.69/100%)

2.Check virt-v2v-input-vmware man page about NFC error 
2.1 #man virt-v2v-inpurt-vmware
 VDDK: ESXi NFC service memory limits
       In the verbose log you may see errors like:

        nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg:
        received NFC error 5 from server: Failed to allocate the
        requested 2097176 bytes

       This seems especially common when there are multiple parallel
       connections open to the VMware server.

       These can be caused by resource limits set on the VMware server.  You
       can increase the limit for the NFC service by editing
       /etc/vmware/hostd/config.xml and adjusting the "<maxMemory>" setting:

        <nfcsvc>
          <path>libnfcsvc.so</path>
          <enabled>true</enabled>
          <maxMemory>50331648</maxMemory>
          <maxStreamMemory>10485760</maxStreamMemory>
        </nfcsvc>

       and restarting the "hostd" service:

        # /etc/init.d/hostd restart

       For more information see https://bugzilla.redhat.com/1614276.

3.Set esxi5.5 host according to step2
3.1 ~ # cat  /etc/vmware/hostd/config.xml  |grep nfcsvc -A 4
      <nfcsvc>
         <path>libnfcsvc.so</path>
         <enabled>true</enabled>
         <maxMemory>50331648</maxMemory>
         <maxStreamMemory>10485760</maxStreamMemory>
      </nfcsvc>  
3.2 ~ #  /etc/init.d/hostd restart
watchdog-hostd: Terminating watchdog process with PID 34023
hostd stopped.
Ramdisk 'hostd' with estimated size of 413MB already exists
hostd started.

4.Convert a guest from ESXi5.5 host via vddk6.7 on rhel8.1 v2v conversion server again
# virt-v2v -ic vpx://root@10.73.75.182/data/10.73.3.19?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io  vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF -o rhv --password-file /tmp/passwd -os 10.66.144.40:/home/nfs_export -b ovirtmgmt esx5.5-win8.1-x86_64
[   0.1] Opening the source -i libvirt -ic vpx://root@10.73.75.182/data/10.73.3.19?no_verify=1 esx5.5-win8.1-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF
[  16.0] Creating an overlay to protect the source from being modified
nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49
nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server
[  91.2] Opening the overlay
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server
[ 174.5] Inspecting the overlay
[ 181.2] Checking for sufficient free disk space in the guest
[ 181.2] Estimating space required on target for each disk
[ 181.2] Converting Windows 8.1 Enterprise to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.  
Firstboot scripts may conflict with PnP.
virt-v2v: warning: there is no QXL driver for this version of Windows (6.3 
x86_64).  virt-v2v looks for this driver in 
/usr/share/virtio-win/virtio-win.iso

The guest will be configured to use a basic VGA display driver.
virt-v2v: This guest has virtio drivers installed.
[ 188.9] Mapping filesystem data to avoid copying unused and blank areas
[ 189.6] Closing the overlay
[ 189.8] Assigning disks to buses
[ 189.8] Checking if the guest needs BIOS or UEFI to boot
[ 189.8] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export
[ 190.0] Copying disk 1/1 to /tmp/v2v.TgtI0U/c980b640-cb59-47f7-a0fe-90312d33deae/images/c78df6f2-9272-4af9-bc62-7fabc909f740/2ca1cb92-6559-4af3-83dc-ea951d2673f5 (raw)
^C  (0.00/100%)

Actual results:
As above description

Expected results:
Suppress NFC error when convert a guest from ESXi5.5 host via vddk6.7 on rhel8.1 fast train

Additional info:
1.Can't reproduce the problem when convert a guest from ESXi5.5 host via vddk6.5 on rhel8.1 v2v conversion server
2.Can't reproduce the problem when convert a guest from ESXi6.0/6.5/6.7 host via vddk6.7 on rhel8.1 v2v conversion server

Comment 1 Richard W.M. Jones 2019-05-13 08:15:04 UTC
https://github.com/libguestfs/nbdkit/blob/master/plugins/vddk/vddk.c#L711

emits an error message while testing if VDDK extents support is broken on the server.
We could suppress this error by temporarily disabling the error handler.

Comment 2 mxie@redhat.com 2019-11-19 09:21:00 UTC
Virt-v2v can't convert guest from esx5.5 with vddk VMware-vix-disklib-6.7.3-14389676.x86_64 tar.gz now

Reproduce the bug with below builds:
virt-v2v-1.40.2-14.module+el8.1.0+4230+0b6e3259.x86_64
libguestfs-1.40.2-14.module+el8.1.0+4230+0b6e3259.x86_64
qemu-kvm-4.1.0-15.module+el8.1.1+4700+209eec8f.x86_64
libvirt-5.6.0-7.module+el8.1.1+4483+2f45aaa2.x86_64

Steps to reproduce:
1. Convert a guest from esx5.5 with vddk 6.7.3 by virt-v2v
#  virt-v2v  -i libvirt  -ic vpx://root@10.73.75.182/data/10.73.3.19/?no_verify=1 -o rhv-upload -of raw -b ovirtmgmt -n ovirtmgmt -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib  -io vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF -oo rhv-cafile=/home/ca.pem -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data esx5.5-rhel6.10-x86_64 -ip /home/passwd 
[   0.5] Opening the source -i libvirt -ic vpx://root@10.73.75.182/data/10.73.3.19/?no_verify=1 esx5.5-rhel6.10-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF
[  12.4] Creating an overlay to protect the source from being modified
nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49
nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvr_Close: Received unexpected message: NFC_SESSION_COMPLETE from server. Expected message: NFC_FSSRVR_CLOSE
[  26.1] Opening the overlay
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_IOEx: Received unexpected message: NFC_SESSION_COMPLETE from server. Expected message: NFC_FSSRVR_MULTIIO_EX
nbdkit: vddk[2]: error: VixDiskLib_Read: Unknown error
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 32.
nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_IOEx: Failed to send io message: The operation experienced a network error (NFC_NETWORK_ERROR)
nbdkit: vddk[2]: error: VixDiskLib_Read: The server refused connection
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe
......
......
[  44.8] Inspecting the overlay
.......
.......
.......
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 32.
nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_IOEx: Failed to send io message: The operation experienced a network error (NFC_NETWORK_ERROR)
nbdkit: vddk[2]: error: VixDiskLib_Read: The server refused connection
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 32.
nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_IOEx: Failed to send io message: The operation experienced a network error (NFC_NETWORK_ERROR)
nbdkit: vddk[2]: error: VixDiskLib_Read: The server refused connection
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 32.
nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_IOEx: Failed to send io message: The operation experienced a network error (NFC_NETWORK_ERROR)
nbdkit: vddk[2]: error: VixDiskLib_Read: The server refused connection
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 32.
nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_IOEx: Failed to send io message: The operation experienced a network error (NFC_NETWORK_ERROR)
nbdkit: vddk[2]: error: VixDiskLib_Read: The server refused connection
virt-v2v: error: inspection could not detect the source guest (or physical 
machine).

Assuming that you are running virt-v2v/virt-p2v on a source which is 
supported (and not, for example, a blank disk), then this should not 
happen.

No root device found in this operating system image.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]
nbdkit: vddk[2]: error: SSL: Unknown SSL Error
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Success
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 0.
nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_Close: Failed to send close message: The operation experienced a network error (NFC_NETWORK_ERROR)
nbdkit: vddk[2]: error: SSL: Unknown SSL Error
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Success
nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 0.
nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR

Result1:
   A lot of NFC ERROR shows during converting and the conversion is failed due to "nbdkit: vddk[2]: error: VixDiskLib_Read: The server refused connection"


2.Convert a guest from esx5.5 with vddk 6.7.0 (VMware-vix-disklib-6.7.0-8173251.x86_64.tar.gz) by virt-v2v
#  virt-v2v  -i libvirt  -ic vpx://root@10.73.75.182/data/10.73.3.19/?no_verify=1 -o rhv-upload -of raw -b ovirtmgmt -n ovirtmgmt -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib  -io vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF -oo rhv-cafile=/home/ca.pem -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data esx5.5-rhel6.10-x86_64 -ip /home/passwd 
[   0.5] Opening the source -i libvirt -ic vpx://root@10.73.75.182/data/10.73.3.19/?no_verify=1 esx5.5-rhel6.10-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF
[  16.5] Creating an overlay to protect the source from being modified
nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49
nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server
[  41.5] Opening the overlay
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49
nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server
[  71.4] Inspecting the overlay
[  89.2] Checking for sufficient free disk space in the guest
[  89.2] Estimating space required on target for each disk
[  89.2] Converting Red Hat Enterprise Linux Workstation release 6.10 (Santiago) to run on KVM
virt-v2v: warning: guest tools directory ‘linux/el6’ is missing from 
the virtio-win directory or ISO.

Guest tools are only provided in the RHV Guest Tools ISO, so this can 
happen if you are using the version of virtio-win which contains just the 
virtio drivers.  In this case only virtio drivers can be installed in the 
guest, and installation of Guest Tools will be skipped.
virt-v2v: This guest has virtio drivers installed.
[ 225.7] Mapping filesystem data to avoid copying unused and blank areas
[ 226.5] Closing the overlay
[ 226.6] Assigning disks to buses
[ 226.6] Checking if the guest needs BIOS or UEFI to boot
[ 226.6] Initializing the target -o rhv-upload -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data
[ 228.0] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/var/tmp/rhvupload.TpBYUG/nbdkit0.sock", "file.export": "/" } (raw)
^C  (0.00/100%)

Result2:
   V2V conversion can be finished although there is NFS error showing during converting.

Comment 3 mxie@redhat.com 2019-11-19 09:23:54 UTC
Created attachment 1637578 [details]
esx5.5-vddk6.7.3.log

Comment 7 Richard W.M. Jones 2020-02-13 09:14:26 UTC
It's nbdkit.  In this function:

https://github.com/libguestfs/nbdkit/blob/4dd31a48a9db61c0a76d7f1c876822b594947057/plugins/vddk/vddk.c#L670

If the server is ESXi 5.5 (which doesn't support extents) then the function
we run to test for extents support will print the following harmless cosmetic
errors:

nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49
nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server

This bug is about requesting a way to suppress these errors from being visible.

In addition there's a further bug that latest VDDK doesn't support
access to ESXi 5.5 at all.  It's unclear if this was deliberate or VMware
simply stopped testing it (since ESXi 5.5 is long out of support).  In any
case, that's a VDDK bug that we cannot do anything about, and therefore there
is no separate bug to track that.

Comment 8 John Ferlan 2020-04-20 13:37:13 UTC
Eric - I leave this for you to handle as I see some upstream changes you posted earlier this month related to vddk support.

Comment 9 Richard W.M. Jones 2020-04-20 16:12:59 UTC
I don't believe this is fixed yet.  See comment 1, comment 7.

Comment 10 Richard W.M. Jones 2020-05-07 11:24:25 UTC
https://www.redhat.com/archives/libguestfs/2020-May/msg00023.html

TBH I think the cure here may be worse than the disease and we
may need to close this bug as WONTFIX.  There's an ongoing internal
discussion about whether to keep supporting VMware ESXi 5.5 which
is the only version that this bug affects.

Comment 11 Richard W.M. Jones 2020-05-07 16:11:15 UTC
https://github.com/libguestfs/nbdkit/commit/47bd6b9b95f9e64489d8b7c84da477a06208b883

I'm going to assign this to AV 8.3.0 so it has plenty of time to "mature"
[in the sense of cheese!] upstream.  It should be a trivial backport on
top of the rebase.


Note You need to log in before you can comment on or make changes to this bug.