Created attachment 1567813 [details] vddk6.7-nfc-error-rhel8.1.log Description of problem: There is NFC error when convert a guest from ESXi5.5 host via vddk6.7 on rhel8.1 fast train Version-Release number of selected component (if applicable): virt-v2v-1.40.2-3.module+el8.1.0+3164+94495c71.x86_64 libguestfs-1.40.2-3.module+el8.1.0+3164+94495c71.x86_64 libvirt-5.3.0-1.module+el8.1.0+3164+94495c71.x86_64 qemu-kvm-4.0.0-0.module+el8.1.0+3169+3c501422.x86_64 nbdkit-1.12.1-1.module+el8.1.0+3164+94495c71.x86_64 kernel-4.18.0-83.el8.x86_64 VMware-vix-disklib-6.7.0-8173251.x86_64.tar.gz How reproducible: 100% Steps to Reproduce: 1.Convert a guest from ESXi5.5 host via vddk6.7 on rhel8.1 v2v conversion server # virt-v2v -ic vpx://root.75.182/data/10.73.3.19?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF -o rhv --password-file /tmp/passwd -os 10.66.144.40:/home/nfs_export -b ovirtmgmt esx5.5-win8.1-x86_64 [ 0.1] Opening the source -i libvirt -ic vpx://root.75.182/data/10.73.3.19?no_verify=1 esx5.5-win8.1-x86_64 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF [ 16.2] Creating an overlay to protect the source from being modified nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49 nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server [ 40.8] Opening the overlay nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49 nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server [ 70.1] Inspecting the overlay [ 77.8] Checking for sufficient free disk space in the guest [ 77.8] Estimating space required on target for each disk [ 77.8] Converting Windows 8.1 Enterprise to run on KVM virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing. Firstboot scripts may conflict with PnP. virt-v2v: warning: there is no QXL driver for this version of Windows (6.3 x86_64). virt-v2v looks for this driver in /usr/share/virtio-win/virtio-win.iso The guest will be configured to use a basic VGA display driver. virt-v2v: This guest has virtio drivers installed. [ 87.0] Mapping filesystem data to avoid copying unused and blank areas [ 87.7] Closing the overlay [ 87.9] Assigning disks to buses [ 87.9] Checking if the guest needs BIOS or UEFI to boot [ 87.9] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export [ 88.2] Copying disk 1/1 to /tmp/v2v.j8tJoW/c980b640-cb59-47f7-a0fe-90312d33deae/images/c8eecdd8-8906-4cfd-8cd2-f018515e868b/6de49971-6544-43d4-8acf-3dd13063f9d5 (raw) nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49 nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server ^C (73.69/100%) 2.Check virt-v2v-input-vmware man page about NFC error 2.1 #man virt-v2v-inpurt-vmware VDDK: ESXi NFC service memory limits In the verbose log you may see errors like: nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 2097176 bytes This seems especially common when there are multiple parallel connections open to the VMware server. These can be caused by resource limits set on the VMware server. You can increase the limit for the NFC service by editing /etc/vmware/hostd/config.xml and adjusting the "<maxMemory>" setting: <nfcsvc> <path>libnfcsvc.so</path> <enabled>true</enabled> <maxMemory>50331648</maxMemory> <maxStreamMemory>10485760</maxStreamMemory> </nfcsvc> and restarting the "hostd" service: # /etc/init.d/hostd restart For more information see https://bugzilla.redhat.com/1614276. 3.Set esxi5.5 host according to step2 3.1 ~ # cat /etc/vmware/hostd/config.xml |grep nfcsvc -A 4 <nfcsvc> <path>libnfcsvc.so</path> <enabled>true</enabled> <maxMemory>50331648</maxMemory> <maxStreamMemory>10485760</maxStreamMemory> </nfcsvc> 3.2 ~ # /etc/init.d/hostd restart watchdog-hostd: Terminating watchdog process with PID 34023 hostd stopped. Ramdisk 'hostd' with estimated size of 413MB already exists hostd started. 4.Convert a guest from ESXi5.5 host via vddk6.7 on rhel8.1 v2v conversion server again # virt-v2v -ic vpx://root.75.182/data/10.73.3.19?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF -o rhv --password-file /tmp/passwd -os 10.66.144.40:/home/nfs_export -b ovirtmgmt esx5.5-win8.1-x86_64 [ 0.1] Opening the source -i libvirt -ic vpx://root.75.182/data/10.73.3.19?no_verify=1 esx5.5-win8.1-x86_64 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF [ 16.0] Creating an overlay to protect the source from being modified nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49 nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server [ 91.2] Opening the overlay nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49 nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server [ 174.5] Inspecting the overlay [ 181.2] Checking for sufficient free disk space in the guest [ 181.2] Estimating space required on target for each disk [ 181.2] Converting Windows 8.1 Enterprise to run on KVM virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing. Firstboot scripts may conflict with PnP. virt-v2v: warning: there is no QXL driver for this version of Windows (6.3 x86_64). virt-v2v looks for this driver in /usr/share/virtio-win/virtio-win.iso The guest will be configured to use a basic VGA display driver. virt-v2v: This guest has virtio drivers installed. [ 188.9] Mapping filesystem data to avoid copying unused and blank areas [ 189.6] Closing the overlay [ 189.8] Assigning disks to buses [ 189.8] Checking if the guest needs BIOS or UEFI to boot [ 189.8] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export [ 190.0] Copying disk 1/1 to /tmp/v2v.TgtI0U/c980b640-cb59-47f7-a0fe-90312d33deae/images/c78df6f2-9272-4af9-bc62-7fabc909f740/2ca1cb92-6559-4af3-83dc-ea951d2673f5 (raw) ^C (0.00/100%) Actual results: As above description Expected results: Suppress NFC error when convert a guest from ESXi5.5 host via vddk6.7 on rhel8.1 fast train Additional info: 1.Can't reproduce the problem when convert a guest from ESXi5.5 host via vddk6.5 on rhel8.1 v2v conversion server 2.Can't reproduce the problem when convert a guest from ESXi6.0/6.5/6.7 host via vddk6.7 on rhel8.1 v2v conversion server
https://github.com/libguestfs/nbdkit/blob/master/plugins/vddk/vddk.c#L711 emits an error message while testing if VDDK extents support is broken on the server. We could suppress this error by temporarily disabling the error handler.
Virt-v2v can't convert guest from esx5.5 with vddk VMware-vix-disklib-6.7.3-14389676.x86_64 tar.gz now Reproduce the bug with below builds: virt-v2v-1.40.2-14.module+el8.1.0+4230+0b6e3259.x86_64 libguestfs-1.40.2-14.module+el8.1.0+4230+0b6e3259.x86_64 qemu-kvm-4.1.0-15.module+el8.1.1+4700+209eec8f.x86_64 libvirt-5.6.0-7.module+el8.1.1+4483+2f45aaa2.x86_64 Steps to reproduce: 1. Convert a guest from esx5.5 with vddk 6.7.3 by virt-v2v # virt-v2v -i libvirt -ic vpx://root.75.182/data/10.73.3.19/?no_verify=1 -o rhv-upload -of raw -b ovirtmgmt -n ovirtmgmt -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF -oo rhv-cafile=/home/ca.pem -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data esx5.5-rhel6.10-x86_64 -ip /home/passwd [ 0.5] Opening the source -i libvirt -ic vpx://root.75.182/data/10.73.3.19/?no_verify=1 esx5.5-rhel6.10-x86_64 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF [ 12.4] Creating an overlay to protect the source from being modified nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49 nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvr_Close: Received unexpected message: NFC_SESSION_COMPLETE from server. Expected message: NFC_FSSRVR_CLOSE [ 26.1] Opening the overlay nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49 nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_IOEx: Received unexpected message: NFC_SESSION_COMPLETE from server. Expected message: NFC_FSSRVR_MULTIIO_EX nbdkit: vddk[2]: error: VixDiskLib_Read: Unknown error nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 32. nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_IOEx: Failed to send io message: The operation experienced a network error (NFC_NETWORK_ERROR) nbdkit: vddk[2]: error: VixDiskLib_Read: The server refused connection nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe ...... ...... [ 44.8] Inspecting the overlay ....... ....... ....... nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 32. nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_IOEx: Failed to send io message: The operation experienced a network error (NFC_NETWORK_ERROR) nbdkit: vddk[2]: error: VixDiskLib_Read: The server refused connection nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 32. nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_IOEx: Failed to send io message: The operation experienced a network error (NFC_NETWORK_ERROR) nbdkit: vddk[2]: error: VixDiskLib_Read: The server refused connection nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 32. nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_IOEx: Failed to send io message: The operation experienced a network error (NFC_NETWORK_ERROR) nbdkit: vddk[2]: error: VixDiskLib_Read: The server refused connection nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 32. nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_IOEx: Failed to send io message: The operation experienced a network error (NFC_NETWORK_ERROR) nbdkit: vddk[2]: error: VixDiskLib_Read: The server refused connection virt-v2v: error: inspection could not detect the source guest (or physical machine). Assuming that you are running virt-v2v/virt-p2v on a source which is supported (and not, for example, a blank disk), then this should not happen. No root device found in this operating system image. If reporting bugs, run virt-v2v with debugging enabled and include the complete output: virt-v2v -v -x [...] nbdkit: vddk[2]: error: SSL: Unknown SSL Error nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Success nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 0. nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_Close: Failed to send close message: The operation experienced a network error (NFC_NETWORK_ERROR) nbdkit: vddk[2]: error: SSL: Unknown SSL Error nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Success nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 0. nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR Result1: A lot of NFC ERROR shows during converting and the conversion is failed due to "nbdkit: vddk[2]: error: VixDiskLib_Read: The server refused connection" 2.Convert a guest from esx5.5 with vddk 6.7.0 (VMware-vix-disklib-6.7.0-8173251.x86_64.tar.gz) by virt-v2v # virt-v2v -i libvirt -ic vpx://root.75.182/data/10.73.3.19/?no_verify=1 -o rhv-upload -of raw -b ovirtmgmt -n ovirtmgmt -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF -oo rhv-cafile=/home/ca.pem -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data esx5.5-rhel6.10-x86_64 -ip /home/passwd [ 0.5] Opening the source -i libvirt -ic vpx://root.75.182/data/10.73.3.19/?no_verify=1 esx5.5-rhel6.10-x86_64 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1E:22:D9:AF:A3:00:09:E7:F5:90:0E:C2:D7:19:CD:6F:90:E5:C8:BF [ 16.5] Creating an overlay to protect the source from being modified nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49 nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server [ 41.5] Opening the overlay nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49 nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server [ 71.4] Inspecting the overlay [ 89.2] Checking for sufficient free disk space in the guest [ 89.2] Estimating space required on target for each disk [ 89.2] Converting Red Hat Enterprise Linux Workstation release 6.10 (Santiago) to run on KVM virt-v2v: warning: guest tools directory ‘linux/el6’ is missing from the virtio-win directory or ISO. Guest tools are only provided in the RHV Guest Tools ISO, so this can happen if you are using the version of virtio-win which contains just the virtio drivers. In this case only virtio drivers can be installed in the guest, and installation of Guest Tools will be skipped. virt-v2v: This guest has virtio drivers installed. [ 225.7] Mapping filesystem data to avoid copying unused and blank areas [ 226.5] Closing the overlay [ 226.6] Assigning disks to buses [ 226.6] Checking if the guest needs BIOS or UEFI to boot [ 226.6] Initializing the target -o rhv-upload -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data [ 228.0] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/var/tmp/rhvupload.TpBYUG/nbdkit0.sock", "file.export": "/" } (raw) ^C (0.00/100%) Result2: V2V conversion can be finished although there is NFS error showing during converting.
Created attachment 1637578 [details] esx5.5-vddk6.7.3.log
It's nbdkit. In this function: https://github.com/libguestfs/nbdkit/blob/4dd31a48a9db61c0a76d7f1c876822b594947057/plugins/vddk/vddk.c#L670 If the server is ESXi 5.5 (which doesn't support extents) then the function we run to test for extents support will print the following harmless cosmetic errors: nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 49 nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvr_Close: received unexpected message 4 from server This bug is about requesting a way to suppress these errors from being visible. In addition there's a further bug that latest VDDK doesn't support access to ESXi 5.5 at all. It's unclear if this was deliberate or VMware simply stopped testing it (since ESXi 5.5 is long out of support). In any case, that's a VDDK bug that we cannot do anything about, and therefore there is no separate bug to track that.
Eric - I leave this for you to handle as I see some upstream changes you posted earlier this month related to vddk support.
I don't believe this is fixed yet. See comment 1, comment 7.
https://www.redhat.com/archives/libguestfs/2020-May/msg00023.html TBH I think the cure here may be worse than the disease and we may need to close this bug as WONTFIX. There's an ongoing internal discussion about whether to keep supporting VMware ESXi 5.5 which is the only version that this bug affects.
https://github.com/libguestfs/nbdkit/commit/47bd6b9b95f9e64489d8b7c84da477a06208b883 I'm going to assign this to AV 8.3.0 so it has plenty of time to "mature" [in the sense of cheese!] upstream. It should be a trivial backport on top of the rebase.
Verify the bug with builds: virt-v2v-1.42.0-7.module+el8.4.0+9398+f376ac33.x86_64 libguestfs-1.44.0-1.module+el8.4.0+9398+f376ac33.x86_64 libvirt-libs-7.0.0-1.module+el8.4.0+9464+3e71831a.x86_64 qemu-kvm-5.2.0-3.module+el8.4.0+9499+42e58f08.x86_64 nbdkit-1.24.0-1.module+el8.4.0+9341+96cf2672.x86_64 virtio-win-1.9.15-0.el8.noarch Steps: 1.Use v2v to convert a guest from ESXi5.5 host to rhv4.4 via vddk6.7.3 # virt-v2v -ic vpx://root.73.148/data/10.73.3.19/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk6.7.3 -io vddk-thumbprint=AA:F5:4C:48:C9:BF:75:1A:94:41:61:4C:D5:EC:DF:46:48:B5:9B:4D -ip /home/passwd -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data -n ovirtmgmt esx5.5-rhel7.9-x86_64 [ 0.5] Opening the source -i libvirt -ic vpx://root.73.148/data/10.73.3.19/?no_verify=1 esx5.5-rhel7.9-x86_64 -it vddk -io vddk-libdir=/home/vddk6.7.3 -io vddk-thumbprint=AA:F5:4C:48:C9:BF:75:1A:94:41:61:4C:D5:EC:DF:46:48:B5:9B:4D [ 2.4] Creating an overlay to protect the source from being modified nbdkit: vddk[1]: error: [NFC ERROR] NfcFssrvr_Close: Received unexpected message: NFC_SESSION_COMPLETE from server. Expected message: NFC_FSSRVR_CLOSE [ 5.6] Opening the overlay nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_IOEx: Received unexpected message: NFC_SESSION_COMPLETE from server. Expected message: NFC_FSSRVR_MULTIIO_EX nbdkit: vddk[2]: error: VixDiskLib_Read: Unknown error nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 32. nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR nbdkit: vddk[2]: error: [NFC ERROR] NfcFssrvr_Close: Failed to send close message: The operation experienced a network error (NFC_NETWORK_ERROR) nbdkit: vddk[2]: error: SSL: Unknown SSL Error nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpSetError: Success nbdkit: vddk[2]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 0. nbdkit: vddk[2]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR [ 17.5] Inspecting the overlay [ 37.3] Checking for sufficient free disk space in the guest [ 37.3] Estimating space required on target for each disk [ 37.3] Converting Red Hat Enterprise Linux Server 7.9 (Maipo) to run on KVM virt-v2v: This guest has virtio drivers installed. [ 156.7] Mapping filesystem data to avoid copying unused and blank areas [ 157.4] Closing the overlay [ 157.6] Assigning disks to buses [ 157.6] Checking if the guest needs BIOS or UEFI to boot [ 157.6] Initializing the target -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data [ 158.9] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/tmp/v2vnbdkit.dnMe63/nbdkit4.sock", "file.export": "/" } (qcow2) nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvr_IOEx: Received unexpected message: NFC_SESSION_COMPLETE from server. Expected message: NFC_FSSRVR_MULTIIO_EX nbdkit: vddk[3]: error: VixDiskLib_Read: Unknown error nbdkit: vddk[3]: error: [NFC ERROR] NfcNetTcpSetError: Broken pipe nbdkit: vddk[3]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 32. nbdkit: vddk[3]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvr_Close: Failed to send close message: The operation experienced a network error (NFC_NETWORK_ERROR) nbdkit: vddk[3]: error: SSL: Unknown SSL Error nbdkit: vddk[3]: error: [NFC ERROR] NfcNetTcpSetError: Success nbdkit: vddk[3]: error: [NFC ERROR] NfcNetTcpWrite: bWritten: -1. Errno: 0. nbdkit: vddk[3]: error: [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR (100.00/100%) [ 738.8] Creating output metadata [ 740.7] Finishing off 2.Use v2v to convert a guest from ESXi5.5 host to rhv4.4 via vddk7.0 # virt-v2v -ic vpx://root.73.148/data/10.73.3.19/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0 -io vddk-thumbprint=AA:F5:4C:48:C9:BF:75:1A:94:41:61:4C:D5:EC:DF:46:48:B5:9B:4D -ip /home/passwd -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data -n ovirtmgmt esx5.5-win2019-x86_64 [ 0.5] Opening the source -i libvirt -ic vpx://root.73.148/data/10.73.3.19/?no_verify=1 esx5.5-win2019-x86_64 -it vddk -io vddk-libdir=/home/vddk7.0 -io vddk-thumbprint=AA:F5:4C:48:C9:BF:75:1A:94:41:61:4C:D5:EC:DF:46:48:B5:9B:4D [ 2.1] Creating an overlay to protect the source from being modified nbdkit: vddk[1]: error: [NFC ERROR]NfcFssrvr_Close: Received unexpected message: NFC_SESSION_COMPLETE from server. Expected message: NFC_FSSRVR_CLOSE [ 5.7] Opening the overlay nbdkit: vddk[2]: error: [NFC ERROR]NfcFssrvr_IOEx: Received unexpected message: NFC_SESSION_COMPLETE from server. Expected message: NFC_FSSRVR_IO_EX nbdkit: vddk[2]: error: VixDiskLib_Read: Unknown error nbdkit: vddk[2]: error: [NFC ERROR]NfcNetTcpSetError: Broken pipe nbdkit: vddk[2]: error: [NFC ERROR]NfcNetTcpWrite: bWritten: -1. Errno: 32. nbdkit: vddk[2]: error: [NFC ERROR]NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR nbdkit: vddk[2]: error: [NFC ERROR]NfcFssrvr_Close: Failed to send close message: The operation experienced a network error (NFC_NETWORK_ERROR) nbdkit: vddk[2]: error: [NFC ERROR]NfcNetTcpSetError: Broken pipe nbdkit: vddk[2]: error: [NFC ERROR]NfcNetTcpWrite: bWritten: -1. Errno: 32. nbdkit: vddk[2]: error: [NFC ERROR]NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR [ 18.3] Inspecting the overlay [ 22.5] Checking for sufficient free disk space in the guest [ 22.5] Estimating space required on target for each disk [ 22.5] Converting Windows Server 2019 Standard to run on KVM virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing. Firstboot scripts may conflict with PnP. virt-v2v: warning: there is no QXL driver for this version of Windows (10.0 x86_64). virt-v2v looks for this driver in /usr/share/virtio-win/virtio-win.iso The guest will be configured to use a basic VGA display driver. virt-v2v: This guest has virtio drivers installed. [ 32.1] Mapping filesystem data to avoid copying unused and blank areas [ 32.5] Closing the overlay [ 32.8] Assigning disks to buses [ 32.8] Checking if the guest needs BIOS or UEFI to boot [ 32.8] Initializing the target -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data [ 34.1] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/tmp/v2vnbdkit.0D8Ayl/nbdkit4.sock", "file.export": "/" } (qcow2) nbdkit: vddk[3]: error: [NFC ERROR]NfcFssrvr_IOEx: Received unexpected message: NFC_SESSION_COMPLETE from server. Expected message: NFC_FSSRVR_IO_EX nbdkit: vddk[3]: error: VixDiskLib_Read: Unknown error nbdkit: vddk[3]: error: [NFC ERROR]NfcNetTcpSetError: Broken pipe nbdkit: vddk[3]: error: [NFC ERROR]NfcNetTcpWrite: bWritten: -1. Errno: 32. nbdkit: vddk[3]: error: [NFC ERROR]NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR nbdkit: vddk[3]: error: [NFC ERROR]NfcFssrvr_Close: Failed to send close message: The operation experienced a network error (NFC_NETWORK_ERROR) nbdkit: vddk[3]: error: [NFC ERROR]NfcNetTcpSetError: Broken pipe nbdkit: vddk[3]: error: [NFC ERROR]NfcNetTcpWrite: bWritten: -1. Errno: 32. nbdkit: vddk[3]: error: [NFC ERROR]NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR (100.00/100%) [ 804.9] Creating output metadata [ 806.7] Finishing off Additional info: 1.There is no NFC error when convert a guest from ESXi5.5 host via vddk6.7.0 # virt-v2v -ic vpx://root.73.148/data/10.73.3.19/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk6.7.0 -io vddk-thumbprint=AA:F5:4C:48:C9:BF:75:1A:94:41:61:4C:D5:EC:DF:46:48:B5:9B:4D -ip /home/passwd -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data -n ovirtmgmt esx5.5-win2019-x86_64 [ 0.5] Opening the source -i libvirt -ic vpx://root.73.148/data/10.73.3.19/?no_verify=1 esx5.5-win2019-x86_64 -it vddk -io vddk-libdir=/home/vddk6.7.0 -io vddk-thumbprint=AA:F5:4C:48:C9:BF:75:1A:94:41:61:4C:D5:EC:DF:46:48:B5:9B:4D [ 2.1] Creating an overlay to protect the source from being modified [ 5.7] Opening the overlay [ 12.9] Inspecting the overlay [ 17.6] Checking for sufficient free disk space in the guest [ 17.6] Estimating space required on target for each disk [ 17.6] Converting Windows Server 2019 Standard to run on KVM virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing. Firstboot scripts may conflict with PnP. virt-v2v: warning: there is no QXL driver for this version of Windows (10.0 x86_64). virt-v2v looks for this driver in /usr/share/virtio-win/virtio-win.iso The guest will be configured to use a basic VGA display driver. virt-v2v: This guest has virtio drivers installed. [ 27.0] Mapping filesystem data to avoid copying unused and blank areas [ 27.4] Closing the overlay [ 27.7] Assigning disks to buses [ 27.7] Checking if the guest needs BIOS or UEFI to boot [ 27.7] Initializing the target -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data [ 29.0] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/tmp/v2vnbdkit.kAEcsD/nbdkit4.sock", "file.export": "/" } (qcow2) ^C (3.01/100%) 2. There is no NFC error when convert a guest from ESXi5.5 host via vddk6.5 # virt-v2v -ic vpx://root.73.148/data/10.73.3.19/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk6.5 -io vddk-thumbprint=AA:F5:4C:48:C9:BF:75:1A:94:41:61:4C:D5:EC:DF:46:48:B5:9B:4D -ip /home/passwd -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data -n ovirtmgmt esx5.5-win2019-x86_64 [ 0.5] Opening the source -i libvirt -ic vpx://root.73.148/data/10.73.3.19/?no_verify=1 esx5.5-win2019-x86_64 -it vddk -io vddk-libdir=/home/vddk6.5 -io vddk-thumbprint=AA:F5:4C:48:C9:BF:75:1A:94:41:61:4C:D5:EC:DF:46:48:B5:9B:4D [ 2.2] Creating an overlay to protect the source from being modified [ 5.3] Opening the overlay [ 12.2] Inspecting the overlay [ 18.2] Checking for sufficient free disk space in the guest [ 18.2] Estimating space required on target for each disk [ 18.2] Converting Windows Server 2019 Standard to run on KVM virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing. Firstboot scripts may conflict with PnP. virt-v2v: warning: there is no QXL driver for this version of Windows (10.0 x86_64). virt-v2v looks for this driver in /usr/share/virtio-win/virtio-win.iso The guest will be configured to use a basic VGA display driver. virt-v2v: This guest has virtio drivers installed. [ 28.3] Mapping filesystem data to avoid copying unused and blank areas [ 28.8] Closing the overlay [ 29.1] Assigning disks to buses [ 29.1] Checking if the guest needs BIOS or UEFI to boot [ 29.1] Initializing the target -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data [ 30.3] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/tmp/v2vnbdkit.6Kbpxt/nbdkit4.sock", "file.export": "/" } (qcow2) ^C (38.22/100%) Hi rjones, Please check step1 and step2, seems the bug hasn't been fixed yet because NFC error info will still show if convert guest from ESXi5.5 host with vddk>=6.7.3, but good news is v2v can convert guest from ESXi5.5 with with vddk>=6.7.3 now (the problem of comment got fixed)
Created attachment 1748745 [details] esx5.5-vddk6.7.3-nbdkit-1.24.0-1.log
I think what's happening is we have successfully suppressed the original error ("Illegal message during fssrvr session"). In that limited sense the bug is fixed. However VDDK 6.7.3 and 7.0 have other problems with the ESXi 5.5 server, although they are just warnings. These are essentially new issues, but they don't affect the conversion from finishing successfully. And coincidentally VDDK 6.7.3 / 7.0 conversions have been fixed to ESXi 5.5. They is most likely because of other changes that we have made upstream (in the rebase) and are nothing to do with this particular bug fix. I'd also like to add that VMware ESXi 5.5 support has ended (in 2018!) Let me ask Martin Tessun if we should continue testing against ESXi 5.5.
According to comment18，v2v has successfully suppressed the original error "Illegal message during fssrvr session", in this sense, the bug has been fixed. Besides, we got confirmation from PM Martin by mail, for ESXi 5.5, v2v will drop testing and support, so move the bug from ON_QA to VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2098