Bug 1879440

Summary:	[DOCS] Import Vm from vSphere failed, Timed out connecting to ESX host on port 902
Product:	Container Native Virtualization (CNV)	Reporter:	Giuseppe Cofano <gcofano>
Component:	V2V	Assignee:	Avital Pinnick <apinnick>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Ilanit Stein <istein>
Severity:	high	Docs Contact:	Avital Pinnick <apinnick>
Priority:	medium
Version:	2.4.0	CC:	aos-bugs, apinnick, cnv-qe-bugs, eparis, fdupont, istein, jokerman, pvauter, rgarcia
Target Milestone:	---
Target Release:	2.5.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Enhancement
Doc Text:	Feature: Add required ports needed to be open between OCPV and VMware to be able to migrate a VM. Reason: Migration may be blocked and fail. Result: Add correct ports needed to be open for VM import. See: https://bugzilla.redhat.com/show_bug.cgi?id=1879440#c2	Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-11-05 13:36:39 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Giuseppe Cofano 2020-09-16 10:05:31 UTC

Description of problem:

When importing a VM from vSphere and following the docs at https://docs.openshift.com/container-platform/4.5/virt/virtual_machines/importing_vms/virt-importing-vmware-vm.html#virt-creating-vddk-image_virt-importing-vmware-vm, hit the following error.


Version-Release number of selected component (if applicable):
CNV 2.4

How reproducible:
Import Vm as in https://docs.openshift.com/container-platform/4.5/virt/virtual_machines/importing_vms/virt-importing-vmware-vm.html#virt-creating-vddk-image_virt-importing-vmware-vm

Steps to Reproduce:
1.
2.
3.

Actual results:

Logs from the pod kubevirt-v2v-conversion:

2020-09-16 08:27:16,958 - root - ERROR - virt-v2v error: qemu-img command failed, see earlier errors
2020-09-16 08:27:16,958 - root - DEBUG - b'nbdkit: vddk[1]: debug: CnxOpenTCPSocket: Timed out connecting to server esx3.mgt.digital-pod3eur.local:902: Operation now in progress\n'
2020-09-16 08:27:16,958 - root - DEBUG - b'nbdkit: vddk[1]: debug: CnxAuthdConnect: Returning false because CnxAuthdConnectTCP failed\n'
2020-09-16 08:27:16,959 - root - DEBUG - b'nbdkit: vddk[1]: debug: CnxConnectAuthd: Returning false because CnxAuthdConnect failed\n'
2020-09-16 08:27:16,959 - root - DEBUG - b'nbdkit: vddk[1]: debug: Cnx_Connect: Returning false because CnxConnectAuthd failed\n'
2020-09-16 08:27:16,959 - root - DEBUG - b'nbdkit: vddk[1]: debug: Cnx_Connect: Error message: Failed to connect to server esx3.mgt.digital-pod3eur.local:902\n'
2020-09-16 08:27:16,959 - root - DEBUG - b'nbdkit: vddk[1]: error: [NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect: Failed to connect to server esx3.mgt.digital-pod3eur.local:902\n'
2020-09-16 08:27:16,959 - root - DEBUG - b'nbdkit: vddk[1]: error: [NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer. Error: Failed to connect to server esx3.mgt.digital-pod3eur.local:902\n'
2020-09-16 08:27:16,959 - root - DEBUG - b'nbdkit: vddk[1]: error: [NFC ERROR] Nfc_BindAndEstablishAuthdCnx2: Failed to create new AuthD connection: The operation completed successfully (NFC_SUCCESS)\n'
2020-09-16 08:27:16,959 - root - DEBUG - b"nbdkit: vddk[1]: debug: NBD_ClientOpen: Couldn't connect to esx3.mgt.digital-pod3eur.local:902 Failed to connect to server esx3.mgt.digital-pod3eur.local:902\n"
2020-09-16 08:27:16,959 - root - DEBUG - b'nbdkit: vddk[1]: debug: DISKLIB-DSCPTR: : "vpxa-nfc://[vsanDS-ClusterA] ac805f5f-829e-c35a-56a6-b026282184e0/ocp4-vmtest.vmdk.digital-pod3eur.local:902" : Failed to open NBD extent.\n'
2020-09-16 08:27:16,959 - root - DEBUG - b'nbdkit: vddk[1]: debug: DISKLIB-LINK  : "vpxa-nfc://[vsanDS-ClusterA] ac805f5f-829e-c35a-56a6-b026282184e0/ocp4-vmtest.vmdk.digital-pod3eur.local:902" : failed to open (NBD_ERR_NETWORK_CONNECT).  \n'
2020-09-16 08:27:16,959 - root - DEBUG - b'nbdkit: vddk[1]: debug: DISKLIB-CHAIN : "vpxa-nfc://[vsanDS-ClusterA] ac805f5f-829e-c35a-56a6-b026282184e0/ocp4-vmtest.vmdk.digital-pod3eur.local:902" : failed to open (NBD_ERR_NETWORK_CONNECT).\n'
2020-09-16 08:27:16,959 - root - DEBUG - b"nbdkit: vddk[1]: debug: DISKLIB-LIB   : Failed to open 'vpxa-nfc://[vsanDS-ClusterA] ac805f5f-829e-c35a-56a6-b026282184e0/ocp4-vmtest.vmdk.digital-pod3eur.local:902' with flags 0xe NBD_ERR_NETWORK_CONNECT (2338).\n"
2020-09-16 08:27:16,959 - root - DEBUG - b'nbdkit: vddk[1]: debug: VixDiskLib: Unable to locate appropriate transport mode to open disk. Error 13 (You do not have access rights to this file) at 5294.\n'
2020-09-16 08:27:16,959 - root - DEBUG - b'nbdkit: vddk[1]: debug: VixDiskLib: VixDiskLib_OpenEx: Cannot open disk [vsanDS-ClusterA] ac805f5f-829e-c35a-56a6-b026282184e0/ocp4-vmtest.vmdk. Error 13 (You do not have access rights to this file) at 5434.\n'
2020-09-16 08:27:16,959 - root - DEBUG - b'nbdkit: vddk[1]: debug: VixDiskLib: VixDiskLib_Open: Cannot open disk [vsanDS-ClusterA] ac805f5f-829e-c35a-56a6-b026282184e0/ocp4-vmtest.vmdk. Error 13 (You do not have access rights to this file) at 5478.\n'
2020-09-16 08:27:16,959 - root - DEBUG - b'nbdkit: vddk[1]: error: VixDiskLib_Open: [vsanDS-ClusterA] ac805f5f-829e-c35a-56a6-b026282184e0/ocp4-vmtest.vmdk: You do not have access rights to this file\n'
2020-09-16 08:27:16,959 - root - DEBUG - b'nbdkit: vddk[1]: debug: VDDK call: VixDiskLib_Disconnect (connection)\n'
2020-09-16 08:27:16,959 - root - DEBUG - b'nbdkit: vddk[1]: debug: VixDiskLib: VixDiskLib_Disconnect: Disconnect.\n'
2020-09-16 08:27:16,959 - root - DEBUG - b'nbdkit: vddk[1]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.\n'
2020-09-16 08:27:16,960 - root - DEBUG - b'nbdkit: vddk[1]: debug: VDDK call: VixDiskLib_FreeConnectParams (params)\n'
2020-09-16 08:27:16,960 - root - DEBUG - b'nbdkit: vddk[1]: debug: VixDiskLib: VixDiskLib_FreeConnectParams: Free connection parameters.\n'
2020-09-16 08:27:16,960 - root - DEBUG - b'nbdkit: vddk[1]: debug: vddk: open returned handle (nil)\n'
2020-09-16 08:27:16,960 - root - DEBUG - b'nbdkit: vddk[1]: debug: vddk: finalize\n'
2020-09-16 08:27:16,960 - root - DEBUG - b'nbdkit: vddk[1]: debug: vddk: close\n'
2020-09-16 08:27:16,960 - root - DEBUG - b'nbdkit: vddk[1]: debug: write: NBD_OPT_ABORT: Broken pipe\n'
2020-09-16 08:27:16,960 - root - DEBUG - b'nbdkit: vddk[1]: debug: vddk: close\n'
2020-09-16 08:27:16,959 - root - DEBUG - b'nbdkit: vddk[1]: debug: Cnx_Connect: Returning false because CnxConnectAuthd failed\n'
2020-09-16 08:27:16,959 - root - DEBUG - b'nbdkit: vddk[1]: debug: Cnx_Connect: Error message: Failed to connect to server esx3.mgt.digital-pod3eur.local:902\n'


Expected results:

IMPORT SUCCESSFUL

Additional info:
Not clear if this is a network issue (no connectivity to the ESX host on the 902) and it should be documented as a requirement on the docs or a missing permissions issue

Comment 1 Sudha Ponnaganti 2020-09-18 01:18:57 UTC

*** Bug 1879441 has been marked as a duplicate of this bug. ***

Comment 2 Brett Thurber 2020-09-21 18:05:20 UTC

Giuseppe, can you provide details on your environment?  Are you testing this on PSI by chance?

Regarding permissions, the VMware account used should have enough access rights to access the VM disk.  administrator or equivalent is recommended.

Regarding required ports/access, it is documented in the IMS product docs however not listed in the OCPV docs:
https://access.redhat.com/documentation/en-us/red_hat_infrastructure_migration_solution/1.2/html/infrastructure_migration_solution_guide/understanding_the_migration_rhv_1-2_vddk#Migration_network_requirements_rhv_1-2_vddk

OCP is slightly different in that there aren't conversion hosts or RHV managers.  Port 902 is required however.  Based on the log it points to a network connectivity issue:
NBD_ClientOpen: Couldn't connect to esx3.mgt.digital-pod3eur.local:902 Failed to connect to server esx3.mgt.digital-pod3eur.local:902\n

Comment 3 Giuseppe Cofano 2020-09-22 08:35:51 UTC

Hi Brett,

it's OCP v4.5, CNV v2.4, vSphere v6.7. OCP is deployed via UPI in a mixed environment, with virtual masters and infra and bare metal workers. What do you mean by PSI?

I'm gonna ask for opening the following ports:

OCP hosts -> ESX hosts: TCP/443
OCP hosts -> ESX hosts: TCP/902
OCP hosts -> ESX hosts: TCP/902
OCP hosts -> vCenter IP: TCP/5840

Can you please confirm before I proceed?

This requirement for the use-case of the import should be added to the CNV documentation in my opinion.

Thanks,
Giuseppe

Comment 5 Brett Thurber 2020-09-30 02:01:23 UTC

Any updates on this BZ?

Comment 6 Fabien Dupont 2020-10-13 12:07:56 UTC

Any update on this BZ?

Comment 7 Giuseppe Cofano 2020-10-13 13:52:23 UTC

Hi guys,

still waiting for confirmation about the set of openings needed:

OCP hosts -> ESX hosts: TCP/443
OCP hosts -> ESX hosts: TCP/902
OCP hosts -> ESX hosts: TCP/902
OCP hosts -> vCenter IP: TCP/5840

I think this should be added to the OCP docs as well

Comment 8 Fabien Dupont 2020-10-26 14:17:44 UTC

Brett confirmed that it was the right set of ports in https://bugzilla.redhat.com/show_bug.cgi?id=1879440#c4.

Comment 11 Avital Pinnick 2020-11-05 13:36:39 UTC

Changes merged.