Bug 1851903

Summary: kuryr cannot access namespaces in /var/run/netns properly
Product: OpenShift Container Platform Reporter: OpenShift BugZilla Robot <openshift-bugzilla-robot>
Component: NetworkingAssignee: MichaƂ Dulko <mdulko>
Networking sub component: kuryr QA Contact: GenadiC <gcheresh>
Status: CLOSED ERRATA Docs Contact:
Severity: low    
Priority: medium CC: pehunt, rlobillo
Version: 4.5   
Target Milestone: ---   
Target Release: 4.4.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-21 10:31:06 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1825339, 1831112    
Bug Blocks: 1838116    

Comment 3 Peter Hunt 2020-07-06 13:53:57 UTC
*** Bug 1825339 has been marked as a duplicate of this bug. ***

Comment 4 rlobillo 2020-07-07 10:42:00 UTC
cannot be verified until https://github.com/openshift/machine-config-operator/pull/1871 is included in OCP4.4 nightlies.

Comment 5 rlobillo 2020-07-07 15:32:23 UTC
back to MODIFIED until the dependent bugzilla is resolved

Comment 9 rlobillo 2020-07-10 08:46:26 UTC
verified on OCP4.4.0-0.nightly-2020-07-09-161409 on OSP16.1 with OVN (RHOS-16.1-RHEL-8-20200701.n.0).

# Installation successful: 

(overcloud) [stack@undercloud-0 ~]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-07-09-161409   True        False         9h      Cluster version is 4.4.0-0.nightly-2020-07-09-161409


# CNI pod can access to worker netns:

(overcloud) [stack@undercloud-0 ~]$ oc get pods -n openshift-kuryr -o wide
NAME                                READY   STATUS    RESTARTS   AGE   IP             NODE                        NOMINATED NODE   READINESS GATES
kuryr-cni-7gxp9                     1/1     Running   1          9h    10.196.0.181   ostest-sttrc-worker-5xzcr   <none>           <none>
kuryr-cni-crk4j                     1/1     Running   0          9h    10.196.0.61    ostest-sttrc-worker-w7bmq   <none>           <none>
kuryr-cni-kc4j4                     1/1     Running   0          9h    10.196.2.6     ostest-sttrc-master-1       <none>           <none>
kuryr-cni-n9qv7                     1/1     Running   2          9h    10.196.2.20    ostest-sttrc-master-2       <none>           <none>
kuryr-cni-tm2kn                     1/1     Running   0          9h    10.196.0.208   ostest-sttrc-worker-cr8k7   <none>           <none>
kuryr-cni-trch9                     1/1     Running   1          9h    10.196.1.209   ostest-sttrc-master-0       <none>           <none>
kuryr-controller-6d95b6c684-fgcd9   1/1     Running   2          9h    10.196.2.6     ostest-sttrc-master-1       <none>           <none>

(overcloud) [stack@undercloud-0 ~]$ ssh -J core.22.116 core.0.181 ip netns list | sort
Warning: Permanently added '10.46.22.116' (ECDSA) to the list of known hosts.
Warning: Permanently added '10.196.0.181' (ECDSA) to the list of known hosts.
09e3ad5b-9de6-4dd2-a93a-d70bdc3c258e (id: 5)
209dfef7-6ab8-4f95-949f-9b471bf7ec82 (id: 0)
279e594b-fde8-405e-8bc6-ac6389f911f1 (id: 13)
3a7e1310-63fd-4b6b-8516-bbe2f49286be (id: 17)
55fe93ef-4b08-4f23-92b9-8f75713cf516 (id: 14)
707d075f-7787-4666-966b-721d568fdf03 (id: 12)
878e0f21-a9cf-4f4e-8b5c-ee0ccd1bffe9 (id: 3)
9cdcaf79-1240-42c4-ab8d-d40fcc62fb62 (id: 11)
a4ad61bb-0995-4f61-abd0-58a46d341988 (id: 2)
a66fba3e-5b7b-4bda-acf1-e5395001f76c
af372a84-77b0-4d66-a47e-b1ecca1d8117 (id: 16)
bb2e63e9-02bb-4049-bf5d-605a449a7cba (id: 1)
c06df8cc-3cab-41a1-9dcd-fcd2b88be475 (id: 4)
d66b0761-d939-48a9-94ff-2b39d8b306fe (id: 10)
d70b46c8-02ca-4f07-bbd3-80e0470abfc9 (id: 9)

(overcloud) [stack@undercloud-0 ~]$ oc rsh -n openshift-kuryr kuryr-cni-7gxp9 ls -l /var/run/netns | awk '{print $9}'| sort 
09e3ad5b-9de6-4dd2-a93a-d70bdc3c258e
209dfef7-6ab8-4f95-949f-9b471bf7ec82
279e594b-fde8-405e-8bc6-ac6389f911f1
3a7e1310-63fd-4b6b-8516-bbe2f49286be
55fe93ef-4b08-4f23-92b9-8f75713cf516
707d075f-7787-4666-966b-721d568fdf03
878e0f21-a9cf-4f4e-8b5c-ee0ccd1bffe9
9cdcaf79-1240-42c4-ab8d-d40fcc62fb62
a4ad61bb-0995-4f61-abd0-58a46d341988
a66fba3e-5b7b-4bda-acf1-e5395001f76c
af372a84-77b0-4d66-a47e-b1ecca1d8117
bb2e63e9-02bb-4049-bf5d-605a449a7cba
c06df8cc-3cab-41a1-9dcd-fcd2b88be475
d66b0761-d939-48a9-94ff-2b39d8b306fe
d70b46c8-02ca-4f07-bbd3-80e0470abfc9

# kuryr-cni PODs mount namespaces on /run/netns directory:

(overcloud) [stack@undercloud-0 ~]$ oc get pods -n openshift-kuryr -o yaml | grep -e mountPath:.*netns -e 'name: kuryr-cni-'
    name: kuryr-cni-7gxp9
      - mountPath: /run/netns
    name: kuryr-cni-crk4j
      - mountPath: /run/netns
    name: kuryr-cni-kc4j4
      - mountPath: /run/netns
    name: kuryr-cni-n9qv7
      - mountPath: /run/netns
    name: kuryr-cni-tm2kn
      - mountPath: /run/netns
    name: kuryr-cni-trch9
      - mountPath: /run/netns

# and there is a link from /run/ to /var/ making it accesible through /var/run/netns:

(overcloud) [stack@undercloud-0 ~]$ for i in $(oc get pods -n openshift-kuryr -l app=kuryr-cni -o NAME); do oc rsh -n openshift-kuryr $i ls -larth /var/run; done
lrwxrwxrwx. 1 root root 6 Jun 22 07:26 /var/run -> ../run
lrwxrwxrwx. 1 root root 6 Jun 22 07:26 /var/run -> ../run
lrwxrwxrwx. 1 root root 6 Jun 22 07:26 /var/run -> ../run
lrwxrwxrwx. 1 root root 6 Jun 22 07:26 /var/run -> ../run
lrwxrwxrwx. 1 root root 6 Jun 22 07:26 /var/run -> ../run
lrwxrwxrwx. 1 root root 6 Jun 22 07:26 /var/run -> ../run

Comment 11 errata-xmlrpc 2020-07-21 10:31:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2913