Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1831112

Summary:	kuryr cannot access namespaces in /var/run/netns properly
Product:	OpenShift Container Platform	Reporter:	Peter Hunt <pehunt>
Component:	Networking	Assignee:	Michał Dulko <mdulko>
Networking sub component:	kuryr	QA Contact:	GenadiC <gcheresh>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	low
Priority:	medium	CC:	rlobillo
Version:	4.5
Target Milestone:	---
Target Release:	4.5.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-07-13 17:34:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1851903

Description Peter Hunt 2020-05-04 16:33:51 UTC

Description of problem:
If a networking plugin needs to access namespaces from a container, they need to be mounted private or rshared in the container. Otherwise they cannot be accessed as namespaces

Version-Release number of selected component (if applicable):
4.5

How reproducible:


Steps to Reproduce:
1. Carry https://github.com/openshift/machine-config-operator/pull/1689 patch (so cri-o has its namespaces in /var/run/netns
2. exec into the kuryr container and run `mount | grep /var/run'

Actual results:
namespaces are not listed as nsfs

Expected results:
namespaces should be listed as nsfs

Comment 4 rlobillo 2020-05-22 10:36:08 UTC

Verified on 4.5.0-0.nightly-2020-05-22-011031 on top of RHOS16 - RHOS_TRUNK-16.0-RHEL-8-20200506.n.2.

CNI pod can access to worker netns:

# worker netns:
(shiftstack) [stack@undercloud-0 ~]$ ssh -J core.22.93 core@$(openstack server show ostest-x7gwn-worker-ndzxn -c addresses -f value | awk -F\= '{print $2}') ip netns list
Warning: Permanently added '10.46.22.93' (ECDSA) to the list of known hosts.
Warning: Permanently added '10.196.0.14' (ECDSA) to the list of known hosts.
c20735a0-1094-47e9-a991-086b09a2458a (id: 58)
205e3c41-bfae-4e7c-8f86-1202668ddd45 (id: 57)
7a730917-5bcf-448a-afec-ce84426b0b32 (id: 45)
bd468054-9fa4-429f-bcd7-42f2b03f290e (id: 37)
99914fb0-2b57-4284-b4f8-5e14873a2824 (id: 23)
ddb5e62d-d245-4e7f-a91e-e5c522220a98 (id: 0)

# links to worker netns from kuryr-CNI POD:
(shiftstack) [stack@undercloud-0 ~]$ oc rsh -n openshift-kuryr kuryr-cni-cbn8g ls -larth /var/run/netns
total 0
drwxr-xr-x. 1 root root  38 May 22 08:16 ..
-r--r--r--. 1 root root   0 May 22 08:16 ddb5e62d-d245-4e7f-a91e-e5c522220a98
-r--r--r--. 1 root root   0 May 22 09:26 99914fb0-2b57-4284-b4f8-5e14873a2824
-r--r--r--. 1 root root   0 May 22 09:38 bd468054-9fa4-429f-bcd7-42f2b03f290e
-r--r--r--. 1 root root   0 May 22 10:05 7a730917-5bcf-448a-afec-ce84426b0b32
-r--r--r--. 1 root root   0 May 22 10:30 205e3c41-bfae-4e7c-8f86-1202668ddd45
-r--r--r--. 1 root root   0 May 22 10:30 c20735a0-1094-47e9-a991-086b09a2458a
drwxr-xr-x. 2 root root 160 May 22 10:30 .

Namespaces are not mounted on /var/run/netns but in /run/netns. There is a linked from /run/ to /var/ so their contents are linked:

1. Cluster up and running:

(shiftstack) [stack@undercloud-0 ~]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-0.nightly-2020-05-22-011031   True        False         62m     Cluster version is 4.5.0-0.nightly-2020-05-22-011031

(shiftstack) [stack@undercloud-0 ~]$ oc get pods -n openshift-kuryr -o wide
NAME                                   READY   STATUS    RESTARTS   AGE    IP              NODE                        NOMINATED NODE   READINESS GATES
kuryr-cni-cbn8g                        1/1     Running   0          86m    10.196.0.14     ostest-x7gwn-worker-ndzxn   <none>           <none>
kuryr-cni-cd59d                        1/1     Running   3          141m   10.196.1.203    ostest-x7gwn-master-2       <none>           <none>
kuryr-cni-cp475                        1/1     Running   5          141m   10.196.2.74     ostest-x7gwn-master-0       <none>           <none>
kuryr-cni-mnskw                        1/1     Running   0          86m    10.196.3.185    ostest-x7gwn-worker-28wb5   <none>           <none>
kuryr-cni-q7n5p                        1/1     Running   4          141m   10.196.0.180    ostest-x7gwn-master-1       <none>           <none>
kuryr-cni-v2hzs                        1/1     Running   1          87m    10.196.1.170    ostest-x7gwn-worker-n2xmb   <none>           <none>
kuryr-controller-76fcdf8bc6-rpp6d      1/1     Running   2          141m   10.196.0.180    ostest-x7gwn-master-1       <none>           <none>
kuryr-dns-admission-controller-49pmg   1/1     Running   0          141m   10.128.65.81    ostest-x7gwn-master-2       <none>           <none>
kuryr-dns-admission-controller-9dphv   1/1     Running   0          141m   10.128.64.244   ostest-x7gwn-master-1       <none>           <none>
kuryr-dns-admission-controller-mdfmd   1/1     Running   0          141m   10.128.64.89    ostest-x7gwn-master-0       <none>           <none>

2. kuryr-cni PODs mount namespaces on /run/netns directory:

(shiftstack) [stack@undercloud-0 ~]$ oc get pods -n openshift-kuryr -o yaml | grep -e mountPath:.*netns -e 'name: kuryr-cni-'
    name: kuryr-cni-cbn8g
      - mountPath: /run/netns
    name: kuryr-cni-cd59d
      - mountPath: /run/netns
    name: kuryr-cni-cp475
      - mountPath: /run/netns
    name: kuryr-cni-mnskw
      - mountPath: /run/netns
    name: kuryr-cni-q7n5p
      - mountPath: /run/netns
    name: kuryr-cni-v2hzs
      - mountPath: /run/netns

3. There is a link from /run/ to /var/run/ so the content of /run/netns and /var/run/netns is identical:

(shiftstack) [stack@undercloud-0 ~]$ oc rsh -n openshift-kuryr kuryr-cni-cbn8g
sh-4.4# mount | grep netns
tmpfs on /run/netns type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
nsfs on /run/netns/ddb5e62d-d245-4e7f-a91e-e5c522220a98 type nsfs (rw,seclabel)
nsfs on /run/netns/99914fb0-2b57-4284-b4f8-5e14873a2824 type nsfs (rw,seclabel)
nsfs on /run/netns/a8f0f69c-6f70-49fb-9b58-38b9e9dbffcd type nsfs (rw,seclabel)
nsfs on /run/netns/05c92b77-afbb-4a89-9289-233781be14dc type nsfs (rw,seclabel)
sh-4.4# df -kh 
Filesystem                            Size  Used Avail Use% Mounted on
overlay                                40G  8.1G   32G  21% /
tmpfs                                  64M     0   64M   0% /dev
tmpfs                                 7.9G     0  7.9G   0% /sys/fs/cgroup
shm                                    64M  8.0K   64M   1% /dev/shm
tmpfs                                 7.9G  4.1M  7.9G   1% /run/netns
/dev/mapper/coreos-luks-root-nocrypt   40G  8.1G   32G  21% /etc/hosts
tmpfs                                 7.9G   28K  7.9G   1% /run/secrets/kubernetes.io/serviceaccount
sh-4.4# ls -alrth /var/run/netns/
total 0
drwxr-xr-x. 1 root root  38 May 22 08:16 ..
-r--r--r--. 1 root root   0 May 22 08:16 ddb5e62d-d245-4e7f-a91e-e5c522220a98
-r--r--r--. 1 root root   0 May 22 09:26 99914fb0-2b57-4284-b4f8-5e14873a2824
-r--r--r--. 1 root root   0 May 22 09:33 05c92b77-afbb-4a89-9289-233781be14dc
-r--r--r--. 1 root root   0 May 22 09:34 fc70b1b8-ef36-472f-9b93-1ad8e4104072
drwxr-xr-x. 2 root root 120 May 22 09:34 .
sh-4.4# ls -larth /var/ | grep run
lrwxrwxrwx. 1 root root    6 Apr 23 05:13 run -> ../run
sh-4.4# ls -arlth /run/netns/
total 0
drwxr-xr-x. 1 root root  38 May 22 08:16 ..
-r--r--r--. 1 root root   0 May 22 08:16 ddb5e62d-d245-4e7f-a91e-e5c522220a98
-r--r--r--. 1 root root   0 May 22 09:26 99914fb0-2b57-4284-b4f8-5e14873a2824
-r--r--r--. 1 root root   0 May 22 09:33 05c92b77-afbb-4a89-9289-233781be14dc
-r--r--r--. 1 root root   0 May 22 09:34 fc70b1b8-ef36-472f-9b93-1ad8e4104072
drwxr-xr-x. 2 root root 120 May 22 09:34 .

Comment 5 errata-xmlrpc 2020-07-13 17:34:48 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409