Bug 1819129 - Cannot install - kuryr-cni pods in CrashLoopBackOff due to crio ns path
Summary: Cannot install - kuryr-cni pods in CrashLoopBackOff due to crio ns path
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.5
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.5.0
Assignee: Michał Dulko
QA Contact: GenadiC
URL:
Whiteboard:
Depends On:
Blocks: 1804734 1808498 1810137 1810501 1810517 1810571 1811022 1846225
TreeView+ depends on / blocked
 
Reported: 2020-03-31 09:39 UTC by Jon Uriarte
Modified: 2020-06-11 08:48 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-04-02 16:33:41 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-network-operator pull 562 0 None closed Bug 1831112: Kuryr: Mount /run/netns to ensure netns access 2020-10-23 08:06:54 UTC

Description Jon Uriarte 2020-03-31 09:39:53 UTC
Description of problem:

OCP 4.5 installer fails during bootstrap phase due to Kuryr-cni pods continuous restarts.

Version-Release number of selected component (if applicable):
OSP 13 2020-03-25.1
4.5.0-0.nightly-2020-03-30-203132

$ oc get pods -n openshift-kuryr
NAME                                   READY   STATUS              RESTARTS   AGE
kuryr-cni-5z27g                        1/1     Running             11         35m
kuryr-cni-f9sv6                        1/1     Running             10         35m
kuryr-cni-sjqjk                        0/1     CrashLoopBackOff    9          35m
kuryr-controller-7d6f7d849b-pkmlc      1/1     Running             0          35m
kuryr-dns-admission-controller-6rwcj   0/1     ContainerCreating   0          35m
kuryr-dns-admission-controller-98cws   0/1     ContainerCreating   0          35m
kuryr-dns-admission-controller-d2vn2   0/1     ContainerCreating   0          35m

kuryr-cni logs:
ERROR kuryr_kubernetes.cni.daemon.service [-] Error when processing delNetwork request. CNI Params: {'CNI_IFNAME': 'eth0', 'CNI_NETNS': '/var/run/crio/ns/8bb3f609-94c5-4910-8d0a-f28f774ad7b1/net', 'CNI_PATH': '/opt/multus/bin:/var/lib/cni/bin', 'CNI_COMMAND': 'DEL', 'CNI_CONTAINERID': '9b3df57af2aea563c6fbcf40e6e9a76e2a38d6b5af3c3aab918fd9e4957d705e', 'CNI_ARGS': 'IgnoreUnknown=true;K8S_POD_NAMESPACE=openshift-kuryr;K8S_POD_NAME=kuryr-dns-admission-controller-98cws;K8S_POD_INFRA_CONTAINER_ID=9b3df57af2aea563c6fbcf40e6e9a76e2a38d6b5af3c3aab918fd9e4957d705e'}.: FileNotFoundError: [Errno 2] No such file or directory: b'/var/run/crio/ns/8bb3f609-94c5-4910-8d0a-f28f774ad7b1'
ERROR kuryr_kubernetes.cni.daemon.service Traceback (most recent call last):
ERROR kuryr_kubernetes.cni.daemon.service   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/cni/daemon/service.py", line 103, in delete
ERROR kuryr_kubernetes.cni.daemon.service     self.plugin.delete(params)  
ERROR kuryr_kubernetes.cni.daemon.service   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/cni/plugins/k8s_cni_registry.py", line 102, in delete
ERROR kuryr_kubernetes.cni.daemon.service     self._do_work(params, b_base.disconnect)
ERROR kuryr_kubernetes.cni.daemon.service   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/cni/plugins/k8s_cni_registry.py", line 160, in _do_work
ERROR kuryr_kubernetes.cni.daemon.service     container_id=params.CNI_CONTAINERID)
ERROR kuryr_kubernetes.cni.daemon.service   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/cni/binding/base.py", line 166, in disconnect
ERROR kuryr_kubernetes.cni.daemon.service     driver.disconnect(vif, ifname, netns, container_id)
ERROR kuryr_kubernetes.cni.daemon.service   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/cni/binding/nested.py", line 118, in disconnect
ERROR kuryr_kubernetes.cni.daemon.service     with b_base.get_ipdb(netns) as c_ipdb:
ERROR kuryr_kubernetes.cni.daemon.service   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/cni/binding/base.py", line 70, in get_ipdb
ERROR kuryr_kubernetes.cni.daemon.service     ipdb = pyroute2.IPDB(nl=pyroute2.NetNS(netns))
ERROR kuryr_kubernetes.cni.daemon.service   File "/usr/lib/python3.6/site-packages/pyroute2/netns/nslink.py", line 172, in __init__
ERROR kuryr_kubernetes.cni.daemon.service     super(NetNS, self).__init__(trnsp_in, trnsp_out)
ERROR kuryr_kubernetes.cni.daemon.service   File "/usr/lib/python3.6/site-packages/pyroute2/iproute/linux.py", line 119, in __init__
ERROR kuryr_kubernetes.cni.daemon.service     super(RTNL_API, self).__init__(*argv, **kwarg)
ERROR kuryr_kubernetes.cni.daemon.service   File "/usr/lib/python3.6/site-packages/pyroute2/remote/__init__.py", line 225, in __init__
ERROR kuryr_kubernetes.cni.daemon.service     raise init['error']
ERROR kuryr_kubernetes.cni.daemon.service FileNotFoundError: [Errno 2] No such file or directory: b'/var/run/crio/ns/8bb3f609-94c5-4910-8d0a-f28f774ad7b1'



How reproducible: always


Steps to Reproduce:
1. Install OSP
2. Install OCP 4.5 with Kuryr

Actual results: Installation failure

Expected results: Successful installation

Comment 1 Jon Uriarte 2020-04-02 16:33:41 UTC
Closing this BZ as this issue was solved with https://github.com/openshift/machine-config-operator/pull/1600.

I could install 4.5.0-0.nightly-2020-04-02-115550 on OSP 13 (2020-03-25.1) and on OSP 16 (RHOS_TRUNK-16.0-RHEL-8-20200324.n.0).


Note You need to log in before you can comment on or make changes to this bug.