Description of problem: Installing crio will add cni configuration files with out adding the cni plugins. When cri-o is installed with out a openshift using it this will cause issues with the kubelet starting a pod due to trying to load the cni bridge plugin. The plugins are also never installed so these files are not needed. # ls /etc/cni/net.d/ | sort -d 100-crio-bridge.conf 200-loopback.conf 80-openshift-network.conf Version-Release number of selected component (if applicable): # rpm -qf /etc/cni/net.d/200-loopback.conf # rpm -qf /etc/cni/net.d/100-crio-bridge.conf cri-o-1.9.10-1.git8723732.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. Yum install cri-o 2. Install openshift before or after the above, no enabling cri-o Actual results: atomic-openshift-node[33346]: E0515 18:40:53.018117 33346 cni.go:259] Error adding network: failed to find plugin "bridge" in path [/opt/bridge/bin /opt/cni/bin] atomic-openshift-node[33346]: E0515 18:40:53.018136 33346 cni.go:227] Error while adding to cni network: failed to find plugin "bridge" in path [/opt/bridge/bin /opt/cni/bin] Expected results: cri-o package install to no add cni configuration files. Additional info: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/#cni If there are multiple CNI configuration files in the directory, the first one in lexicographic order of file name is used. # ls /etc/cni/net.d/ | sort -d 100-crio-bridge.conf 200-loopback.conf 80-openshift-network.conf Plugins Loaded: https://github.com/openshift/origin/blob/release-3.9/vendor/k8s.io/kubernetes/pkg/kubelet/network/cni/cni.go#L257 https://github.com/openshift/origin/blob/release-3.9/vendor/github.com/containernetworking/cni/libcni/conf.go#L151-L171 When crio is enabled via the OpenShift installer anisble deletes the cni plugin configuration files: - This is the only play that will make sure those files are absent: https://github.com/openshift/openshift-ansible/blob/release-3.9/roles/container_runtime/tasks/package_crio.yml#L40-L46 - Its called from this play: https://github.com/openshift/openshift-ansible/blob/release-3.9/playbooks/container-runtime/private/config.yml#L39-L45
Also this when cri-o is enabled on an OpenShift node and then cri-o is updated it will add these back on the host. # rm /etc/cni/net.d/200-loopback.conf # rm /etc/cni/net.d/100-crio-bridge.conf # yum update cri-o # ls /etc/cni/net.d/ 100-crio-bridge.conf 200-loopback.conf 80-openshift-network.conf Causing the OpenShift node to fail to start pods with erro "Error while adding to cni network: failed to find plugin "bridge" in path [/opt/bridge/bin /opt/cni/bin]" Manual deletion is needed then.
mmm I'm fine removing those configurations and provide them as another cri-o-cni-configs RPM. We can suggest people to install this new package when running CRI-O w/o openshift or with kube. Lokesh, can we ship such an RPM containing just CNI config files? Mrunal, Dan, wdyt?
I believe this is not a 3.10 blocker though. The installer takes care of the network just fine, this is an edge case when CRI-O is installed prior to installing openshift. I'm removing this from 3.10 but feel free to move it back if you think so.
Hi, We have another customer asking to see if it is possible to add cri-o to the atomic-openshift-excluder or create one like we have for docker. Every time we update system and cri-o gets updated it will add the files already mentioned in this bugzila. Or you can rename the files because now following that order the atomic-openshift-node service will fail since 80-openshift-network.conf gets in the last place. I'm uploading an example I did with OCP 3.11 repos. Thanks,
Created attachment 1510852 [details] Yum update cri-o example
(In reply to Antonio Murdaca from comment #2) > mmm I'm fine removing those configurations and provide them as another > cri-o-cni-configs RPM. We can suggest people to install this new package > when running CRI-O w/o openshift or with kube. > Lokesh, can we ship such an RPM containing just CNI config files? > Mrunal, Dan, wdyt? Frantisek, can you please make this change ^
Lokesh can you take this over.
OCP 3.6-3.10 is no longer on full support [1]. Marking CLOSED DEFERRED. If you have a customer case with a support exception or have reproduced on 3.11+, please reopen and include those details. When reopening, please set the Target Release to the appropriate version where needed. [1]: https://access.redhat.com/support/policy/updates/openshift
Hello, This is for OCP 3.11 which is the current version on the customer and I was able to reproduce this issue recently.
Verified with cri-o-1.11.16-0.4.dev.rhaos3.11.git3f89eba.el7 # rpm -ql cri-o /etc/crictl.yaml /etc/crio /etc/crio/crio.conf /etc/crio/seccomp.json /etc/sysconfig/crio-network /etc/sysconfig/crio-storage /usr/bin/crio /usr/lib/systemd/system/cri-o.service /usr/lib/systemd/system/crio-shutdown.service /usr/lib/systemd/system/crio.service /usr/libexec/crio /usr/libexec/crio/conmon /usr/libexec/crio/pause /usr/share/doc/cri-o-1.11.16 /usr/share/doc/cri-o-1.11.16/README.md /usr/share/licenses/cri-o-1.11.16 /usr/share/licenses/cri-o-1.11.16/LICENSE /usr/share/man/man5/crio.conf.5.gz /usr/share/man/man8/crio.8.gz /usr/share/oci-umount /usr/share/oci-umount/oci-umount.d /usr/share/oci-umount/oci-umount.d/crio-umount.conf /var/lib/containers # rpm -qa|grep -i cri-o cri-o-1.11.16-0.4.dev.rhaos3.11.git3f89eba.el7.x86_64
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:4050