Bug 1731597 - containerd/config.toml bin_dir /usr/libexec/cni/ instead of /opt/cni/bin [NEEDINFO]
Summary: containerd/config.toml bin_dir /usr/libexec/cni/ instead of /opt/cni/bin
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: containerd
Version: 32
Hardware: All
OS: All
Target Milestone: ---
Assignee: Olivier Lemasle
QA Contact: Fedora Extras Quality Assurance
Depends On:
TreeView+ depends on / blocked
Reported: 2019-07-19 22:46 UTC by Michael Vorburger.ch
Modified: 2020-05-04 20:44 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed:
Type: Bug
carl: needinfo? (lsm5)

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github kubernetes kubeadm issues 1677 0 'None' closed kubeadm on Fedora 30 with containerd and flannel is (surprisingly badly) broken 2020-11-26 12:06:35 UTC

Description Michael Vorburger.ch 2019-07-19 22:46:22 UTC
Description of problem: If you install Kubernetes on a simple single node cluster on a baremetal Fedora 30 Minimal server with "dnf install kubernetes-cni containerd" (from Fedora) created by "kubeadm init --pod-network-cidr=" following https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/, after get to the step where you do the step on https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/ for Installing a pod network add-on, you'll eventually get to a point where the kubedns Pod remains stuck in STATUS ContainerCreating.  A kubectl -n kube-system describe pod coredns-... and/or kubectl -n kube-system logs coredns-... will show:

   failed to find plugin "loopback" in path [/opt/cni/bin]

This is because two packages, both shipped by Fedora, disagree on where Container Networking Interface (CNI) binaries should be:

containernetworking-plugins (AKA kubernetes-cni) puts loopback (as well as bandwidth  bridge  dhcp  firewall  flannel  host-device  host-local  ipvlan  loopback  macvlan  portmap  ptp  sample  sbr  static  tuning  vlan)  into /usr/libexec/cni/.

containerd by default expects them in /opt/cni/bin.

I can locally work around this by adding this to /etc/containerd/config.toml:

    bin_dir = "/usr/libexec/cni/"

It seems to me that this should be included out of the box in the package.

If someone would like to guide me how to contribute this as a fix to Fedora, if there is a agreement that this would be the right way to fix this, then I may be interested in learning how to do this via this bug.

Version-Release number of selected component (if applicable):
* containerd 1.2.4-1.fc30 @fedora
* containernetworking-plugins 0.8.1-1.fc30 @updates

Comment 1 Michael Vorburger.ch 2019-07-19 22:59:01 UTC
I should probably mention that I have installed current Kubernetes 1.15.1 using "yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes" as per kubernetes.io/docs, after adding /etc/yum.repos.d/kubernetes.repo which pulls from https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64, so this is not using the kubernetes package from Fedora.. just because at 1.13.5-1.fc30 that seems way behind, and pulls that old docker 2:1.13.1-67 and what not, instead of modern containerd.  

Hopefully it's not crazy to expect that "dnf install containerd containernetworking-plugins" provides a consistent base in Fedora on top of which one can install Kubernetes from kubernetes.io instead of the old Fedora package.

Comment 2 Carl George 2019-08-26 20:53:01 UTC
I agree that if containerd is looking for those binaries it should be aware of the path Fedora ships them in.  The quick fix would be to include your config workaround in the default config file, but I think it would be better to patch the code to look in the right place.  If you have a patch file that does that, you can submit it as a pull request to https://src.fedoraproject.org/rpms/containerd.

Should containerd also depend on containernetworking-plugins?

Lokesh, I'd love your input here as well since you maintain containernetworking-plugins.

Comment 3 Ben Cotton 2020-04-30 20:19:42 UTC
This message is a reminder that Fedora 30 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 30 on 2020-05-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '30'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 30 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Note You need to log in before you can comment on or make changes to this bug.