Bug 1731597 - containerd/config.toml bin_dir /usr/libexec/cni/ instead of /opt/cni/bin [NEEDINFO]
Summary: containerd/config.toml bin_dir /usr/libexec/cni/ instead of /opt/cni/bin
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: containerd
Version: 30
Hardware: All
OS: All
unspecified
high
Target Milestone: ---
Assignee: Carl George
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-19 22:46 UTC by Michael Vorburger.ch
Modified: 2019-08-26 20:53 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
carl: needinfo? (lsm5)


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github kubernetes kubeadm issues 1677 'None' 'closed' 'kubeadm on Fedora 30 with containerd and flannel is (surprisingly badly) broken' 2019-11-20 16:05:58 UTC

Description Michael Vorburger.ch 2019-07-19 22:46:22 UTC
Description of problem: If you install Kubernetes on a simple single node cluster on a baremetal Fedora 30 Minimal server with "dnf install kubernetes-cni containerd" (from Fedora) created by "kubeadm init --pod-network-cidr=10.10.0.0/16" following https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/, after get to the step where you do the step on https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/ for Installing a pod network add-on, you'll eventually get to a point where the kubedns Pod remains stuck in STATUS ContainerCreating.  A kubectl -n kube-system describe pod coredns-... and/or kubectl -n kube-system logs coredns-... will show:

   failed to find plugin "loopback" in path [/opt/cni/bin]

This is because two packages, both shipped by Fedora, disagree on where Container Networking Interface (CNI) binaries should be:

containernetworking-plugins (AKA kubernetes-cni) puts loopback (as well as bandwidth  bridge  dhcp  firewall  flannel  host-device  host-local  ipvlan  loopback  macvlan  portmap  ptp  sample  sbr  static  tuning  vlan)  into /usr/libexec/cni/.

containerd by default expects them in /opt/cni/bin.

I can locally work around this by adding this to /etc/containerd/config.toml:

[plugins.cri]
  [plugins.cri.cni]
    bin_dir = "/usr/libexec/cni/"

It seems to me that this should be included out of the box in the package.

If someone would like to guide me how to contribute this as a fix to Fedora, if there is a agreement that this would be the right way to fix this, then I may be interested in learning how to do this via this bug.

Version-Release number of selected component (if applicable):
* containerd 1.2.4-1.fc30 @fedora
* containernetworking-plugins 0.8.1-1.fc30 @updates

Comment 1 Michael Vorburger.ch 2019-07-19 22:59:01 UTC
I should probably mention that I have installed current Kubernetes 1.15.1 using "yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes" as per kubernetes.io/docs, after adding /etc/yum.repos.d/kubernetes.repo which pulls from https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64, so this is not using the kubernetes package from Fedora.. just because at 1.13.5-1.fc30 that seems way behind, and pulls that old docker 2:1.13.1-67 and what not, instead of modern containerd.  

Hopefully it's not crazy to expect that "dnf install containerd containernetworking-plugins" provides a consistent base in Fedora on top of which one can install Kubernetes from kubernetes.io instead of the old Fedora package.

Comment 2 Carl George 2019-08-26 20:53:01 UTC
I agree that if containerd is looking for those binaries it should be aware of the path Fedora ships them in.  The quick fix would be to include your config workaround in the default config file, but I think it would be better to patch the code to look in the right place.  If you have a patch file that does that, you can submit it as a pull request to https://src.fedoraproject.org/rpms/containerd.

Should containerd also depend on containernetworking-plugins?

Lokesh, I'd love your input here as well since you maintain containernetworking-plugins.


Note You need to log in before you can comment on or make changes to this bug.