Created attachment 1448992 [details] crio.log Description of problem: After https://github.com/openshift/openshift-ansible/pull/8618 merged, it will install podman in node, this will cause to crio to use podman cni network, which conflict with openshift-sdn, then all pods will keep ContainerCreating status and with error 'failed to find plugin "bridge" in path [/opt/cni/bin/ /opt/bridge/bin]' Version-Release number of selected component (if applicable): atomic-openshift--3.10.0-0.64.0 openshift-ansible-3.10.0-0.64.0.git.20.48df973.el7.noarch.rpm cri-o-1.10.2-1.rhaos3.10.git095e88c.el7.x86_64 podman-0.4.1-4.gitb51d327.el7.x86_64 RHEL 7.5 How reproducible: always Steps to Reproduce: 1. Deploy OCP v3.10.0-0.64.0 with cri-o enabled then check pod status root@ip-172-18-14-146 ~]# oc get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default docker-registry-1-deploy 0/1 ContainerCreating 0 43m [root@ip-172-18-14-146 ~]# oc describe pod docker-registry-1-deploy Name: docker-registry-1-deploy Namespace: default Node: ip-172-18-14-48.ec2.internal/172.18.14.48 Start Time: Thu, 07 Jun 2018 23:37:09 -0400 Labels: openshift.io/deployer-pod-for.name=docker-registry-1 Annotations: openshift.io/deployment-config.name=docker-registry openshift.io/deployment.name=docker-registry-1 openshift.io/scc=restricted Status: Pending IP: Containers: deployment: Container ID: Image: registry.reg-aws.openshift.com:443/openshift3/ose-deployer:v3.10.0-0.64.0 Image ID: Port: <none> Host Port: <none> State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment: OPENSHIFT_DEPLOYMENT_NAME: docker-registry-1 OPENSHIFT_DEPLOYMENT_NAMESPACE: default Mounts: /var/run/secrets/kubernetes.io/serviceaccount from deployer-token-p4fxw (ro) Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: deployer-token-p4fxw: Type: Secret (a volume populated by a Secret) SecretName: deployer-token-p4fxw Optional: false QoS Class: BestEffort Node-Selectors: registry=enabled role=node Tolerations: <none> Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 42m default-scheduler Successfully assigned docker-registry-1-deploy to ip-172-18-14-48.ec2.internal Warning FailedCreatePodSandBox 42m kubelet, ip-172-18-14-48.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(5c74334796a33e72ec25c7804db2ef423a0631ecbeaa81e53a327f3d68b9f0de): failed to find plugin "loopback" in path [/opt/cni/bin/ /opt/loopback/bin] Warning FailedCreatePodSandBox 42m kubelet, ip-172-18-14-48.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(c9aa6a55460ee5151a8d347d822a2cf58fffc6a02481354df356d7666107ddf6): failed to find plugin "loopback" in path [/opt/cni/bin/ /opt/loopback/bin] Warning FailedCreatePodSandBox 42m kubelet, ip-172-18-14-48.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(6f6ae503ea6049e1e0f41fe569b22d4f7f374c6cc56d8f73c0de07ce11516162): failed to find plugin "loopback" in path [/opt/cni/bin/ /opt/loopback/bin] Warning FailedCreatePodSandBox 42m kubelet, ip-172-18-14-48.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(cebec6c353c4e632ca7887e2ec657d97104b6b7c80d55de36071ba7fa7dd15e9): failed to find plugin "bridge" in path [/opt/cni/bin/ /opt/bridge/bin] Warning FailedCreatePodSandBox 42m kubelet, ip-172-18-14-48.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(6f398f2c26504de4e01af61b204f246606d78b7b6428fc935e3760fde99bbd40): failed to find plugin "bridge" in path [/opt/cni/bin/ /opt/bridge/bin] Warning FailedCreatePodSandBox 41m kubelet, ip-172-18-14-48.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(6911691de666a1922cf2b586e4c32c682db2fde2d917ae7670205c4811f15289): failed to find plugin "bridge" in path [/opt/cni/bin/ /opt/bridge/bin] Warning FailedCreatePodSandBox 41m kubelet, ip-172-18-14-48.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(093179c9d3b43501232961a37d71f444cbab703356b9076be533c03491ca0b65): failed to find plugin "bridge" in path [/opt/cni/bin/ /opt/bridge/bin] Warning FailedCreatePodSandBox 41m kubelet, ip-172-18-14-48.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(89cca132c07b3aaa92d40dc9937269867d8e268257c270e4b70ffeaf9a99d90c): failed to find plugin "bridge" in path [/opt/cni/bin/ /opt/bridge/bin] Warning FailedCreatePodSandBox 41m kubelet, ip-172-18-14-48.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(f28907afdcd619cc8731f4e5e965657fb142f4a3f99289409a79eba138092d2c): failed to find plugin "bridge" in path [/opt/cni/bin/ /opt/bridge/bin] Warning FailedCreatePodSandBox 41m kubelet, ip-172-18-14-48.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(91d1cf23758f88939b8a2d7f0e1fb4e289f6ad80a036d37f500d699c07b9c804): failed to find plugin "bridge" in path [/opt/cni/bin/ /opt/bridge/bin] Warning FailedCreatePodSandBox 40m kubelet, ip-172-18-14-48.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(8e9e32ee024f034ae1b48d1848c32eb2ce7f4082ea09fd9e18b2a4529235c9a5): failed to find plugin "bridge" in path [/opt/cni/bin/ /opt/bridge/bin] Warning FailedCreatePodSandBox 2m (x191 over 40m) kubelet, ip-172-18-14-48.ec2.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(1cd587cbb140f9ddb70cfb444a57412489335912c75342cad48bf35053c3a8a0): failed to find plugin "bridge" in path [/opt/cni/bin/ /opt/bridge/bin] Actual Result: Pod couldn't be deployed Expected Result: Pod can be running Additional info:
https://github.com/openshift/openshift-ansible/pull/8686 to remove podman from the install/upgrade, we added it prematurely There's ongoing discussion in https://bugzilla.redhat.com/show_bug.cgi?id=1583591 as to whether or not we should add it and when.
The latest version is atomic-openshift-3.10.0-0.64.0.git.0.8285be6.el7.x86_64.rpm Wait for new build.
In openshift-ansible-3.10.0-0.65.0 and later
Verified on openshift-ansible-3.10.0-0.66.0.git.79.68197f9.el7.noarch.rpm