Bug 1589013

Summary: [CRI-O] podman cni network conflict with openshift-sdn
Product: OpenShift Container Platform Reporter: DeShuai Ma <dma>
Component: InstallerAssignee: Scott Dodson <sdodson>
Status: CLOSED CURRENTRELEASE QA Contact: DeShuai Ma <dma>
Severity: high Docs Contact:
Priority: high    
Version: 3.10.0CC: aos-bugs, hongli, jokerman, mmccomas, wmeng, wsun
Target Milestone: ---   
Target Release: 3.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
 Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-18 17:06:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1583591    
Bug Blocks: 1585038    
Attachments:
Description Flags
crio.log none

Description DeShuai Ma 2018-06-08 08:20:49 UTC 
Created attachment 1448992 [details]
crio.log

Description of problem:
After https://github.com/openshift/openshift-ansible/pull/8618 merged, it will install podman in node, this will cause to crio to use podman cni network, which conflict with  openshift-sdn, then all pods will keep ContainerCreating status and with error 'failed to find plugin "bridge" in path [/opt/cni/bin/ /opt/bridge/bin]'

Version-Release number of selected component (if applicable):
atomic-openshift--3.10.0-0.64.0
openshift-ansible-3.10.0-0.64.0.git.20.48df973.el7.noarch.rpm
cri-o-1.10.2-1.rhaos3.10.git095e88c.el7.x86_64
podman-0.4.1-4.gitb51d327.el7.x86_64
RHEL 7.5

How reproducible:
always

Steps to Reproduce:
1. Deploy OCP v3.10.0-0.64.0 with cri-o enabled then check pod status

root@ip-172-18-14-146 ~]# oc get pods --all-namespaces
NAMESPACE               NAME                                               READY     STATUS              RESTARTS   AGE
default                 docker-registry-1-deploy                           0/1       ContainerCreating   0          43m


[root@ip-172-18-14-146 ~]# oc describe pod docker-registry-1-deploy
Name:         docker-registry-1-deploy
Namespace:    default
Node:         ip-172-18-14-48.ec2.internal/172.18.14.48
Start Time:   Thu, 07 Jun 2018 23:37:09 -0400
Labels:       openshift.io/deployer-pod-for.name=docker-registry-1
Annotations:  openshift.io/deployment-config.name=docker-registry
              openshift.io/deployment.name=docker-registry-1
              openshift.io/scc=restricted
Status:       Pending
IP:           
Containers:
  deployment:
    Container ID:   
    Image:          registry.reg-aws.openshift.com:443/openshift3/ose-deployer:v3.10.0-0.64.0
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:
      OPENSHIFT_DEPLOYMENT_NAME:       docker-registry-1
      OPENSHIFT_DEPLOYMENT_NAMESPACE:  default
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from deployer-token-p4fxw (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  deployer-token-p4fxw:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  deployer-token-p4fxw
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  registry=enabled
                 role=node
Tolerations:     <none>
Events:
  Type     Reason                  Age                 From                                   Message
  ----     ------                  ----                ----                                   -------
   Normal   Scheduled               42m                  default-scheduler                      Successfully assigned  docker-registry-1-deploy to ip-172-18-14-48.ec2.internal
   Warning  FailedCreatePodSandBox  42m                 kubelet,  ip-172-18-14-48.ec2.internal  Failed create pod sandbox: rpc error: code  = Unknown desc = failed to create pod network sandbox  k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(5c74334796a33e72ec25c7804db2ef423a0631ecbeaa81e53a327f3d68b9f0de):  failed to find plugin "loopback" in path [/opt/cni/bin/  /opt/loopback/bin]
   Warning  FailedCreatePodSandBox  42m                 kubelet,  ip-172-18-14-48.ec2.internal  Failed create pod sandbox: rpc error: code  = Unknown desc = failed to create pod network sandbox  k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(c9aa6a55460ee5151a8d347d822a2cf58fffc6a02481354df356d7666107ddf6):  failed to find plugin "loopback" in path [/opt/cni/bin/  /opt/loopback/bin]
   Warning  FailedCreatePodSandBox  42m                 kubelet,  ip-172-18-14-48.ec2.internal  Failed create pod sandbox: rpc error: code  = Unknown desc = failed to create pod network sandbox  k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(6f6ae503ea6049e1e0f41fe569b22d4f7f374c6cc56d8f73c0de07ce11516162):  failed to find plugin "loopback" in path [/opt/cni/bin/  /opt/loopback/bin]
   Warning  FailedCreatePodSandBox  42m                 kubelet,  ip-172-18-14-48.ec2.internal  Failed create pod sandbox: rpc error: code  = Unknown desc = failed to create pod network sandbox  k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(cebec6c353c4e632ca7887e2ec657d97104b6b7c80d55de36071ba7fa7dd15e9):  failed to find plugin "bridge" in path [/opt/cni/bin/ /opt/bridge/bin]
   Warning  FailedCreatePodSandBox  42m                 kubelet,  ip-172-18-14-48.ec2.internal  Failed create pod sandbox: rpc error: code  = Unknown desc = failed to create pod network sandbox  k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(6f398f2c26504de4e01af61b204f246606d78b7b6428fc935e3760fde99bbd40):  failed to find plugin "bridge" in path [/opt/cni/bin/ /opt/bridge/bin]
   Warning  FailedCreatePodSandBox  41m                 kubelet,  ip-172-18-14-48.ec2.internal  Failed create pod sandbox: rpc error: code  = Unknown desc = failed to create pod network sandbox  k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(6911691de666a1922cf2b586e4c32c682db2fde2d917ae7670205c4811f15289):  failed to find plugin "bridge" in path [/opt/cni/bin/ /opt/bridge/bin]
   Warning  FailedCreatePodSandBox  41m                 kubelet,  ip-172-18-14-48.ec2.internal  Failed create pod sandbox: rpc error: code  = Unknown desc = failed to create pod network sandbox  k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(093179c9d3b43501232961a37d71f444cbab703356b9076be533c03491ca0b65):  failed to find plugin "bridge" in path [/opt/cni/bin/ /opt/bridge/bin]
   Warning  FailedCreatePodSandBox  41m                 kubelet,  ip-172-18-14-48.ec2.internal  Failed create pod sandbox: rpc error: code  = Unknown desc = failed to create pod network sandbox  k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(89cca132c07b3aaa92d40dc9937269867d8e268257c270e4b70ffeaf9a99d90c):  failed to find plugin "bridge" in path [/opt/cni/bin/ /opt/bridge/bin]
   Warning  FailedCreatePodSandBox  41m                 kubelet,  ip-172-18-14-48.ec2.internal  Failed create pod sandbox: rpc error: code  = Unknown desc = failed to create pod network sandbox  k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(f28907afdcd619cc8731f4e5e965657fb142f4a3f99289409a79eba138092d2c):  failed to find plugin "bridge" in path [/opt/cni/bin/ /opt/bridge/bin]
   Warning  FailedCreatePodSandBox  41m                 kubelet,  ip-172-18-14-48.ec2.internal  Failed create pod sandbox: rpc error: code  = Unknown desc = failed to create pod network sandbox  k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(91d1cf23758f88939b8a2d7f0e1fb4e289f6ad80a036d37f500d699c07b9c804):  failed to find plugin "bridge" in path [/opt/cni/bin/ /opt/bridge/bin]
   Warning  FailedCreatePodSandBox  40m                 kubelet,  ip-172-18-14-48.ec2.internal  Failed create pod sandbox: rpc error: code  = Unknown desc = failed to create pod network sandbox  k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(8e9e32ee024f034ae1b48d1848c32eb2ce7f4082ea09fd9e18b2a4529235c9a5):  failed to find plugin "bridge" in path [/opt/cni/bin/ /opt/bridge/bin]
   Warning  FailedCreatePodSandBox  2m (x191 over 40m)  kubelet,  ip-172-18-14-48.ec2.internal  (combined from similar events): Failed  create pod sandbox: rpc error: code = Unknown desc = failed to create  pod network sandbox  k8s_docker-registry-1-deploy_default_37f9546c-6acd-11e8-a09b-0e5f74f0e954_0(1cd587cbb140f9ddb70cfb444a57412489335912c75342cad48bf35053c3a8a0):  failed to find plugin "bridge" in path [/opt/cni/bin/ /opt/bridge/bin]


Actual Result:
Pod couldn't be deployed 

Expected Result:
Pod can be running

Additional info:
Comment 1 Scott Dodson 2018-06-08 12:46:34 UTC 
https://github.com/openshift/openshift-ansible/pull/8686 to remove podman from the install/upgrade, we added it prematurely

There's ongoing discussion in https://bugzilla.redhat.com/show_bug.cgi?id=1583591 as to whether or not we should add it and when.
Comment 3 DeShuai Ma 2018-06-11 02:10:12 UTC 
The latest version is atomic-openshift-3.10.0-0.64.0.git.0.8285be6.el7.x86_64.rpm   
Wait for new build.
Comment 4 Scott Dodson 2018-06-12 12:52:25 UTC 
In openshift-ansible-3.10.0-0.65.0 and later
Comment 5 DeShuai Ma 2018-06-13 07:53:35 UTC 
Verified on openshift-ansible-3.10.0-0.66.0.git.79.68197f9.el7.noarch.rpm