Bug 1623989

Description of problem:

Openshift-on-Openstack installer playbooks end successfully, with no errors, but some pods in the default namespace remain in Error status.
It's currently happening in almost 100% of the deployments. The pod that fails is somehow aleatory, but there is always at least one pod that ends up in Error status.
In this case the docker-registry could not be deployed:

$ oc get pods -o wide                                                                                                                                                                        
NAME                       READY     STATUS              RESTARTS   AGE
docker-registry-1-deploy   1/1       Running             0          10m
docker-registry-1-rg9jj    0/1       ContainerCreating   0          2m
registry-console-1-mdh8z   1/1       Running             0          21s
router-1-v5dv4             1/1       Running             0          2m

$ oc logs docker-registry-1-rg9jj
time="2018-08-30T14:52:15Z" level=warning msg="Ignoring unrecognized environment variable REGISTRY_CONSOLE_PORT" 
time="2018-08-30T14:52:15Z" level=warning msg="Ignoring unrecognized environment variable REGISTRY_CONSOLE_PORT_9000_TCP" 
time="2018-08-30T14:52:15Z" level=warning msg="Ignoring unrecognized environment variable REGISTRY_CONSOLE_PORT_9000_TCP_ADDR" 
time="2018-08-30T14:52:15Z" level=warning msg="Ignoring unrecognized environment variable REGISTRY_CONSOLE_PORT_9000_TCP_PORT" 
time="2018-08-30T14:52:15Z" level=warning msg="Ignoring unrecognized environment variable REGISTRY_CONSOLE_PORT_9000_TCP_PROTO" 
time="2018-08-30T14:52:15Z" level=warning msg="Ignoring unrecognized environment variable REGISTRY_CONSOLE_SERVICE_HOST" 
time="2018-08-30T14:52:15Z" level=warning msg="Ignoring unrecognized environment variable REGISTRY_CONSOLE_SERVICE_PORT" 
time="2018-08-30T14:52:15Z" level=warning msg="Ignoring unrecognized environment variable REGISTRY_CONSOLE_SERVICE_PORT_REGISTRY_CONSOLE" 
time="2018-08-30T14:52:15.919779213Z" level=info msg="start registry" distribution_version=v2.6.2+unknown go.version=go1.9.4 instance.id=1ba98ace-b855-4205-927f-bf36899de2b7 openshift_version=v3.10.34 
time="2018-08-30T14:52:15.920140907Z" level=info msg="quota enforcement disabled" go.version=go1.9.4 instance.id=1ba98ace-b855-4205-927f-bf36899de2b7 
time="2018-08-30T14:52:15.921843358Z" level=info msg="redis not configured" go.version=go1.9.4 instance.id=1ba98ace-b855-4205-927f-bf36899de2b7 
time="2018-08-30T14:52:15.941752053Z" level=info msg="using inmemory blob descriptor cache" go.version=go1.9.4 instance.id=1ba98ace-b855-4205-927f-bf36899de2b7 
time="2018-08-30T14:52:15.941783908Z" level=debug msg="configured \"openshift\" access controller" go.version=go1.9.4 instance.id=1ba98ace-b855-4205-927f-bf36899de2b7 
time="2018-08-30T14:52:15.941813082Z" level=debug msg="configured token endpoint at \"/openshift/token\"" go.version=go1.9.4 instance.id=1ba98ace-b855-4205-927f-bf36899de2b7 
time="2018-08-30T14:52:15.942707687Z" level=info msg="listening on :5000, tls" go.version=go1.9.4 instance.id=1ba98ace-b855-4205-927f-bf36899de2b7 
time="2018-08-30T14:52:15.943766611Z" level=info msg="Starting upload purge in 40m0s" go.version=go1.9.4 instance.id=1ba98ace-b855-4205-927f-bf36899de2b7 

There seems to be some kind of error when deploying the deploymentconfig that makes it fail, and thus the deploy pod is set to Error.
As the docker-registry-1-rg9jj pod is deleted no more logs could be retrieved, the above logs were taken during the second that docker-registry-1-rg9jj pod was running.

[openshift@master-0 ~]$ oc get pods
NAME                       READY     STATUS    RESTARTS   AGE
docker-registry-1-deploy   0/1       Error     0          30m
registry-console-1-mdh8z   1/1       Running   0          19m
router-1-v5dv4             1/1       Running   0          21m

The workaround to recover from this situation is to re-trigger the pod(s) creation:
1- Edit the deploymentconfig of each pod in Error status:
  $ oc get dc
  NAME               REVISION   DESIRED   CURRENT   TRIGGERED BY
  docker-registry    1          1         1         config       <---
  registry-console   1          1         1         config
  router             1          1         1         config

  $ oc edit dc docker-registry
   Change (for example) the name inside the spec section:
        name: set it to registry-1

2- Check a new pod is deployed:
  $ oc get pods
  NAME                       READY     STATUS    RESTARTS   AGE
  docker-registry-1-deploy   0/1       Error     0          43m
  docker-registry-2-8cp2m    1/1       Running   0          1m
  registry-console-1-mdh8z   1/1       Running   0          32m
  router-1-v5dv4             1/1       Running   0          34m

3- Delete the replicationcontroller of the pod(s) in Error:
  $ oc get rc
  NAME                 DESIRED   CURRENT   READY     AGE
  docker-registry-1    0         0         0         45m
  docker-registry-2    1         1         1         4m
  registry-console-1   1         1         1         45m
  router-1             1         1         1         45m
  
  $ oc delete rc docker-registry-1
  replicationcontroller "docker-registry-1" deleted

4- Check all the pods are running
  $ oc get pods
  NAME                       READY     STATUS    RESTARTS   AGE
  docker-registry-2-8cp2m    1/1       Running   0          4m
  registry-console-1-mdh8z   1/1       Running   0          35m
  router-1-v5dv4             1/1       Running   0          38m

5- These steps must be followed for all the pods in Error status

Version-Release number of the following components:
rpm -q openshift-ansible

openshift-ansible-3.10.34-1.git.0.48df172None.noarch

rpm -q ansible
ansible-2.4.6.0-1.el7ae.noarch

ansible --version
ansible 2.4.6.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/cloud-user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Feb 20 2018, 09:19:12) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]


How reproducible: Almost 100% of the times

Steps to Reproduce:
1. Deploy OpenStack with Octavia (OSP13)
2. Deploy a DNS server and the Ansible host in the overcloud
3. Download OCP rpm and configure:
   - OpenStack (inventory/group_vars/all.yml)
   - OpenShift (inventory/group_vars/OSEv3.yml)
4. Run Openshift-on-Openstack playbooks:
- ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/prerequisites.yml
- ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/provision.yml
- ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory red-hat-ca.yml
- ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory repos.yml
- ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/install.yml

5. Check the pods are in Running status:
  $ oc get pods --all-namespaces

Actual results:
Some pods in the default namespace are in Error
[openshift@master-0 ~]$ oc get pods -o wide
NAME                       READY     STATUS    RESTARTS   AGE       IP              NODE
docker-registry-1-deploy   0/1       Error     0          14m       10.11.0.5       infra-node-0.openshift.example.com
registry-console-1-mdh8z   1/1       Running   0          4m        10.11.0.23      master-0.openshift.example.com
router-1-v5dv4             1/1       Running   0          6m        192.168.99.13   infra-node-0.openshift.example.com

Expected results:
All the pods in Running status


Additional info:

$ oc describe pod docker-registry-1-deploy
Name:         docker-registry-1-deploy
Namespace:    default
Node:         infra-node-0.openshift.example.com/192.168.99.13
Start Time:   Thu, 30 Aug 2018 10:51:29 -0400
Labels:       openshift.io/deployer-pod-for.name=docker-registry-1
Annotations:  openshift.io/deployment-config.name=docker-registry
              openshift.io/deployment.name=docker-registry-1
              openshift.io/scc=restricted
              openstack.org/kuryr-vif={"versioned_object.data": {"active": true, "address": "fa:16:3e:18:d9:a8", "has_traffic_filtering": false, "id": "b5ecb7ba-1cdd-46cf-9f0b-388e5a74ef50", "network": {"version
ed_...
Status:       Failed
IP:           10.11.0.5
Containers:
  deployment:
    Container ID:   docker://63a980461b4445798046c45a725e557a641d5aba13318b43581b46cccf2a0958
    Image:          registry.reg-aws.openshift.com:443/openshift3/ose-deployer:v3.10.34
    Image ID:       docker-pullable://registry.reg-aws.openshift.com:443/openshift3/ose-deployer@sha256:208bd20062c144da5115d30aca966cf30619912855e709b663bafdfad8b5357a
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 30 Aug 2018 10:51:31 -0400
      Finished:     Thu, 30 Aug 2018 10:52:23 -0400
    Ready:          False
    Restart Count:  0
    Environment:
      OPENSHIFT_DEPLOYMENT_NAME:       docker-registry-1
      OPENSHIFT_DEPLOYMENT_NAMESPACE:  default
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from deployer-token-pmhh9 (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  deployer-token-pmhh9:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  deployer-token-pmhh9
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  node-role.kubernetes.io/infra=true
Tolerations:     <none>
Events:
  Type     Reason            Age                 From                                         Message
  ----     ------            ----                ----                                         -------
  Warning  FailedScheduling  31m (x26 over 37m)  default-scheduler                            0/4 nodes are available: 4 node(s) were not ready.
  Normal   Pulled            29m                 kubelet, infra-node-0.openshift.example.com  Container image "registry.reg-aws.openshift.com:443/openshift3/ose-deployer:v3.10.34" already present on machine
  Normal   Created           29m                 kubelet, infra-node-0.openshift.example.com  Created container
  Normal   Started           29m                 kubelet, infra-node-0.openshift.example.com  Started container