Bug 1656411

Description of problem:

After the reboot of the controller node, haproxy container does not come up properly.

We can see these messages in the logs:

Dec 04 21:32:16 ctrl03 docker(haproxy-bundle-docker-2)[12379]: INFO: haproxy-bundle-docker-2
Dec 04 21:32:16 ctrl03 docker(haproxy-bundle-docker-2)[12400]: NOTICE: Cleaning up inactive container, haproxy-bundle-docker-2.
Dec 04 21:32:16 ctrl03 docker(haproxy-bundle-docker-2)[12425]: INFO: haproxy-bundle-docker-2
Dec 04 21:32:16 ctrl03 crmd[7407]:   notice: Result of stop operation for haproxy-bundle-docker-2 on ctrl03: 0 (ok)


Dec  4 21:32:13 ctrl03 docker(haproxy-bundle-docker-2)[10359]: ERROR: Newly created docker container exited after start
Dec  4 21:32:13 ctrl03 lrmd[7401]:  notice: haproxy-bundle-docker-2_start_0:10359:stderr [ ocf-exit-reason:waiting on monitor_cmd to pass after start ]
Dec  4 21:32:13ctrl03 lrmd[7401]:  notice: haproxy-bundle-docker-2_start_0:10359:stderr [ ocf-exit-reason:Newly created docker container exited after start ]
Dec  4 21:32:13 ctrl03 crmd[7407]:  notice: Result of start operation for haproxy-bundle-docker-2 on ctrl03: 1 (unknown error)
Dec  4 21:32:13 ctrl03 crmd[7407]:  notice: ctrl03-haproxy-bundle-docker-2_start_0:99 [ ocf-exit-reason:waiting on monitor_cmd to pass after start\nocf-exit-reason:Newly created docker container exited after start\n ]
Version-Release number of selected component (if applicable):


There is already a very similiar bugzilla #1565376 which includes the fix in:

puppet-tripleo-8.3.2-2.el7ost

but this environment uses:

puppet-tripleo-8.3.4-5.el7ost.noarch                        Wed Sep  5 18:33:25 2018

and the issue persists.

Actual results:

The HAProxy container does not come up after reboot.

Expected results:

The HAProxy container does come up after reboot.

Additional info:

 Docker container set: rabbitmq-bundle [10.20.12.10:8787/rhosp13/openstack-rabbitmq:pcmklatest]
   rabbitmq-bundle-0    (ocf::heartbeat:rabbitmq-cluster):      Started os2-prd-ctrl01
   rabbitmq-bundle-1    (ocf::heartbeat:rabbitmq-cluster):      Started os2-prd-ctrl02
   rabbitmq-bundle-2    (ocf::heartbeat:rabbitmq-cluster):      Started os2-prd-ctrl03
 Docker container set: galera-bundle [10.20.12.10:8787/rhosp13/openstack-mariadb:pcmklatest]
   galera-bundle-0      (ocf::heartbeat:galera):        Master os2-prd-ctrl01
   galera-bundle-1      (ocf::heartbeat:galera):        Master os2-prd-ctrl02
   galera-bundle-2      (ocf::heartbeat:galera):        Master os2-prd-ctrl03
 Docker container set: redis-bundle [10.20.12.10:8787/rhosp13/openstack-redis:pcmklatest]
   redis-bundle-0       (ocf::heartbeat:redis): Master os2-prd-ctrl01
   redis-bundle-1       (ocf::heartbeat:redis): Slave os2-prd-ctrl02
   redis-bundle-2       (ocf::heartbeat:redis): Slave os2-prd-ctrl03
 ip-10.20.12.105        (ocf::heartbeat:IPaddr2):       Started os2-prd-ctrl01
 ip-10.20.14.10 (ocf::heartbeat:IPaddr2):       Started os2-prd-ctrl02
 ip-10.20.10.103        (ocf::heartbeat:IPaddr2):       Started os2-prd-ctrl02
 ip-10.20.10.105        (ocf::heartbeat:IPaddr2):       Started os2-prd-ctrl01
 ip-10.20.15.103        (ocf::heartbeat:IPaddr2):       Started os2-prd-ctrl02
 ip-10.20.13.106        (ocf::heartbeat:IPaddr2):       Started os2-prd-ctrl01
 Docker container set: haproxy-bundle [10.20.12.10:8787/rhosp13/openstack-haproxy:pcmklatest]
   haproxy-bundle-docker-0      (ocf::heartbeat:docker):        Started os2-prd-ctrl01
   haproxy-bundle-docker-1      (ocf::heartbeat:docker):        Started os2-prd-ctrl02
   haproxy-bundle-docker-2      (ocf::heartbeat:docker):        Stopped