Bug 1656411 - HAProxy container does not come up properly after controller reboot due to radosgw not binding to any ip [NEEDINFO]
Summary: HAProxy container does not come up properly after controller reboot due to ra...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-haproxy
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: RHOS Maint
QA Contact: nlevinki
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-05 13:04 UTC by ojanas
Modified: 2019-01-09 14:37 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-01-09 14:37:35 UTC
Target Upstream Version:
yrabl: needinfo? (ojanas)


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1565376 urgent CLOSED [Deployment] haproxy containers stopped on two controllers on fresh deployment 2020-10-14 00:28:05 UTC

Description ojanas 2018-12-05 13:04:16 UTC
Description of problem:

After the reboot of the controller node, haproxy container does not come up properly.

We can see these messages in the logs:

Dec 04 21:32:16 ctrl03 docker(haproxy-bundle-docker-2)[12379]: INFO: haproxy-bundle-docker-2
Dec 04 21:32:16 ctrl03 docker(haproxy-bundle-docker-2)[12400]: NOTICE: Cleaning up inactive container, haproxy-bundle-docker-2.
Dec 04 21:32:16 ctrl03 docker(haproxy-bundle-docker-2)[12425]: INFO: haproxy-bundle-docker-2
Dec 04 21:32:16 ctrl03 crmd[7407]:   notice: Result of stop operation for haproxy-bundle-docker-2 on ctrl03: 0 (ok)


Dec  4 21:32:13 ctrl03 docker(haproxy-bundle-docker-2)[10359]: ERROR: Newly created docker container exited after start
Dec  4 21:32:13 ctrl03 lrmd[7401]:  notice: haproxy-bundle-docker-2_start_0:10359:stderr [ ocf-exit-reason:waiting on monitor_cmd to pass after start ]
Dec  4 21:32:13ctrl03 lrmd[7401]:  notice: haproxy-bundle-docker-2_start_0:10359:stderr [ ocf-exit-reason:Newly created docker container exited after start ]
Dec  4 21:32:13 ctrl03 crmd[7407]:  notice: Result of start operation for haproxy-bundle-docker-2 on ctrl03: 1 (unknown error)
Dec  4 21:32:13 ctrl03 crmd[7407]:  notice: ctrl03-haproxy-bundle-docker-2_start_0:99 [ ocf-exit-reason:waiting on monitor_cmd to pass after start\nocf-exit-reason:Newly created docker container exited after start\n ]
Version-Release number of selected component (if applicable):


There is already a very similiar bugzilla #1565376 which includes the fix in:

puppet-tripleo-8.3.2-2.el7ost

but this environment uses:

puppet-tripleo-8.3.4-5.el7ost.noarch                        Wed Sep  5 18:33:25 2018

and the issue persists.

Actual results:

The HAProxy container does not come up after reboot.

Expected results:

The HAProxy container does come up after reboot.

Additional info:

 Docker container set: rabbitmq-bundle [10.20.12.10:8787/rhosp13/openstack-rabbitmq:pcmklatest]
   rabbitmq-bundle-0    (ocf::heartbeat:rabbitmq-cluster):      Started os2-prd-ctrl01
   rabbitmq-bundle-1    (ocf::heartbeat:rabbitmq-cluster):      Started os2-prd-ctrl02
   rabbitmq-bundle-2    (ocf::heartbeat:rabbitmq-cluster):      Started os2-prd-ctrl03
 Docker container set: galera-bundle [10.20.12.10:8787/rhosp13/openstack-mariadb:pcmklatest]
   galera-bundle-0      (ocf::heartbeat:galera):        Master os2-prd-ctrl01
   galera-bundle-1      (ocf::heartbeat:galera):        Master os2-prd-ctrl02
   galera-bundle-2      (ocf::heartbeat:galera):        Master os2-prd-ctrl03
 Docker container set: redis-bundle [10.20.12.10:8787/rhosp13/openstack-redis:pcmklatest]
   redis-bundle-0       (ocf::heartbeat:redis): Master os2-prd-ctrl01
   redis-bundle-1       (ocf::heartbeat:redis): Slave os2-prd-ctrl02
   redis-bundle-2       (ocf::heartbeat:redis): Slave os2-prd-ctrl03
 ip-10.20.12.105        (ocf::heartbeat:IPaddr2):       Started os2-prd-ctrl01
 ip-10.20.14.10 (ocf::heartbeat:IPaddr2):       Started os2-prd-ctrl02
 ip-10.20.10.103        (ocf::heartbeat:IPaddr2):       Started os2-prd-ctrl02
 ip-10.20.10.105        (ocf::heartbeat:IPaddr2):       Started os2-prd-ctrl01
 ip-10.20.15.103        (ocf::heartbeat:IPaddr2):       Started os2-prd-ctrl02
 ip-10.20.13.106        (ocf::heartbeat:IPaddr2):       Started os2-prd-ctrl01
 Docker container set: haproxy-bundle [10.20.12.10:8787/rhosp13/openstack-haproxy:pcmklatest]
   haproxy-bundle-docker-0      (ocf::heartbeat:docker):        Started os2-prd-ctrl01
   haproxy-bundle-docker-1      (ocf::heartbeat:docker):        Started os2-prd-ctrl02
   haproxy-bundle-docker-2      (ocf::heartbeat:docker):        Stopped

Comment 8 John Fulton 2018-12-19 14:30:37 UTC
Please provide the version of ceph-ansible that you're using.

Comment 11 John Fulton 2019-01-09 14:37:35 UTC
We're unable to reproduce this with the current release and we haven't gotten your logs. If you feel we're mistaken feel free to re-open and provide the requested information.


Note You need to log in before you can comment on or make changes to this bug.