Created attachment 1484914 [details] docker inspect ceph rgw container Description of problem: Deploying the overcloud with ceph-ansible and ceph-rgw enabled fails during the startup of the haproxy container because the port 8080 is already used by the ceph rgw container. The rgw is binding on all ip addresses (instead of the restricted storage network) so haproxy isn't able to start. Version-Release number of selected component (if applicable): OSP13 puddle 2018-09-13.1 ceph-ansible-3.1.3-1.el7cp.noarch # ceph --version ceph version 12.2.4-42.el7cp (f73642baacccbf2a3c254d1fb5f0317b933b28cf) luminous (stable) How reproducible: 100% Steps to Reproduce: 1. Deploy the overcloud with ceph-ansible and ceph-rgw enabled $ openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-rgw.yaml Actual results: Haproxy container fails to start because the port 8080 is already used by ceph rgw container. Sep 19 13:54:31 overcloud-controller-0 docker(haproxy-bundle-docker-0)[106801]: INFO: running container haproxy-bundle-docker-0 for the first time Sep 19 13:54:31 overcloud-controller-0 systemd: Started libcontainer container 023409a88a7d6131c84b9599b89e83028cde89021cd03c142c6af9acc242917b. Sep 19 13:54:31 overcloud-controller-0 systemd: Starting libcontainer container 023409a88a7d6131c84b9599b89e83028cde89021cd03c142c6af9acc242917b. Sep 19 13:54:31 overcloud-controller-0 journal: + sudo -E kolla_set_configs Sep 19 13:54:31 overcloud-controller-0 journal: INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json Sep 19 13:54:31 overcloud-controller-0 journal: INFO:__main__:Validating config file Sep 19 13:54:31 overcloud-controller-0 journal: INFO:__main__:Kolla config strategy set to: COPY_ALWAYS Sep 19 13:54:31 overcloud-controller-0 journal: INFO:__main__:Copying service configuration files Sep 19 13:54:31 overcloud-controller-0 journal: INFO:__main__:Deleting /etc/haproxy/haproxy.cfg Sep 19 13:54:31 overcloud-controller-0 journal: INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/haproxy/haproxy.cfg to /etc/haproxy/haproxy.cfg Sep 19 13:54:31 overcloud-controller-0 journal: INFO:__main__:Writing out command to execute Sep 19 13:54:31 overcloud-controller-0 journal: INFO:__main__:Setting permission for /var/lib/haproxy Sep 19 13:54:31 overcloud-controller-0 journal: INFO:__main__:Setting permission for /var/lib/haproxy/stats Sep 19 13:54:31 overcloud-controller-0 journal: ++ cat /run_command Sep 19 13:54:31 overcloud-controller-0 journal: + CMD='/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg' Sep 19 13:54:31 overcloud-controller-0 journal: + ARGS= Sep 19 13:54:31 overcloud-controller-0 journal: + [[ ! -n '' ]] Sep 19 13:54:31 overcloud-controller-0 journal: + . kolla_extend_start Sep 19 13:54:31 overcloud-controller-0 journal: + echo 'Running command: '\''/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg'\''' Sep 19 13:54:31 overcloud-controller-0 journal: + exec /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg Sep 19 13:54:31 overcloud-controller-0 journal: Running command: '/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg' Sep 19 13:54:31 overcloud-controller-0 journal: <7>haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -Ds Sep 19 13:54:31 overcloud-controller-0 journal: [WARNING] 261/175431 (10) : parsing [/etc/haproxy/haproxy.cfg:128] : 'option httplog' not usable with proxy 'nova_metadata' (needs 'mode http'). Falling back to 'option tcplog'. Sep 19 13:54:31 overcloud-controller-0 haproxy[10]: Proxy cinder started. Sep 19 13:54:31 overcloud-controller-0 journal: [ALERT] 261/175431 (10) : Starting proxy ceph_rgw: cannot bind socket [192.168.24.12:8080] Sep 19 13:54:31 overcloud-controller-0 haproxy[10]: Proxy glance_api started. Sep 19 13:54:31 overcloud-controller-0 haproxy[10]: Proxy haproxy.stats started. Sep 19 13:54:31 overcloud-controller-0 journal: <5>haproxy-systemd-wrapper: exit, haproxy RC=1 Sep 19 13:54:31 overcloud-controller-0 haproxy[10]: Proxy heat_api started. Sep 19 13:54:31 overcloud-controller-0 haproxy[10]: Proxy heat_cfn started. Sep 19 13:54:31 overcloud-controller-0 haproxy[10]: Proxy horizon started. Sep 19 13:54:31 overcloud-controller-0 haproxy[10]: Proxy keystone_admin started. Sep 19 13:54:31 overcloud-controller-0 haproxy[10]: Proxy keystone_public started. Sep 19 13:54:31 overcloud-controller-0 haproxy[10]: Proxy mysql started. Sep 19 13:54:31 overcloud-controller-0 haproxy[10]: Proxy neutron started. Sep 19 13:54:31 overcloud-controller-0 haproxy[10]: Proxy nova_metadata started. Sep 19 13:54:31 overcloud-controller-0 haproxy[10]: Proxy nova_novncproxy started. Sep 19 13:54:31 overcloud-controller-0 haproxy[10]: Proxy nova_osapi started. Sep 19 13:54:31 overcloud-controller-0 haproxy[10]: Proxy nova_placement started. Sep 19 13:54:31 overcloud-controller-0 haproxy[10]: Proxy redis started. Sep 19 13:54:31 overcloud-controller-0 docker(haproxy-bundle-docker-0)[106801]: INFO: monitor cmd exit code = 1 Expected results: Haproxy container is able to start. The same configuration was working using puddle 2018-09-11.1 Additional info: ceph rgw service in the container binds on all ip: /usr/bin/radosgw --cluster ceph --setuser ceph --setgroup ceph -d -n client.rgw.overcloud-controller-0 -k /var/lib/ceph/radosgw/ceph-rgw.overcloud-controller-0/keyring --rgw-socket-path= --rgw-zonegroup= --rgw-zone= --rgw-frontends=civetweb port=0.0.0.0:8080 The ip binding in the ceph configuration is the good one but is not honored in the container : # /etc/ceph/ceph.conf [client.rgw.overcloud-controller-0] host = overcloud-controller-0 keyring = /var/lib/ceph/radosgw/ceph-rgw.overcloud-controller-0/keyring log file = /var/log/ceph/ceph-rgw-overcloud-controller-0.log rgw frontends = civetweb port=192.168.24.15:8080 num_threads=100