Description of problem: On each controller node after OSP12 HA deployment (undercloud) [stack@undercloud-0 ~]$ ssh heat-admin.24.10 "sudo clustercheck" HTTP/1.1 503 Service Unavailable Content-Type: text/plain Connection: close Content-Length: 36 Galera cluster node is not synced. (undercloud) [stack@undercloud-0 ~]$ ssh heat-admin.24.10 "sudo pcs status" Cluster name: tripleo_cluster Stack: corosync Current DC: controller-2 (version 1.1.16-12.el7-94ff4df) - partition with quorum Last updated: Fri Jul 7 14:32:39 2017 Last change: Fri Jul 7 14:13:23 2017 by root via cibadmin on controller-2 12 nodes configured 33 resources configured Online: [ controller-0 controller-1 controller-2 ] GuestOnline: [ galera-bundle-0@controller-0 galera-bundle-1@controller-1 galera-bundle-2@controller-2 rabbitmq-bundle-0@controller-0 rabbitmq-bundle-1@controller-1 rabbitmq-bundle-2@controller-2 redis-bundle-0@controller-0 redis-bundle-1@controller-1 redis-bundle-2@controller-2 ] Full list of resources: Docker container set: rabbitmq-bundle [192.168.24.1:8787/rhosp12/openstack-rabbitmq-docker:2017-07-02.2] rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Started controller-0 rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Started controller-1 rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Started controller-2 Docker container set: galera-bundle [192.168.24.1:8787/rhosp12/openstack-mariadb-docker:2017-07-02.2] galera-bundle-0 (ocf::heartbeat:galera): Master controller-0 galera-bundle-1 (ocf::heartbeat:galera): Master controller-1 galera-bundle-2 (ocf::heartbeat:galera): Master controller-2 Docker container set: redis-bundle [192.168.24.1:8787/rhosp12/openstack-redis-docker:2017-07-02.2] redis-bundle-0 (ocf::heartbeat:redis): Master controller-0 redis-bundle-1 (ocf::heartbeat:redis): Slave controller-1 redis-bundle-2 (ocf::heartbeat:redis): Slave controller-2 ip-192.168.24.15 (ocf::heartbeat:IPaddr2): Started controller-0 ip-192.168.24.11 (ocf::heartbeat:IPaddr2): Started controller-1 Docker container set: haproxy-bundle [192.168.24.1:8787/rhosp12/openstack-haproxy-docker:2017-07-02.2] haproxy-bundle-docker-0 (ocf::heartbeat:docker): Started controller-2 haproxy-bundle-docker-1 (ocf::heartbeat:docker): Started controller-0 haproxy-bundle-docker-2 (ocf::heartbeat:docker): Started controller-1 openstack-cinder-volume (systemd:openstack-cinder-volume): Started controller-2 Version-Release number of selected component (if applicable): OSP12 openstack-glance.noarch 1:15.0.0-0.20170623215940.8188eca.el7ost openstack-heat-api.noarch 1:9.0.0-0.20170628061551.30c87ba.el7ost openstack-heat-api-cfn.noarch 1:9.0.0-0.20170628061551.30c87ba.el7ost openstack-heat-common.noarch 1:9.0.0-0.20170628061551.30c87ba.el7ost openstack-heat-engine.noarch 1:9.0.0-0.20170628061551.30c87ba.el7ost openstack-ironic-api.noarch 1:8.0.1-0.20170628015314.3521203.el7ost openstack-ironic-common.noarch 1:8.0.1-0.20170628015314.3521203.el7ost openstack-ironic-conductor.noarch openstack-ironic-inspector.noarch openstack-keystone.noarch 1:12.0.0-0.20170628062203.e4c0c8d.el7ost openstack-mistral-api.noarch 5.0.0-0.20170627151830.fe1a1e1.el7ost openstack-mistral-common.noarch 5.0.0-0.20170627151830.fe1a1e1.el7ost openstack-mistral-engine.noarch 5.0.0-0.20170627151830.fe1a1e1.el7ost openstack-mistral-executor.noarch openstack-neutron.noarch 1:11.0.0-0.20170628060509.9a72bfe.el7ost openstack-neutron-common.noarch 1:11.0.0-0.20170628060509.9a72bfe.el7ost openstack-neutron-ml2.noarch 1:11.0.0-0.20170628060509.9a72bfe.el7ost openstack-neutron-openvswitch.noarch openstack-nova-api.noarch 1:16.0.0-0.20170627215041.el7ost openstack-nova-common.noarch 1:16.0.0-0.20170627215041.el7ost openstack-nova-compute.noarch 1:16.0.0-0.20170627215041.el7ost openstack-nova-conductor.noarch 1:16.0.0-0.20170627215041.el7ost openstack-nova-placement-api.noarch openstack-nova-scheduler.noarch 1:16.0.0-0.20170627215041.el7ost openstack-puppet-modules.noarch 1:10.0.0-0.20170315222135.0333c73.el7.1 openstack-selinux.noarch 0.8.8-0.20170622195307.74ddc0e.el7ost openstack-swift-account.noarch 2.14.1-0.20170622024006.2d18ecd.el7ost openstack-swift-container.noarch 2.14.1-0.20170622024006.2d18ecd.el7ost openstack-swift-object.noarch 2.14.1-0.20170622024006.2d18ecd.el7ost openstack-swift-proxy.noarch 2.14.1-0.20170622024006.2d18ecd.el7ost openstack-tempest.noarch 1:16.0.1-0.20170628065146.f6288d7.el7ost openstack-tripleo-common.noarch 7.2.1-0.20170628001837.5a6423b.el7ost openstack-tripleo-common-containers.noarch openstack-tripleo-heat-templates.noarch openstack-tripleo-image-elements.noarch openstack-tripleo-puppet-elements.noarch openstack-tripleo-ui.noarch 7.1.1-0.20170627161345.68e8684.el7ost openstack-tripleo-validations.noarch openstack-zaqar.noarch 1:5.0.0-0.20170626084003.b1cd63c.el7ost puppet-openstack_extras.noarch 11.2.0-0.20170626060428.de67a39.el7ost puppet-openstacklib.noarch 11.2.0-0.20170626053453.8264861.el7ost python-openstack-mistral.noarch 5.0.0-0.20170627151830.fe1a1e1.el7ost python-openstackclient.noarch 3.11.0-0.20170613232431.c69304e.el7ost python-openstacksdk.noarch 0.9.17-0.20170621195806.7946243.el7ost openstack-aodh-api.noarch 4.0.1-0.20170626182538.ba461b4.el7ost openstack-aodh-common.noarch 4.0.1-0.20170626182538.ba461b4.el7ost openstack-aodh-compat.noarch 4.0.1-0.20170626182538.ba461b4.el7ost openstack-aodh-evaluator.noarch 4.0.1-0.20170626182538.ba461b4.el7ost openstack-aodh-expirer.noarch 4.0.1-0.20170626182538.ba461b4.el7ost openstack-aodh-listener.noarch 4.0.1-0.20170626182538.ba461b4.el7ost openstack-aodh-notifier.noarch 4.0.1-0.20170626182538.ba461b4.el7ost openstack-ceilometer-api.noarch 1:8.1.0-0.20170627214628.f53ab18.el7ost openstack-ceilometer-central.noarch openstack-ceilometer-collector.noarch openstack-ceilometer-common.noarch openstack-ceilometer-compute.noarch openstack-ceilometer-ipmi.noarch 1:8.1.0-0.20170627214628.f53ab18.el7ost openstack-ceilometer-notification.noarch openstack-ceilometer-polling.noarch openstack-cinder.noarch 1:11.0.0-0.20170628012330.3046f75.el7ost openstack-dashboard.noarch 1:12.0.0-0.20170628063727.4c78a3a.el7ost openstack-dashboard-theme.noarch 12.0.0-2.el7ost rhelosp-12.0-puddle openstack-designate-agent.noarch 1:5.0.0-0.20170612143112.9a2f65e.el7ost openstack-designate-api.noarch 1:5.0.0-0.20170612143112.9a2f65e.el7ost openstack-designate-central.noarch openstack-designate-common.noarch openstack-designate-mdns.noarch 1:5.0.0-0.20170612143112.9a2f65e.el7ost openstack-designate-pool-manager.noarch openstack-designate-producer.noarch openstack-designate-sink.noarch 1:5.0.0-0.20170612143112.9a2f65e.el7ost openstack-designate-worker.noarch openstack-designate-zone-manager.noarch openstack-ec2-api.noarch 5.0.0-0.20170615161910.0e36272.el7ost openstack-gnocchi-api.noarch 4.0.0-0.20170612072714.a852437.el7ost openstack-gnocchi-common.noarch 4.0.0-0.20170612072714.a852437.el7ost openstack-gnocchi-indexer-sqlalchemy.noarch openstack-gnocchi-metricd.noarch 4.0.0-0.20170612072714.a852437.el7ost openstack-gnocchi-statsd.noarch 4.0.0-0.20170612072714.a852437.el7ost openstack-heat-agents.noarch 1.2.1-0.20170621230844.964c41e.el7ost openstack-heat-api-cloudwatch.noarch openstack-heat-monolith.noarch 1:9.0.0-0.20170628061551.30c87ba.el7ost openstack-ironic-inspector-doc.noarch openstack-ironic-python-agent.noarch openstack-ironic-ui.noarch 2.3.1-0.20170627163537.7ce520e.el7ost openstack-ironic-ui-doc.noarch 2.3.1-0.20170627163537.7ce520e.el7ost openstack-manila.noarch 1:5.0.0-0.20170627164152.5a23d63.el7ost openstack-manila-share.noarch 1:5.0.0-0.20170627164152.5a23d63.el7ost openstack-manila-ui.noarch 2.9.1-0.20170627113433.4bdf21e.el7ost openstack-mistral-all.noarch 5.0.0-0.20170627151830.fe1a1e1.el7ost openstack-mistral-event-engine.noarch openstack-neutron-bigswitch-agent.noarch openstack-neutron-bigswitch-lldp.noarch openstack-neutron-fwaas.noarch 1:11.0.0-0.20170627163845.034c18c.el7ost openstack-neutron-l2gw-agent.noarch openstack-neutron-lbaas.noarch 1:11.0.0-0.20170627061233.2c054e0.el7ost openstack-neutron-lbaas-ui.noarch openstack-neutron-lbaas-ui-doc.noarch openstack-neutron-linuxbridge.noarch openstack-neutron-macvtap-agent.noarch openstack-neutron-metering-agent.noarch openstack-neutron-rpc-server.noarch openstack-neutron-sriov-nic-agent.noarch openstack-nova.noarch 1:16.0.0-0.20170627215041.el7ost openstack-nova-cells.noarch 1:16.0.0-0.20170627215041.el7ost openstack-nova-console.noarch 1:16.0.0-0.20170627215041.el7ost openstack-nova-migration.noarch 1:16.0.0-0.20170627215041.el7ost openstack-nova-network.noarch 1:16.0.0-0.20170627215041.el7ost openstack-nova-novncproxy.noarch 1:16.0.0-0.20170627215041.el7ost openstack-nova-serialproxy.noarch openstack-nova-spicehtml5proxy.noarch openstack-octavia-amphora-agent.noarch openstack-octavia-api.noarch 1.0.0-0.20170628055307.3ccd8a3.el7ost openstack-octavia-common.noarch 1.0.0-0.20170628055307.3ccd8a3.el7ost openstack-octavia-diskimage-create.noarch openstack-octavia-health-manager.noarch openstack-octavia-housekeeping.noarch openstack-octavia-worker.noarch 1.0.0-0.20170628055307.3ccd8a3.el7ost openstack-packstack.noarch 1:10.1.0-0.20170623170421.2052808.el7ost openstack-packstack-puppet.noarch openstack-panko-api.noarch 2.0.1-0.20170627053457.ff94b1a.el7ost openstack-panko-common.noarch 2.0.1-0.20170627053457.ff94b1a.el7ost openstack-rally.noarch 0.9.1-0.20170526140855.4bddc74.el7ost openstack-sahara.noarch 1:7.0.0-0.20170627184752.f589965.el7ost openstack-sahara-api.noarch 1:7.0.0-0.20170627184752.f589965.el7ost openstack-sahara-common.noarch 1:7.0.0-0.20170627184752.f589965.el7ost openstack-sahara-doc.noarch 1:7.0.0-0.20170627184752.f589965.el7ost openstack-sahara-engine.noarch 1:7.0.0-0.20170627184752.f589965.el7ost openstack-sahara-tests.noarch 0.4.2-0.20170627174535.b13eb25.el7ost openstack-sahara-ui.noarch 7.0.0-0.20170627173546.5f07d19.el7ost openstack-selinux-devel.noarch 0.8.8-0.20170622195307.74ddc0e.el7ost openstack-selinux-test.noarch 0.8.8-0.20170622195307.74ddc0e.el7ost openstack-swift-doc.noarch 2.14.1-0.20170622024006.2d18ecd.el7ost openstack-swift-plugin-swift3.noarch openstack-tempest-all.noarch 1:16.0.1-0.20170628065146.f6288d7.el7ost openstack-tripleo.noarch 0.0.8-0.2.4de13b3git.el7ost openstack-tripleo-common-container-base.noarch openstack-tripleo-common-devtools.noarch openstack-tripleo-doc.noarch 0.0.8-0.2.4de13b3git.el7ost openstack-tripleo-heat-templates-compat.noarch openstack-tripleo-validations-tests.noarch openstack-utils.noarch 2017.1-1.el7ost rhelosp-12.0-puddle python-django-openstack-auth.noarch python-openstackclient-doc.noarch python-openstackdocstheme.noarch 1.5.0-3.el7ost rhelosp-12.0-puddle python-openstackdocstheme-doc.noarch python-openstacksdk-tests.noarch 0.9.17-0.20170621195806.7946243.el7ost
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.
That is because the clustercheck service now runs in the clustercheck container. Calling the clustercheck script from the host will try to connect to the mariadb server with credentials from /etc/sysconfig/clustercheck on the host. This will fail on new OSP12 install since this file is no longer populated. For containers, the clustercheck service is now server by a xinetd process running in the clustercheck container. [root@overcloud-controller-0 ~]# docker ps | grep clustercheck 39719ce62eda 192.168.24.1:8787/rhosp12/openstack-mariadb-docker:2017-07-02.2 "kolla_start" 6 hours ago Up 6 hours clustercheck The service is exposed on port :9200 on every controller node: [root@overcloud-controller-0 ~]# ss -tnlp src :9200 State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 64 172.17.0.12:9200 *:* users:(("xinetd",pid=37920,fd=5)) You would access it via a simple curl: [root@overcloud-controller-0 ~]# curl 172.17.0.12:9200 Galera cluster node is synced.
confirm