This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1468621 - after deploy OSP12 HA clustercheck command return " Galera cluster node is not synced."
after deploy OSP12 HA clustercheck command return " Galera cluster node is no...
Status: CLOSED NOTABUG
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-containers (Show other bugs)
12.0 (Pike)
Unspecified All
unspecified Severity unspecified
: rc
: ---
Assigned To: Dan Prince
Omri Hochman
Andrew Burden
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-07 10:53 EDT by Artem Hrechanychenko
Modified: 2017-07-07 13:02 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-07-07 13:02:00 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Artem Hrechanychenko 2017-07-07 10:53:23 EDT
Description of problem:
On each controller node after OSP12 HA deployment 

(undercloud) [stack@undercloud-0 ~]$ ssh heat-admin@192.168.24.10 "sudo clustercheck"
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain
Connection: close
Content-Length: 36

Galera cluster node is not synced.

(undercloud) [stack@undercloud-0 ~]$ ssh heat-admin@192.168.24.10 "sudo pcs status"
Cluster name: tripleo_cluster
Stack: corosync
Current DC: controller-2 (version 1.1.16-12.el7-94ff4df) - partition with quorum
Last updated: Fri Jul  7 14:32:39 2017
Last change: Fri Jul  7 14:13:23 2017 by root via cibadmin on controller-2

12 nodes configured
33 resources configured

Online: [ controller-0 controller-1 controller-2 ]
GuestOnline: [ galera-bundle-0@controller-0 galera-bundle-1@controller-1 galera-bundle-2@controller-2 rabbitmq-bundle-0@controller-0 rabbitmq-bundle-1@controller-1 rabbitmq-bundle-2@controller-2 redis-bundle-0@controller-0 redis-bundle-1@controller-1 redis-bundle-2@controller-2 ]

Full list of resources:

 Docker container set: rabbitmq-bundle [192.168.24.1:8787/rhosp12/openstack-rabbitmq-docker:2017-07-02.2]
   rabbitmq-bundle-0	(ocf::heartbeat:rabbitmq-cluster):	Started controller-0
   rabbitmq-bundle-1	(ocf::heartbeat:rabbitmq-cluster):	Started controller-1
   rabbitmq-bundle-2	(ocf::heartbeat:rabbitmq-cluster):	Started controller-2
 Docker container set: galera-bundle [192.168.24.1:8787/rhosp12/openstack-mariadb-docker:2017-07-02.2]
   galera-bundle-0	(ocf::heartbeat:galera):	Master controller-0
   galera-bundle-1	(ocf::heartbeat:galera):	Master controller-1
   galera-bundle-2	(ocf::heartbeat:galera):	Master controller-2
 Docker container set: redis-bundle [192.168.24.1:8787/rhosp12/openstack-redis-docker:2017-07-02.2]
   redis-bundle-0	(ocf::heartbeat:redis):	Master controller-0
   redis-bundle-1	(ocf::heartbeat:redis):	Slave controller-1
   redis-bundle-2	(ocf::heartbeat:redis):	Slave controller-2
 ip-192.168.24.15	(ocf::heartbeat:IPaddr2):	Started controller-0
 ip-192.168.24.11	(ocf::heartbeat:IPaddr2):	Started controller-1
 Docker container set: haproxy-bundle [192.168.24.1:8787/rhosp12/openstack-haproxy-docker:2017-07-02.2]
   haproxy-bundle-docker-0	(ocf::heartbeat:docker):	Started controller-2
   haproxy-bundle-docker-1	(ocf::heartbeat:docker):	Started controller-0
   haproxy-bundle-docker-2	(ocf::heartbeat:docker):	Started controller-1
 openstack-cinder-volume	(systemd:openstack-cinder-volume):	Started controller-2


Version-Release number of selected component (if applicable):
OSP12
openstack-glance.noarch          1:15.0.0-0.20170623215940.8188eca.el7ost
openstack-heat-api.noarch        1:9.0.0-0.20170628061551.30c87ba.el7ost
openstack-heat-api-cfn.noarch    1:9.0.0-0.20170628061551.30c87ba.el7ost
openstack-heat-common.noarch     1:9.0.0-0.20170628061551.30c87ba.el7ost
openstack-heat-engine.noarch     1:9.0.0-0.20170628061551.30c87ba.el7ost
openstack-ironic-api.noarch      1:8.0.1-0.20170628015314.3521203.el7ost
openstack-ironic-common.noarch   1:8.0.1-0.20170628015314.3521203.el7ost
openstack-ironic-conductor.noarch
openstack-ironic-inspector.noarch
openstack-keystone.noarch        1:12.0.0-0.20170628062203.e4c0c8d.el7ost
openstack-mistral-api.noarch     5.0.0-0.20170627151830.fe1a1e1.el7ost
openstack-mistral-common.noarch  5.0.0-0.20170627151830.fe1a1e1.el7ost
openstack-mistral-engine.noarch  5.0.0-0.20170627151830.fe1a1e1.el7ost
openstack-mistral-executor.noarch
openstack-neutron.noarch         1:11.0.0-0.20170628060509.9a72bfe.el7ost
openstack-neutron-common.noarch  1:11.0.0-0.20170628060509.9a72bfe.el7ost
openstack-neutron-ml2.noarch     1:11.0.0-0.20170628060509.9a72bfe.el7ost
openstack-neutron-openvswitch.noarch
openstack-nova-api.noarch        1:16.0.0-0.20170627215041.el7ost
openstack-nova-common.noarch     1:16.0.0-0.20170627215041.el7ost
openstack-nova-compute.noarch    1:16.0.0-0.20170627215041.el7ost
openstack-nova-conductor.noarch  1:16.0.0-0.20170627215041.el7ost
openstack-nova-placement-api.noarch
openstack-nova-scheduler.noarch  1:16.0.0-0.20170627215041.el7ost
openstack-puppet-modules.noarch  1:10.0.0-0.20170315222135.0333c73.el7.1
openstack-selinux.noarch         0.8.8-0.20170622195307.74ddc0e.el7ost
openstack-swift-account.noarch   2.14.1-0.20170622024006.2d18ecd.el7ost
openstack-swift-container.noarch 2.14.1-0.20170622024006.2d18ecd.el7ost
openstack-swift-object.noarch    2.14.1-0.20170622024006.2d18ecd.el7ost
openstack-swift-proxy.noarch     2.14.1-0.20170622024006.2d18ecd.el7ost
openstack-tempest.noarch         1:16.0.1-0.20170628065146.f6288d7.el7ost
openstack-tripleo-common.noarch  7.2.1-0.20170628001837.5a6423b.el7ost
openstack-tripleo-common-containers.noarch
openstack-tripleo-heat-templates.noarch
openstack-tripleo-image-elements.noarch
openstack-tripleo-puppet-elements.noarch
openstack-tripleo-ui.noarch      7.1.1-0.20170627161345.68e8684.el7ost
openstack-tripleo-validations.noarch
openstack-zaqar.noarch           1:5.0.0-0.20170626084003.b1cd63c.el7ost
puppet-openstack_extras.noarch   11.2.0-0.20170626060428.de67a39.el7ost
puppet-openstacklib.noarch       11.2.0-0.20170626053453.8264861.el7ost
python-openstack-mistral.noarch  5.0.0-0.20170627151830.fe1a1e1.el7ost
python-openstackclient.noarch    3.11.0-0.20170613232431.c69304e.el7ost
python-openstacksdk.noarch       0.9.17-0.20170621195806.7946243.el7ost
openstack-aodh-api.noarch        4.0.1-0.20170626182538.ba461b4.el7ost
openstack-aodh-common.noarch     4.0.1-0.20170626182538.ba461b4.el7ost
openstack-aodh-compat.noarch     4.0.1-0.20170626182538.ba461b4.el7ost
openstack-aodh-evaluator.noarch  4.0.1-0.20170626182538.ba461b4.el7ost
openstack-aodh-expirer.noarch    4.0.1-0.20170626182538.ba461b4.el7ost
openstack-aodh-listener.noarch   4.0.1-0.20170626182538.ba461b4.el7ost
openstack-aodh-notifier.noarch   4.0.1-0.20170626182538.ba461b4.el7ost
openstack-ceilometer-api.noarch  1:8.1.0-0.20170627214628.f53ab18.el7ost
openstack-ceilometer-central.noarch
openstack-ceilometer-collector.noarch
openstack-ceilometer-common.noarch
openstack-ceilometer-compute.noarch
openstack-ceilometer-ipmi.noarch 1:8.1.0-0.20170627214628.f53ab18.el7ost
openstack-ceilometer-notification.noarch
openstack-ceilometer-polling.noarch
openstack-cinder.noarch          1:11.0.0-0.20170628012330.3046f75.el7ost
openstack-dashboard.noarch       1:12.0.0-0.20170628063727.4c78a3a.el7ost
openstack-dashboard-theme.noarch 12.0.0-2.el7ost        rhelosp-12.0-puddle     
openstack-designate-agent.noarch 1:5.0.0-0.20170612143112.9a2f65e.el7ost
openstack-designate-api.noarch   1:5.0.0-0.20170612143112.9a2f65e.el7ost
openstack-designate-central.noarch
openstack-designate-common.noarch
openstack-designate-mdns.noarch  1:5.0.0-0.20170612143112.9a2f65e.el7ost
openstack-designate-pool-manager.noarch
openstack-designate-producer.noarch
openstack-designate-sink.noarch  1:5.0.0-0.20170612143112.9a2f65e.el7ost
openstack-designate-worker.noarch
openstack-designate-zone-manager.noarch
openstack-ec2-api.noarch         5.0.0-0.20170615161910.0e36272.el7ost
openstack-gnocchi-api.noarch     4.0.0-0.20170612072714.a852437.el7ost
openstack-gnocchi-common.noarch  4.0.0-0.20170612072714.a852437.el7ost
openstack-gnocchi-indexer-sqlalchemy.noarch
openstack-gnocchi-metricd.noarch 4.0.0-0.20170612072714.a852437.el7ost
openstack-gnocchi-statsd.noarch  4.0.0-0.20170612072714.a852437.el7ost
openstack-heat-agents.noarch     1.2.1-0.20170621230844.964c41e.el7ost
openstack-heat-api-cloudwatch.noarch
openstack-heat-monolith.noarch   1:9.0.0-0.20170628061551.30c87ba.el7ost
openstack-ironic-inspector-doc.noarch
openstack-ironic-python-agent.noarch
openstack-ironic-ui.noarch       2.3.1-0.20170627163537.7ce520e.el7ost
openstack-ironic-ui-doc.noarch   2.3.1-0.20170627163537.7ce520e.el7ost
openstack-manila.noarch          1:5.0.0-0.20170627164152.5a23d63.el7ost
openstack-manila-share.noarch    1:5.0.0-0.20170627164152.5a23d63.el7ost
openstack-manila-ui.noarch       2.9.1-0.20170627113433.4bdf21e.el7ost
openstack-mistral-all.noarch     5.0.0-0.20170627151830.fe1a1e1.el7ost
openstack-mistral-event-engine.noarch
openstack-neutron-bigswitch-agent.noarch
openstack-neutron-bigswitch-lldp.noarch
openstack-neutron-fwaas.noarch   1:11.0.0-0.20170627163845.034c18c.el7ost
openstack-neutron-l2gw-agent.noarch
openstack-neutron-lbaas.noarch   1:11.0.0-0.20170627061233.2c054e0.el7ost
openstack-neutron-lbaas-ui.noarch
openstack-neutron-lbaas-ui-doc.noarch
openstack-neutron-linuxbridge.noarch
openstack-neutron-macvtap-agent.noarch
openstack-neutron-metering-agent.noarch
openstack-neutron-rpc-server.noarch
openstack-neutron-sriov-nic-agent.noarch
openstack-nova.noarch            1:16.0.0-0.20170627215041.el7ost
openstack-nova-cells.noarch      1:16.0.0-0.20170627215041.el7ost
openstack-nova-console.noarch    1:16.0.0-0.20170627215041.el7ost
openstack-nova-migration.noarch  1:16.0.0-0.20170627215041.el7ost
openstack-nova-network.noarch    1:16.0.0-0.20170627215041.el7ost
openstack-nova-novncproxy.noarch 1:16.0.0-0.20170627215041.el7ost
openstack-nova-serialproxy.noarch
openstack-nova-spicehtml5proxy.noarch
openstack-octavia-amphora-agent.noarch
openstack-octavia-api.noarch     1.0.0-0.20170628055307.3ccd8a3.el7ost
openstack-octavia-common.noarch  1.0.0-0.20170628055307.3ccd8a3.el7ost
openstack-octavia-diskimage-create.noarch
openstack-octavia-health-manager.noarch
openstack-octavia-housekeeping.noarch
openstack-octavia-worker.noarch  1.0.0-0.20170628055307.3ccd8a3.el7ost
openstack-packstack.noarch       1:10.1.0-0.20170623170421.2052808.el7ost
openstack-packstack-puppet.noarch
openstack-panko-api.noarch       2.0.1-0.20170627053457.ff94b1a.el7ost
openstack-panko-common.noarch    2.0.1-0.20170627053457.ff94b1a.el7ost
openstack-rally.noarch           0.9.1-0.20170526140855.4bddc74.el7ost
openstack-sahara.noarch          1:7.0.0-0.20170627184752.f589965.el7ost
openstack-sahara-api.noarch      1:7.0.0-0.20170627184752.f589965.el7ost
openstack-sahara-common.noarch   1:7.0.0-0.20170627184752.f589965.el7ost
openstack-sahara-doc.noarch      1:7.0.0-0.20170627184752.f589965.el7ost
openstack-sahara-engine.noarch   1:7.0.0-0.20170627184752.f589965.el7ost
openstack-sahara-tests.noarch    0.4.2-0.20170627174535.b13eb25.el7ost
openstack-sahara-ui.noarch       7.0.0-0.20170627173546.5f07d19.el7ost
openstack-selinux-devel.noarch   0.8.8-0.20170622195307.74ddc0e.el7ost
openstack-selinux-test.noarch    0.8.8-0.20170622195307.74ddc0e.el7ost
openstack-swift-doc.noarch       2.14.1-0.20170622024006.2d18ecd.el7ost
openstack-swift-plugin-swift3.noarch
openstack-tempest-all.noarch     1:16.0.1-0.20170628065146.f6288d7.el7ost
openstack-tripleo.noarch         0.0.8-0.2.4de13b3git.el7ost
openstack-tripleo-common-container-base.noarch
openstack-tripleo-common-devtools.noarch
openstack-tripleo-doc.noarch     0.0.8-0.2.4de13b3git.el7ost
openstack-tripleo-heat-templates-compat.noarch
openstack-tripleo-validations-tests.noarch
openstack-utils.noarch           2017.1-1.el7ost        rhelosp-12.0-puddle     
python-django-openstack-auth.noarch
python-openstackclient-doc.noarch
python-openstackdocstheme.noarch 1.5.0-3.el7ost         rhelosp-12.0-puddle     
python-openstackdocstheme-doc.noarch
python-openstacksdk-tests.noarch 0.9.17-0.20170621195806.7946243.el7ost
Comment 1 Red Hat Bugzilla Rules Engine 2017-07-07 10:53:28 EDT
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.
Comment 2 Damien Ciabrini 2017-07-07 11:50:54 EDT
That is because the clustercheck service now runs in the clustercheck container.

Calling the clustercheck script from the host will try to connect to the mariadb server with credentials from /etc/sysconfig/clustercheck on the host. This will fail on new OSP12 install since this file is no longer populated.

For containers, the clustercheck service is now server by a xinetd process running in the clustercheck container. 

[root@overcloud-controller-0 ~]# docker ps | grep clustercheck
39719ce62eda        192.168.24.1:8787/rhosp12/openstack-mariadb-docker:2017-07-02.2                     "kolla_start"            6 hours ago         Up 6 hours                              clustercheck

The service is exposed on port :9200 on every controller node:

[root@overcloud-controller-0 ~]# ss -tnlp src :9200
State       Recv-Q Send-Q                                                            Local Address:Port                                                                           Peer Address:Port              
LISTEN      0      64                                                                  172.17.0.12:9200                                                                                      *:*                   users:(("xinetd",pid=37920,fd=5))

You would access it via a simple curl:

[root@overcloud-controller-0 ~]# curl 172.17.0.12:9200
Galera cluster node is synced.
Comment 3 Artem Hrechanychenko 2017-07-07 13:02:00 EDT
confirm

Note You need to log in before you can comment on or make changes to this bug.