Bug 1358345

Summary: Add a base constraint so gnocchi services should start after galera
Product: Red Hat OpenStack Reporter: Pradeep Kilambi <pkilambi>
Component: openstack-tripleo-heat-templatesAssignee: Pradeep Kilambi <pkilambi>
Status: CLOSED ERRATA QA Contact: Yurii Prokulevych <yprokule>
Severity: high Docs Contact:
Priority: unspecified    
Version: 9.0 (Mitaka)CC: achernet, fbaudin, jason.dobies, mburns, rhel-osp-director-maint, tvignaud
Target Milestone: gaKeywords: Triaged
Target Release: 9.0 (Mitaka)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-2.0.0-23.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-11 11:36:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Pradeep Kilambi 2016-07-20 14:42:39 UTC
When we restart the cluster sometime statsd comes up before galera causing connection issues and fails. Add a constraint to order so statsd starts after openstack-core-clone. 

Relevant traceback from yurii:

2016-07-20 11:57:54.399 6045 CRITICAL gnocchi [-] StorageError: Unable to start coordinator: Error while reading from socket: ('Connection closed by server.',)
2016-07-20 11:57:54.399 6045 ERROR gnocchi Traceback (most recent call last):
2016-07-20 11:57:54.399 6045 ERROR gnocchi   File "/usr/bin/gnocchi-statsd", line 10, in <module>
2016-07-20 11:57:54.399 6045 ERROR gnocchi     sys.exit(statsd())
2016-07-20 11:57:54.399 6045 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 75, in statsd
2016-07-20 11:57:54.399 6045 ERROR gnocchi     statsd_service.start()
2016-07-20 11:57:54.399 6045 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/statsd.py", line 174, in start
2016-07-20 11:57:54.399 6045 ERROR gnocchi     stats = Stats(conf)
2016-07-20 11:57:54.399 6045 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/statsd.py", line 38, in __init__
2016-07-20 11:57:54.399 6045 ERROR gnocchi     self.storage = storage.get_driver(self.conf)
2016-07-20 11:57:54.399 6045 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 155, in get_driver
2016-07-20 11:57:54.399 6045 ERROR gnocchi     return get_driver_class(conf)(conf.storage)
2016-07-20 11:57:54.399 6045 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 61, in __init__
2016-07-20 11:57:54.399 6045 ERROR gnocchi     super(CephStorage, self).__init__(conf)
2016-07-20 11:57:54.399 6045 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py", line 61, in __init__
2016-07-20 11:57:54.399 6045 ERROR gnocchi     raise storage.StorageError("Unable to start coordinator: %s" % e)
2016-07-20 11:57:54.399 6045 ERROR gnocchi StorageError: Unable to start coordinator: Error while reading from socket: ('Connection closed by server.',)
2016-07-20 11:57:54.399 6045 ERROR gnocchi

Comment 4 Yurii Prokulevych 2016-07-29 12:01:24 UTC
New constraint added :
----------------------
pcs constraint order | grep gnocchi
  start openstack-core-clone then start openstack-gnocchi-metricd-clone (kind:Mandatory)
  start openstack-gnocchi-metricd-clone then start openstack-gnocchi-statsd-clone (kind:Mandatory)

Verified with openstack-tripleo-heat-templates-2.0.0-24.el7ost.noarch

Deploy command:
openstack  overcloud deploy --libvirt-type qemu \
    --ntp-server clock.redhat.com --templates \
    --control-scale 3 --compute-scale 1 --ceph-storage-scale 1 \
    -e storage-environment.yaml 2>&1

cat storage-environment.yaml
parameter_defaults:
  CinderEnableIscsiBackend: false
  CinderEnableRbdBackend: true
  NovaEnableRbdBackend: false
  GlanceBackend: rbd
  GnocchiBackend: file

Comment 6 errata-xmlrpc 2016-08-11 11:36:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1599.html