Bug 1434344 - openstack-gnocchi-statsd.service can sometimes start only on 1/3 controllers when using swift as backend
Summary: openstack-gnocchi-statsd.service can sometimes start only on 1/3 controllers ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: 12.0 (Pike)
Assignee: Emilien Macchi
QA Contact: Gurenko Alex
URL:
Whiteboard:
Depends On:
Blocks: 1568466
TreeView+ depends on / blocked
 
Reported: 2017-03-21 10:18 UTC by Marius Cornea
Modified: 2018-04-17 14:56 UTC (History)
9 users (show)

Fixed In Version: openstack-tripleo-heat-templates-7.0.1-0.20170927205937.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1568466 (view as bug list)
Environment:
Last Closed: 2017-12-13 21:18:38 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 505403 0 None MERGED Run gnocchi statsd and metrcd at step 5 2020-09-29 13:30:41 UTC
Red Hat Product Errata RHEA-2017:3462 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 12.0 Enhancement Advisory 2018-02-16 01:43:25 UTC

Description Marius Cornea 2017-03-21 10:18:17 UTC
Description of problem:
openstack-gnocchi-statsd.service can sometimes start only on 1/3 controllers when using swift as backend.

Deployment with 3 controllers, 1 compute node and 2 nodes running Heat only services. After deployment:

controller-0 | SUCCESS | rc=0 >>
active


cmd: systemctl is-active openstack-gnocchi-statsd

start: 2017-03-21 10:09:59.074116

end: 2017-03-21 10:09:59.085453

delta: 0:00:00.011337

stdout: active
controller-1 | SUCCESS | rc=0 >>
active


cmd: systemctl is-active openstack-gnocchi-statsd

start: 2017-03-21 10:09:59.305255

end: 2017-03-21 10:09:59.327560

delta: 0:00:00.022305

stdout: active
controller-2 | FAILED | rc=3 >>
failed


cmd: systemctl is-active openstack-gnocchi-statsd

start: 2017-03-21 10:09:59.319084

end: 2017-03-21 10:09:59.337314

delta: 0:00:00.018230

stdout: failed

Checking the logs on the nodes where the service is failed we can see:

[root@controller-2 heat-admin]# tail /var/log/gnocchi/statsd.log 
2017-03-20 22:01:54.502 95788 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1635, in _retry
2017-03-20 22:01:54.502 95788 ERROR gnocchi     self.url, self.token = self.get_auth()
2017-03-20 22:01:54.502 95788 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1587, in get_auth
2017-03-20 22:01:54.502 95788 ERROR gnocchi     timeout=self.timeout)
2017-03-20 22:01:54.502 95788 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 662, in get_auth
2017-03-20 22:01:54.502 95788 ERROR gnocchi     auth_version=auth_version)
2017-03-20 22:01:54.502 95788 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 580, in get_auth_keystone
2017-03-20 22:01:54.502 95788 ERROR gnocchi     raise ClientException(msg)
2017-03-20 22:01:54.502 95788 ERROR gnocchi ClientException: Unauthorized. Check username, password and tenant name/id.
2017-03-20 22:01:54.502 95788 ERROR gnocchi 


Version-Release number of selected component (if applicable):
openstack-gnocchi-indexer-sqlalchemy-3.0.4-3.el7ost.noarch
openstack-gnocchi-statsd-3.0.4-3.el7ost.noarch
openstack-gnocchi-common-3.0.4-3.el7ost.noarch
openstack-gnocchi-metricd-3.0.4-3.el7ost.noarch
puppet-gnocchi-9.5.0-1.el7ost.noarch
python-gnocchi-3.0.4-3.el7ost.noarch
openstack-gnocchi-api-3.0.4-3.el7ost.noarch
openstack-gnocchi-carbonara-3.0.4-3.el7ost.noarch
python-gnocchiclient-2.6.0-1.el7ost.noarch
openstack-tripleo-heat-templates-6.0.0-0.20170307170102.3134785.0rc2.el7ost.noarch

How reproducible:
Only on this kind of deployment so far.

Steps to Reproduce:
1. Deploy OSP10 with 3 controllers, 1 compute and 2 nodes running Heat only services
2. Check openstack-gnocchi-statsd service on all controllers

Actual results:
Service is failed on 2/3 controllers.

Expected results:
Service gets started on all controllers.

Additional info:
Adding the sosreports. 

Workaround: manually start the service.

Comment 14 errata-xmlrpc 2017-12-13 21:18:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462


Note You need to log in before you can comment on or make changes to this bug.