Bug 1416349 - Workers ain't set for metricd in gnocchi.conf
Summary: Workers ain't set for metricd in gnocchi.conf
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 11.0 (Ocata)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: Upstream M3
: 11.0 (Ocata)
Assignee: Pradeep Kilambi
QA Contact: Sasha Smolyak
URL:
Whiteboard:
: 1416179 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-25 10:41 UTC by Yurii Prokulevych
Modified: 2017-05-17 19:43 UTC (History)
12 users (show)

Fixed In Version: openstack-tripleo-heat-templates-6.0.0-0.20170130212245.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-05-17 19:43:04 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 425347 0 None None None 2017-01-25 20:06:31 UTC
Red Hat Product Errata RHEA-2017:1245 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 11.0 Bug Fix and Enhancement Advisory 2017-05-17 23:01:50 UTC

Description Yurii Prokulevych 2017-01-25 10:41:46 UTC
Description of problem:
-----------------------
openstack-gnocchi-metricd.service fails to start on controllers:

systemctl status openstack-gnocchi-metricd.service 
● openstack-gnocchi-metricd.service - OpenStack gnocchi metricd service
   Loaded: loaded (/usr/lib/systemd/system/openstack-gnocchi-metricd.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Wed 2017-01-25 10:27:30 UTC; 7min ago
 Main PID: 700461 (code=exited, status=1/FAILURE)

Jan 25 10:27:30 overcloud-controller-1.localdomain systemd[1]: openstack-gnocchi-metricd.service: main process exited, code=exited, status=1/FAILURE
Jan 25 10:27:30 overcloud-controller-1.localdomain systemd[1]: Unit openstack-gnocchi-metricd.service entered failed state.
Jan 25 10:27:30 overcloud-controller-1.localdomain systemd[1]: openstack-gnocchi-metricd.service failed.
Jan 25 10:27:30 overcloud-controller-1.localdomain systemd[1]: openstack-gnocchi-metricd.service holdoff time over, scheduling restart.
Jan 25 10:27:30 overcloud-controller-1.localdomain systemd[1]: start request repeated too quickly for openstack-gnocchi-metricd.service
Jan 25 10:27:30 overcloud-controller-1.localdomain systemd[1]: Failed to start OpenStack gnocchi metricd service.
Jan 25 10:27:30 overcloud-controller-1.localdomain systemd[1]: Unit openstack-gnocchi-metricd.service entered failed state.
Jan 25 10:27:30 overcloud-controller-1.localdomain systemd[1]: openstack-gnocchi-metricd.service failed.

Excerpt form gnocchi/metricd.log
--------------------------------
2017-01-25 10:27:30.249 700461 WARNING oslo_config.cfg [-] Option "project_id" from group "statsd" is deprecated for removal.  Its value may be silently ignored in the future.
2017-01-25 10:27:30.274 700461 CRITICAL gnocchi [-] TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'
2017-01-25 10:27:30.274 700461 ERROR gnocchi Traceback (most recent call last):
2017-01-25 10:27:30.274 700461 ERROR gnocchi   File "/usr/bin/gnocchi-metricd", line 10, in <module>
2017-01-25 10:27:30.274 700461 ERROR gnocchi     sys.exit(metricd())
2017-01-25 10:27:30.274 700461 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 346, in metricd
2017-01-25 10:27:30.274 700461 ERROR gnocchi     MetricdServiceManager(conf).run()
2017-01-25 10:27:30.274 700461 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 316, in run
2017-01-25 10:27:30.274 700461 ERROR gnocchi     super(MetricdServiceManager, self).run()
2017-01-25 10:27:30.274 700461 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/cotyledon/_service_manager.py", line 205, in run
2017-01-25 10:27:30.274 700461 ERROR gnocchi     self._wait_forever()
2017-01-25 10:27:30.274 700461 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/cotyledon/_utils.py", line 148, in _wait_forever
2017-01-25 10:27:30.274 700461 ERROR gnocchi     self._on_wakeup()
2017-01-25 10:27:30.274 700461 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/cotyledon/_service_manager.py", line 213, in _on_wakeup
2017-01-25 10:27:30.274 700461 ERROR gnocchi     self._adjust_workers()
2017-01-25 10:27:30.274 700461 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/cotyledon/_service_manager.py", line 279, in _adjust_workers
2017-01-25 10:27:30.274 700461 ERROR gnocchi     self._start_worker(service_id, worker_id)
2017-01-25 10:27:30.274 700461 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/cotyledon/_service_manager.py", line 329, in _start_worker
2017-01-25 10:27:30.274 700461 ERROR gnocchi     self._slowdown_respawn_if_needed()
2017-01-25 10:27:30.274 700461 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/cotyledon/_service_manager.py", line 320, in _slowdown_respawn_if_needed
2017-01-25 10:27:30.274 700461 ERROR gnocchi     expected_children = sum(s.workers for s in self._services.values())
2017-01-25 10:27:30.274 700461 ERROR gnocchi TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'
2017-01-25 10:27:30.274 700461 ERROR gnocchi 

Metricd section from gnocchi.conf doesn't have value set for parameter worker:

[metricd]
workers = 


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
python-gnocchiclient-2.8.2-0.20170118224916.f14fe0e.el7ost.noarch
openstack-gnocchi-statsd-3.1.0-0.20170121210255.d06792b.el7ost.noarch
python-gnocchi-3.1.0-0.20170121210255.d06792b.el7ost.noarch
openstack-gnocchi-metricd-3.1.0-0.20170121210255.d06792b.el7ost.noarch
puppet-gnocchi-10.1.0-0.20170121054618.765ff3f.el7ost.noarch
openstack-gnocchi-indexer-sqlalchemy-3.1.0-0.20170121210255.d06792b.el7ost.noarch
openstack-gnocchi-api-3.1.0-0.20170121210255.d06792b.el7ost.noarch
openstack-gnocchi-common-3.1.0-0.20170121210255.d06792b.el7ost.noarch

openstack-heat-templates-0.0.1-0.20170109231310.01b1768.el7ost.noarch
openstack-tripleo-heat-templates-6.0.0-0.20170120220933.36d493c.el7ost.noarch

python-cotyledon-1.6.3-1.el7ost.noarch

Additional info:
----------------
manually setting workers parameter bypasses the issue

Comment 1 Marius Cornea 2017-01-25 11:45:21 UTC
*** Bug 1416179 has been marked as a duplicate of this bug. ***

Comment 4 Mehdi ABAAKOUK 2017-01-25 15:01:15 UTC
I have enhanced the error handling of cotyledon for this case: 

https://github.com/sileht/cotyledon/commit/e23d36560597648b7a14062eafa3de29a342b96c

And I have marked some Gnocchi options as required where None is not a valid value: https://review.openstack.org/425223

This will allow to catch this kind of issue earlier in the integration process.

Comment 9 Sasha Smolyak 2017-02-23 11:28:46 UTC
Checked on the latest build: openstack-tripleo-heat-templates-6.0.0-0.20170218023452.edbaaa9.el7ost.noarch
the service is running successfully

Comment 10 errata-xmlrpc 2017-05-17 19:43:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245


Note You need to log in before you can comment on or make changes to this bug.