Bug 1015878 - openstack-ceilometer: After installation ceilometer not functioning due to 'missing module named kombu'.
openstack-ceilometer: After installation ceilometer not functioning due to 'm...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-ceilometer (Show other bugs)
4.0
x86_64 Linux
high Severity high
: rc
: 4.0
Assigned To: Eoghan Glynn
Nir Magnezi
: TestBlocker
Depends On:
Blocks: 967310
  Show dependency treegraph
 
Reported: 2013-10-06 09:11 EDT by Omri Hochman
Modified: 2013-12-19 19:25 EST (History)
9 users (show)

See Also:
Fixed In Version: openstack-ceilometer-2013.2-2.el6ost
Doc Type: Bug Fix
Doc Text:
Cause: The client library for RabbitMQ was unconditionally imported by Ceilometer, regardless of the actual AMQP provider in use. Consequence: When only QPID is installed, the Ceilometer service failed to start, producing an ImportError. Fix: Importing the kombu library is now conditional on RabbitMQ being the selected AMQP provider. Result: The Ceilometer services start successfully in a QPID-only installation.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-12-19 19:25:59 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
ceilometer-api.log (53.32 KB, text/x-log)
2013-10-06 09:26 EDT, Omri Hochman
no flags Details
ceilometer-compute.log (25.42 KB, text/x-log)
2013-10-06 09:27 EDT, Omri Hochman
no flags Details
answer_file (13.08 KB, text/plain)
2013-10-20 08:19 EDT, Omri Hochman
no flags Details
pasckatck_setup.log (10.39 KB, text/x-log)
2013-10-20 08:20 EDT, Omri Hochman
no flags Details
ceilomeret.conf (21.28 KB, text/plain)
2013-10-22 08:12 EDT, Omri Hochman
no flags Details
ceilomeret-dist.conf (275 bytes, text/plain)
2013-10-22 08:14 EDT, Omri Hochman
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1244698 None None None Never

  None (edit)
Description Omri Hochman 2013-10-06 09:11:17 EDT
openstack-ceilometer: After installation ceilometer not functioning due to 'missing module named kombu'.  

Environment (havana - puddle 2013-10-03.3) :
----------------------------------------------
[root@cougar15 /(keystone_admin)]# rpm -qa | grep ceilometer
python-ceilometer-2013.2-0.11.rc1.el6ost.noarch
openstack-ceilometer-central-2013.2-0.11.rc1.el6ost.noarch
openstack-ceilometer-common-2013.2-0.11.rc1.el6ost.noarch
openstack-ceilometer-api-2013.2-0.11.rc1.el6ost.noarch
python-ceilometerclient-1.0.0-2.el6ost.noarch
openstack-ceilometer-compute-2013.2-0.11.rc1.el6ost.noarch
openstack-ceilometer-collector-2013.2-0.11.rc1.el6ost.noarch
openstack-nova-2013.2-0.24

Steps: 
-------
1) Install RHOS Havana with ceilometer - using packstack.
2) Boot Instance. 
3) Attempt to perform - 'celiometer meter-list'

Results: 
---------
- 'celiometer meter-list' - return no results.

Notes:
------
-  all ceilometer services are up and running. 
-  restart ceilometer services does not help 

ceilometer-compute.log:
------------------------
2013-10-06 13:35:30.555 3988 INFO ceilometer.openstack.common.rpc.impl_qpid [-] Connected to AMQP server on 10.35.160.119:5672
2013-10-06 13:35:30.589 3988 INFO ceilometer.pipeline [-] Pipeline config: [{'meters': ['*'], 'interval': 600, 'transformers': None, 'name': 'meter_pipeline', 'publishers': ['rpc://']}, {'me
ters': ['cpu'], 'interval': 600, 'transformers': [{'name': 'rate_of_change', 'parameters': {'target': {'scale': '100.0 / (10**9 * (resource_metadata.cpu_number or 1))', 'type': 'gauge', 'nam
e': 'cpu_util', 'unit': '%'}}}], 'name': 'cpu_pipeline', 'publishers': ['rpc://']}]
2013-10-06 13:35:30.602 3988 ERROR stevedore.extension [-] Could not load 'rpc': No module named kombu
2013-10-06 13:35:30.602 3988 ERROR stevedore.extension [-] No module named kombu
2013-10-06 13:35:30.602 3988 TRACE stevedore.extension Traceback (most recent call last):
2013-10-06 13:35:30.602 3988 TRACE stevedore.extension   File "/usr/lib/python2.6/site-packages/stevedore/extension.py", line 89, in _load_plugins
2013-10-06 13:35:30.602 3988 TRACE stevedore.extension     invoke_kwds,
2013-10-06 13:35:30.602 3988 TRACE stevedore.extension   File "/usr/lib/python2.6/site-packages/stevedore/named.py", line 57, in _load_one_plugin
2013-10-06 13:35:30.602 3988 TRACE stevedore.extension     ep, invoke_on_load, invoke_args, invoke_kwds,
2013-10-06 13:35:30.602 3988 TRACE stevedore.extension   File "/usr/lib/python2.6/site-packages/stevedore/extension.py", line 101, in _load_one_plugin
2013-10-06 13:35:30.602 3988 TRACE stevedore.extension     plugin = ep.load()
2013-10-06 13:35:30.602 3988 TRACE stevedore.extension   File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1948, in load
2013-10-06 13:35:30.602 3988 TRACE stevedore.extension     entry = __import__(self.module_name, globals(),globals(), ['__name__'])
2013-10-06 13:35:30.602 3988 TRACE stevedore.extension   File "/usr/lib/python2.6/site-packages/ceilometer/publisher/rpc.py", line 61, in <module>
2013-10-06 13:35:30.602 3988 TRACE stevedore.extension     'ceilometer.openstack.common.rpc.impl_kombu')
2013-10-06 13:35:30.602 3988 TRACE stevedore.extension   File "/usr/lib/python2.6/site-packages/oslo/config/cfg.py", line 1754, in import_opt
2013-10-06 13:35:30.602 3988 TRACE stevedore.extension     __import__(module_str)
2013-10-06 13:35:30.602 3988 TRACE stevedore.extension   File "/usr/lib/python2.6/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 26, in <module>
2013-10-06 13:35:30.602 3988 TRACE stevedore.extension     import kombu
2013-10-06 13:35:30.602 3988 TRACE stevedore.extension ImportError: No module named kombu
2013-10-06 13:35:30.602 3988 TRACE stevedore.extension 
2013-10-06 13:35:30.606 3988 ERROR ceilometer.pipeline [-] Unable to load publisher rpc://
2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline Traceback (most recent call last):
2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline   File "/usr/lib/python2.6/site-packages/ceilometer/pipeline.py", line 126, in __init__
2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline     self.publishers.append(publisher.get_publisher(p))
2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline   File "/usr/lib/python2.6/site-packages/ceilometer/publisher/__init__.py", line 33, in get_publisher
2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline     loaded_driver = driver.DriverManager(namespace, parse_result.scheme)
2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline   File "/usr/lib/python2.6/site-packages/stevedore/driver.py", line 35, in __init__
2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline     (namespace, name))
2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline RuntimeError: No 'ceilometer.publisher' driver found, looking for 'rpc'
2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline 
2013-10-06 13:35:30.607 3988 ERROR stevedore.extension [-] Could not load 'rpc': No module named kombu
2013-10-06 13:35:30.608 3988 ERROR stevedore.extension [-] No module named kombu
2013-10-06 13:35:30.608 3988 TRACE stevedore.extension Traceback (most recent call last):
2013-10-06 13:35:30.608 3988 TRACE stevedore.extension   File "/usr/lib/python2.6/site-packages/stevedore/extension.py", line 89, in _load_plugins
2013-10-06 13:35:30.608 3988 TRACE stevedore.extension     invoke_kwds,
2013-10-06 13:35:30.608 3988 TRACE stevedore.extension   File "/usr/lib/python2.6/site-packages/stevedore/named.py", line 57, in _load_one_plugin
2013-10-06 13:35:30.608 3988 TRACE stevedore.extension     ep, invoke_on_load, invoke_args, invoke_kwds,
2013-10-06 13:35:30.608 3988 TRACE stevedore.extension   File "/usr/lib/python2.6/site-packages/stevedore/extension.py", line 101, in _load_one_plugin
2013-10-06 13:35:30.608 3988 TRACE stevedore.extension     plugin = ep.load()
2013-10-06 13:35:30.608 3988 TRACE stevedore.extension   File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1948, in load
2013-10-06 13:35:30.608 3988 TRACE stevedore.extension     entry = __import__(self.module_name, globals(),globals(), ['__name__'])
2013-10-06 13:35:30.608 3988 TRACE stevedore.extension   File "/usr/lib/python2.6/site-packages/ceilometer/publisher/rpc.py", line 61, in <module>
2013-10-06 13:35:30.608 3988 TRACE stevedore.extension     'ceilometer.openstack.common.rpc.impl_kombu')
2013-10-06 13:35:30.608 3988 TRACE stevedore.extension   File "/usr/lib/python2.6/site-packages/oslo/config/cfg.py", line 1754, in import_opt
2013-10-06 13:35:30.608 3988 TRACE stevedore.extension     __import__(module_str)
2013-10-06 13:35:30.608 3988 TRACE stevedore.extension   File "/usr/lib/python2.6/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 26, in <module>
2013-10-06 13:35:30.608 3988 TRACE stevedore.extension     import kombu
2013-10-06 13:35:30.608 3988 TRACE stevedore.extension ImportError: No module named kombu
2013-10-06 13:35:30.608 3988 TRACE stevedore.extension 
2013-10-06 13:35:30.609 3988 ERROR ceilometer.pipeline [-] Unable to load publisher rpc://
2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline Traceback (most recent call last):
2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline   File "/usr/lib/python2.6/site-packages/ceilometer/pipeline.py", line 126, in __init__
2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline     self.publishers.append(publisher.get_publisher(p))
2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline   File "/usr/lib/python2.6/site-packages/ceilometer/publisher/__init__.py", line 33, in get_publisher
2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline     loaded_driver = driver.DriverManager(namespace, parse_result.scheme)
2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline   File "/usr/lib/python2.6/site-packages/stevedore/driver.py", line 35, in __init__
2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline     (namespace, name))
2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline RuntimeError: No 'ceilometer.publisher' driver found, looking for 'rpc'
2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline 
2013-10-06 13:35:30.609 3988 INFO ceilometer.pipeline [-] Pipeline cpu_pipeline: Setup transformer instance rate_of_change with parameter {'target': {'scale': '100.0 / (10**9 * (resource_metadata.cpu_number or 1))', 'type': 'gauge', 'name': 'cpu_util', 'unit': '%'}}
2013-10-06 13:35:30.621 3988 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.35.160.119
2013-10-06 13:35:30.865 3988 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.35.160.119
2013-10-06 13:35:32.399 3988 AUDIT ceilometer.pipeline [-] Flush pipeline meter_pipeline
2013-10-06 13:35:32.400 3988 AUDIT ceilometer.pipeline [-] Flush pipeline cpu_pipeline
Comment 2 Omri Hochman 2013-10-06 09:26:36 EDT
Created attachment 808497 [details]
ceilometer-api.log
Comment 3 Omri Hochman 2013-10-06 09:27:25 EDT
Created attachment 808498 [details]
ceilometer-compute.log
Comment 6 Pádraig Brady 2013-10-07 07:57:16 EDT
kombu is not supported in RHOS.
Eoghan can we config around this dependency?
Comment 7 Pádraig Brady 2013-10-07 09:44:22 EDT
I should note that I did try openstack-ceilometer-compute-2013.2-0.12.rc1.el6.noarch  from RDO on RHEL 6.4, which should not differ significantly from the above tested package, and it started without issue with no python-kombu installed
Comment 8 Pádraig Brady 2013-10-07 10:27:39 EDT
So I tried the latest RHOS build on RHEL 6.4 (openstack-ceilometer-2013.2-0.12.rc1.el6ost) and couldn't reproduce the issue. As mentioned this shouldn't be significantly different from openstack-ceilometer-2013.2-0.11.rc1.el6ost in this regard, so I suspect config differences are triggering this.

Can you attach /usr/share/ceilometer/ceilometer-dist.conf and /etc/ceilometer/ceilometer.conf

thanks.
Comment 9 Omri Hochman 2013-10-10 11:24:06 EDT
Yes - It happened to me only on RHEL 6.5 (on 6.4 it was fine).
Comment 10 Lon Hohberger 2013-10-15 23:01:44 EDT
I can't reproduce this on 6.4 or 6.5.
Comment 11 Omri Hochman 2013-10-17 15:19:44 EDT
(In reply to Lon Hohberger from comment #10)
> I can't reproduce this on 6.4 or 6.5.
Comment 12 Nir Magnezi 2013-10-20 05:59:17 EDT
(In reply to Lon Hohberger from comment #10)
> I can't reproduce this on 6.4 or 6.5.

Hi Lon,

I managed to reproduce this issue with RHEL 6.5
Used: packstack --allinone

Checked Ceilometer Logs:

grep -i error /var/log/celimeter/*.log

central.log:2013-10-20 12:54:14.335 25169 WARNING ceilometer.central.manager [-] Continue after error from storage.objects: [Errno 111] ECONNREFUSED
central.log:2013-10-20 12:54:14.335 25169 ERROR ceilometer.central.manager [-] [Errno 111] ECONNREFUSED
central.log:2013-10-20 12:54:14.335 25169 TRACE ceilometer.central.manager     raise error, msg
central.log:2013-10-20 12:54:14.335 25169 TRACE ceilometer.central.manager error: [Errno 111] ECONNREFUSED
compute.log:2013-10-20 12:33:16.065 13754 ERROR stevedore.extension [-] Could not load 'rpc': No module named kombu
compute.log:2013-10-20 12:33:16.066 13754 ERROR stevedore.extension [-] No module named kombu
compute.log:2013-10-20 12:33:16.066 13754 TRACE stevedore.extension ImportError: No module named kombu
compute.log:2013-10-20 12:33:16.067 13754 ERROR ceilometer.pipeline [-] Unable to load publisher rpc://
compute.log:2013-10-20 12:33:16.067 13754 TRACE ceilometer.pipeline RuntimeError: No 'ceilometer.publisher' driver found, looking for 'rpc'
Comment 13 Omri Hochman 2013-10-20 08:18:32 EDT
Reproduces on my environment (rhel 6.5 with puddle 2013-10-15.1)  : 
--------------------------------------------------------------------

[root@ ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.5 Beta (Santiago)

[root@ ~]# rpm -qa | grep ceilometer
openstack-ceilometer-common-2013.2-0.12.rc2.el6ost.noarch
openstack-ceilometer-collector-2013.2-0.12.rc2.el6ost.noarch
python-ceilometer-2013.2-0.12.rc2.el6ost.noarch
openstack-ceilometer-api-2013.2-0.12.rc2.el6ost.noarch
python-ceilometerclient-1.0.6-1.el6ost.noarch
openstack-ceilometer-compute-2013.2-0.12.rc2.el6ost.noarch
openstack-ceilometer-central-2013.2-0.12.rc2.el6ost.noarch


[root@ ~]# rpm -qa | grep packstack
openstack-packstack-2013.2.1-0.6.dev763.el6ost.noarch
Comment 14 Omri Hochman 2013-10-20 08:19:16 EDT
just in case I'm adding packstack -  setup.log + answer_file.
Comment 15 Omri Hochman 2013-10-20 08:19:50 EDT
Created attachment 814217 [details]
answer_file
Comment 16 Omri Hochman 2013-10-20 08:20:17 EDT
Created attachment 814218 [details]
pasckatck_setup.log
Comment 17 Pádraig Brady 2013-10-20 13:31:50 EDT
Can you attach the configs as requested in comment 8
Comment 18 Pádraig Brady 2013-10-21 10:02:35 EDT
May be caused by https://review.openstack.org/#/c/35604/
Specifically therein the unconditional:

+cfg.CONF.import_opt('rabbit_max_retries',
+                    'ceilometer.openstack.common.rpc.impl_kombu')
Comment 19 Pádraig Brady 2013-10-21 12:05:34 EDT
Temp workaround to be addressed more generally upstream:
https://code.engineering.redhat.com/gerrit/gitweb?p=ceilometer.git;a=commitdiff;h=c3e0978c0
Comment 21 Omri Hochman 2013-10-22 08:12:37 EDT
Created attachment 814956 [details]
ceilomeret.conf
Comment 22 Omri Hochman 2013-10-22 08:14:34 EDT
Created attachment 814957 [details]
ceilomeret-dist.conf
Comment 23 Eoghan Glynn 2013-10-25 11:52:12 EDT
More complete fix proposed upstream as:

  https://review.openstack.org/53889

Once this lands on master, it can be cherry-picked onto stable/havana upstream and then replace the temporary workaround:

  https://code.engineering.redhat.com/gerrit/gitweb?p=ceilometer.git;a=commitdiff;h=c3e0978c0
Comment 25 Nir Magnezi 2013-11-11 08:19:02 EST
Verified NVR:
# rpm -qa | grep openstack-ceilometer
openstack-ceilometer-central-2013.2-1.el6ost.noarch
openstack-ceilometer-api-2013.2-1.el6ost.noarch
openstack-ceilometer-compute-2013.2-1.el6ost.noarch
openstack-ceilometer-common-2013.2-1.el6ost.noarch
openstack-ceilometer-collector-2013.2-1.el6ost.noarch

Puddle 2013-11-08.1

Verification Steps:
1. Install OpenStack
2. Verify that 'meter-list' works (as tested in Comment #0)
# ceilometer meter-list
+----------------------------+-------+-----------+--------------------------------------+---------+----------------------------------+
| Name                       | Type  | Unit      | Resource ID                          | User ID | Project ID                       |
+----------------------------+-------+-----------+--------------------------------------+---------+----------------------------------+
| image                      | gauge | image     | 9b3d9a0f-968f-4cc6-b896-a6e520eef9f7 | None    | fc4b66bccefa4d12b3b02f5b2852f64c |
| image.size                 | gauge | B         | 9b3d9a0f-968f-4cc6-b896-a6e520eef9f7 | None    | fc4b66bccefa4d12b3b02f5b2852f64c |
| storage.objects            | gauge | object    | 80c44e301b0744dda33062dcc9d807f8     | None    | 80c44e301b0744dda33062dcc9d807f8 |
| storage.objects            | gauge | object    | ea4656c175ea476399fe86f10b2cc48a     | None    | ea4656c175ea476399fe86f10b2cc48a |
| storage.objects            | gauge | object    | f1ae831adfd34984be1004421afbc77f     | None    | f1ae831adfd34984be1004421afbc77f |
| storage.objects            | gauge | object    | fc4b66bccefa4d12b3b02f5b2852f64c     | None    | fc4b66bccefa4d12b3b02f5b2852f64c |
| storage.objects.containers | gauge | container | 80c44e301b0744dda33062dcc9d807f8     | None    | 80c44e301b0744dda33062dcc9d807f8 |
| storage.objects.containers | gauge | container | ea4656c175ea476399fe86f10b2cc48a     | None    | ea4656c175ea476399fe86f10b2cc48a |
| storage.objects.containers | gauge | container | f1ae831adfd34984be1004421afbc77f     | None    | f1ae831adfd34984be1004421afbc77f |
| storage.objects.containers | gauge | container | fc4b66bccefa4d12b3b02f5b2852f64c     | None    | fc4b66bccefa4d12b3b02f5b2852f64c |
| storage.objects.size       | gauge | B         | 80c44e301b0744dda33062dcc9d807f8     | None    | 80c44e301b0744dda33062dcc9d807f8 |
| storage.objects.size       | gauge | B         | ea4656c175ea476399fe86f10b2cc48a     | None    | ea4656c175ea476399fe86f10b2cc48a |
| storage.objects.size       | gauge | B         | f1ae831adfd34984be1004421afbc77f     | None    | f1ae831adfd34984be1004421afbc77f |
| storage.objects.size       | gauge | B         | fc4b66bccefa4d12b3b02f5b2852f64c     | None    | fc4b66bccefa4d12b3b02f5b2852f64c |
+----------------------------+-------+-----------+--------------------------------------+---------+----------------------------------+

3. Verified that there are no errors in Celimeter logs
4. Verified that all Ceilomter services are up
# for svc in api central collector compute ; do service openstack-ceilometer-$svc status ; done
openstack-ceilometer-api (pid  22755) is running...
openstack-ceilometer-central (pid  22725) is running...
openstack-ceilometer-collector (pid  22696) is running...
openstack-ceilometer-compute (pid  6091) is running...
Comment 26 Eoghan Glynn 2013-11-22 10:20:27 EST
More permanent fix landed on master upstream:

  http://github.com/openstack/ceilometer/commit/a8e12c04

and proposed upstream as a backport to stable/havana:

  https://review.openstack.org/57932

Once the latter lands upstream, the temporary workaround in commit c3e0978c can be reverted.
Comment 27 Eoghan Glynn 2013-11-25 08:13:55 EST
More permanent fix landed on stable/havana upstream:

  http://github.com/openstack/ceilometer/commit/9f0a54a4
Comment 28 Eoghan Glynn 2013-11-25 08:15:07 EST
Temporary work-around revert proposed to internal internal gerrit:

  https://code.engineering.redhat.com/gerrit/16234

and corresponding backport of the more permanent fix:

  https://code.engineering.redhat.com/gerrit/16235
Comment 34 errata-xmlrpc 2013-12-19 19:25:59 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2013-1859.html

Note You need to log in before you can comment on or make changes to this bug.