openstack-ceilometer: After installation ceilometer not functioning due to 'missing module named kombu'. Environment (havana - puddle 2013-10-03.3) : ---------------------------------------------- [root@cougar15 /(keystone_admin)]# rpm -qa | grep ceilometer python-ceilometer-2013.2-0.11.rc1.el6ost.noarch openstack-ceilometer-central-2013.2-0.11.rc1.el6ost.noarch openstack-ceilometer-common-2013.2-0.11.rc1.el6ost.noarch openstack-ceilometer-api-2013.2-0.11.rc1.el6ost.noarch python-ceilometerclient-1.0.0-2.el6ost.noarch openstack-ceilometer-compute-2013.2-0.11.rc1.el6ost.noarch openstack-ceilometer-collector-2013.2-0.11.rc1.el6ost.noarch openstack-nova-2013.2-0.24 Steps: ------- 1) Install RHOS Havana with ceilometer - using packstack. 2) Boot Instance. 3) Attempt to perform - 'celiometer meter-list' Results: --------- - 'celiometer meter-list' - return no results. Notes: ------ - all ceilometer services are up and running. - restart ceilometer services does not help ceilometer-compute.log: ------------------------ 2013-10-06 13:35:30.555 3988 INFO ceilometer.openstack.common.rpc.impl_qpid [-] Connected to AMQP server on 10.35.160.119:5672 2013-10-06 13:35:30.589 3988 INFO ceilometer.pipeline [-] Pipeline config: [{'meters': ['*'], 'interval': 600, 'transformers': None, 'name': 'meter_pipeline', 'publishers': ['rpc://']}, {'me ters': ['cpu'], 'interval': 600, 'transformers': [{'name': 'rate_of_change', 'parameters': {'target': {'scale': '100.0 / (10**9 * (resource_metadata.cpu_number or 1))', 'type': 'gauge', 'nam e': 'cpu_util', 'unit': '%'}}}], 'name': 'cpu_pipeline', 'publishers': ['rpc://']}] 2013-10-06 13:35:30.602 3988 ERROR stevedore.extension [-] Could not load 'rpc': No module named kombu 2013-10-06 13:35:30.602 3988 ERROR stevedore.extension [-] No module named kombu 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension Traceback (most recent call last): 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/stevedore/extension.py", line 89, in _load_plugins 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension invoke_kwds, 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/stevedore/named.py", line 57, in _load_one_plugin 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension ep, invoke_on_load, invoke_args, invoke_kwds, 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/stevedore/extension.py", line 101, in _load_one_plugin 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension plugin = ep.load() 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1948, in load 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension entry = __import__(self.module_name, globals(),globals(), ['__name__']) 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/ceilometer/publisher/rpc.py", line 61, in <module> 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension 'ceilometer.openstack.common.rpc.impl_kombu') 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/oslo/config/cfg.py", line 1754, in import_opt 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension __import__(module_str) 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 26, in <module> 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension import kombu 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension ImportError: No module named kombu 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension 2013-10-06 13:35:30.606 3988 ERROR ceilometer.pipeline [-] Unable to load publisher rpc:// 2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline Traceback (most recent call last): 2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline File "/usr/lib/python2.6/site-packages/ceilometer/pipeline.py", line 126, in __init__ 2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline self.publishers.append(publisher.get_publisher(p)) 2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline File "/usr/lib/python2.6/site-packages/ceilometer/publisher/__init__.py", line 33, in get_publisher 2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline loaded_driver = driver.DriverManager(namespace, parse_result.scheme) 2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline File "/usr/lib/python2.6/site-packages/stevedore/driver.py", line 35, in __init__ 2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline (namespace, name)) 2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline RuntimeError: No 'ceilometer.publisher' driver found, looking for 'rpc' 2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline 2013-10-06 13:35:30.607 3988 ERROR stevedore.extension [-] Could not load 'rpc': No module named kombu 2013-10-06 13:35:30.608 3988 ERROR stevedore.extension [-] No module named kombu 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension Traceback (most recent call last): 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/stevedore/extension.py", line 89, in _load_plugins 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension invoke_kwds, 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/stevedore/named.py", line 57, in _load_one_plugin 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension ep, invoke_on_load, invoke_args, invoke_kwds, 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/stevedore/extension.py", line 101, in _load_one_plugin 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension plugin = ep.load() 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1948, in load 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension entry = __import__(self.module_name, globals(),globals(), ['__name__']) 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/ceilometer/publisher/rpc.py", line 61, in <module> 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension 'ceilometer.openstack.common.rpc.impl_kombu') 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/oslo/config/cfg.py", line 1754, in import_opt 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension __import__(module_str) 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 26, in <module> 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension import kombu 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension ImportError: No module named kombu 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension 2013-10-06 13:35:30.609 3988 ERROR ceilometer.pipeline [-] Unable to load publisher rpc:// 2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline Traceback (most recent call last): 2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline File "/usr/lib/python2.6/site-packages/ceilometer/pipeline.py", line 126, in __init__ 2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline self.publishers.append(publisher.get_publisher(p)) 2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline File "/usr/lib/python2.6/site-packages/ceilometer/publisher/__init__.py", line 33, in get_publisher 2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline loaded_driver = driver.DriverManager(namespace, parse_result.scheme) 2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline File "/usr/lib/python2.6/site-packages/stevedore/driver.py", line 35, in __init__ 2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline (namespace, name)) 2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline RuntimeError: No 'ceilometer.publisher' driver found, looking for 'rpc' 2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline 2013-10-06 13:35:30.609 3988 INFO ceilometer.pipeline [-] Pipeline cpu_pipeline: Setup transformer instance rate_of_change with parameter {'target': {'scale': '100.0 / (10**9 * (resource_metadata.cpu_number or 1))', 'type': 'gauge', 'name': 'cpu_util', 'unit': '%'}} 2013-10-06 13:35:30.621 3988 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.35.160.119 2013-10-06 13:35:30.865 3988 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.35.160.119 2013-10-06 13:35:32.399 3988 AUDIT ceilometer.pipeline [-] Flush pipeline meter_pipeline 2013-10-06 13:35:32.400 3988 AUDIT ceilometer.pipeline [-] Flush pipeline cpu_pipeline
Created attachment 808497 [details] ceilometer-api.log
Created attachment 808498 [details] ceilometer-compute.log
kombu is not supported in RHOS. Eoghan can we config around this dependency?
I should note that I did try openstack-ceilometer-compute-2013.2-0.12.rc1.el6.noarch from RDO on RHEL 6.4, which should not differ significantly from the above tested package, and it started without issue with no python-kombu installed
So I tried the latest RHOS build on RHEL 6.4 (openstack-ceilometer-2013.2-0.12.rc1.el6ost) and couldn't reproduce the issue. As mentioned this shouldn't be significantly different from openstack-ceilometer-2013.2-0.11.rc1.el6ost in this regard, so I suspect config differences are triggering this. Can you attach /usr/share/ceilometer/ceilometer-dist.conf and /etc/ceilometer/ceilometer.conf thanks.
Yes - It happened to me only on RHEL 6.5 (on 6.4 it was fine).
I can't reproduce this on 6.4 or 6.5.
(In reply to Lon Hohberger from comment #10) > I can't reproduce this on 6.4 or 6.5.
(In reply to Lon Hohberger from comment #10) > I can't reproduce this on 6.4 or 6.5. Hi Lon, I managed to reproduce this issue with RHEL 6.5 Used: packstack --allinone Checked Ceilometer Logs: grep -i error /var/log/celimeter/*.log central.log:2013-10-20 12:54:14.335 25169 WARNING ceilometer.central.manager [-] Continue after error from storage.objects: [Errno 111] ECONNREFUSED central.log:2013-10-20 12:54:14.335 25169 ERROR ceilometer.central.manager [-] [Errno 111] ECONNREFUSED central.log:2013-10-20 12:54:14.335 25169 TRACE ceilometer.central.manager raise error, msg central.log:2013-10-20 12:54:14.335 25169 TRACE ceilometer.central.manager error: [Errno 111] ECONNREFUSED compute.log:2013-10-20 12:33:16.065 13754 ERROR stevedore.extension [-] Could not load 'rpc': No module named kombu compute.log:2013-10-20 12:33:16.066 13754 ERROR stevedore.extension [-] No module named kombu compute.log:2013-10-20 12:33:16.066 13754 TRACE stevedore.extension ImportError: No module named kombu compute.log:2013-10-20 12:33:16.067 13754 ERROR ceilometer.pipeline [-] Unable to load publisher rpc:// compute.log:2013-10-20 12:33:16.067 13754 TRACE ceilometer.pipeline RuntimeError: No 'ceilometer.publisher' driver found, looking for 'rpc'
Reproduces on my environment (rhel 6.5 with puddle 2013-10-15.1) : -------------------------------------------------------------------- [root@ ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.5 Beta (Santiago) [root@ ~]# rpm -qa | grep ceilometer openstack-ceilometer-common-2013.2-0.12.rc2.el6ost.noarch openstack-ceilometer-collector-2013.2-0.12.rc2.el6ost.noarch python-ceilometer-2013.2-0.12.rc2.el6ost.noarch openstack-ceilometer-api-2013.2-0.12.rc2.el6ost.noarch python-ceilometerclient-1.0.6-1.el6ost.noarch openstack-ceilometer-compute-2013.2-0.12.rc2.el6ost.noarch openstack-ceilometer-central-2013.2-0.12.rc2.el6ost.noarch [root@ ~]# rpm -qa | grep packstack openstack-packstack-2013.2.1-0.6.dev763.el6ost.noarch
just in case I'm adding packstack - setup.log + answer_file.
Created attachment 814217 [details] answer_file
Created attachment 814218 [details] pasckatck_setup.log
Can you attach the configs as requested in comment 8
May be caused by https://review.openstack.org/#/c/35604/ Specifically therein the unconditional: +cfg.CONF.import_opt('rabbit_max_retries', + 'ceilometer.openstack.common.rpc.impl_kombu')
Temp workaround to be addressed more generally upstream: https://code.engineering.redhat.com/gerrit/gitweb?p=ceilometer.git;a=commitdiff;h=c3e0978c0
Created attachment 814956 [details] ceilomeret.conf
Created attachment 814957 [details] ceilomeret-dist.conf
More complete fix proposed upstream as: https://review.openstack.org/53889 Once this lands on master, it can be cherry-picked onto stable/havana upstream and then replace the temporary workaround: https://code.engineering.redhat.com/gerrit/gitweb?p=ceilometer.git;a=commitdiff;h=c3e0978c0
Verified NVR: # rpm -qa | grep openstack-ceilometer openstack-ceilometer-central-2013.2-1.el6ost.noarch openstack-ceilometer-api-2013.2-1.el6ost.noarch openstack-ceilometer-compute-2013.2-1.el6ost.noarch openstack-ceilometer-common-2013.2-1.el6ost.noarch openstack-ceilometer-collector-2013.2-1.el6ost.noarch Puddle 2013-11-08.1 Verification Steps: 1. Install OpenStack 2. Verify that 'meter-list' works (as tested in Comment #0) # ceilometer meter-list +----------------------------+-------+-----------+--------------------------------------+---------+----------------------------------+ | Name | Type | Unit | Resource ID | User ID | Project ID | +----------------------------+-------+-----------+--------------------------------------+---------+----------------------------------+ | image | gauge | image | 9b3d9a0f-968f-4cc6-b896-a6e520eef9f7 | None | fc4b66bccefa4d12b3b02f5b2852f64c | | image.size | gauge | B | 9b3d9a0f-968f-4cc6-b896-a6e520eef9f7 | None | fc4b66bccefa4d12b3b02f5b2852f64c | | storage.objects | gauge | object | 80c44e301b0744dda33062dcc9d807f8 | None | 80c44e301b0744dda33062dcc9d807f8 | | storage.objects | gauge | object | ea4656c175ea476399fe86f10b2cc48a | None | ea4656c175ea476399fe86f10b2cc48a | | storage.objects | gauge | object | f1ae831adfd34984be1004421afbc77f | None | f1ae831adfd34984be1004421afbc77f | | storage.objects | gauge | object | fc4b66bccefa4d12b3b02f5b2852f64c | None | fc4b66bccefa4d12b3b02f5b2852f64c | | storage.objects.containers | gauge | container | 80c44e301b0744dda33062dcc9d807f8 | None | 80c44e301b0744dda33062dcc9d807f8 | | storage.objects.containers | gauge | container | ea4656c175ea476399fe86f10b2cc48a | None | ea4656c175ea476399fe86f10b2cc48a | | storage.objects.containers | gauge | container | f1ae831adfd34984be1004421afbc77f | None | f1ae831adfd34984be1004421afbc77f | | storage.objects.containers | gauge | container | fc4b66bccefa4d12b3b02f5b2852f64c | None | fc4b66bccefa4d12b3b02f5b2852f64c | | storage.objects.size | gauge | B | 80c44e301b0744dda33062dcc9d807f8 | None | 80c44e301b0744dda33062dcc9d807f8 | | storage.objects.size | gauge | B | ea4656c175ea476399fe86f10b2cc48a | None | ea4656c175ea476399fe86f10b2cc48a | | storage.objects.size | gauge | B | f1ae831adfd34984be1004421afbc77f | None | f1ae831adfd34984be1004421afbc77f | | storage.objects.size | gauge | B | fc4b66bccefa4d12b3b02f5b2852f64c | None | fc4b66bccefa4d12b3b02f5b2852f64c | +----------------------------+-------+-----------+--------------------------------------+---------+----------------------------------+ 3. Verified that there are no errors in Celimeter logs 4. Verified that all Ceilomter services are up # for svc in api central collector compute ; do service openstack-ceilometer-$svc status ; done openstack-ceilometer-api (pid 22755) is running... openstack-ceilometer-central (pid 22725) is running... openstack-ceilometer-collector (pid 22696) is running... openstack-ceilometer-compute (pid 6091) is running...
More permanent fix landed on master upstream: http://github.com/openstack/ceilometer/commit/a8e12c04 and proposed upstream as a backport to stable/havana: https://review.openstack.org/57932 Once the latter lands upstream, the temporary workaround in commit c3e0978c can be reverted.
More permanent fix landed on stable/havana upstream: http://github.com/openstack/ceilometer/commit/9f0a54a4
Temporary work-around revert proposed to internal internal gerrit: https://code.engineering.redhat.com/gerrit/16234 and corresponding backport of the more permanent fix: https://code.engineering.redhat.com/gerrit/16235
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2013-1859.html