Bug 1015878
| Summary: | openstack-ceilometer: After installation ceilometer not functioning due to 'missing module named kombu'. | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Omri Hochman <ohochman> | ||||||||||||||
| Component: | openstack-ceilometer | Assignee: | Eoghan Glynn <eglynn> | ||||||||||||||
| Status: | CLOSED ERRATA | QA Contact: | Nir Magnezi <nmagnezi> | ||||||||||||||
| Severity: | high | Docs Contact: | |||||||||||||||
| Priority: | high | ||||||||||||||||
| Version: | 4.0 | CC: | ddomingo, eglynn, hateya, jruzicka, ohochman, pbrady, sradvan, yeylon | ||||||||||||||
| Target Milestone: | rc | Keywords: | TestBlocker | ||||||||||||||
| Target Release: | 4.0 | ||||||||||||||||
| Hardware: | x86_64 | ||||||||||||||||
| OS: | Linux | ||||||||||||||||
| Whiteboard: | |||||||||||||||||
| Fixed In Version: | openstack-ceilometer-2013.2-2.el6ost | Doc Type: | Bug Fix | ||||||||||||||
| Doc Text: |
Cause: The client library for RabbitMQ was unconditionally imported by Ceilometer, regardless of the actual AMQP provider in use.
Consequence: When only QPID is installed, the Ceilometer service failed to start, producing an ImportError.
Fix: Importing the kombu library is now conditional on RabbitMQ being the selected AMQP provider.
Result: The Ceilometer services start successfully in a QPID-only installation.
|
Story Points: | --- | ||||||||||||||
| Clone Of: | Environment: | ||||||||||||||||
| Last Closed: | 2013-12-20 00:25:59 UTC | Type: | Bug | ||||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||||
| Documentation: | --- | CRM: | |||||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||||
| Embargoed: | |||||||||||||||||
| Bug Depends On: | |||||||||||||||||
| Bug Blocks: | 967310 | ||||||||||||||||
| Attachments: |
|
||||||||||||||||
Created attachment 808497 [details]
ceilometer-api.log
Created attachment 808498 [details]
ceilometer-compute.log
kombu is not supported in RHOS. Eoghan can we config around this dependency? I should note that I did try openstack-ceilometer-compute-2013.2-0.12.rc1.el6.noarch from RDO on RHEL 6.4, which should not differ significantly from the above tested package, and it started without issue with no python-kombu installed So I tried the latest RHOS build on RHEL 6.4 (openstack-ceilometer-2013.2-0.12.rc1.el6ost) and couldn't reproduce the issue. As mentioned this shouldn't be significantly different from openstack-ceilometer-2013.2-0.11.rc1.el6ost in this regard, so I suspect config differences are triggering this. Can you attach /usr/share/ceilometer/ceilometer-dist.conf and /etc/ceilometer/ceilometer.conf thanks. Yes - It happened to me only on RHEL 6.5 (on 6.4 it was fine). I can't reproduce this on 6.4 or 6.5. (In reply to Lon Hohberger from comment #10) > I can't reproduce this on 6.4 or 6.5. (In reply to Lon Hohberger from comment #10) > I can't reproduce this on 6.4 or 6.5. Hi Lon, I managed to reproduce this issue with RHEL 6.5 Used: packstack --allinone Checked Ceilometer Logs: grep -i error /var/log/celimeter/*.log central.log:2013-10-20 12:54:14.335 25169 WARNING ceilometer.central.manager [-] Continue after error from storage.objects: [Errno 111] ECONNREFUSED central.log:2013-10-20 12:54:14.335 25169 ERROR ceilometer.central.manager [-] [Errno 111] ECONNREFUSED central.log:2013-10-20 12:54:14.335 25169 TRACE ceilometer.central.manager raise error, msg central.log:2013-10-20 12:54:14.335 25169 TRACE ceilometer.central.manager error: [Errno 111] ECONNREFUSED compute.log:2013-10-20 12:33:16.065 13754 ERROR stevedore.extension [-] Could not load 'rpc': No module named kombu compute.log:2013-10-20 12:33:16.066 13754 ERROR stevedore.extension [-] No module named kombu compute.log:2013-10-20 12:33:16.066 13754 TRACE stevedore.extension ImportError: No module named kombu compute.log:2013-10-20 12:33:16.067 13754 ERROR ceilometer.pipeline [-] Unable to load publisher rpc:// compute.log:2013-10-20 12:33:16.067 13754 TRACE ceilometer.pipeline RuntimeError: No 'ceilometer.publisher' driver found, looking for 'rpc' Reproduces on my environment (rhel 6.5 with puddle 2013-10-15.1) : -------------------------------------------------------------------- [root@ ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.5 Beta (Santiago) [root@ ~]# rpm -qa | grep ceilometer openstack-ceilometer-common-2013.2-0.12.rc2.el6ost.noarch openstack-ceilometer-collector-2013.2-0.12.rc2.el6ost.noarch python-ceilometer-2013.2-0.12.rc2.el6ost.noarch openstack-ceilometer-api-2013.2-0.12.rc2.el6ost.noarch python-ceilometerclient-1.0.6-1.el6ost.noarch openstack-ceilometer-compute-2013.2-0.12.rc2.el6ost.noarch openstack-ceilometer-central-2013.2-0.12.rc2.el6ost.noarch [root@ ~]# rpm -qa | grep packstack openstack-packstack-2013.2.1-0.6.dev763.el6ost.noarch just in case I'm adding packstack - setup.log + answer_file. Created attachment 814217 [details]
answer_file
Created attachment 814218 [details]
pasckatck_setup.log
Can you attach the configs as requested in comment 8 May be caused by https://review.openstack.org/#/c/35604/ Specifically therein the unconditional: +cfg.CONF.import_opt('rabbit_max_retries', + 'ceilometer.openstack.common.rpc.impl_kombu') Temp workaround to be addressed more generally upstream: https://code.engineering.redhat.com/gerrit/gitweb?p=ceilometer.git;a=commitdiff;h=c3e0978c0 Created attachment 814956 [details]
ceilomeret.conf
Created attachment 814957 [details]
ceilomeret-dist.conf
More complete fix proposed upstream as: https://review.openstack.org/53889 Once this lands on master, it can be cherry-picked onto stable/havana upstream and then replace the temporary workaround: https://code.engineering.redhat.com/gerrit/gitweb?p=ceilometer.git;a=commitdiff;h=c3e0978c0 Verified NVR: # rpm -qa | grep openstack-ceilometer openstack-ceilometer-central-2013.2-1.el6ost.noarch openstack-ceilometer-api-2013.2-1.el6ost.noarch openstack-ceilometer-compute-2013.2-1.el6ost.noarch openstack-ceilometer-common-2013.2-1.el6ost.noarch openstack-ceilometer-collector-2013.2-1.el6ost.noarch Puddle 2013-11-08.1 Verification Steps: 1. Install OpenStack 2. Verify that 'meter-list' works (as tested in Comment #0) # ceilometer meter-list +----------------------------+-------+-----------+--------------------------------------+---------+----------------------------------+ | Name | Type | Unit | Resource ID | User ID | Project ID | +----------------------------+-------+-----------+--------------------------------------+---------+----------------------------------+ | image | gauge | image | 9b3d9a0f-968f-4cc6-b896-a6e520eef9f7 | None | fc4b66bccefa4d12b3b02f5b2852f64c | | image.size | gauge | B | 9b3d9a0f-968f-4cc6-b896-a6e520eef9f7 | None | fc4b66bccefa4d12b3b02f5b2852f64c | | storage.objects | gauge | object | 80c44e301b0744dda33062dcc9d807f8 | None | 80c44e301b0744dda33062dcc9d807f8 | | storage.objects | gauge | object | ea4656c175ea476399fe86f10b2cc48a | None | ea4656c175ea476399fe86f10b2cc48a | | storage.objects | gauge | object | f1ae831adfd34984be1004421afbc77f | None | f1ae831adfd34984be1004421afbc77f | | storage.objects | gauge | object | fc4b66bccefa4d12b3b02f5b2852f64c | None | fc4b66bccefa4d12b3b02f5b2852f64c | | storage.objects.containers | gauge | container | 80c44e301b0744dda33062dcc9d807f8 | None | 80c44e301b0744dda33062dcc9d807f8 | | storage.objects.containers | gauge | container | ea4656c175ea476399fe86f10b2cc48a | None | ea4656c175ea476399fe86f10b2cc48a | | storage.objects.containers | gauge | container | f1ae831adfd34984be1004421afbc77f | None | f1ae831adfd34984be1004421afbc77f | | storage.objects.containers | gauge | container | fc4b66bccefa4d12b3b02f5b2852f64c | None | fc4b66bccefa4d12b3b02f5b2852f64c | | storage.objects.size | gauge | B | 80c44e301b0744dda33062dcc9d807f8 | None | 80c44e301b0744dda33062dcc9d807f8 | | storage.objects.size | gauge | B | ea4656c175ea476399fe86f10b2cc48a | None | ea4656c175ea476399fe86f10b2cc48a | | storage.objects.size | gauge | B | f1ae831adfd34984be1004421afbc77f | None | f1ae831adfd34984be1004421afbc77f | | storage.objects.size | gauge | B | fc4b66bccefa4d12b3b02f5b2852f64c | None | fc4b66bccefa4d12b3b02f5b2852f64c | +----------------------------+-------+-----------+--------------------------------------+---------+----------------------------------+ 3. Verified that there are no errors in Celimeter logs 4. Verified that all Ceilomter services are up # for svc in api central collector compute ; do service openstack-ceilometer-$svc status ; done openstack-ceilometer-api (pid 22755) is running... openstack-ceilometer-central (pid 22725) is running... openstack-ceilometer-collector (pid 22696) is running... openstack-ceilometer-compute (pid 6091) is running... More permanent fix landed on master upstream: http://github.com/openstack/ceilometer/commit/a8e12c04 and proposed upstream as a backport to stable/havana: https://review.openstack.org/57932 Once the latter lands upstream, the temporary workaround in commit c3e0978c can be reverted. More permanent fix landed on stable/havana upstream: http://github.com/openstack/ceilometer/commit/9f0a54a4 Temporary work-around revert proposed to internal internal gerrit: https://code.engineering.redhat.com/gerrit/16234 and corresponding backport of the more permanent fix: https://code.engineering.redhat.com/gerrit/16235 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2013-1859.html |
openstack-ceilometer: After installation ceilometer not functioning due to 'missing module named kombu'. Environment (havana - puddle 2013-10-03.3) : ---------------------------------------------- [root@cougar15 /(keystone_admin)]# rpm -qa | grep ceilometer python-ceilometer-2013.2-0.11.rc1.el6ost.noarch openstack-ceilometer-central-2013.2-0.11.rc1.el6ost.noarch openstack-ceilometer-common-2013.2-0.11.rc1.el6ost.noarch openstack-ceilometer-api-2013.2-0.11.rc1.el6ost.noarch python-ceilometerclient-1.0.0-2.el6ost.noarch openstack-ceilometer-compute-2013.2-0.11.rc1.el6ost.noarch openstack-ceilometer-collector-2013.2-0.11.rc1.el6ost.noarch openstack-nova-2013.2-0.24 Steps: ------- 1) Install RHOS Havana with ceilometer - using packstack. 2) Boot Instance. 3) Attempt to perform - 'celiometer meter-list' Results: --------- - 'celiometer meter-list' - return no results. Notes: ------ - all ceilometer services are up and running. - restart ceilometer services does not help ceilometer-compute.log: ------------------------ 2013-10-06 13:35:30.555 3988 INFO ceilometer.openstack.common.rpc.impl_qpid [-] Connected to AMQP server on 10.35.160.119:5672 2013-10-06 13:35:30.589 3988 INFO ceilometer.pipeline [-] Pipeline config: [{'meters': ['*'], 'interval': 600, 'transformers': None, 'name': 'meter_pipeline', 'publishers': ['rpc://']}, {'me ters': ['cpu'], 'interval': 600, 'transformers': [{'name': 'rate_of_change', 'parameters': {'target': {'scale': '100.0 / (10**9 * (resource_metadata.cpu_number or 1))', 'type': 'gauge', 'nam e': 'cpu_util', 'unit': '%'}}}], 'name': 'cpu_pipeline', 'publishers': ['rpc://']}] 2013-10-06 13:35:30.602 3988 ERROR stevedore.extension [-] Could not load 'rpc': No module named kombu 2013-10-06 13:35:30.602 3988 ERROR stevedore.extension [-] No module named kombu 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension Traceback (most recent call last): 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/stevedore/extension.py", line 89, in _load_plugins 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension invoke_kwds, 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/stevedore/named.py", line 57, in _load_one_plugin 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension ep, invoke_on_load, invoke_args, invoke_kwds, 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/stevedore/extension.py", line 101, in _load_one_plugin 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension plugin = ep.load() 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1948, in load 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension entry = __import__(self.module_name, globals(),globals(), ['__name__']) 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/ceilometer/publisher/rpc.py", line 61, in <module> 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension 'ceilometer.openstack.common.rpc.impl_kombu') 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/oslo/config/cfg.py", line 1754, in import_opt 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension __import__(module_str) 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 26, in <module> 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension import kombu 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension ImportError: No module named kombu 2013-10-06 13:35:30.602 3988 TRACE stevedore.extension 2013-10-06 13:35:30.606 3988 ERROR ceilometer.pipeline [-] Unable to load publisher rpc:// 2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline Traceback (most recent call last): 2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline File "/usr/lib/python2.6/site-packages/ceilometer/pipeline.py", line 126, in __init__ 2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline self.publishers.append(publisher.get_publisher(p)) 2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline File "/usr/lib/python2.6/site-packages/ceilometer/publisher/__init__.py", line 33, in get_publisher 2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline loaded_driver = driver.DriverManager(namespace, parse_result.scheme) 2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline File "/usr/lib/python2.6/site-packages/stevedore/driver.py", line 35, in __init__ 2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline (namespace, name)) 2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline RuntimeError: No 'ceilometer.publisher' driver found, looking for 'rpc' 2013-10-06 13:35:30.606 3988 TRACE ceilometer.pipeline 2013-10-06 13:35:30.607 3988 ERROR stevedore.extension [-] Could not load 'rpc': No module named kombu 2013-10-06 13:35:30.608 3988 ERROR stevedore.extension [-] No module named kombu 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension Traceback (most recent call last): 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/stevedore/extension.py", line 89, in _load_plugins 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension invoke_kwds, 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/stevedore/named.py", line 57, in _load_one_plugin 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension ep, invoke_on_load, invoke_args, invoke_kwds, 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/stevedore/extension.py", line 101, in _load_one_plugin 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension plugin = ep.load() 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1948, in load 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension entry = __import__(self.module_name, globals(),globals(), ['__name__']) 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/ceilometer/publisher/rpc.py", line 61, in <module> 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension 'ceilometer.openstack.common.rpc.impl_kombu') 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/oslo/config/cfg.py", line 1754, in import_opt 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension __import__(module_str) 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension File "/usr/lib/python2.6/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 26, in <module> 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension import kombu 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension ImportError: No module named kombu 2013-10-06 13:35:30.608 3988 TRACE stevedore.extension 2013-10-06 13:35:30.609 3988 ERROR ceilometer.pipeline [-] Unable to load publisher rpc:// 2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline Traceback (most recent call last): 2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline File "/usr/lib/python2.6/site-packages/ceilometer/pipeline.py", line 126, in __init__ 2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline self.publishers.append(publisher.get_publisher(p)) 2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline File "/usr/lib/python2.6/site-packages/ceilometer/publisher/__init__.py", line 33, in get_publisher 2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline loaded_driver = driver.DriverManager(namespace, parse_result.scheme) 2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline File "/usr/lib/python2.6/site-packages/stevedore/driver.py", line 35, in __init__ 2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline (namespace, name)) 2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline RuntimeError: No 'ceilometer.publisher' driver found, looking for 'rpc' 2013-10-06 13:35:30.609 3988 TRACE ceilometer.pipeline 2013-10-06 13:35:30.609 3988 INFO ceilometer.pipeline [-] Pipeline cpu_pipeline: Setup transformer instance rate_of_change with parameter {'target': {'scale': '100.0 / (10**9 * (resource_metadata.cpu_number or 1))', 'type': 'gauge', 'name': 'cpu_util', 'unit': '%'}} 2013-10-06 13:35:30.621 3988 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.35.160.119 2013-10-06 13:35:30.865 3988 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.35.160.119 2013-10-06 13:35:32.399 3988 AUDIT ceilometer.pipeline [-] Flush pipeline meter_pipeline 2013-10-06 13:35:32.400 3988 AUDIT ceilometer.pipeline [-] Flush pipeline cpu_pipeline