Bug 1293799

Summary: Ceilometer compute agent does not provide any meters, stats
Product: Red Hat OpenStack Reporter: Pratik Pravin Bandarkar <pbandark>
Component: openstack-ceilometerAssignee: Pradeep Kilambi <pkilambi>
Status: CLOSED CURRENTRELEASE QA Contact: Sasha Smolyak <ssmolyak>
Severity: high Docs Contact:
Priority: high    
Version: 7.0 (Kilo)CC: eglynn, jruzicka, pkilambi, sputhenp, srevivo
Target Milestone: ---Keywords: Triaged, Unconfirmed, ZStream
Target Release: 7.0 (Kilo)   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-20 15:04:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pratik Pravin Bandarkar 2015-12-23 06:37:45 UTC
Description of problem:

`ceilometer meter-list` do not show meters from the compute nodes. 

From "ceilometer/compute.log" i can see 
<snip>
2015-12-22 21:55:03.618 1304 ERROR ceilometer.agent.base [-] Unable to discover resources: ('Connection aborted.', error(101, 'ENETUNREACH'))
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base Traceback (most recent call last):
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/ceilometer/agent/base.py", line 346, in discover
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base     discovered = discoverer.discover(self, param)
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/ceilometer/compute/discovery.py", line 37, in discover
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base     instances = self.nova_cli.instance_get_all_by_host(cfg.CONF.host)
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/ceilometer/nova_client.py", line 48, in with_logging
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base     return func(*args, **kwargs)
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/ceilometer/nova_client.py", line 145, in instance_get_all_by_host
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base     search_opts=search_opts))
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 583, in list
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base     return self._list("/servers%s%s" % (detail, query_string), "servers")
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 64, in _list
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base     _resp, body = self.api.client.get(url)
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 450, in get
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base     return self._cs_request(url, 'GET', **kwargs)
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 428, in _cs_request
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base     resp, body = self._time_request(url, method, **kwargs)
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 397, in _time_request
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base     resp, body = self.request(url, method, **kwargs)
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 368, in request
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base     **kwargs)
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base     response = session.request(method=method, url=url, **kwargs)
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base     resp = self.send(prep, **send_kwargs)
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base     r = adapter.send(request, **kwargs)
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base     raise ConnectionError(err, request=request)
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base ConnectionError: ('Connection aborted.', error(101, 'ENETUNREACH'))
2015-12-22 21:55:03.618 1304 TRACE ceilometer.agent.base 
2015-12-22 21:55:03.619 1304 INFO ceilometer.agent.base [-] Skip polling pollster network.outgoing.bytes, no resources found
</snip>


- "openstack-ceilometer-compute" service is starting on the compute node without any issue.



Version-Release number of selected component (if applicable):
- RHOS7 deployed using director.
- openstack-ceilometer-compute-2015.1.1-1.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1.Deploy HA setup using director.
2.check `ceilometer meter-list` output. 
3.monitor "/var/log/ceilometer/compute.log" file for the error.

Actual results:
Ceilometer compute agent does not provide any meters, stats

Expected results:
ceilometer should display all meters.

Additional info:

Comment 3 Eoghan Glynn 2015-12-23 14:30:08 UTC
As a work-around, please set:

  os_endpoint_type=internalURL

in the /etc/ceilometer/ceilometer.conf on each on the compute nodes, and then restart the ceilometer compute agent on each node:

  $ sudo service openstack-ceilometer-compute restart

This should be set by default by director.

Comment 6 Sadique Puthen 2015-12-25 18:19:44 UTC
This is handled at Director side through bug https://bugzilla.redhat.com/show_bug.cgi?id=1262025

Patch is here: https://review.openstack.org/#/c/240538/

Request a backport to OSP7-D if needed.