Bug 1655652 - ceilometer.nova_client HTTP Error 503
Summary: ceilometer.nova_client HTTP Error 503
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: nova-maint
QA Contact: nova-maint
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-03 16:08 UTC by Noam Manos
Modified: 2019-09-09 15:37 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-12-27 09:20:05 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Noam Manos 2018-12-03 16:08:53 UTC
Description of problem:
On OSP14 fresh deployment with basic tempest neutron tests, I'm seeing ceilometer.nova_client HTTP Error 503

ceilometer/central.log:

novaclient.v2.client [-] REQ: curl -g -i -X GET http://172.17.1.11:8774/v2.1/servers/detail?all_tenants=True -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA1}daf2739c3d72123ea012e3c94ad0e869e3a3c0dc" -H "X-OpenStack-Nova-API-Version: 2.1" _http_log_request /usr/lib/python2.7/site-packages/keystoneauth1/session.py:448

novaclient.v2.client [-] RESP: [503] Cache-Control: no-cache Connection: close Content-Type: text/html _http_log_response /usr/lib/python2.7/site-packages/keystoneauth1/session.py:479

----

ceilometer/agent-notification.log:

ceilometer.publisher.gnocchi [-] No gnocchi definition for event type: capacity.backend publish_events /usr/lib/python2.7/site-packages/ceilometer/publisher/gnocchi.py:484


Version-Release number of selected component (if applicable):
OSP 14, 2018-11-21.2

How reproducible:
Always

Steps to Reproduce:
1. Install OSP14
2. Run neutron tempest tests
3.

Actual results:

*** /var/log/containers/ceilometer/central.log ***

2018-11-22 11:06:38.573 26 INFO ceilometer.agent [-] Config file: {'sources': [{'interval': 300, 'meters': ['cpu', 'cpu_l3_cache', 'memory.usage', 'network.incoming.bytes', 'network.incoming.packets', 'network.outgoing.bytes', 'network.outgoing.packets', 'disk.device.read.bytes', 'disk.device.read.requests', 'disk.device.write.bytes', 'disk.device.write.requests', 'hardware.cpu.util', 'hardware.memory.used', 'hardware.memory.total', 'hardware.memory.buffer', 'hardware.memory.cached', 'hardware.memory.swap.avail', 'hardware.memory.swap.total', 'hardware.system_stats.io.outgoing.blocks', 'hardware.system_stats.io.incoming.blocks', 'hardware.network.ip.incoming.datagrams', 'hardware.network.ip.outgoing.datagrams'], 'name': 'some_pollsters'}]}
2018-11-22 11:06:41.565 26 DEBUG novaclient.v2.client [-] REQ: curl -g -i -X GET http://172.17.1.11:8774/v2.1/servers/detail?all_tenants=True -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA1}daf2739c3d72123ea012e3c94ad0e869e3a3c0dc" -H "X-OpenStack-Nova-API-Version: 2.1" _http_log_request /usr/lib/python2.7/site-packages/keystoneauth1/session.py:448
2018-11-22 11:06:41.587 26 DEBUG novaclient.v2.client [-] RESP: [503] Cache-Control: no-cache Connection: close Content-Type: text/html _http_log_response /usr/lib/python2.7/site-packages/keystoneauth1/session.py:479
2018-11-22 11:06:41.587 26 DEBUG novaclient.v2.client [-] RESP BODY: Omitted, Content-Type is set to text/html. Only application/json responses have their bodies logged. _http_log_response /usr/lib/python2.7/site-packages/keystoneauth1/session.py:511
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client [-] Unknown Error (HTTP 503): ClientException: Unknown Error (HTTP 503)
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client Traceback (most recent call last):
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/ceilometer/nova_client.py", line 39, in with_logging
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/ceilometer/nova_client.py", line 157, in instance_get_all
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client     search_opts=search_opts)
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 858, in list
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client     "servers")
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 257, in _list
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client     resp, body = self.api.client.get(url)
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 328, in get
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client     return self.request(url, 'GET', **kwargs)
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client     raise exceptions.from_response(resp, body, url, method)
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client ClientException: Unknown Error (HTTP 503)
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client 
2018-11-22 11:06:41.591 26 DEBUG ceilometer.polling.manager [-] Skip pollster hardware.memory.cached, no resources found this cycle poll_and_notify /usr/lib/python2.7/site-packages/ceilometer/polling/manager.py:189


*** /var/log/containers/ceilometer/agent-notification.log ***

2018-11-22 12:53:15.516 26 DEBUG ceilometer.publisher.gnocchi [-] Resource 66734959-92e1-4f4c-8d11-a0f4d2df1f16 updated _update_resource /usr/lib/python2.7/site-packages/ceilometer/publisher/gnocchi.py:448
2018-11-22 12:54:00.262 26 DEBUG ceilometer.publisher.gnocchi [-] No gnocchi definition for event type: capacity.backend publish_events /usr/lib/python2.7/site-packages/ceilometer/publisher/gnocchi.py:484
2018-11-22 12:54:01.194 26 DEBUG ceilometer.publisher.gnocchi [-] 5 measures posted against 5 metrics through 1 resources batch_measures /usr/lib/python2.7/site-packages/ceilometer/publisher/gnocchi.py:440
2018-11-22 12:55:02.020 26 DEBUG ceilometer.publisher.gnocchi [-] 5 measures posted against 5 metrics through 1 resources batch_measures /usr/lib/python2.7/site-packages/ceilometer/publisher/gnocchi.py:440
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event [-] Fail to process a notification: KeyError: 0
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event Traceback (most recent call last):
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event   File "/usr/lib/python2.7/site-packages/ceilometer/pipeline/event.py", line 59, in process_notifications
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event     event = self.event_converter.to_event(priority, message)
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event   File "/usr/lib/python2.7/site-packages/ceilometer/event/converter.py", line 276, in to_event
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event     return edef.to_event(priority, notification_body)
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event   File "/usr/lib/python2.7/site-packages/ceilometer/event/converter.py", line 152, in to_event
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event     traits = [trait for trait in traits if trait is not None]
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event   File "/usr/lib/python2.7/site-packages/ceilometer/event/converter.py", line 150, in <genexpr>
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event     for t in self.traits)
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event   File "/usr/lib/python2.7/site-packages/ceilometer/event/converter.py", line 58, in to_trait
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event     value = self.parse(notification_body)
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event   File "/usr/lib/python2.7/site-packages/ceilometer/declarative.py", line 116, in parse
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event     values = self.getter(obj)
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event   File "/usr/lib/python2.7/site-packages/jsonpath_rw/jsonpath.py", line 226, in find
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event     for subdata in self.left.find(datum)
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event   File "/usr/lib/python2.7/site-packages/jsonpath_rw/jsonpath.py", line 228, in find
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event     for submatch in self.right.find(subdata)]
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event   File "/usr/lib/python2.7/site-packages/jsonpath_rw/jsonpath.py", line 444, in find
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event     return [DatumInContext(datum.value[self.index], path=self, context=datum)]
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event KeyError: 0
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event 


Expected results:
No ceilometer errors

Additional info:

There's an upstream bug also - https://bugs.launchpad.net/ceilometer/+bug/1764451

--- Additional comment from Martin Magr on 2018-12-03 10:44:39 EST ---

Originated from bug 1652685:

Those two bugs are not connected. Could you please open new bug on  "ceilometer.nova_client HTTP Error 503" for DFG:Compute? We will keep this BZ open for the later issue.

Comment 1 Matthew Booth 2018-12-13 15:47:24 UTC
Based on the linked launchpad I'm initially assuming this is related to TLS, and therefore the nova logs probably aren't interesting. Is the error that ceilometer is using the wrong endpoint, or that nova isn't available on the expected endpoint? Or TLS configuration of ceilometer, perhaps?

Comment 2 Lee Yarwood 2018-12-13 15:58:34 UTC
Actually the LP might not be relevant, can we get nova-api logs please?

Comment 6 Noam Manos 2018-12-27 09:20:05 UTC
The issue was not reproduced on later puddles, closing.


Note You need to log in before you can comment on or make changes to this bug.