Description of problem:
On OSP14 fresh deployment with basic tempest neutron tests, I'm seeing ceilometer.nova_client HTTP Error 503

ceilometer/central.log:

novaclient.v2.client [-] REQ: curl -g -i -X GET http://172.17.1.11:8774/v2.1/servers/detail?all_tenants=True -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA1}daf2739c3d72123ea012e3c94ad0e869e3a3c0dc" -H "X-OpenStack-Nova-API-Version: 2.1" _http_log_request /usr/lib/python2.7/site-packages/keystoneauth1/session.py:448

novaclient.v2.client [-] RESP: [503] Cache-Control: no-cache Connection: close Content-Type: text/html _http_log_response /usr/lib/python2.7/site-packages/keystoneauth1/session.py:479

----

ceilometer/agent-notification.log:

ceilometer.publisher.gnocchi [-] No gnocchi definition for event type: capacity.backend publish_events /usr/lib/python2.7/site-packages/ceilometer/publisher/gnocchi.py:484


Version-Release number of selected component (if applicable):
OSP 14, 2018-11-21.2

How reproducible:
Always

Steps to Reproduce:
1. Install OSP14
2. Run neutron tempest tests
3.

Actual results:

*** /var/log/containers/ceilometer/central.log ***

2018-11-22 11:06:38.573 26 INFO ceilometer.agent [-] Config file: {'sources': [{'interval': 300, 'meters': ['cpu', 'cpu_l3_cache', 'memory.usage', 'network.incoming.bytes', 'network.incoming.packets', 'network.outgoing.bytes', 'network.outgoing.packets', 'disk.device.read.bytes', 'disk.device.read.requests', 'disk.device.write.bytes', 'disk.device.write.requests', 'hardware.cpu.util', 'hardware.memory.used', 'hardware.memory.total', 'hardware.memory.buffer', 'hardware.memory.cached', 'hardware.memory.swap.avail', 'hardware.memory.swap.total', 'hardware.system_stats.io.outgoing.blocks', 'hardware.system_stats.io.incoming.blocks', 'hardware.network.ip.incoming.datagrams', 'hardware.network.ip.outgoing.datagrams'], 'name': 'some_pollsters'}]}
2018-11-22 11:06:41.565 26 DEBUG novaclient.v2.client [-] REQ: curl -g -i -X GET http://172.17.1.11:8774/v2.1/servers/detail?all_tenants=True -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA1}daf2739c3d72123ea012e3c94ad0e869e3a3c0dc" -H "X-OpenStack-Nova-API-Version: 2.1" _http_log_request /usr/lib/python2.7/site-packages/keystoneauth1/session.py:448
2018-11-22 11:06:41.587 26 DEBUG novaclient.v2.client [-] RESP: [503] Cache-Control: no-cache Connection: close Content-Type: text/html _http_log_response /usr/lib/python2.7/site-packages/keystoneauth1/session.py:479
2018-11-22 11:06:41.587 26 DEBUG novaclient.v2.client [-] RESP BODY: Omitted, Content-Type is set to text/html. Only application/json responses have their bodies logged. _http_log_response /usr/lib/python2.7/site-packages/keystoneauth1/session.py:511
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client [-] Unknown Error (HTTP 503): ClientException: Unknown Error (HTTP 503)
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client Traceback (most recent call last):
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/ceilometer/nova_client.py", line 39, in with_logging
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/ceilometer/nova_client.py", line 157, in instance_get_all
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client     search_opts=search_opts)
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 858, in list
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client     "servers")
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 257, in _list
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client     resp, body = self.api.client.get(url)
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 328, in get
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client     return self.request(url, 'GET', **kwargs)
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client     raise exceptions.from_response(resp, body, url, method)
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client ClientException: Unknown Error (HTTP 503)
2018-11-22 11:06:41.588 26 ERROR ceilometer.nova_client 
2018-11-22 11:06:41.591 26 DEBUG ceilometer.polling.manager [-] Skip pollster hardware.memory.cached, no resources found this cycle poll_and_notify /usr/lib/python2.7/site-packages/ceilometer/polling/manager.py:189


*** /var/log/containers/ceilometer/agent-notification.log ***

2018-11-22 12:53:15.516 26 DEBUG ceilometer.publisher.gnocchi [-] Resource 66734959-92e1-4f4c-8d11-a0f4d2df1f16 updated _update_resource /usr/lib/python2.7/site-packages/ceilometer/publisher/gnocchi.py:448
2018-11-22 12:54:00.262 26 DEBUG ceilometer.publisher.gnocchi [-] No gnocchi definition for event type: capacity.backend publish_events /usr/lib/python2.7/site-packages/ceilometer/publisher/gnocchi.py:484
2018-11-22 12:54:01.194 26 DEBUG ceilometer.publisher.gnocchi [-] 5 measures posted against 5 metrics through 1 resources batch_measures /usr/lib/python2.7/site-packages/ceilometer/publisher/gnocchi.py:440
2018-11-22 12:55:02.020 26 DEBUG ceilometer.publisher.gnocchi [-] 5 measures posted against 5 metrics through 1 resources batch_measures /usr/lib/python2.7/site-packages/ceilometer/publisher/gnocchi.py:440
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event [-] Fail to process a notification: KeyError: 0
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event Traceback (most recent call last):
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event   File "/usr/lib/python2.7/site-packages/ceilometer/pipeline/event.py", line 59, in process_notifications
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event     event = self.event_converter.to_event(priority, message)
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event   File "/usr/lib/python2.7/site-packages/ceilometer/event/converter.py", line 276, in to_event
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event     return edef.to_event(priority, notification_body)
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event   File "/usr/lib/python2.7/site-packages/ceilometer/event/converter.py", line 152, in to_event
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event     traits = [trait for trait in traits if trait is not None]
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event   File "/usr/lib/python2.7/site-packages/ceilometer/event/converter.py", line 150, in <genexpr>
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event     for t in self.traits)
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event   File "/usr/lib/python2.7/site-packages/ceilometer/event/converter.py", line 58, in to_trait
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event     value = self.parse(notification_body)
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event   File "/usr/lib/python2.7/site-packages/ceilometer/declarative.py", line 116, in parse
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event     values = self.getter(obj)
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event   File "/usr/lib/python2.7/site-packages/jsonpath_rw/jsonpath.py", line 226, in find
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event     for subdata in self.left.find(datum)
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event   File "/usr/lib/python2.7/site-packages/jsonpath_rw/jsonpath.py", line 228, in find
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event     for submatch in self.right.find(subdata)]
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event   File "/usr/lib/python2.7/site-packages/jsonpath_rw/jsonpath.py", line 444, in find
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event     return [DatumInContext(datum.value[self.index], path=self, context=datum)]
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event KeyError: 0
2018-11-22 12:55:05.026 26 ERROR ceilometer.pipeline.event 


Expected results:
No ceilometer errors

Additional info:

There's an upstream bug also - https://bugs.launchpad.net/ceilometer/+bug/1764451

--- Additional comment from Martin Magr on 2018-12-03 10:44:39 EST ---

Originated from bug 1652685:

Those two bugs are not connected. Could you please open new bug on  "ceilometer.nova_client HTTP Error 503" for DFG:Compute? We will keep this BZ open for the later issue.