Bug 1260842 - queries to ceilometer are too slow for horizon and ceilometer client
queries to ceilometer are too slow for horizon and ceilometer client
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-ceilometer (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: 8.0 (Liberty)
Assigned To: Eoghan Glynn
Yurii Prokulevych
Depends On:
  Show dependency treegraph
Reported: 2015-09-07 23:20 EDT by August Simonelli
Modified: 2015-09-08 01:32 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2015-09-08 01:32:05 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description August Simonelli 2015-09-07 23:20:01 EDT
Description of problem:

When running ceilometer sample-list i get the following error:

[stack@ospd2 ~ (overcloud-site2)] $ ceilometer --debug sample-list
DEBUG (session) REQ: curl -g -i -X GET -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO (connectionpool) Starting new HTTP connection (1):
DEBUG (connectionpool) "GET /v2.0/ HTTP/1.1" 200 336
DEBUG (session) RESP: [200] content-length: 336 vary: X-Auth-Token connection: keep-alive date: Tue, 08 Sep 2015 01:21:48 GMT content-type: application/json x-openstack-request-id: req-9949e5fd-ab78-423c-b38f-3172fd20f76a
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG (v2) Making authentication request to
DEBUG (connectionpool) "POST /v2.0/tokens HTTP/1.1" 200 4303
DEBUG (client) REQ: curl -g -i -X 'GET' '' -H 'User-Agent: ceilometerclient.openstack.common.apiclient' -H 'X-Auth-Token: {SHA1}9ebecaa3b61611b162c3e21004365e345e25c635'
INFO (connectionpool) Starting new HTTP connection (1):
Traceback (most recent call last):
  File "/usr/bin/ceilometer", line 10, in <module>
  File "/usr/lib/python2.7/site-packages/ceilometerclient/shell.py", line 297, in main
  File "/usr/lib/python2.7/site-packages/ceilometerclient/shell.py", line 246, in main
    args.func(client, args)
  File "/usr/lib/python2.7/site-packages/ceilometerclient/v2/shell.py", line 134, in do_sample_list
    return _do_sample_list(cc, args)
  File "/usr/lib/python2.7/site-packages/ceilometerclient/v2/shell.py", line 160, in _do_sample_list
    samples = cc.new_samples.list(**fields)
  File "/usr/lib/python2.7/site-packages/ceilometerclient/v2/samples.py", line 75, in list
    return self._list(options.build_url("/v2/samples", q, params))
  File "/usr/lib/python2.7/site-packages/ceilometerclient/common/base.py", line 68, in _list
    resp = self.api.get(url)
  File "/usr/lib/python2.7/site-packages/ceilometerclient/openstack/common/apiclient/client.py", line 355, in get
    return self.client_request("GET", url, **kwargs)
  File "/usr/lib/python2.7/site-packages/ceilometerclient/openstack/common/apiclient/client.py", line 345, in client_request
    self, method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/ceilometerclient/openstack/common/apiclient/client.py", line 261, in client_request
    method, self.concat_url(endpoint, url), **kwargs)
  File "/usr/lib/python2.7/site-packages/ceilometerclient/openstack/common/apiclient/client.py", line 189, in request
    resp = self.http.request(method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))

This happens after 60 seconds.

Version-Release number of selected component (if applicable):

How reproducible:
Happens every time.

Steps to Reproduce:
1. load up ceilometer with a lot of samples
2. run ceilometer sample-list

Actual results:
requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))

Expected results:
A list of all samples

Additional info:
Comment 3 August Simonelli 2015-09-07 23:24:00 EDT
The 60sec failure time made me think this is something else. 

I bypassed haproxy by going directly to the ip it was balancing with:

 ceilometer --debug --os-endpoint sample-list

and it worked.

So haproxy.cfg i added a timeout to ceilometer:

listen ceilometer
  timeout  server 2m
  server overcloud-controller-0 check fall 5 inter 2000 rise 2

the default was 1m as set higher in the file by OSP director.

When i did this I could get to my samples but it takes a while:

ceilometer --debug meter-list

real    3m7.002s
user    1m0.061s
sys     0m3.088s

either ceilometer needs to be quicker or haproxy needs to be changed.
this also breaks horizon. i'll log another BZ for that.
Comment 4 August Simonelli 2015-09-07 23:33:35 EDT
see https://bugzilla.redhat.com/show_bug.cgi?id=1260844 for how this affects horizon
Comment 5 August Simonelli 2015-09-07 23:37:02 EDT
by a big query i guess i mean this:

[root@overcloud-controller-0 ~ (overcloud-site2)] $ time ceilometer sample-list| wc -l

real    2m54.349s
user    0m59.231s
sys     0m1.548s
Comment 6 August Simonelli 2015-09-08 01:32:05 EDT
changing haproxy does solve this. so i don't see this as a bug but rather a misconfig on my part. closing.

Note You need to log in before you can comment on or make changes to this bug.