Description of problem: When running ceilometer sample-list i get the following error: [stack@ospd2 ~ (overcloud-site2)] $ ceilometer --debug sample-list DEBUG (session) REQ: curl -g -i -X GET http://10.0.14.20:5000/v2.0/ -H "Accept: application/json" -H "User-Agent: python-keystoneclient" INFO (connectionpool) Starting new HTTP connection (1): 10.0.14.20 DEBUG (connectionpool) "GET /v2.0/ HTTP/1.1" 200 336 DEBUG (session) RESP: [200] content-length: 336 vary: X-Auth-Token connection: keep-alive date: Tue, 08 Sep 2015 01:21:48 GMT content-type: application/json x-openstack-request-id: req-9949e5fd-ab78-423c-b38f-3172fd20f76a RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://10.0.14.20:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}} DEBUG (v2) Making authentication request to http://10.0.14.20:5000/v2.0/tokens DEBUG (connectionpool) "POST /v2.0/tokens HTTP/1.1" 200 4303 DEBUG (client) REQ: curl -g -i -X 'GET' 'http://10.0.14.20:8777/v2/samples' -H 'User-Agent: ceilometerclient.openstack.common.apiclient' -H 'X-Auth-Token: {SHA1}9ebecaa3b61611b162c3e21004365e345e25c635' INFO (connectionpool) Starting new HTTP connection (1): 10.0.14.20 Traceback (most recent call last): File "/usr/bin/ceilometer", line 10, in <module> sys.exit(main()) File "/usr/lib/python2.7/site-packages/ceilometerclient/shell.py", line 297, in main CeilometerShell().main(args) File "/usr/lib/python2.7/site-packages/ceilometerclient/shell.py", line 246, in main args.func(client, args) File "/usr/lib/python2.7/site-packages/ceilometerclient/v2/shell.py", line 134, in do_sample_list return _do_sample_list(cc, args) File "/usr/lib/python2.7/site-packages/ceilometerclient/v2/shell.py", line 160, in _do_sample_list samples = cc.new_samples.list(**fields) File "/usr/lib/python2.7/site-packages/ceilometerclient/v2/samples.py", line 75, in list return self._list(options.build_url("/v2/samples", q, params)) File "/usr/lib/python2.7/site-packages/ceilometerclient/common/base.py", line 68, in _list resp = self.api.get(url) File "/usr/lib/python2.7/site-packages/ceilometerclient/openstack/common/apiclient/client.py", line 355, in get return self.client_request("GET", url, **kwargs) File "/usr/lib/python2.7/site-packages/ceilometerclient/openstack/common/apiclient/client.py", line 345, in client_request self, method, url, **kwargs) File "/usr/lib/python2.7/site-packages/ceilometerclient/openstack/common/apiclient/client.py", line 261, in client_request method, self.concat_url(endpoint, url), **kwargs) File "/usr/lib/python2.7/site-packages/ceilometerclient/openstack/common/apiclient/client.py", line 189, in request resp = self.http.request(method, url, **kwargs) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",)) This happens after 60 seconds. Version-Release number of selected component (if applicable): How reproducible: Happens every time. Steps to Reproduce: 1. load up ceilometer with a lot of samples 2. run ceilometer sample-list 3. Actual results: requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",)) Expected results: A list of all samples Additional info:
The 60sec failure time made me think this is something else. I bypassed haproxy by going directly to the ip it was balancing with: ceilometer --debug --os-endpoint http://10.0.1.23:8777/ sample-list and it worked. So haproxy.cfg i added a timeout to ceilometer: listen ceilometer timeout server 2m bind 10.0.1.20:8777 bind 10.0.14.20:8777 server overcloud-controller-0 10.0.1.23:8777 check fall 5 inter 2000 rise 2 the default was 1m as set higher in the file by OSP director. When i did this I could get to my samples but it takes a while: ceilometer --debug meter-list real 3m7.002s user 1m0.061s sys 0m3.088s either ceilometer needs to be quicker or haproxy needs to be changed. this also breaks horizon. i'll log another BZ for that.
see https://bugzilla.redhat.com/show_bug.cgi?id=1260844 for how this affects horizon
by a big query i guess i mean this: [root@overcloud-controller-0 ~ (overcloud-site2)] $ time ceilometer sample-list| wc -l 135519 real 2m54.349s user 0m59.231s sys 0m1.548s
changing haproxy does solve this. so i don't see this as a bug but rather a misconfig on my part. closing.