Bug 1575886 - Telemetry standalone node: Get error Error 500 when trying to access measures of gnocchi metrics
Summary: Telemetry standalone node: Get error Error 500 when trying to access measures...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: gnocchi
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: z3
: 13.0 (Queens)
Assignee: Pradeep Kilambi
QA Contact: Sasha Smolyak
URL:
Whiteboard:
Depends On:
Blocks: 1458303
TreeView+ depends on / blocked
 
Reported: 2018-05-08 07:51 UTC by Sasha Smolyak
Modified: 2019-08-28 06:34 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-28 06:34:02 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Gerrithub.io 415501 0 None None None 2018-07-11 21:36:34 UTC

Description Sasha Smolyak 2018-05-08 07:51:21 UTC
Description of problem:
When deploying Telemetry on standalone node, get error Error 500 when trying to access measures of gnocchi metrics. The table of resources is accessible, the resource is shown, but measures aren't reachable

Version-Release number of selected component (if applicable):
gnocchi-common-4.2.3-2.el7ost.noarch
python2-collectd-gnocchi-1.7.1-1.el7ost.noarch
gnocchi-metricd-4.2.3-2.el7ost.noarch
python2-gnocchiclient-7.0.1-1.el7ost.noarch
puppet-gnocchi-12.4.0-0.20180329032858.5dfa350.el7ost.noarch
gnocchi-statsd-4.2.3-2.el7ost.noarch
gnocchi-api-4.2.3-2.el7ost.noarch
python-gnocchi-4.2.3-2.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy Telemetry in standalone node, configuration 3 controllers, 2 compute, 1 telemetry
2. Create instance in overcloud
3. Try to reach cpu measures of the instance

Actual results:
(overcloud) [stack@undercloud-0 ~]$ openstack metric measures show --resource-id 59b85149-595c-45f3-8621-71a041ee604d cpu
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at 
 [no address given] to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>
 (HTTP 500)


Expected results:
The measures are shown

Additional info:
Logs on telemetry-0 node:

1. /var/log/containers/ceilometer/central.log

2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client [-] Unknown Error (HTTP 503): ClientException: Unknown Error (HTTP 503)
2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client Traceback (most recent call last):
2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/ceilometer/nova_client.py", line 39, in with_logging
2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client     return func(*args, **kwargs)
2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/ceilometer/nova_client.py", line 157, in instance_get_all
2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client     search_opts=search_opts)
2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 884, in list
2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client     "servers")
2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 254, in _list
2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client     resp, body = self.api.client.get(url)
2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 304, in get
2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client     return self.request(url, 'GET', **kwargs)
2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client     raise exceptions.from_response(resp, body, url, method)
2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client ClientException: Unknown Error (HTTP 503)
2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client 



2. /var/log/containers/ceilometer/ceilometer-upgrade.log

2018-05-07 18:52:40.968 11 CRITICAL ceilometer [-] Unhandled error: ClientException: <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
 (HTTP 503)
2018-05-07 18:52:40.968 11 ERROR ceilometer Traceback (most recent call last):
2018-05-07 18:52:40.968 11 ERROR ceilometer   File "/usr/bin/ceilometer-upgrade", line 10, in <module>
2018-05-07 18:52:40.968 11 ERROR ceilometer     sys.exit(upgrade())
2018-05-07 18:52:40.968 11 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/ceilometer/cmd/storage.py", line 60, in upgrade
2018-05-07 18:52:40.968 11 ERROR ceilometer     )(gnocchi_client.upgrade_resource_types, conf)
2018-05-07 18:52:40.968 11 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 295, in call
2018-05-07 18:52:40.968 11 ERROR ceilometer     start_time=start_time)
2018-05-07 18:52:40.968 11 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 252, in iter
2018-05-07 18:52:40.968 11 ERROR ceilometer     return fut.result()
2018-05-07 18:52:40.968 11 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result
2018-05-07 18:52:40.968 11 ERROR ceilometer     return self.__get_result()
2018-05-07 18:52:40.968 11 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 298, in call
2018-05-07 18:52:40.968 11 ERROR ceilometer     result = fn(*args, **kwargs)
2018-05-07 18:52:40.968 11 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/ceilometer/gnocchi_client.py", line 210, in upgrade_resource_types
2018-05-07 18:52:40.968 11 ERROR ceilometer     gnocchi_version = version.LooseVersion(gnocchi.build.get())
2018-05-07 18:52:40.968 11 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/gnocchiclient/v1/build.py", line 19, in get
2018-05-07 18:52:40.968 11 ERROR ceilometer     return self._get("").json().get("build", "unknown")
2018-05-07 18:52:40.968 11 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/gnocchiclient/v1/base.py", line 37, in _get
2018-05-07 18:52:40.968 11 ERROR ceilometer     return self.client.api.get(*args, **kwargs)
2018-05-07 18:52:40.968 11 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 304, in get
2018-05-07 18:52:40.968 11 ERROR ceilometer     return self.request(url, 'GET', **kwargs)
2018-05-07 18:52:40.968 11 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/gnocchiclient/client.py", line 52, in request
2018-05-07 18:52:40.968 11 ERROR ceilometer     raise exceptions.from_response(resp, method)
2018-05-07 18:52:40.968 11 ERROR ceilometer ClientException: <html><body><h1>503 Service Unavailable</h1>
2018-05-07 18:52:40.968 11 ERROR ceilometer No server is available to handle this request.
2018-05-07 18:52:40.968 11 ERROR ceilometer </body></html>
2018-05-07 18:52:40.968 11 ERROR ceilometer  (HTTP 503)
2018-05-07 18:52:40.968 11 ERROR ceilometer 



3. /var/log/containers/gnocchi/gnocchi-metricd.log is full of:

ConnectionError: HTTPConnectionPool(host='172.17.3.19', port=8080): Max retries exceeded with url: /v1/AUTH_fefae8d58008484bafa18c43a8e073a5/gnocchi.26fb92f7-17ef-44a6-a296-a2d8c03456c8/none_v3 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fca527aba50>: Failed to establish a new connection: [Errno 113] No route to host',))
2018-05-08 06:09:14,182 [26] ERROR    gnocchi.storage: Error processing new measures
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 505, in process_new_measures
    self._compute_and_store_timeseries(metric, measures)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 532, in _compute_and_store_timeseries
    metric, block_size=block_size, back_window=back_window)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 154, in _get_unaggregated_timeserie_and_unserialize
    metric)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/swift.py", line 193, in _get_unaggregated_timeserie
    self._build_unaggregated_timeserie_path(version))
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1799, in get_object
    headers=headers)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1691, in _retry
    service_token=self.service_token, **kwargs)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1167, in get_object
    conn.request(method, path, '', headers)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 439, in request
    files=files, **self.requests_args)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 422, in _request
    return self.request_session.request(*arg, **kwarg)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 518, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 639, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 502, in send
    raise ConnectionError(e, request=request)

Checked the port list:
(undercloud) [stack@undercloud-0 ~]$ openstack port list
+--------------------------------------+-------------------------------+-------------------+------------------------------------------------------------------------------+--------+
| ID                                   | Name                          | MAC Address       | Fixed IP Addresses                                                           | Status |
+--------------------------------------+-------------------------------+-------------------+------------------------------------------------------------------------------+--------+
| 005180a5-e5f1-448b-981d-4915e174ff86 | storage_management_virtual_ip | fa:16:3e:7e:a7:c3 | ip_address='172.17.4.16', subnet_id='e9f9baa1-e52c-4f0d-bc9b-fd425c6d0aee'   | DOWN   |
| 025a7340-9297-4b30-86e0-02f3142e104c |                               | fa:16:3e:7b:69:5c | ip_address='172.17.3.15', subnet_id='8c344347-9e3f-413c-a8f7-501d358059fe'   | DOWN   |
| 03d39f0e-e4c6-467a-9dce-d6e4eb0a6b50 | public_virtual_ip             | fa:16:3e:52:3f:11 | ip_address='10.0.0.108', subnet_id='55b3d577-a8ce-41db-bd9a-829df581fa7b'    | DOWN   |
| 08d27275-f9bd-413e-a23b-a1a10038ad65 |                               | fa:16:3e:aa:d4:90 | ip_address='10.0.0.105', subnet_id='55b3d577-a8ce-41db-bd9a-829df581fa7b'    | DOWN   |
| 102f7a35-e396-4939-8945-4eb11898a166 |                               | fa:16:3e:04:35:1e | ip_address='172.17.2.12', subnet_id='151f1c16-aa6a-4638-91b5-fc026a45e418'   | DOWN   |
| 107c329e-a557-4433-be53-557dfc649eca |                               | fa:16:3e:b8:64:af | ip_address='172.17.1.17', subnet_id='0ae4a483-32d9-482a-8f90-c59437c75b53'   | DOWN   |
| 18ec9b75-5a5e-47da-ad57-aeebc2cbc7db |                               | fa:16:3e:b0:96:7b | ip_address='172.17.1.10', subnet_id='0ae4a483-32d9-482a-8f90-c59437c75b53'   | DOWN   |
| 1b027db3-d97d-4284-bf29-bc642cd43ad8 | internal_api_virtual_ip       | fa:16:3e:c9:3a:e8 | ip_address='172.17.1.18', subnet_id='0ae4a483-32d9-482a-8f90-c59437c75b53'   | DOWN   |
| 1bb43183-32ac-4877-aa8b-843ae7cbefe2 |                               | fa:16:3e:c0:a1:29 | ip_address='172.17.2.17', subnet_id='151f1c16-aa6a-4638-91b5-fc026a45e418'   | DOWN   |
| 1d17a6a3-c5d2-4c43-887f-329e54588b2f |                               | fa:16:3e:c6:8f:ae | ip_address='172.17.2.20', subnet_id='151f1c16-aa6a-4638-91b5-fc026a45e418'   | DOWN   |
| 24c6a659-bef1-4a10-bd5a-d2a669a3172b |                               | fa:16:3e:18:bf:83 | ip_address='172.17.3.10', subnet_id='8c344347-9e3f-413c-a8f7-501d358059fe'   | DOWN   |
| 256543ba-3c42-42c3-bafe-b7fd9849763b |                               | fa:16:3e:84:02:2b | ip_address='172.17.1.24', subnet_id='0ae4a483-32d9-482a-8f90-c59437c75b53'   | DOWN   |
| 301d5cc1-2dbd-4755-9316-9cf3605b3326 | Controller-port-0             | 52:54:00:be:fc:69 | ip_address='192.168.24.12', subnet_id='744a3905-1c5d-46a5-8581-fcb9b5f49468' | ACTIVE |
| 3219345a-fffc-4442-826c-0f2f604a27e2 |                               | fa:16:3e:f3:6e:b0 | ip_address='172.17.4.12', subnet_id='e9f9baa1-e52c-4f0d-bc9b-fd425c6d0aee'   | DOWN   |
| 3ed65534-0abf-41bf-8a2d-d1f0b8787adc | Controller-port-0             | 52:54:00:a4:62:51 | ip_address='192.168.24.16', subnet_id='744a3905-1c5d-46a5-8581-fcb9b5f49468' | ACTIVE |
| 41282afb-48f8-4cf4-ad64-a00868e3116d | Compute-port-0                | 52:54:00:f0:c9:6d | ip_address='192.168.24.9', subnet_id='744a3905-1c5d-46a5-8581-fcb9b5f49468'  | ACTIVE |
| 46246bc9-f72b-497b-a053-33652646bad8 |                               | fa:16:3e:24:a9:9c | ip_address='172.17.3.21', subnet_id='8c344347-9e3f-413c-a8f7-501d358059fe'   | DOWN   |
| 4b453fc4-9a12-49d1-9806-46da9a464e64 |                               | fa:16:3e:05:6d:00 | ip_address='10.0.0.102', subnet_id='55b3d577-a8ce-41db-bd9a-829df581fa7b'    | DOWN   |
| 4b9a10a2-8b01-4654-aa77-0fd0fa59929b |                               | fa:16:3e:2a:b1:11 | ip_address='192.168.24.5', subnet_id='744a3905-1c5d-46a5-8581-fcb9b5f49468'  | ACTIVE |
| 68d81add-50f4-4251-9a70-c52de436fba3 | control_virtual_ip            | fa:16:3e:36:2d:78 | ip_address='192.168.24.11', subnet_id='744a3905-1c5d-46a5-8581-fcb9b5f49468' | DOWN   |
| 6cd4775e-4edb-4872-a3b1-36cb47944115 |                               | fa:16:3e:29:e1:dd | ip_address='10.0.0.111', subnet_id='55b3d577-a8ce-41db-bd9a-829df581fa7b'    | DOWN   |
| 8540844e-2142-4b8b-be4e-523835c89936 | redis_virtual_ip              | fa:16:3e:59:3a:34 | ip_address='172.17.1.12', subnet_id='0ae4a483-32d9-482a-8f90-c59437c75b53'   | DOWN   |
| 88739c93-4d6c-4e95-af6d-0f2d0220ef28 |                               | fa:16:3e:54:8d:66 | ip_address='172.17.3.12', subnet_id='8c344347-9e3f-413c-a8f7-501d358059fe'   | DOWN   |
| ad0438bd-0b56-4c3c-85ae-c7484249c806 |                               | fa:16:3e:fc:b4:50 | ip_address='172.17.4.17', subnet_id='e9f9baa1-e52c-4f0d-bc9b-fd425c6d0aee'   | DOWN   |
| b00f83e3-f0cb-4a88-b1cf-adfcefa10d76 | storage_virtual_ip            | fa:16:3e:af:a6:7a | ip_address='172.17.3.19', subnet_id='8c344347-9e3f-413c-a8f7-501d358059fe'   | DOWN   |
| b7037ede-b7a8-4428-835c-01589752ccae |                               | fa:16:3e:6b:62:bb | ip_address='172.17.3.18', subnet_id='8c344347-9e3f-413c-a8f7-501d358059fe'   | DOWN   |
| bdc772aa-8db9-4a69-829a-6b694b88b116 |                               | fa:16:3e:3b:56:d4 | ip_address='172.17.1.15', subnet_id='0ae4a483-32d9-482a-8f90-c59437c75b53'   | DOWN   |
| becb92ed-42e9-4865-8d04-375dfad4db69 |                               | fa:16:3e:79:06:0e | ip_address='172.17.2.16', subnet_id='151f1c16-aa6a-4638-91b5-fc026a45e418'   | DOWN   |
| c6f8fb13-df1f-47f5-af48-feebf0d1f0ad | Telemetry-port-0              | 52:54:00:c8:4a:ec | ip_address='192.168.24.8', subnet_id='744a3905-1c5d-46a5-8581-fcb9b5f49468'  | ACTIVE |
| cf755bc4-75c9-4952-8617-d21b89eebe50 | Compute-port-0                | 52:54:00:c6:2a:6d | ip_address='192.168.24.6', subnet_id='744a3905-1c5d-46a5-8581-fcb9b5f49468'  | ACTIVE |
| d69938a1-21e0-4a01-9cfb-a6d9f6d838e7 |                               | fa:16:3e:9f:a9:eb | ip_address='172.17.1.21', subnet_id='0ae4a483-32d9-482a-8f90-c59437c75b53'   | DOWN   |
| e01a0dc6-cbbc-491e-bc5f-7f5cc7ed2680 |                               | fa:16:3e:de:81:5a | ip_address='172.17.4.15', subnet_id='e9f9baa1-e52c-4f0d-bc9b-fd425c6d0aee'   | DOWN   |
| e5ad48a7-d872-4ce7-b653-412d43c35418 | Controller-port-0             | 52:54:00:e9:f1:be | ip_address='192.168.24.18', subnet_id='744a3905-1c5d-46a5-8581-fcb9b5f49468' | ACTIVE |
| e8365b0e-3527-4468-999b-bc0eb9108dce |                               | fa:16:3e:9e:7d:2c | ip_address='172.17.1.19', subnet_id='0ae4a483-32d9-482a-8f90-c59437c75b53'   | DOWN   |
| ee30c50d-7429-4fd5-a955-ae6a8cb78ef7 |                               | fa:16:3e:f4:36:3a | ip_address='172.17.2.11', subnet_id='151f1c16-aa6a-4638-91b5-fc026a45e418'   | DOWN   |
+--------------------------------------+-------------------------------+-------------------+------------------------------------------------------------------------------+--------+

Comment 6 Sasha Smolyak 2018-11-18 08:23:21 UTC
Sure, let's reopen

Comment 7 Mehdi ABAAKOUK 2018-11-29 15:55:26 UTC
Can you open another one, because at first glance, Nova placement is unhealthy and this is unrelated to telemetry.


Note You need to log in before you can comment on or make changes to this bug.