Bug 1575886
| Summary: | Telemetry standalone node: Get error Error 500 when trying to access measures of gnocchi metrics | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Sasha Smolyak <ssmolyak> |
| Component: | gnocchi | Assignee: | Pradeep Kilambi <pkilambi> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Sasha Smolyak <ssmolyak> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 13.0 (Queens) | CC: | apevec, jjoyce, jschluet, lhh, mrunge, nmanos, ssmolyak |
| Target Milestone: | z3 | Keywords: | TestOnly, Triaged, ZStream |
| Target Release: | 13.0 (Queens) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-08-28 06:34:02 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1458303 | ||
Sure, let's reopen Can you open another one, because at first glance, Nova placement is unhealthy and this is unrelated to telemetry. |
Description of problem: When deploying Telemetry on standalone node, get error Error 500 when trying to access measures of gnocchi metrics. The table of resources is accessible, the resource is shown, but measures aren't reachable Version-Release number of selected component (if applicable): gnocchi-common-4.2.3-2.el7ost.noarch python2-collectd-gnocchi-1.7.1-1.el7ost.noarch gnocchi-metricd-4.2.3-2.el7ost.noarch python2-gnocchiclient-7.0.1-1.el7ost.noarch puppet-gnocchi-12.4.0-0.20180329032858.5dfa350.el7ost.noarch gnocchi-statsd-4.2.3-2.el7ost.noarch gnocchi-api-4.2.3-2.el7ost.noarch python-gnocchi-4.2.3-2.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy Telemetry in standalone node, configuration 3 controllers, 2 compute, 1 telemetry 2. Create instance in overcloud 3. Try to reach cpu measures of the instance Actual results: (overcloud) [stack@undercloud-0 ~]$ openstack metric measures show --resource-id 59b85149-595c-45f3-8621-71a041ee604d cpu <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> (HTTP 500) Expected results: The measures are shown Additional info: Logs on telemetry-0 node: 1. /var/log/containers/ceilometer/central.log 2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client [-] Unknown Error (HTTP 503): ClientException: Unknown Error (HTTP 503) 2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client Traceback (most recent call last): 2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/ceilometer/nova_client.py", line 39, in with_logging 2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client return func(*args, **kwargs) 2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/ceilometer/nova_client.py", line 157, in instance_get_all 2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client search_opts=search_opts) 2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 884, in list 2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client "servers") 2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 254, in _list 2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client resp, body = self.api.client.get(url) 2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 304, in get 2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client return self.request(url, 'GET', **kwargs) 2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request 2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client raise exceptions.from_response(resp, body, url, method) 2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client ClientException: Unknown Error (HTTP 503) 2018-05-07 18:51:08.504 25 ERROR ceilometer.nova_client 2. /var/log/containers/ceilometer/ceilometer-upgrade.log 2018-05-07 18:52:40.968 11 CRITICAL ceilometer [-] Unhandled error: ClientException: <html><body><h1>503 Service Unavailable</h1> No server is available to handle this request. </body></html> (HTTP 503) 2018-05-07 18:52:40.968 11 ERROR ceilometer Traceback (most recent call last): 2018-05-07 18:52:40.968 11 ERROR ceilometer File "/usr/bin/ceilometer-upgrade", line 10, in <module> 2018-05-07 18:52:40.968 11 ERROR ceilometer sys.exit(upgrade()) 2018-05-07 18:52:40.968 11 ERROR ceilometer File "/usr/lib/python2.7/site-packages/ceilometer/cmd/storage.py", line 60, in upgrade 2018-05-07 18:52:40.968 11 ERROR ceilometer )(gnocchi_client.upgrade_resource_types, conf) 2018-05-07 18:52:40.968 11 ERROR ceilometer File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 295, in call 2018-05-07 18:52:40.968 11 ERROR ceilometer start_time=start_time) 2018-05-07 18:52:40.968 11 ERROR ceilometer File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 252, in iter 2018-05-07 18:52:40.968 11 ERROR ceilometer return fut.result() 2018-05-07 18:52:40.968 11 ERROR ceilometer File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result 2018-05-07 18:52:40.968 11 ERROR ceilometer return self.__get_result() 2018-05-07 18:52:40.968 11 ERROR ceilometer File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 298, in call 2018-05-07 18:52:40.968 11 ERROR ceilometer result = fn(*args, **kwargs) 2018-05-07 18:52:40.968 11 ERROR ceilometer File "/usr/lib/python2.7/site-packages/ceilometer/gnocchi_client.py", line 210, in upgrade_resource_types 2018-05-07 18:52:40.968 11 ERROR ceilometer gnocchi_version = version.LooseVersion(gnocchi.build.get()) 2018-05-07 18:52:40.968 11 ERROR ceilometer File "/usr/lib/python2.7/site-packages/gnocchiclient/v1/build.py", line 19, in get 2018-05-07 18:52:40.968 11 ERROR ceilometer return self._get("").json().get("build", "unknown") 2018-05-07 18:52:40.968 11 ERROR ceilometer File "/usr/lib/python2.7/site-packages/gnocchiclient/v1/base.py", line 37, in _get 2018-05-07 18:52:40.968 11 ERROR ceilometer return self.client.api.get(*args, **kwargs) 2018-05-07 18:52:40.968 11 ERROR ceilometer File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 304, in get 2018-05-07 18:52:40.968 11 ERROR ceilometer return self.request(url, 'GET', **kwargs) 2018-05-07 18:52:40.968 11 ERROR ceilometer File "/usr/lib/python2.7/site-packages/gnocchiclient/client.py", line 52, in request 2018-05-07 18:52:40.968 11 ERROR ceilometer raise exceptions.from_response(resp, method) 2018-05-07 18:52:40.968 11 ERROR ceilometer ClientException: <html><body><h1>503 Service Unavailable</h1> 2018-05-07 18:52:40.968 11 ERROR ceilometer No server is available to handle this request. 2018-05-07 18:52:40.968 11 ERROR ceilometer </body></html> 2018-05-07 18:52:40.968 11 ERROR ceilometer (HTTP 503) 2018-05-07 18:52:40.968 11 ERROR ceilometer 3. /var/log/containers/gnocchi/gnocchi-metricd.log is full of: ConnectionError: HTTPConnectionPool(host='172.17.3.19', port=8080): Max retries exceeded with url: /v1/AUTH_fefae8d58008484bafa18c43a8e073a5/gnocchi.26fb92f7-17ef-44a6-a296-a2d8c03456c8/none_v3 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fca527aba50>: Failed to establish a new connection: [Errno 113] No route to host',)) 2018-05-08 06:09:14,182 [26] ERROR gnocchi.storage: Error processing new measures Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 505, in process_new_measures self._compute_and_store_timeseries(metric, measures) File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 532, in _compute_and_store_timeseries metric, block_size=block_size, back_window=back_window) File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 154, in _get_unaggregated_timeserie_and_unserialize metric) File "/usr/lib/python2.7/site-packages/gnocchi/storage/swift.py", line 193, in _get_unaggregated_timeserie self._build_unaggregated_timeserie_path(version)) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1799, in get_object headers=headers) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1691, in _retry service_token=self.service_token, **kwargs) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1167, in get_object conn.request(method, path, '', headers) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 439, in request files=files, **self.requests_args) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 422, in _request return self.request_session.request(*arg, **kwarg) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 518, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 639, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 502, in send raise ConnectionError(e, request=request) Checked the port list: (undercloud) [stack@undercloud-0 ~]$ openstack port list +--------------------------------------+-------------------------------+-------------------+------------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+-------------------------------+-------------------+------------------------------------------------------------------------------+--------+ | 005180a5-e5f1-448b-981d-4915e174ff86 | storage_management_virtual_ip | fa:16:3e:7e:a7:c3 | ip_address='172.17.4.16', subnet_id='e9f9baa1-e52c-4f0d-bc9b-fd425c6d0aee' | DOWN | | 025a7340-9297-4b30-86e0-02f3142e104c | | fa:16:3e:7b:69:5c | ip_address='172.17.3.15', subnet_id='8c344347-9e3f-413c-a8f7-501d358059fe' | DOWN | | 03d39f0e-e4c6-467a-9dce-d6e4eb0a6b50 | public_virtual_ip | fa:16:3e:52:3f:11 | ip_address='10.0.0.108', subnet_id='55b3d577-a8ce-41db-bd9a-829df581fa7b' | DOWN | | 08d27275-f9bd-413e-a23b-a1a10038ad65 | | fa:16:3e:aa:d4:90 | ip_address='10.0.0.105', subnet_id='55b3d577-a8ce-41db-bd9a-829df581fa7b' | DOWN | | 102f7a35-e396-4939-8945-4eb11898a166 | | fa:16:3e:04:35:1e | ip_address='172.17.2.12', subnet_id='151f1c16-aa6a-4638-91b5-fc026a45e418' | DOWN | | 107c329e-a557-4433-be53-557dfc649eca | | fa:16:3e:b8:64:af | ip_address='172.17.1.17', subnet_id='0ae4a483-32d9-482a-8f90-c59437c75b53' | DOWN | | 18ec9b75-5a5e-47da-ad57-aeebc2cbc7db | | fa:16:3e:b0:96:7b | ip_address='172.17.1.10', subnet_id='0ae4a483-32d9-482a-8f90-c59437c75b53' | DOWN | | 1b027db3-d97d-4284-bf29-bc642cd43ad8 | internal_api_virtual_ip | fa:16:3e:c9:3a:e8 | ip_address='172.17.1.18', subnet_id='0ae4a483-32d9-482a-8f90-c59437c75b53' | DOWN | | 1bb43183-32ac-4877-aa8b-843ae7cbefe2 | | fa:16:3e:c0:a1:29 | ip_address='172.17.2.17', subnet_id='151f1c16-aa6a-4638-91b5-fc026a45e418' | DOWN | | 1d17a6a3-c5d2-4c43-887f-329e54588b2f | | fa:16:3e:c6:8f:ae | ip_address='172.17.2.20', subnet_id='151f1c16-aa6a-4638-91b5-fc026a45e418' | DOWN | | 24c6a659-bef1-4a10-bd5a-d2a669a3172b | | fa:16:3e:18:bf:83 | ip_address='172.17.3.10', subnet_id='8c344347-9e3f-413c-a8f7-501d358059fe' | DOWN | | 256543ba-3c42-42c3-bafe-b7fd9849763b | | fa:16:3e:84:02:2b | ip_address='172.17.1.24', subnet_id='0ae4a483-32d9-482a-8f90-c59437c75b53' | DOWN | | 301d5cc1-2dbd-4755-9316-9cf3605b3326 | Controller-port-0 | 52:54:00:be:fc:69 | ip_address='192.168.24.12', subnet_id='744a3905-1c5d-46a5-8581-fcb9b5f49468' | ACTIVE | | 3219345a-fffc-4442-826c-0f2f604a27e2 | | fa:16:3e:f3:6e:b0 | ip_address='172.17.4.12', subnet_id='e9f9baa1-e52c-4f0d-bc9b-fd425c6d0aee' | DOWN | | 3ed65534-0abf-41bf-8a2d-d1f0b8787adc | Controller-port-0 | 52:54:00:a4:62:51 | ip_address='192.168.24.16', subnet_id='744a3905-1c5d-46a5-8581-fcb9b5f49468' | ACTIVE | | 41282afb-48f8-4cf4-ad64-a00868e3116d | Compute-port-0 | 52:54:00:f0:c9:6d | ip_address='192.168.24.9', subnet_id='744a3905-1c5d-46a5-8581-fcb9b5f49468' | ACTIVE | | 46246bc9-f72b-497b-a053-33652646bad8 | | fa:16:3e:24:a9:9c | ip_address='172.17.3.21', subnet_id='8c344347-9e3f-413c-a8f7-501d358059fe' | DOWN | | 4b453fc4-9a12-49d1-9806-46da9a464e64 | | fa:16:3e:05:6d:00 | ip_address='10.0.0.102', subnet_id='55b3d577-a8ce-41db-bd9a-829df581fa7b' | DOWN | | 4b9a10a2-8b01-4654-aa77-0fd0fa59929b | | fa:16:3e:2a:b1:11 | ip_address='192.168.24.5', subnet_id='744a3905-1c5d-46a5-8581-fcb9b5f49468' | ACTIVE | | 68d81add-50f4-4251-9a70-c52de436fba3 | control_virtual_ip | fa:16:3e:36:2d:78 | ip_address='192.168.24.11', subnet_id='744a3905-1c5d-46a5-8581-fcb9b5f49468' | DOWN | | 6cd4775e-4edb-4872-a3b1-36cb47944115 | | fa:16:3e:29:e1:dd | ip_address='10.0.0.111', subnet_id='55b3d577-a8ce-41db-bd9a-829df581fa7b' | DOWN | | 8540844e-2142-4b8b-be4e-523835c89936 | redis_virtual_ip | fa:16:3e:59:3a:34 | ip_address='172.17.1.12', subnet_id='0ae4a483-32d9-482a-8f90-c59437c75b53' | DOWN | | 88739c93-4d6c-4e95-af6d-0f2d0220ef28 | | fa:16:3e:54:8d:66 | ip_address='172.17.3.12', subnet_id='8c344347-9e3f-413c-a8f7-501d358059fe' | DOWN | | ad0438bd-0b56-4c3c-85ae-c7484249c806 | | fa:16:3e:fc:b4:50 | ip_address='172.17.4.17', subnet_id='e9f9baa1-e52c-4f0d-bc9b-fd425c6d0aee' | DOWN | | b00f83e3-f0cb-4a88-b1cf-adfcefa10d76 | storage_virtual_ip | fa:16:3e:af:a6:7a | ip_address='172.17.3.19', subnet_id='8c344347-9e3f-413c-a8f7-501d358059fe' | DOWN | | b7037ede-b7a8-4428-835c-01589752ccae | | fa:16:3e:6b:62:bb | ip_address='172.17.3.18', subnet_id='8c344347-9e3f-413c-a8f7-501d358059fe' | DOWN | | bdc772aa-8db9-4a69-829a-6b694b88b116 | | fa:16:3e:3b:56:d4 | ip_address='172.17.1.15', subnet_id='0ae4a483-32d9-482a-8f90-c59437c75b53' | DOWN | | becb92ed-42e9-4865-8d04-375dfad4db69 | | fa:16:3e:79:06:0e | ip_address='172.17.2.16', subnet_id='151f1c16-aa6a-4638-91b5-fc026a45e418' | DOWN | | c6f8fb13-df1f-47f5-af48-feebf0d1f0ad | Telemetry-port-0 | 52:54:00:c8:4a:ec | ip_address='192.168.24.8', subnet_id='744a3905-1c5d-46a5-8581-fcb9b5f49468' | ACTIVE | | cf755bc4-75c9-4952-8617-d21b89eebe50 | Compute-port-0 | 52:54:00:c6:2a:6d | ip_address='192.168.24.6', subnet_id='744a3905-1c5d-46a5-8581-fcb9b5f49468' | ACTIVE | | d69938a1-21e0-4a01-9cfb-a6d9f6d838e7 | | fa:16:3e:9f:a9:eb | ip_address='172.17.1.21', subnet_id='0ae4a483-32d9-482a-8f90-c59437c75b53' | DOWN | | e01a0dc6-cbbc-491e-bc5f-7f5cc7ed2680 | | fa:16:3e:de:81:5a | ip_address='172.17.4.15', subnet_id='e9f9baa1-e52c-4f0d-bc9b-fd425c6d0aee' | DOWN | | e5ad48a7-d872-4ce7-b653-412d43c35418 | Controller-port-0 | 52:54:00:e9:f1:be | ip_address='192.168.24.18', subnet_id='744a3905-1c5d-46a5-8581-fcb9b5f49468' | ACTIVE | | e8365b0e-3527-4468-999b-bc0eb9108dce | | fa:16:3e:9e:7d:2c | ip_address='172.17.1.19', subnet_id='0ae4a483-32d9-482a-8f90-c59437c75b53' | DOWN | | ee30c50d-7429-4fd5-a955-ae6a8cb78ef7 | | fa:16:3e:f4:36:3a | ip_address='172.17.2.11', subnet_id='151f1c16-aa6a-4638-91b5-fc026a45e418' | DOWN | +--------------------------------------+-------------------------------+-------------------+------------------------------------------------------------------------------+--------+