Bug 1904092
| Summary: | "gnocchi status" command returns HTTP 500 error in RHOSP 16.1 deployments | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alex Stupnikov <astupnik> | ||||||
| Component: | openstack-gnocchi | Assignee: | Martin Magr <mmagr> | ||||||
| Status: | CLOSED DUPLICATE | QA Contact: | Leonid Natapov <lnatapov> | ||||||
| Severity: | medium | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 16.1 (Train) | CC: | apevec, jschluet, lhh, michal.vasko, pkilambi | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | All | ||||||||
| OS: | All | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2020-12-09 16:14:25 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
Created attachment 1736102 [details]
redis_trace.log
Created attachment 1736103 [details]
typeerror_trace.log
I have attached two error traces from /var/log/containers/httpd/gnocchi-api/gnocchi_wsgi_error.log (all data is collected in my lab and doesn't contain anything private): 1. redis_trace.log: Dec 3 14:05:55 controller-0 haproxy[15]: 10.0.0.24:47704 [03/Dec/2020:14:05:54.704] gnocchi gnocchi/controller-0.internalapi.redhat.local 0/0/0/473/473 500 707 - - ---- 129/1/0/0/0 0/0 "GET /v1/status?details=False HTTP/1.1" 2. typeerror_trace.log: Dec 3 12:46:27 controller-0 haproxy[15]: 10.0.0.24:59222 [03/Dec/2020:12:46:27.357] gnocchi gnocchi/controller-0.internalapi.redhat.local 0/0/0/449/449 500 707 - - ---- 91/1/0/0/0 0/0 "GET /v1/status?details=False HTTP/1.1" |
Description of problem: Customer reported the following problem for RHOSP 16.1 deployment where Gnocchi is configured to use RBD to store metrics. I also was able to reproduce it in my lab. Every "gnocchi status" command fails with HTTP 500 error [1]. It is interesting that this problem actually could have two different causes: 1. "TypeError: keys must be a string", which occurs because json encoder is not able to encode some variable. 2. "redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)", which likely occurs because Redis connection from cache(?) is timed out. Second cause is reproduced when "gnocchi status" command is issued after some period of time. It looks like gnocchi cached active connections to Redis, but fails to resume it. I can see two possible problems here: a. Gnocchi and redis configurations are likely not aligned in RHOSP b. Gnocchi fails to process request instead of trying to re-initiate cached connection First problem is reproduced in all other cases. I will use file attachements to provide Python traces from /var/log/containers/httpd/gnocchi-api/gnocchi_wsgi_error.log [1] $ gnocchi status <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> (HTTP 500)