Description of problem: 502 proxy errors appear randomly on the appliance Version-Release number of selected component (if applicable): 5.9.3 How reproducible: randomly Steps to Reproduce: 1. configure the appliance to have multiple webui threads 2. use as normal 3. Actual results: after some time while displaying the following popup shows up: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ URL https://testos.billing.ru/api/service_catalogs/4/service_templates/1 Status 502 Proxy Error Content-Type text/html; charset=iso-8859-1 Data <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>502 Proxy Error</title> </head><body> <h1>Proxy Error</h1> <p>The proxy server received an invalid response from an upstream server.<br /> The proxy server could not handle the request <em><a href="/api/service_catalogs/4/service_templates/1">POST /api/service_catalogs/4/service_templates/1</a></em>.<p> Reason: <strong>Error reading from remote server</strong></p></p> </body></html> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Expected results: system continues to work as normal Additional info: might be related to bz#1595832
Felix, this sounds very similar to bug #1595832 but we need to look at logs to know for sure. Please provide the normal set of logs. Some troubleshooting information is here: http://talk.manageiq.org/t/502-proxy-error-after-migrating-from-fine-to-gaprindashvili/3238/18 We need to find out if there are any Started lines in the production logs that don't have Completed, and if so, figure out which PID and threadids never complete the requests. From there, grep the PID and TID in the production.log to find what requests aren't finishing. If it's an api/webservice worker handling /api/* and it's one of the first requests, you might have hit the deadlock problem resolved by #1595832
To summarize: This DEFINITELY looks like bug #1595832. 2 /api requests within 1 second of each other, neither finishing. No other obvious requests occur before this time. Please try 5.9.4.0 [----] I, [2018-07-13T16:22:20.542402 #24250:3cbcf20] INFO -- : Started GET "/api/notifications?expand=resources&attributes=details&sort_by=id&sort_order=desc&limit=100" for 127.0.0.1 at 2018-07-13 16:22:20 +0300 [----] I, [2018-07-13T16:22:21.486351 #24250:3cbd100] INFO -- : Started GET "/api/auth?requester_type=ws" for 127.0.0.1 at 2018-07-13 16:22:21 +0300 Do you mind if we close as duplicate of bug #1595832?
(In reply to Joe Rafaniello from comment #8) > To summarize: > > This DEFINITELY looks like bug #1595832. 2 /api requests within 1 second of > each other, neither finishing. No other obvious requests occur before this > time. > > Please try 5.9.4.0 > > [----] I, [2018-07-13T16:22:20.542402 #24250:3cbcf20] INFO -- : Started GET > "/api/ > notifications?expand=resources&attributes=details&sort_by=id&sort_order=desc& > limit=100" for 127.0.0.1 at 2018-07-13 16:22:20 +0300 > > [----] I, [2018-07-13T16:22:21.486351 #24250:3cbd100] INFO -- : Started GET > "/api/auth?requester_type=ws" for 127.0.0.1 at 2018-07-13 16:22:21 +0300 > > Do you mind if we close as duplicate of bug #1595832? go ahead! is there a version of 5.9.4 I can share with the customer, how long before the errata, approximatively?
Felix, the bug has been verified so it's only a matter of time before we ship 5.9.4.0. I asked around about a date and will update when I have it. You can also ask for a hotfix for the 5.9 clone of bug #1595832, here in bug #1600191
Felix, the current GA date for 5.9.4 is 08/23/2018, although this could change.
*** This bug has been marked as a duplicate of bug 1595832 ***