Bug 1609849

Summary:	502 Proxy Error
Product:	Red Hat CloudForms Management Engine	Reporter:	Felix Dewaleyne <fdewaley>
Component:	Appliance	Assignee:	Joe Rafaniello <jrafanie>
Status:	CLOSED DUPLICATE	QA Contact:	Dave Johnson <dajohnso>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	5.9.0	CC:	abellott, dmetzger, fdewaley, jrafanie, obarenbo
Target Milestone:	GA
Target Release:	5.9.5
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-08-01 16:32:22 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Felix Dewaleyne 2018-07-30 15:45:16 UTC

Description of problem:
502 proxy errors appear randomly on the appliance 

Version-Release number of selected component (if applicable):
5.9.3

How reproducible:
randomly

Steps to Reproduce:
1. configure the appliance to have multiple webui threads
2. use as normal
3.

Actual results:
after some time while displaying the following popup shows up:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 URL https://testos.billing.ru/api/service_catalogs/4/service_templates/1

Status 502 Proxy Error

Content-Type text/html; charset=iso-8859-1

Data <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>502 Proxy Error</title> </head><body> <h1>Proxy Error</h1> <p>The proxy server received an invalid response from an upstream server.<br /> The proxy server could not handle the request <em><a href="/api/service_catalogs/4/service_templates/1">POST&nbsp;/api/service_catalogs/4/service_templates/1</a></em>.<p> Reason: <strong>Error reading from remote server</strong></p></p> </body></html>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Expected results:
system continues to work as normal 

Additional info:
might be related to bz#1595832

Comment 2 Joe Rafaniello 2018-07-31 17:08:26 UTC

Felix, this sounds very similar to bug #1595832 but we need to look at logs to know for sure.  Please provide the normal set of logs.

Some troubleshooting information is here: http://talk.manageiq.org/t/502-proxy-error-after-migrating-from-fine-to-gaprindashvili/3238/18

We need to find out if there are any Started lines in the production logs that don't have Completed, and if so, figure out which PID and threadids never complete the requests.  From there, grep the PID and TID in the production.log to find what requests aren't finishing.  If it's an api/webservice worker handling /api/* and it's one of the first requests, you might have hit the deadlock problem resolved by #1595832

Comment 8 Joe Rafaniello 2018-08-01 16:20:18 UTC

To summarize:

This DEFINITELY looks like bug #1595832.  2 /api requests within 1 second of each other, neither finishing.  No other obvious requests occur before this time.

Please try 5.9.4.0

[----] I, [2018-07-13T16:22:20.542402 #24250:3cbcf20]  INFO -- : Started GET "/api/notifications?expand=resources&attributes=details&sort_by=id&sort_order=desc&limit=100" for 127.0.0.1 at 2018-07-13 16:22:20 +0300

[----] I, [2018-07-13T16:22:21.486351 #24250:3cbd100]  INFO -- : Started GET "/api/auth?requester_type=ws" for 127.0.0.1 at 2018-07-13 16:22:21 +0300

Do you mind if we close as duplicate of bug #1595832?

Comment 9 Felix Dewaleyne 2018-08-01 16:23:54 UTC

(In reply to Joe Rafaniello from comment #8)
> To summarize:
> 
> This DEFINITELY looks like bug #1595832.  2 /api requests within 1 second of
> each other, neither finishing.  No other obvious requests occur before this
> time.
> 
> Please try 5.9.4.0
> 
> [----] I, [2018-07-13T16:22:20.542402 #24250:3cbcf20]  INFO -- : Started GET
> "/api/
> notifications?expand=resources&attributes=details&sort_by=id&sort_order=desc&
> limit=100" for 127.0.0.1 at 2018-07-13 16:22:20 +0300
> 
> [----] I, [2018-07-13T16:22:21.486351 #24250:3cbd100]  INFO -- : Started GET
> "/api/auth?requester_type=ws" for 127.0.0.1 at 2018-07-13 16:22:21 +0300
> 
> Do you mind if we close as duplicate of bug #1595832?

go ahead! is there a version of 5.9.4 I can share with the customer, how long before the errata, approximatively?

Comment 10 Joe Rafaniello 2018-08-01 16:29:55 UTC

Felix, the bug has been verified so it's only a matter of time before we ship 5.9.4.0.  I asked around about a date and will update when I have it.  You can also ask for a hotfix for the 5.9 clone of bug #1595832, here in bug #1600191

Comment 11 Joe Rafaniello 2018-08-01 16:31:21 UTC

Felix, the current GA date for 5.9.4 is 08/23/2018, although this could change.

Comment 12 Joe Rafaniello 2018-08-01 16:32:22 UTC


*** This bug has been marked as a duplicate of bug 1595832 ***