Bug 1435403

Summary: [dev-preview] Routes aren't sending back any data
Product: OpenShift Online Reporter: Will Gordon <wgordon>
Component: RoutingAssignee: Ben Bennett <bbennett>
Status: CLOSED CURRENTRELEASE QA Contact: zhaozhanqi <zzhao>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.xCC: aos-bugs, eparis, gburges, imraxy, jfiala, kingdon.b
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-24 18:01:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Will Gordon 2017-03-23 17:52:50 UTC
Description of problem:
Routes in Dev Preview are not sending any data when browsed. Chrome is displaying ERR_EMPTY_RESPONSE

Version-Release number of selected component (if applicable):
OpenShift Master: v3.4.1.8 (online version 3.4.1.14)
Kubernetes Master: v1.4.0+776c994

How reproducible:
always

Steps to Reproduce:
1. Deploy a new application (oc new-app openshift/php~https://github.com/wgordon17/openshift.org#deploy)
2. Wait for pod to deploy
3. Browse route (https://openshift-org-wgordon-test.44fs.preview.openshiftapps.com)

Actual results:
ERR_EMPTY_RESPONSE from browser

Expected results:
Page content to show

Additional info:

Comment 1 Eric Paris 2017-03-23 17:57:44 UTC
Can I get

master logs
controller logs
node logs from the node running a pod which is supposed to be backing this route
oc get pod -o yaml   for a pod backing this route
oc get events --namespace=$user_namespace for a pod backing this route
oc get route -o yaml for the route
the router logs

can you curl the pod from the node it is running on?
can you curl it from inside the router pod?

what else....

Comment 2 Will Gordon 2017-03-23 18:04:20 UTC
oc get pod -o yaml openshift-org-1-vftw4
http://pastebin.test.redhat.com/467931

oc get events
http://pastebin.test.redhat.com/467933

oc get route -o yaml openshift-org
http://pastebin.test.redhat.com/467934

curl from inside the pod
sh-4.2$ curl https://openshift-org-wgordon-test.44fs.preview.openshiftapps.com                                                                                                      
curl: (35) Encountered end of file

Comment 3 Eric Paris 2017-03-23 18:20:07 UTC
wgordon says that `curl -v 10.1.25.45:8080` from inside another pod in the same namespace works correctly.

Comment 4 Eric Paris 2017-03-23 18:20:44 UTC
waiting on gburges to run the same curl test from inside the router pod.
also waiting on router logs.

Comment 5 Will Gordon 2017-03-23 18:22:05 UTC
curl by IP address, from another pod within the same project successfully works
http://pastebin.test.redhat.com/467948 (HTML purposefully excluded from pastebin)

curl within the pod to localhost also successfully works
http://pastebin.test.redhat.com/467952 (HTML purposefully excluded from pastebin)

Comment 6 Eric Paris 2017-03-23 18:24:10 UTC
we have proven the app is working.  wgordan hit it inside his NS
gburges hit is from inside the router pod

# oc exec router-36-47gl2 curl 10.1.25.45:8080
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0<!DOCTYPE html>
<html lang="en">
<head>
[snip]
  </body>
</html>

Comment 8 Will Gordon 2017-03-23 19:04:09 UTC
Pod was re-deployed; latest oc get pod (http://pastebin.test.redhat.com/467973)

Comment 9 Will Gordon 2017-03-23 19:05:48 UTC
oc get route: http://pastebin.test.redhat.com/467975
oc get service: http://pastebin.test.redhat.com/467977

Comment 10 Will Gordon 2017-03-23 19:49:21 UTC
New Deployment following router kill

oc get route: http://pastebin.test.redhat.com/467993
oc get pod: http://pastebin.test.redhat.com/467995
oc get ep: http://pastebin.test.redhat.com/467996

Comment 11 Rakesh Sharam 2017-03-24 10:45:45 UTC
Facing the same issue, the AWS load balancer CNAME also not responding 

http://nodejs-mongo-persistent-demo-node-js.44fs.preview.openshiftapps.com/
 

preview-infra-1165939487.us-east-1.elb.amazonaws.com


tried removing the whole project and recreating but no luck :(

Comment 12 Kingdon Barrett 2017-03-24 13:01:21 UTC
I've gone through the same debugging steps, and I'm having the same issue also.

My app is a very basic Ruby on Rails app, I can rsh into the pod and curl localhost:8080, and I get the expected response, but it is not accessible at the URL exposed by route:

http://coolers-env-chmbr-20-10-c.44fs.preview.openshiftapps.com/

I am also on the developer preview.  I am a new user, coming from Kubernetes and GKE.

Comment 13 Ben Bennett 2017-03-24 18:01:53 UTC
This was caused by the change for https://bugzilla.redhat.com/show_bug.cgi?id=1430105

The router was expecting the haproxy "proxy protocol" and ELB was not speaking it properly.

The fix was to run:
  oc env dc/router ROUTER_USE_PROXY_PROTOCOL=false

Comment 14 Kingdon Barrett 2017-03-24 18:14:33 UTC
Fixed for me, thank you!

Comment 15 Will Gordon 2017-03-24 18:58:32 UTC
*** Bug 1435407 has been marked as a duplicate of this bug. ***