Bug 1653633

Summary: [Next_gen_installer] user cannot access the routes
Product: OpenShift Container Platform Reporter: Hongan Li <hongli>
Component: NetworkingAssignee: Dan Mace <dmace>
Networking sub component: router QA Contact: Hongan Li <hongli>
Status: CLOSED WORKSFORME Docs Contact:
Severity: high    
Priority: high CC: aos-bugs, wsun, xxia
Version: 4.1.0Keywords: TestBlocker
Target Milestone: ---   
Target Release: 4.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-12-07 06:04:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Hongan Li 2018-11-27 10:24:46 UTC 
Description of problem:
The user cannot access the routes

Version-Release number of selected component (if applicable):
# bin/openshift-install version
bin/openshift-install v0.4.0-10-ge15d801ad69481da18d409bec5fa1c7bd7998f3a
Terraform v0.11.8

$ oc version
oc v4.0.0-0.63.0
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://qe-test-api.devcluster.openshift.com:6443
kubernetes v1.11.0+d4cacc0

How reproducible:
always

Steps to Reproduce:
1. Launch the cluster via aws provider.
2. create your project, pod, svc and route.
oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/caddy-docker.json
oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/edge/service_unsecure.json
oc expose svc service-unsecure

3. check the route
$ oc get route -n hongli
NAME               HOST/PORT                                                  PATH      SERVICES           PORT      TERMINATION   WILDCARD
service-unsecure   service-unsecure-hongli.router.default.svc.cluster.local             service-unsecure   http                    None


Actual results:
The user cannot access the route.

Expected results:
The route should be accessed from both outside cluster and inside cluster.

Additional info:
1. the route sub-domain cannot be configured, and no DNS for the domain.
2. checked the default router service and found it use LB and node port, but the node port didn't respond to any request.
3. resolve the lb to IP then use lb's IP as resolved IP when curling and get the response, seems it is a workaround but not suitable for regression testing.  

$ oc get svc -n openshift-ingress
NAME             TYPE           CLUSTER-IP    EXTERNAL-IP                                                              PORT(S)                      AGE
router-default   LoadBalancer   10.3.24.243   ac9e63418f1e811e88beb0ae40ba6a02-984794043.us-east-1.elb.amazonaws.com   80:30944/TCP,443:32470/TCP   8h

$ nslookup ac9e63418f1e811e88beb0ae40ba6a02-984794043.us-east-1.elb.amazonaws.com
Server:		127.0.0.1
Address:	127.0.0.1#53

Non-authoritative answer:
Name:	ac9e63418f1e811e88beb0ae40ba6a02-984794043.us-east-1.elb.amazonaws.com
Address: 50.17.1.72
Name:	ac9e63418f1e811e88beb0ae40ba6a02-984794043.us-east-1.elb.amazonaws.com
Address: 34.232.235.247
Name:	ac9e63418f1e811e88beb0ae40ba6a02-984794043.us-east-1.elb.amazonaws.com
Address: 107.21.31.144

$ curl --resolve service-unsecure-hongli.router.default.svc.cluster.local:80:50.17.1.72 http://service-unsecure-hongli.router.default.svc.cluster.local
Hello-OpenShift-1 http-8080
Comment 1 Hongan Li 2018-12-07 05:57:26 UTC 
routing subdomain issue was fixed by PR: https://github.com/openshift/cluster-openshift-apiserver-operator/pull/63
Comment 2 Hongan Li 2018-12-07 06:04:18 UTC 
Checked with version v0.5.0-master-14-g8c504c011e5ce8c28c9fb383e5861e1c70353c82 on AWS, the routes have resolvable subdomain and are reachable now.

$ oc get route -n hongli
NAME               HOST/PORT                                                               PATH      SERVICES           PORT      TERMINATION   WILDCARD
service-unsecure   service-unsecure-hongli.apps.hongli.origin-ci-int-aws.dev.rhcloud.com             service-unsecure   http                    None

$ oc get ingresses.config.openshift.io/cluster -o yaml
apiVersion: config.openshift.io/v1
kind: Ingress
metadata:
  creationTimestamp: 2018-12-06T01:16:26Z
  generation: 1
  name: cluster
  resourceVersion: "236"
  selfLink: /apis/config.openshift.io/v1/ingresses/cluster
  uid: 8da1b8d4-f8f4-11e8-a6ff-06b256c6bf30
spec:
  domain: apps.hongli.origin-ci-int-aws.dev.rhcloud.com
status: {}