Bug 1285731 - Missing routes when creating http and https routes to the same host
Missing routes when creating http and https routes to the same host
Status: CLOSED WORKSFORME
Product: OpenShift Container Platform
Classification: Red Hat
Component: Routing (Show other bugs)
3.1.0
x86_64 Linux
unspecified Severity high
: ---
: ---
Assigned To: Ram Ranganathan
zhaozhanqi
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-26 06:00 EST by Dave McCormick
Modified: 2015-12-11 13:45 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-11 13:45:21 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dave McCormick 2015-11-26 06:00:09 EST
Description of problem:

I want to enable routing to an application via both http or https routes: -

http://helloworld.paas.dev.iggroup.local
https://helloworld.paas.dev.iggroup.local

(*.paas.dev.iggroup.local routes to our openshift routers)

Creating a Route with tls section will ONLY create a secure route and so I try to create two routes: -

- apiVersion: v1
  kind: Route
  metadata:
    name: helloworld-route-https
  spec:
    host: helloworld.paas.dev.iggroup.local
    to:
      kind: Service
      name: helloworld-service
    tls:
      termination: edge
- apiVersion: v1
  kind: Route
  metadata:
    name: helloworld-route-http
  spec:
    host: helloworld.paas.dev.iggroup.local
    to:
      kind: Service
      name: helloworld-service

I am allowed to create them: -

NAME                     HOST/PORT                           PATH      SERVICE              LABELS    TLS TERMINATION
helloworld-route-http    helloworld.paas.dev.iggroup.local             helloworld-service
helloworld-route-https   helloworld.paas.dev.iggroup.local             helloworld-service             edge

So it looks as though the two routes have been accepted.
The host on the two routes is exactly the same - 'helloworld.paas.dev.iggroup.local'.  Within the router container only the ha_proxy configuration map 'os_http_be.map' has been updated: -

helloworld.paas.dev.iggroup.local hello-openshift_helloworld-route-http

and only the following backend in ha_proxy has been configured: -

backend be_http_hello-openshift_helloworld-route-http

  mode http
  option redispatch
  option forwardfor
  balance leastconn
  timeout check 5000ms
  http-request set-header X-Forwarded-Host %[req.hdr(host)]
  http-request set-header X-Forwarded-Port %[dst_port]
  http-request set-header X-Forwarded-Proto https if { ssl_fc }

    cookie OPENSHIFT_hello-openshift_helloworld-route-http_SERVERID insert indirect nocache httponly
    http-request set-header X-Forwarded-Proto http

  http-request set-header Forwarded for=%[src],host=%[req.hdr(host)],proto=%[req.hdr(X-Forwarded-Proto)]

  server 10.1.1.7:8080 10.1.1.7:8080 check inter 5000ms cookie 10.1.1.7:8080

  server 10.1.3.6:8080 10.1.3.6:8080 check inter 5000ms cookie 10.1.3.6:8080

Nothing has been added to the map 'os_edge_http_be.map' which should be updated with the helloworld-route-https route.
The be_http_hello-openshift_helloworld-route-http backend is only accessible via http (port 80).

The following backend is missing: -


os_edge_http_be.map
helloworld.paas.dev.iggroup.local hello-openshift_helloworld-route-https

backend be_edge_http_hello-openshift_helloworld-route-https

  mode http
  option redispatch
  option forwardfor
  balance leastconn
  timeout check 5000ms
  http-request set-header X-Forwarded-Host %[req.hdr(host)]
  http-request set-header X-Forwarded-Port %[dst_port]
  http-request set-header X-Forwarded-Proto https if { ssl_fc }

    cookie OPENSHIFT_EDGE_hello-openshift_helloworld-route-https_SERVERID insert indirect nocache httponly secure

  http-request set-header Forwarded for=%[src],host=%[req.hdr(host)],proto=%[req.hdr(X-Forwarded-Proto)]

  server 10.1.1.7:8080 10.1.1.7:8080 check inter 5000ms cookie 10.1.1.7:8080

  server 10.1.3.6:8080 10.1.3.6:8080 check inter 5000ms cookie 10.1.3.6:8080


Version-Release number of selected component (if applicable):
ose-haproxy-router:v3.1.0.4

How reproducible:
Every time.

Expected results:
I would ideally like to configure a single route that allows me to define availability on httpd, https or BOTH.
Alternatively, if I must create two routes then creating an insecure route and a secure route to the same host should result in valid ha_proxy configuration and both working routes.


Additional info:
Comment 1 zhaozhanqi 2015-11-26 20:28:52 EST
you can add the following to your route to fixed your issue.

insecureEdgeTerminationPolicy: Allow

for detail, you can see https://github.com/openshift/origin/pull/5224
or https://github.com/openshift/openshift-docs/blob/master/architecture/core_concepts/routes.adoc#secured-routes

Example 7. A Secured Route Using Edge Termination Allowing HTTP Traffic
apiVersion: v1
kind: Route
metadata:
  name: route-edge-secured-allow-insecure
spec:
  host: www.example.com
  to:
    kind: Service
    name: service-name
  tls:
    termination:                   edge   (1)
    insecureEdgeTerminationPolicy: Allow  (2)
Comment 2 Dave McCormick 2015-12-01 07:50:27 EST
Hi

When I try these settings...

apiVersion: v1
kind: Route
metadata:
  name: helloworld-route
spec:
  host: helloworld.paas.dev.iggroup.local
  to:
    kind: Service
    name: helloworld-service
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Allow

and browse to http://helloworld.paas.dev.iggroup.local I get a 503 error.  This is the only route.

backend be_edge_http_davetest_helloworld-route

  mode http
  option redispatch
  option forwardfor
  balance leastconn
  timeout check 5000ms
  http-request set-header X-Forwarded-Host %[req.hdr(host)]
  http-request set-header X-Forwarded-Port %[dst_port]
  http-request set-header X-Forwarded-Proto https if { ssl_fc }

    cookie OPENSHIFT_EDGE_davetest_helloworld-route_SERVERID insert indirect nocache httponly secure

  http-request set-header Forwarded for=%[src],host=%[req.hdr(host)],proto=%[req.hdr(X-Forwarded-Proto)]

  server 10.1.0.5:8080 10.1.0.5:8080 check inter 5000ms cookie 10.1.0.5:8080

  server 10.1.1.3:8080 10.1.1.3:8080 check inter 5000ms cookie 10.1.1.3:8080

There is nothing in the os_edge_http_expose.map map which looks as though there perhaps should be...

Inside the router:- 
ls -al
total 72
drwxrwxrwx. 2 root    root     4096 Dec  1 07:43 .
drwxrwxrwx. 6 haproxy haproxy    68 Nov 11 20:17 ..
-rwxrwxrwx. 1 root    root     2042 Nov 11 19:59 default_pub_keys.pem
-rwxrwxrwx. 1 root    root      103 Nov 11 20:16 error-page-503.html
-rwxrwxrwx. 1 root    root    16766 Nov 11 20:16 haproxy-config.template
-rwxrwxrwx. 1 root    root     9759 Dec  1 07:42 haproxy.config
-rwxrwxrwx. 1 root    root       73 Dec  1 07:42 os_edge_http_be.map
-rwxrwxrwx. 1 root    root       12 Dec  1 07:42 os_edge_http_expose.map
-rwxrwxrwx. 1 root    root       12 Dec  1 07:42 os_edge_http_redirect.map
-rwxrwxrwx. 1 root    root       12 Dec  1 07:42 os_http_be.map
-rwxrwxrwx. 1 root    root       12 Dec  1 07:42 os_reencrypt.map
-rwxrwxrwx. 1 root    root       12 Dec  1 07:42 os_sni_passthrough.map
-rwxrwxrwx. 1 root    root       12 Dec  1 07:42 os_tcp_be.map

Only os_edge_http_be.map has any content..
helloworld.paas.dev.iggroup.local davetest_helloworld-route

This suggests that we should also have something in the map os_edge_http_expose.map so that a back-end is made available...

frontend public
  bind :80
  mode http
  tcp-request inspect-delay 5s
  tcp-request content accept if HTTP

  # check if we need to redirect/force using https.
  acl secure_redirect base,map_beg(/var/lib/haproxy/conf/os_edge_http_redirect.map) -m found
  redirect scheme https if secure_redirect

  # Check if it is an edge route exposed insecurely.
  acl edge_http_expose base,map_beg(/var/lib/haproxy/conf/os_edge_http_expose.map) -m found
  use_backend be_edge_http_%[base,map_beg(/var/lib/haproxy/conf/os_edge_http_expose.map)] if edge_http_expose

  # map to http backend
  # Search from most specific to general path (host case).
  # Note: If no match, haproxy uses the default_backend, no other
  #       use_backend directives below this will be processed.
  use_backend be_http_%[base,map_beg(/var/lib/haproxy/conf/os_http_be.map)]

  default_backend openshift_default

regards



Dave
Comment 5 Ram Ranganathan 2015-12-02 18:07:44 EST
@dave.mccormick what version of OpenShift Origin are you running?  From the latest code, this works for me. There should be an allowed insecure route entry in os_edge_http_expose.map - looks like your file is empty.

Here's an example repo that has the different types of routes.
   https://github.com/ramr/nodejs-header-echo

Instructions are in the README for how to create the different types of routes. 
The allow http example uses this json for defining the routes:
https://github.com/ramr/nodejs-header-echo/blob/master/openshift/edge-secured-allow-http-route.json

And this test to ensure it works:
$  #  Check edge terminated route with insecure allowed (http + https).
$  curl --resolve allow-http.header.test:80:127.0.0.1 -vvv  \
        -H "test: allow-http" -k http://allow-http.header.test
$  curl --resolve allow-http.header.test:443:127.0.0.1 -vvv  \
        -H "test: allow-https" -k https://allow-http.header.test

Note: You will either need to run that on the node the router is running on. Or if you want to run it from outside, replace 127.0.0.1 with the IP address of the router.

Give that a try and let me know if that works for you. If not, we can debug your environment and see what's amiss. Thanks.
Comment 6 Ram Ranganathan 2015-12-09 16:11:17 EST
@dave.mccormick - any updates? Thx
Comment 7 Dave McCormick 2015-12-10 03:28:43 EST
Hi Ram

Sorry for the slow reply.  Thanks for the link to your repository, your example edge-secured-allow-http-route.json looks very similar but not quite the same as mine: -

apiVersion: v1
kind: Route
metadata:
  name: helloworld-route
spec:
  host: helloworld.paas.dev.iggroup.local
  to:
    kind: Service
    name: helloworld-service
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Allow

The difference is that I am wanting to allow the HTTPS route to fall back and use the default wildcard certificate that I have defined on my router (which I'm told I should be able to do and works for a simple https route).  Can you test with the same set up as me above?

I'm using openshift enterprise v3.1.0.4 (router is also labled ose-haproxy-router:v3.1.0.4).  Are you using a newer release?  Should I be using a newer one?

You are correct - the issue does seem to be the missing entry in the os_edge_http_expose.map.  That is my bug/issue.  I don't know why when I create the route as above it don't get both routes, I think that my syntax looks good and so it feels like a bug the fact that the map file is not being updated.

regards


Dave
Comment 8 Ram Ranganathan 2015-12-10 22:05:24 EST
Hi Dave,
   Hmm, that is weird. v3.1.0.4 should work - it works for me with this json file: 

$ cat  _rrtests/nodejs-header-echo/dave.json
{
  "kind": "Route",
  "apiVersion": "v1",
  "metadata": {
    "name": "header-test-http-ig-allow"
  },
  "id": "header-test-http-ig-allow",
  "spec": {
    "host": "ig-allow-http.header.test",
    "to": {
      "kind": "Service",
      "name": "header-test-insecure"
    },
    "tls": {
      "termination": "edge",
      "insecureEdgeTerminationPolicy": "Allow"
    }
  }
}

$ oc create -f _rrtests/nodejs-header-echo/dave.json

$ oc get routes
NAME                        HOST/PORT                   PATH      SERVICE                LABELS    INSECURE POLICY   TLS TERMINATION
header-test-http-ig-allow   ig-allow-http.header.test             header-test-insecure             Allow             edge


# grepped for brevity ...
$  curl --resolve ig-allow-http.header.test:443:127.0.0.1 -s -H "test: ig-allow-http" -k https://ig-allow-http.header.test | grep 'test:'
  test: ig-allow-http
$  curl --resolve ig-allow-http.header.test:80:127.0.0.1 -s -H "test: ig-allow-http" -k http://ig-allow-http.header.test | grep 'test:'
  test: ig-allow-http
Comment 10 Ram Ranganathan 2015-12-10 22:11:06 EST
Hi Dave,
   Could you please attach the haproxy-config file and the output of docker inspect. 

docker inspect $(docker ps | grep haproxy-router | awk '{print $1}')
docker exec $(docker ps | grep haproxy-router | awk '{print $1}') cat /var/lib/haproxy/conf/haproxy.config


   Thanks in advance,

Ram//
Comment 11 zhaozhanqi 2015-12-11 00:19:12 EST
hi, ram & Dave


this also works for me on my ose env with the #c8 dava.json

# oc version
oc v3.1.0.4-16-g112fcc4
kubernetes v1.1.0-origin-1107-g4c8e6f4

********router images:********
openshift3/ose-haproxy-router          v3.1.0.4            81c50204da05        4 weeks ago         410.2 MB

**************in router container:**********
# grep -i "zzhao" *
haproxy.config:backend be_edge_http_zzhao_header-test-http-ig-allow
haproxy.config:    cookie OPENSHIFT_EDGE_zzhao_header-test-http-ig-allow_SERVERID insert indirect nocache httponly secure
os_edge_http_be.map:ig-allow-http.header.test zzhao_header-test-http-ig-allow
os_edge_http_expose.map:ig-allow-http.header.test zzhao_header-test-http-ig-allow

*******************test the route*******************************
[root@dhcp-136-93 nodejs-header-echo]# curl --resolve ig-allow-http.header.test:443:10.14.6.137 https://ig-allow-http.header.test -k -s -H "test: ig-allow-http" |grep 'test:'
  test: ig-allow-http
[root@dhcp-136-93 nodejs-header-echo]# curl --resolve ig-allow-http.header.test:80:10.14.6.137 http://ig-allow-http.header.test -k -s -H "test: ig-allow-http" |grep 'test:'
  test: ig-allow-http
Comment 12 Dave McCormick 2015-12-11 03:44:29 EST
Hi

That weird!  I must be doing something wrong.  I was just collecting the information again and ... now it is working! LOL - that's annoying!

I think there might be two possibilities for the change: -

1.  I updated the Openshift platform yesterday with latest errata etc. and so restarted all masters and nodes.
2.  I deleted the project and created again using the latest oc: -

oc v3.1.0.4-5-gebe80f5
kubernetes v1.1.0-origin-1107-g4c8e6f4

Previously I had been creating using a slightly older one I had on a dev box assuming that it would not matter?

oc v3.0.2.0-17-g701346b
kubernetes v1.1.0-alpha.0-1605-g44c91b1

I assume that the API is consistent across these two (seen as it is versioned)?

Apologies for the run around - I can only insist that I WAS seeing the broken behaviour in my previous testing but it seems to be working as expected now.

regards


Dave
Comment 13 Ram Ranganathan 2015-12-11 13:45:21 EST
@zhaozhanqi, Thanks for double checking.

@Dave, no worries - glad its working for you now. Would be have been good to get to the crux of the issue. I suspect it was the restart that likely fixed it but can't say for certain. Will run some tests internally to see if it can be reproduced.
Those gremlins in the machine must be really chuckling!

Yes, the API is backwards compatible as it is versioned.

Am going to close this out for now - please feel free to open it if you run into this again. Thx

Note You need to log in before you can comment on or make changes to this bug.