Description of problem: I want to enable routing to an application via both http or https routes: - http://helloworld.paas.dev.iggroup.local https://helloworld.paas.dev.iggroup.local (*.paas.dev.iggroup.local routes to our openshift routers) Creating a Route with tls section will ONLY create a secure route and so I try to create two routes: - - apiVersion: v1 kind: Route metadata: name: helloworld-route-https spec: host: helloworld.paas.dev.iggroup.local to: kind: Service name: helloworld-service tls: termination: edge - apiVersion: v1 kind: Route metadata: name: helloworld-route-http spec: host: helloworld.paas.dev.iggroup.local to: kind: Service name: helloworld-service I am allowed to create them: - NAME HOST/PORT PATH SERVICE LABELS TLS TERMINATION helloworld-route-http helloworld.paas.dev.iggroup.local helloworld-service helloworld-route-https helloworld.paas.dev.iggroup.local helloworld-service edge So it looks as though the two routes have been accepted. The host on the two routes is exactly the same - 'helloworld.paas.dev.iggroup.local'. Within the router container only the ha_proxy configuration map 'os_http_be.map' has been updated: - helloworld.paas.dev.iggroup.local hello-openshift_helloworld-route-http and only the following backend in ha_proxy has been configured: - backend be_http_hello-openshift_helloworld-route-http mode http option redispatch option forwardfor balance leastconn timeout check 5000ms http-request set-header X-Forwarded-Host %[req.hdr(host)] http-request set-header X-Forwarded-Port %[dst_port] http-request set-header X-Forwarded-Proto https if { ssl_fc } cookie OPENSHIFT_hello-openshift_helloworld-route-http_SERVERID insert indirect nocache httponly http-request set-header X-Forwarded-Proto http http-request set-header Forwarded for=%[src],host=%[req.hdr(host)],proto=%[req.hdr(X-Forwarded-Proto)] server 10.1.1.7:8080 10.1.1.7:8080 check inter 5000ms cookie 10.1.1.7:8080 server 10.1.3.6:8080 10.1.3.6:8080 check inter 5000ms cookie 10.1.3.6:8080 Nothing has been added to the map 'os_edge_http_be.map' which should be updated with the helloworld-route-https route. The be_http_hello-openshift_helloworld-route-http backend is only accessible via http (port 80). The following backend is missing: - os_edge_http_be.map helloworld.paas.dev.iggroup.local hello-openshift_helloworld-route-https backend be_edge_http_hello-openshift_helloworld-route-https mode http option redispatch option forwardfor balance leastconn timeout check 5000ms http-request set-header X-Forwarded-Host %[req.hdr(host)] http-request set-header X-Forwarded-Port %[dst_port] http-request set-header X-Forwarded-Proto https if { ssl_fc } cookie OPENSHIFT_EDGE_hello-openshift_helloworld-route-https_SERVERID insert indirect nocache httponly secure http-request set-header Forwarded for=%[src],host=%[req.hdr(host)],proto=%[req.hdr(X-Forwarded-Proto)] server 10.1.1.7:8080 10.1.1.7:8080 check inter 5000ms cookie 10.1.1.7:8080 server 10.1.3.6:8080 10.1.3.6:8080 check inter 5000ms cookie 10.1.3.6:8080 Version-Release number of selected component (if applicable): ose-haproxy-router:v3.1.0.4 How reproducible: Every time. Expected results: I would ideally like to configure a single route that allows me to define availability on httpd, https or BOTH. Alternatively, if I must create two routes then creating an insecure route and a secure route to the same host should result in valid ha_proxy configuration and both working routes. Additional info:
you can add the following to your route to fixed your issue. insecureEdgeTerminationPolicy: Allow for detail, you can see https://github.com/openshift/origin/pull/5224 or https://github.com/openshift/openshift-docs/blob/master/architecture/core_concepts/routes.adoc#secured-routes Example 7. A Secured Route Using Edge Termination Allowing HTTP Traffic apiVersion: v1 kind: Route metadata: name: route-edge-secured-allow-insecure spec: host: www.example.com to: kind: Service name: service-name tls: termination: edge (1) insecureEdgeTerminationPolicy: Allow (2)
Hi When I try these settings... apiVersion: v1 kind: Route metadata: name: helloworld-route spec: host: helloworld.paas.dev.iggroup.local to: kind: Service name: helloworld-service tls: termination: edge insecureEdgeTerminationPolicy: Allow and browse to http://helloworld.paas.dev.iggroup.local I get a 503 error. This is the only route. backend be_edge_http_davetest_helloworld-route mode http option redispatch option forwardfor balance leastconn timeout check 5000ms http-request set-header X-Forwarded-Host %[req.hdr(host)] http-request set-header X-Forwarded-Port %[dst_port] http-request set-header X-Forwarded-Proto https if { ssl_fc } cookie OPENSHIFT_EDGE_davetest_helloworld-route_SERVERID insert indirect nocache httponly secure http-request set-header Forwarded for=%[src],host=%[req.hdr(host)],proto=%[req.hdr(X-Forwarded-Proto)] server 10.1.0.5:8080 10.1.0.5:8080 check inter 5000ms cookie 10.1.0.5:8080 server 10.1.1.3:8080 10.1.1.3:8080 check inter 5000ms cookie 10.1.1.3:8080 There is nothing in the os_edge_http_expose.map map which looks as though there perhaps should be... Inside the router:- ls -al total 72 drwxrwxrwx. 2 root root 4096 Dec 1 07:43 . drwxrwxrwx. 6 haproxy haproxy 68 Nov 11 20:17 .. -rwxrwxrwx. 1 root root 2042 Nov 11 19:59 default_pub_keys.pem -rwxrwxrwx. 1 root root 103 Nov 11 20:16 error-page-503.html -rwxrwxrwx. 1 root root 16766 Nov 11 20:16 haproxy-config.template -rwxrwxrwx. 1 root root 9759 Dec 1 07:42 haproxy.config -rwxrwxrwx. 1 root root 73 Dec 1 07:42 os_edge_http_be.map -rwxrwxrwx. 1 root root 12 Dec 1 07:42 os_edge_http_expose.map -rwxrwxrwx. 1 root root 12 Dec 1 07:42 os_edge_http_redirect.map -rwxrwxrwx. 1 root root 12 Dec 1 07:42 os_http_be.map -rwxrwxrwx. 1 root root 12 Dec 1 07:42 os_reencrypt.map -rwxrwxrwx. 1 root root 12 Dec 1 07:42 os_sni_passthrough.map -rwxrwxrwx. 1 root root 12 Dec 1 07:42 os_tcp_be.map Only os_edge_http_be.map has any content.. helloworld.paas.dev.iggroup.local davetest_helloworld-route This suggests that we should also have something in the map os_edge_http_expose.map so that a back-end is made available... frontend public bind :80 mode http tcp-request inspect-delay 5s tcp-request content accept if HTTP # check if we need to redirect/force using https. acl secure_redirect base,map_beg(/var/lib/haproxy/conf/os_edge_http_redirect.map) -m found redirect scheme https if secure_redirect # Check if it is an edge route exposed insecurely. acl edge_http_expose base,map_beg(/var/lib/haproxy/conf/os_edge_http_expose.map) -m found use_backend be_edge_http_%[base,map_beg(/var/lib/haproxy/conf/os_edge_http_expose.map)] if edge_http_expose # map to http backend # Search from most specific to general path (host case). # Note: If no match, haproxy uses the default_backend, no other # use_backend directives below this will be processed. use_backend be_http_%[base,map_beg(/var/lib/haproxy/conf/os_http_be.map)] default_backend openshift_default regards Dave
@dave.mccormick what version of OpenShift Origin are you running? From the latest code, this works for me. There should be an allowed insecure route entry in os_edge_http_expose.map - looks like your file is empty. Here's an example repo that has the different types of routes. https://github.com/ramr/nodejs-header-echo Instructions are in the README for how to create the different types of routes. The allow http example uses this json for defining the routes: https://github.com/ramr/nodejs-header-echo/blob/master/openshift/edge-secured-allow-http-route.json And this test to ensure it works: $ # Check edge terminated route with insecure allowed (http + https). $ curl --resolve allow-http.header.test:80:127.0.0.1 -vvv \ -H "test: allow-http" -k http://allow-http.header.test $ curl --resolve allow-http.header.test:443:127.0.0.1 -vvv \ -H "test: allow-https" -k https://allow-http.header.test Note: You will either need to run that on the node the router is running on. Or if you want to run it from outside, replace 127.0.0.1 with the IP address of the router. Give that a try and let me know if that works for you. If not, we can debug your environment and see what's amiss. Thanks.
@dave.mccormick - any updates? Thx
Hi Ram Sorry for the slow reply. Thanks for the link to your repository, your example edge-secured-allow-http-route.json looks very similar but not quite the same as mine: - apiVersion: v1 kind: Route metadata: name: helloworld-route spec: host: helloworld.paas.dev.iggroup.local to: kind: Service name: helloworld-service tls: termination: edge insecureEdgeTerminationPolicy: Allow The difference is that I am wanting to allow the HTTPS route to fall back and use the default wildcard certificate that I have defined on my router (which I'm told I should be able to do and works for a simple https route). Can you test with the same set up as me above? I'm using openshift enterprise v3.1.0.4 (router is also labled ose-haproxy-router:v3.1.0.4). Are you using a newer release? Should I be using a newer one? You are correct - the issue does seem to be the missing entry in the os_edge_http_expose.map. That is my bug/issue. I don't know why when I create the route as above it don't get both routes, I think that my syntax looks good and so it feels like a bug the fact that the map file is not being updated. regards Dave
Hi Dave, Hmm, that is weird. v3.1.0.4 should work - it works for me with this json file: $ cat _rrtests/nodejs-header-echo/dave.json { "kind": "Route", "apiVersion": "v1", "metadata": { "name": "header-test-http-ig-allow" }, "id": "header-test-http-ig-allow", "spec": { "host": "ig-allow-http.header.test", "to": { "kind": "Service", "name": "header-test-insecure" }, "tls": { "termination": "edge", "insecureEdgeTerminationPolicy": "Allow" } } } $ oc create -f _rrtests/nodejs-header-echo/dave.json $ oc get routes NAME HOST/PORT PATH SERVICE LABELS INSECURE POLICY TLS TERMINATION header-test-http-ig-allow ig-allow-http.header.test header-test-insecure Allow edge # grepped for brevity ... $ curl --resolve ig-allow-http.header.test:443:127.0.0.1 -s -H "test: ig-allow-http" -k https://ig-allow-http.header.test | grep 'test:' test: ig-allow-http $ curl --resolve ig-allow-http.header.test:80:127.0.0.1 -s -H "test: ig-allow-http" -k http://ig-allow-http.header.test | grep 'test:' test: ig-allow-http
Hi Dave, Could you please attach the haproxy-config file and the output of docker inspect. docker inspect $(docker ps | grep haproxy-router | awk '{print $1}') docker exec $(docker ps | grep haproxy-router | awk '{print $1}') cat /var/lib/haproxy/conf/haproxy.config Thanks in advance, Ram//
hi, ram & Dave this also works for me on my ose env with the #c8 dava.json # oc version oc v3.1.0.4-16-g112fcc4 kubernetes v1.1.0-origin-1107-g4c8e6f4 ********router images:******** openshift3/ose-haproxy-router v3.1.0.4 81c50204da05 4 weeks ago 410.2 MB **************in router container:********** # grep -i "zzhao" * haproxy.config:backend be_edge_http_zzhao_header-test-http-ig-allow haproxy.config: cookie OPENSHIFT_EDGE_zzhao_header-test-http-ig-allow_SERVERID insert indirect nocache httponly secure os_edge_http_be.map:ig-allow-http.header.test zzhao_header-test-http-ig-allow os_edge_http_expose.map:ig-allow-http.header.test zzhao_header-test-http-ig-allow *******************test the route******************************* [root@dhcp-136-93 nodejs-header-echo]# curl --resolve ig-allow-http.header.test:443:10.14.6.137 https://ig-allow-http.header.test -k -s -H "test: ig-allow-http" |grep 'test:' test: ig-allow-http [root@dhcp-136-93 nodejs-header-echo]# curl --resolve ig-allow-http.header.test:80:10.14.6.137 http://ig-allow-http.header.test -k -s -H "test: ig-allow-http" |grep 'test:' test: ig-allow-http
Hi That weird! I must be doing something wrong. I was just collecting the information again and ... now it is working! LOL - that's annoying! I think there might be two possibilities for the change: - 1. I updated the Openshift platform yesterday with latest errata etc. and so restarted all masters and nodes. 2. I deleted the project and created again using the latest oc: - oc v3.1.0.4-5-gebe80f5 kubernetes v1.1.0-origin-1107-g4c8e6f4 Previously I had been creating using a slightly older one I had on a dev box assuming that it would not matter? oc v3.0.2.0-17-g701346b kubernetes v1.1.0-alpha.0-1605-g44c91b1 I assume that the API is consistent across these two (seen as it is versioned)? Apologies for the run around - I can only insist that I WAS seeing the broken behaviour in my previous testing but it seems to be working as expected now. regards Dave
@zhaozhanqi, Thanks for double checking. @Dave, no worries - glad its working for you now. Would be have been good to get to the crux of the issue. I suspect it was the restart that likely fixed it but can't say for certain. Will run some tests internally to see if it can be reproduced. Those gremlins in the machine must be really chuckling! Yes, the API is backwards compatible as it is versioned. Am going to close this out for now - please feel free to open it if you run into this again. Thx