Red Hat Bugzilla – Bug 1391338
[networking_public_222]reencrypt route cannot work well for wildcard routes
Last modified: 2017-03-08 13:43 EST
Description of problem: When enable wildcard route.Create the reencrypt route which cannot work well (return 503) and also make other routes cannot work (return 503) Version-Release number of selected component (if applicable): openshift version openshift v3.4.0.19+346a31d kubernetes v1.4.0+776c994 etcd 3.1.0-rc.0 /openshift3/ose-haproxy-router v3.4.0.19 6e54d63e6bc9 13 hours ago 628.7 MB How reproducible: always Steps to Reproduce: 1. setup router and make sure is running 2. enable the wildcard route # oc env dc router ROUTER_ALLOW_WILDCARD_ROUTES=true 3. Create pod/service/reencript route using user1 $oc create -f https://raw.githubusercontent.com/openshift-qe/v3- testfiles/master/routing/caddy-docker.json $ oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/passthrough/service_secure.json $ oc reate -f https://raw.githubusercontent.com/zhaozhanqi/subdomain/master/route_reen.json 4. access the route curl --resolve 111.test-reen.example.com:443:$router_ip https://111.test-reen.example.com -k -v Actual results: step 4 return 503 Expected results: step 4 should return 200 and return 'Hello-OpenShift-1 https-8443' Additional info: 1. $ cat os_reencrypt.map [^\.]*\.example\.com(|/.*) default_route-reencrypt 2. $cat os_wildcard_domain.map [^\.]*\.example\.com 1 3. backend be_secure_default_route-reencrypt mode http option redispatch balance leastconn timeout check 5000ms http-request set-header X-Forwarded-Host %[req.hdr(host)] http-request set-header X-Forwarded-Port %[dst_port] http-request set-header X-Forwarded-Proto http if !{ ssl_fc } http-request set-header X-Forwarded-Proto https if { ssl_fc } http-request set-header Forwarded for=%[src];host=%[req.hdr(host)];proto=%[req.hdr(X-Forwarded-Proto)] cookie c5b3d58c01c6c072ae5e3c0de5eb7c16 insert indirect nocache httponly secure server 36b6ac76d21320f22a1f22ecc6f68a3b 10.128.0.20:8443 ssl check inter 5000ms verify required ca-file /var/lib/haproxy/router/cacerts/default_route-reencrypt.pem cookie 36b6ac76d21320f22a1f22ecc6f68a3b weight 100
@zhaozhanqi - this works fine for me - I took your route file and used my service (and changed the DestinationCACertificate to match my service) and it works fine. Here's my test case (copied from yours but pointing to my secure service): $ wget -q https://raw.githubusercontent.com/zhaozhanqi/subdomain/master/route_reen.json $ diff route_reen.json route-reen1.json | cut -c 1-80 8c8 < "host": "test-reen.example.com", --- > "host": "xyz.reencrypt.it", 12c12 < "name": "service-secure" --- > "name": "header-test-secure" 19c19 < "destinationCACertificate": "-----BEGIN CERTIFICATE--- --- > "destinationCACertificate": "-----BEGIN CERTIFICATE--- $ curl -k -s -o /dev/null -w "%{http_code}" --resolve www222.reencrypt.it:443:127.0.0.1 https://www222.reencrypt.it/ && echo "" 503 $ # above 503 error expected - no route exists, so let's create one ... $ oc create -f route-reen1.json route "route-reencrypt" created $ curl -k -s -o /dev/null -w "%{http_code}" --resolve www222.reencrypt.it:443:127.0.0.1 https://www222.reencrypt.it/ && echo "" 200 $ # 200 ok - as the route's added now - wildcard route so used a name www222 for xyz $ curl -k -s -o /dev/null -w "%{http_code}" --resolve abcdef.reencrypt.it:443:127.0.0.1 https://abcdef.reencrypt.it/ && echo "" 200 And tried with different host names: $ for i in `seq 5`; do echo "host: www$i.reencrypt.it - response code: $(curl -k -s -o /dev/null -w "%{http_code}" --resolve www$i.reencrypt.it:443:127.0.0.1 https://www$i.reencrypt.it/) "; done host: www1.reencrypt.it - response code: 200 host: www2.reencrypt.it - response code: 200 host: www3.reencrypt.it - response code: 200 host: www4.reencrypt.it - response code: 200 host: www5.reencrypt.it - response code: 200
Aah, one thing I can think of is that your destination ca certificate does not match the CA used to sign the pod/endpoint's (server in the haproxy config) certificate. So haproxy marks that server as down because of: server ... required ca-file /var/lib/haproxy/router/cacerts/default_route-reencrypt.pem Make sure that the destination ca certificate matches - otherwise you will see a 503 because haproxy can't talk to the backend pod. Please reopen if you still have an issue. Thx
Hi ,Ram I find the steps of reproducing this issue. 1. oc create -f https://raw.githubusercontent.com/zhaozhanqi/subdomain/master/route_pass.json 2. oc create -f https://raw.githubusercontent.com/zhaozhanqi/subdomain/master/route_reen.json When those two routes are created. the route will return 503.
Well, if both routes are set, then the oldest one would take precedence since both are wildcard routes for *.example.com - in your case it would be the passthrough route. Maybe the correct reencrypt route test here is to use a different subdomain name for the route_reen.json file ala test-reen.second.com ?
Please note currently we don't block routes within the same namespace - so this is working as expected. Both wildcard routes would be allowed as they are part of the same namespace. I guess in a way if we do fix the second one being rejected, then the fix for this bugz would be a duplicate of bugz: https://bugzilla.redhat.com/show_bug.cgi?id=1391878 Still mulling re: the rejection ... stay tuned.
Fixes in PR: https://github.com/openshift/origin/pull/11849
This has been merged into ose and is in OSE v3.4.0.25 or newer.
Verified this bug on OSE v3.4.0.25.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0066