Bug 1614727

Summary: [W21FD0v0] router dynamic changes does not work after adding custom blueprints route
Product: OpenShift Container Platform Reporter: Hongan Li <hongli>
Component: NetworkingAssignee: Ram Ranganathan <ramr>
Networking sub component: router QA Contact: zhaozhanqi <zzhao>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: aos-bugs, dmace
Version: 3.11.0   
Target Milestone: ---   
Target Release: 3.11.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-11 07:24:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Hongan Li 2018-08-10 09:59:24 UTC 
Description of problem:
If set env ROUTER_HAPROXY_CONFIG_MANAGER and ROUTER_BLUEPRINT_ROUTE_NAMESPACE firstly then add custom blueprints route, the haproxy router dynamic change does not work.

Version-Release number of selected component (if applicable):
openshift v3.11.0-0.11.0
kubernetes v1.11.0+d4cacc0

How reproducible:
always

Steps to Reproduce:
1. create new project "blueprints".
$ oc new-project blueprints

2. enable router dynamic changes and set blueprint namespace to "blueprints".
$ oc set env dc/router ROUTER_HAPROXY_CONFIG_MANAGER=true ROUTER_BLUEPRINT_ROUTE_NAMESPACE=blueprints

3. create svc and reencrypt route in namespace/project blueprints
$ oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/reencrypt/service_secure.json
$ oc create route reencrypt blueprints-reen --service=service-secure --dest-ca-cert=route_reencrypt_dest.ca -n blueprints

4. create your project, pod, svc and route
$ oc new-project hongli
$ oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/caddy-docker.json
$ oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/reencrypt/service_secure.json
$ oc create route reencrypt reen-route --service=service-secure --dest-ca-cert=route_reencrypt_dest.ca -n hongli

Actual results:
step4: router reloaded after creating reencrypt route

Expected results:
should no router reloaded in step4

Additional info:
1. if create the reencrypt route in blueprint project firstly, then enable ROUTER_HAPROXY_CONFIG_MANAGER and ROUTER_BLUEPRINT_ROUTE_NAMESPACE, it will not shows 'router reloaded' in the last step.
 
2. route_reencrypt_dest.ca can be downloaded from: https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/reencrypt/route_reencrypt_dest.ca
Comment 1 Ram Ranganathan 2018-08-13 22:02:40 UTC 
@hongli aargh found the bug, it is basically using the `blueprints` namespace 
when we dynamically add a blueprint. I fixed that (and on deleting a blueprint as well).

Fix is in PR: https://github.com/openshift/origin/pull/20630


But once the fix is in, one reason I can see something similar happening is  because of the order of processing of routes.
 
1. A new route is added to the blueprint namespace.
2. Router watch shows a new route added, so it adds a new "route" 
   (non-blueprint) for the re-encrypt route. 
   _Note:_ This is done before the blueprint watch process this route.
           So no matching blueprint route exists, so it needs a reload.

2. Router blueprint watch shows a new route and so it adds a new 
    blueprint route. Subsequent add/deletes should not require a reload (until
    we exhaust the blueprint pool/dynamic servers).


So once the fix is in, this might be a better test to see if we can reproduce a reload via a loop: 

$ # add blueprint route  (route to blueprint namespace) and wait for it to
$ # to be picked up.
$ # And then: 
$ for i in `seq 5`; do
   oc delete route reen-route
   oc create route reencrypt reen-route --service=service-secure --dest-ca-cert=route_reencrypt_dest.ca -n hongli

   curl <route-host>; sleep 10
done

and you can tail the router logs in another windows to verify that a reload 
doesn't occur.
Comment 2 Hongan Li 2018-08-14 06:58:11 UTC 
(In reply to Ram Ranganathan from comment #1)
> @hongli aargh found the bug, it is basically using the `blueprints`
> namespace 
> when we dynamically add a blueprint. I fixed that (and on deleting a
> blueprint as well).
> 
> Fix is in PR: https://github.com/openshift/origin/pull/20630
> 
Thank you Ram, will test it per your comments when the fix is in.
Comment 4 Hongan Li 2018-08-23 08:37:48 UTC 
verified in atomic-openshift-3.11.0-0.20.0.git.0.d80d8ad.el7 and the issue has been fixed.
Comment 6 errata-xmlrpc 2018-10-11 07:24:38 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2652