Bug 1614727 - [W21FD0v0] router dynamic changes does not work after adding custom blueprints route
Summary: [W21FD0v0] router dynamic changes does not work after adding custom blueprint...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.11.0
Assignee: Ram Ranganathan
QA Contact: zhaozhanqi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-08-10 09:59 UTC by Hongan Li
Modified: 2022-08-04 22:20 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-11 07:24:38 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github https://github.com/openshift origin pull 20630 0 None None None 2020-08-01 03:06:50 UTC
Red Hat Product Errata RHBA-2018:2652 0 None None None 2018-10-11 07:25:04 UTC

Description Hongan Li 2018-08-10 09:59:24 UTC
Description of problem:
If set env ROUTER_HAPROXY_CONFIG_MANAGER and ROUTER_BLUEPRINT_ROUTE_NAMESPACE firstly then add custom blueprints route, the haproxy router dynamic change does not work.

Version-Release number of selected component (if applicable):
openshift v3.11.0-0.11.0
kubernetes v1.11.0+d4cacc0

How reproducible:
always

Steps to Reproduce:
1. create new project "blueprints".
$ oc new-project blueprints

2. enable router dynamic changes and set blueprint namespace to "blueprints".
$ oc set env dc/router ROUTER_HAPROXY_CONFIG_MANAGER=true ROUTER_BLUEPRINT_ROUTE_NAMESPACE=blueprints

3. create svc and reencrypt route in namespace/project blueprints
$ oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/reencrypt/service_secure.json
$ oc create route reencrypt blueprints-reen --service=service-secure --dest-ca-cert=route_reencrypt_dest.ca -n blueprints

4. create your project, pod, svc and route
$ oc new-project hongli
$ oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/caddy-docker.json
$ oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/reencrypt/service_secure.json
$ oc create route reencrypt reen-route --service=service-secure --dest-ca-cert=route_reencrypt_dest.ca -n hongli

Actual results:
step4: router reloaded after creating reencrypt route

Expected results:
should no router reloaded in step4

Additional info:
1. if create the reencrypt route in blueprint project firstly, then enable ROUTER_HAPROXY_CONFIG_MANAGER and ROUTER_BLUEPRINT_ROUTE_NAMESPACE, it will not shows 'router reloaded' in the last step.
 
2. route_reencrypt_dest.ca can be downloaded from: https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/reencrypt/route_reencrypt_dest.ca

Comment 1 Ram Ranganathan 2018-08-13 22:02:40 UTC
@hongli aargh found the bug, it is basically using the `blueprints` namespace 
when we dynamically add a blueprint. I fixed that (and on deleting a blueprint as well).

Fix is in PR: https://github.com/openshift/origin/pull/20630


But once the fix is in, one reason I can see something similar happening is  because of the order of processing of routes.
 
1. A new route is added to the blueprint namespace.
2. Router watch shows a new route added, so it adds a new "route" 
   (non-blueprint) for the re-encrypt route. 
   _Note:_ This is done before the blueprint watch process this route.
           So no matching blueprint route exists, so it needs a reload.

2. Router blueprint watch shows a new route and so it adds a new 
    blueprint route. Subsequent add/deletes should not require a reload (until
    we exhaust the blueprint pool/dynamic servers).


So once the fix is in, this might be a better test to see if we can reproduce a reload via a loop: 

$ # add blueprint route  (route to blueprint namespace) and wait for it to
$ # to be picked up.
$ # And then: 
$ for i in `seq 5`; do
   oc delete route reen-route
   oc create route reencrypt reen-route --service=service-secure --dest-ca-cert=route_reencrypt_dest.ca -n hongli

   curl <route-host>; sleep 10
done

and you can tail the router logs in another windows to verify that a reload 
doesn't occur.

Comment 2 Hongan Li 2018-08-14 06:58:11 UTC
(In reply to Ram Ranganathan from comment #1)
> @hongli aargh found the bug, it is basically using the `blueprints`
> namespace 
> when we dynamically add a blueprint. I fixed that (and on deleting a
> blueprint as well).
> 
> Fix is in PR: https://github.com/openshift/origin/pull/20630
> 
Thank you Ram, will test it per your comments when the fix is in.

Comment 4 Hongan Li 2018-08-23 08:37:48 UTC
verified in atomic-openshift-3.11.0-0.20.0.git.0.d80d8ad.el7 and the issue has been fixed.

Comment 6 errata-xmlrpc 2018-10-11 07:24:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2652


Note You need to log in before you can comment on or make changes to this bug.