Bug 1915079 - Canary controller should not periodically rotate the canary route endpoint for performance reasons
Summary: Canary controller should not periodically rotate the canary route endpoint fo...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.7
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.7.0
Assignee: Stephen Greene
QA Contact: Arvind iyengar
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-01-11 21:58 UTC by Stephen Greene
Modified: 2022-08-04 22:30 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-24 15:51:51 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-ingress-operator pull 525 0 None closed Bug 1915079: Disable canary route churn by default 2021-01-18 07:05:23 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:52:13 UTC

Description Stephen Greene 2021-01-11 21:58:06 UTC
Description of problem:
Currently in 4.7, the new canary route is periodically modified by the canary controller so that the canary route switches which endpoint it hits. This is great for verifying whether or not the router has wedged, but since router reloads have a performance cost, this should not be enabled by default (especially for larger clusters already impacted by router reload performance issues). 

To resolve this BZ, add an annotation option for the default ingress controller that makes the canary route rotation functionality opt-in (disabled by default).


Version-Release number of selected component (if applicable):

4.7
How reproducible:
100%

Steps to Reproduce:
1. View ingress-operator logs on a 4.7 cluster


Actual results:
Observe that the canary route is periodically rotated every 5 minutes by default.

Expected results:
The canary route is not rotated once it is created, unless the canary-route-rotation annotation on the default ingress controller is set to true.


Additional info:

Comment 2 Arvind iyengar 2021-01-19 11:16:22 UTC
Verfied in "4.7.0-0.nightly-2021-01-18-214951" release payload. With this release, enabling the route rotation with the new annotation option, the rotation is triggered periodically:
-----
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2021-01-18-214951   True        False         142m    Cluster version is 4.7.0-0.nightly-2021-01-18-214951

With the value set to "true" which is default applied:
$ oc -n openshift-ingress-operator get  ingresscontroller default -o yaml 
         apiVersion: operator.openshift.io/v1
         kind: IngressController
         metadata:
           annotations:
             ingress.operator.openshift.io/rotate-canary-route: "true" <---
           creationTimestamp: "2021-01-19T08:19:33Z"
           finalizers:
           - ingresscontroller.operator.openshift.io/finalizer-ingresscontroller
           generation: 1
           managedFields:

$ oc -n openshift-ingress logs deployment.apps/router-default  --tail 20
I0119 10:12:29.201234       1 router.go:578] template "msg"="router reloaded"  "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I0119 10:19:29.344200       1 router.go:578] template "msg"="router reloaded"  "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I0119 10:26:29.503635       1 router.go:578] template "msg"="router reloaded"  "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I0119 10:33:29.707338       1 router.go:578] template "msg"="router reloaded"  "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I0119 10:40:29.857865       1 router.go:578] template "msg"="router reloaded"  "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I0119 10:47:30.045171       1 router.go:578] template "msg"="router reloaded"  "output"=" - Checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"

$ oc -n openshift-ingress-operator logs deployment.apps/ingress-operator -c ingress-operator --tail 20
2021-01-19T10:12:28.962Z	INFO	operator.canary_controller	canary/controller.go:363	updated canary route	{"namespace": "openshift-ingress-canary", "name": "canary"}
2021-01-19T10:19:29.108Z	INFO	operator.canary_controller	canary/controller.go:363	updated canary route	{"namespace": "openshift-ingress-canary", "name": "canary"}
2021-01-19T10:26:29.267Z	INFO	operator.canary_controller	canary/controller.go:363	updated canary route	{"namespace": "openshift-ingress-canary", "name": "canary"}
2021-01-19T10:33:29.473Z	INFO	operator.canary_controller	canary/controller.go:363	updated canary route	{"namespace": "openshift-ingress-canary", "name": "canary"}
2021-01-19T10:40:29.622Z	INFO	operator.canary_controller	canary/controller.go:363	updated canary route	{"namespace": "openshift-ingress-canary", "name": "canary"}
2021-01-19T10:47:29.809Z	INFO	operator.canary_controller	canary/controller.go:363	updated canary route	{"namespace": "openshift-ingress-canary", "name": "canary"}   
-----

Whereas setting it "False" or removing the annotation from the controller section, the reload no more occurs.

Comment 5 errata-xmlrpc 2021-02-24 15:51:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.