Cause:
There was a race condition in which, after a routeSelector update, the ingress controlller would clear the route status before the router deployment finished rolling out, resulting in the status being repopulated incorrectly.
Consequence:
Route status was incorrect showing admitted when in reality it was not admitted.
Fix:
Two fixes:
1. Update status detection to not use K8S object cache to help avoid using stale data
2. Check generation id on Route Deployment to determine if it is in the process of rolling out.
Result:
Route status is consistently cleared with a routeSelector update.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Moderate: OpenShift Container Platform 4.11.5 bug fix and security update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2022:6536
melvinjoseph@mjoseph-mac Downloads % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.0-0.ci.test-2022-07-25-161831-ci-ln-sqsrx9t-latest True False 19m Cluster version is 4.11.0-0.ci.test-2022-07-25-161831-ci-ln-sqsrx9t-latest melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-ingress-canary canary | jq '.status' { "ingress": [ { "conditions": [ { "lastTransitionTime": "2022-07-25T16:34:31Z", "status": "True", "type": "Admitted" } ], "host": "canary-openshift-ingress-canary.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com", "routerCanonicalHostname": "router-default.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com", "routerName": "default", "wildcardPolicy": "None" } ] } melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress get pods NAME READY STATUS RESTARTS AGE router-default-5b5c46d7db-h6z8s 1/1 Running 0 35m router-default-5b5c46d7db-r69rb 1/1 Running 0 35m melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress-operator patch ingresscontroller/default --type=merge --patch='{"spec":{"routeSelector":{"matchLabels":{"type":"shard"}}}}'; ingresscontroller.operator.openshift.io/default patched melvinjoseph@mjoseph-mac Downloads % oc get route -A NAMESPACE NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD openshift-authentication oauth-openshift oauth-openshift.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com oauth-openshift 6443 passthrough/Redirect None openshift-console console console-openshift-console.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com console https reencrypt/Redirect None openshift-console downloads downloads-openshift-console.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com downloads http edge/Redirect None openshift-ingress-canary canary canary-openshift-ingress-canary.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com ingress-canary 8080 edge/Redirect None openshift-monitoring alertmanager-main alertmanager-main-openshift-monitoring.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com /api alertmanager-main web reencrypt/Redirect None openshift-monitoring prometheus-k8s prometheus-k8s-openshift-monitoring.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com /api prometheus-k8s web reencrypt/Redirect None openshift-monitoring prometheus-k8s-federate prometheus-k8s-federate-openshift-monitoring.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com /federate prometheus-k8s web reencrypt/Redirect None openshift-monitoring thanos-querier thanos-querier-openshift-monitoring.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com /api thanos-querier web reencrypt/Redirect None melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-console console | jq '.status' {} melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-authentication oauth-openshift | jq '.status' {} melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-ingress-canary canary | jq '.status' {} melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress get pods NAME READY STATUS RESTARTS AGE router-default-57c76d4d95-j7b7c 1/1 Running 0 85s router-default-57c76d4d95-jfj4c 1/1 Running 0 85s router-default-5b5c46d7db-h6z8s 0/1 Terminating 0 36m melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress-operator patch ingresscontroller/default --type=merge --patch='{"spec":{"routeSelector":{"matchLabels":{"type":null}}}}'; ingresscontroller.operator.openshift.io/default patched melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-authentication oauth-openshift | jq '.status' { "ingress": [ { "conditions": [ { "lastTransitionTime": "2022-07-25T17:06:01Z", "status": "True", "type": "Admitted" } ], "host": "oauth-openshift.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com", "routerCanonicalHostname": "router-default.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com", "routerName": "default", "wildcardPolicy": "None" } ] } melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress get pods NAME READY STATUS RESTARTS AGE router-default-57c76d4d95-j7b7c 1/1 Terminating 0 117s router-default-57c76d4d95-jfj4c 1/1 Running 0 117s router-default-66b96c6d46-92f76 1/1 Running 0 23s router-default-66b96c6d46-pbxlh 1/1 Running 0 23s melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-authentication oauth-openshift | jq '.status' { "ingress": [ { "conditions": [ { "lastTransitionTime": "2022-07-25T17:06:01Z", "status": "True", "type": "Admitted" } ], "host": "oauth-openshift.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com", "routerCanonicalHostname": "router-default.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com", "routerName": "default", "wildcardPolicy": "None" } ] } melvinjoseph@mjoseph-mac Downloads %