Bug 2110528 - Route status isn't always getting cleared with routeSelector updates
Summary: Route status isn't always getting cleared with routeSelector updates
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.11
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.11.z
Assignee: Grant Spence
QA Contact: Melvin Joseph
URL:
Whiteboard:
Depends On: 2101878
Blocks: 2108214
TreeView+ depends on / blocked
 
Reported: 2022-07-25 14:31 UTC by OpenShift BugZilla Robot
Modified: 2022-09-20 16:35 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: There was a race condition in which, after a routeSelector update, the ingress controlller would clear the route status before the router deployment finished rolling out, resulting in the status being repopulated incorrectly. Consequence: Route status was incorrect showing admitted when in reality it was not admitted. Fix: Two fixes: 1. Update status detection to not use K8S object cache to help avoid using stale data 2. Check generation id on Route Deployment to determine if it is in the process of rolling out. Result: Route status is consistently cleared with a routeSelector update.
Clone Of:
Environment:
Last Closed: 2022-09-20 16:34:44 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-ingress-operator pull 813 0 None open [release-4.11] Bug 2110528: Fix another issue with route status clearing race condition caused by not validating generat... 2022-08-22 17:57:49 UTC
Red Hat Product Errata RHSA-2022:6536 0 None None None 2022-09-20 16:35:00 UTC

Comment 1 Melvin Joseph 2022-07-25 17:10:10 UTC 
melvinjoseph@mjoseph-mac Downloads % oc get clusterversion
NAME      VERSION                                                   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.ci.test-2022-07-25-161831-ci-ln-sqsrx9t-latest   True        False         19m     Cluster version is 4.11.0-0.ci.test-2022-07-25-161831-ci-ln-sqsrx9t-latest
melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-ingress-canary   canary  | jq '.status'
{
  "ingress": [
    {
      "conditions": [
        {
          "lastTransitionTime": "2022-07-25T16:34:31Z",
          "status": "True",
          "type": "Admitted"
        }
      ],
      "host": "canary-openshift-ingress-canary.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com",
      "routerCanonicalHostname": "router-default.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com",
      "routerName": "default",
      "wildcardPolicy": "None"
    }
  ]
}
melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress get pods
NAME                              READY   STATUS    RESTARTS   AGE
router-default-5b5c46d7db-h6z8s   1/1     Running   0          35m
router-default-5b5c46d7db-r69rb   1/1     Running   0          35m

melvinjoseph@mjoseph-mac Downloads % 
melvinjoseph@mjoseph-mac Downloads % 
melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress-operator patch ingresscontroller/default --type=merge --patch='{"spec":{"routeSelector":{"matchLabels":{"type":"shard"}}}}';
ingresscontroller.operator.openshift.io/default patched


melvinjoseph@mjoseph-mac Downloads % oc get route -A 
NAMESPACE                  NAME                      HOST/PORT                                                                                                 PATH        SERVICES            PORT    TERMINATION            WILDCARD
openshift-authentication   oauth-openshift           oauth-openshift.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com                                            oauth-openshift     6443    passthrough/Redirect   None
openshift-console          console                   console-openshift-console.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com                                  console             https   reencrypt/Redirect     None
openshift-console          downloads                 downloads-openshift-console.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com                                downloads           http    edge/Redirect          None
openshift-ingress-canary   canary                    canary-openshift-ingress-canary.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com                            ingress-canary      8080    edge/Redirect          None
openshift-monitoring       alertmanager-main         alertmanager-main-openshift-monitoring.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com         /api        alertmanager-main   web     reencrypt/Redirect     None
openshift-monitoring       prometheus-k8s            prometheus-k8s-openshift-monitoring.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com            /api        prometheus-k8s      web     reencrypt/Redirect     None
openshift-monitoring       prometheus-k8s-federate   prometheus-k8s-federate-openshift-monitoring.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com   /federate   prometheus-k8s      web     reencrypt/Redirect     None
openshift-monitoring       thanos-querier            thanos-querier-openshift-monitoring.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com            /api        thanos-querier      web     reencrypt/Redirect     None
melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-console console   | jq '.status'
{}
melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-authentication oauth-openshift   | jq '.status'
{}
melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-ingress-canary   canary  | jq '.status'                                                                           
{}

melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress get pods                                         
NAME                              READY   STATUS        RESTARTS   AGE
router-default-57c76d4d95-j7b7c   1/1     Running       0          85s
router-default-57c76d4d95-jfj4c   1/1     Running       0          85s
router-default-5b5c46d7db-h6z8s   0/1     Terminating   0          36m
melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress-operator patch ingresscontroller/default --type=merge --patch='{"spec":{"routeSelector":{"matchLabels":{"type":null}}}}';
ingresscontroller.operator.openshift.io/default patched
melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-authentication oauth-openshift   | jq '.status'                                                                
{
  "ingress": [
    {
      "conditions": [
        {
          "lastTransitionTime": "2022-07-25T17:06:01Z",
          "status": "True",
          "type": "Admitted"
        }
      ],
      "host": "oauth-openshift.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com",
      "routerCanonicalHostname": "router-default.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com",
      "routerName": "default",
      "wildcardPolicy": "None"
    }
  ]
}
melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress get pods                                                                                                                 
NAME                              READY   STATUS        RESTARTS   AGE
router-default-57c76d4d95-j7b7c   1/1     Terminating   0          117s
router-default-57c76d4d95-jfj4c   1/1     Running       0          117s
router-default-66b96c6d46-92f76   1/1     Running       0          23s
router-default-66b96c6d46-pbxlh   1/1     Running       0          23s
melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-authentication oauth-openshift   | jq '.status'                                                                
{
  "ingress": [
    {
      "conditions": [
        {
          "lastTransitionTime": "2022-07-25T17:06:01Z",
          "status": "True",
          "type": "Admitted"
        }
      ],
      "host": "oauth-openshift.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com",
      "routerCanonicalHostname": "router-default.apps.ci-ln-sqsrx9t-72292.origin-ci-int-gce.dev.rhcloud.com",
      "routerName": "default",
      "wildcardPolicy": "None"
    }
  ]
}
melvinjoseph@mjoseph-mac Downloads %
Comment 6 errata-xmlrpc 2022-09-20 16:34:44 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.11.5 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:6536
Note You need to log in before you can comment on or make changes to this bug.