Description of problem: https://bugzilla.redhat.com/show_bug.cgi?id=1944851# introduced a fix to clearing route status, however, QA found that it wasn't consistently working. OpenShift release version: 4.11 Cluster Platform: All How reproducible: 25%-50% of the time Steps to Reproduce (in detail): 1. Create a new cluster 2. Shard the default ingress controller: oc -n openshift-ingress-operator patch ingresscontroller/default --type=merge --patch='{"spec":{"routeSelector":{"matchLabels":{"type":"shard"}}}}'; 3. Wait about 30-60 seconds 4. Is canary route status cleared? oc get route -n openshift-ingress-canary -o yaml Actual results: Sometimes, canary/console/auth route don't clear status when sharding the default ingress controller Expected results: Status should always clear if the route is no longer admitted. Impact of the problem: No impact other than it's confusing to developers to debug route status Additional info: ** Please do not disregard the report template; filling the template out as much as possible will allow us to help you. Please consider attaching a must-gather archive (via `oc adm must-gather`). Please review must-gather contents for sensitive information before attaching any must-gathers to a bugzilla report. You may also mark the bug private if you wish.
melvinjoseph@mjoseph-mac Downloads % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.12.0-0.nightly-2022-07-19-125123 True False 5h6m Cluster version is 4.12.0-0.nightly-2022-07-19-125123 melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-ingress-canary canary | jq '.status' { "ingress": [ { "conditions": [ { "lastTransitionTime": "2022-07-20T12:53:15Z", "status": "True", "type": "Admitted" } ], "host": "canary-openshift-ingress-canary.apps.mjoseph-re5.qe.gcp.devcluster.openshift.com", "routerCanonicalHostname": "router-default.apps.mjoseph-re5.qe.gcp.devcluster.openshift.com", "routerName": "default", "wildcardPolicy": "None" } ] } melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress get pods NAME READY STATUS RESTARTS AGE router-default-6f475d9ff9-5fz6b 1/1 Running 0 3m56s router-default-6f475d9ff9-w28nr 1/1 Running 0 3m56s melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress-operator patch ingresscontroller/default --type=merge --patch='{"spec":{"routeSelector":{"matchLabels":{"type":"shard"}}}}'; ingresscontroller.operator.openshift.io/default patched melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress get pods NAME READY STATUS RESTARTS AGE router-default-657b758c8b-4cpfg 0/1 Running 0 4s router-default-657b758c8b-5qhnh 0/1 Running 0 5s router-default-6f475d9ff9-5fz6b 1/1 Terminating 0 4m11s router-default-6f475d9ff9-w28nr 1/1 Running 0 4m11s melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress get pods NAME READY STATUS RESTARTS AGE router-default-657b758c8b-4cpfg 1/1 Running 0 97s router-default-657b758c8b-5qhnh 1/1 Running 0 98s router-default-6f475d9ff9-w28nr 0/1 Terminating 0 5m44s melvinjoseph@mjoseph-mac Downloads % oc get route -A NAMESPACE NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD openshift-authentication oauth-openshift oauth-openshift.apps.mjoseph-re5.qe.gcp.devcluster.openshift.com oauth-openshift 6443 passthrough/Redirect None openshift-console console console-openshift-console.apps.mjoseph-re5.qe.gcp.devcluster.openshift.com console https reencrypt/Redirect None openshift-console downloads downloads-openshift-console.apps.mjoseph-re5.qe.gcp.devcluster.openshift.com downloads http edge/Redirect None openshift-ingress-canary canary canary-openshift-ingress-canary.apps.mjoseph-re5.qe.gcp.devcluster.openshift.com ingress-canary 8080 edge/Redirect None openshift-monitoring alertmanager-main alertmanager-main-openshift-monitoring.apps.mjoseph-re5.qe.gcp.devcluster.openshift.com /api alertmanager-main web reencrypt/Redirect None openshift-monitoring prometheus-k8s prometheus-k8s-openshift-monitoring.apps.mjoseph-re5.qe.gcp.devcluster.openshift.com /api prometheus-k8s web reencrypt/Redirect None openshift-monitoring prometheus-k8s-federate prometheus-k8s-federate-openshift-monitoring.apps.mjoseph-re5.qe.gcp.devcluster.openshift.com /federate prometheus-k8s web reencrypt/Redirect None openshift-monitoring thanos-querier thanos-querier-openshift-monitoring.apps.mjoseph-re5.qe.gcp.devcluster.openshift.com /api thanos-querier web reencrypt/Redirect None melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress get pods NAME READY STATUS RESTARTS AGE router-default-657b758c8b-4cpfg 1/1 Running 0 111s router-default-657b758c8b-5qhnh 1/1 Running 0 112s melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-ingress-canary canary | jq '.status' { "ingress": [ { "conditions": [ { "lastTransitionTime": "2022-07-20T12:57:19Z", "status": "True", "type": "Admitted" } ], "host": "canary-openshift-ingress-canary.apps.mjoseph-re5.qe.gcp.devcluster.openshift.com", "routerCanonicalHostname": "router-default.apps.mjoseph-re5.qe.gcp.devcluster.openshift.com", "routerName": "default", "wildcardPolicy": "None" } ] } Hence verified
melvinjoseph@mjoseph-mac Downloads % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.0-0.ci.test-2022-07-22-052833-ci-ln-hhbncbk-latest True False 16m Cluster version is 4.11.0-0.ci.test-2022-07-22-052833-ci-ln-hhbncbk-latest melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-ingress-canary canary | jq '.status' { "ingress": [ { "conditions": [ { "lastTransitionTime": "2022-07-22T05:50:06Z", "status": "True", "type": "Admitted" } ], "host": "canary-openshift-ingress-canary.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerCanonicalHostname": "router-default.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerName": "default", "wildcardPolicy": "None" } ] } melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress get pods NAME READY STATUS RESTARTS AGE router-default-5869df5488-54k74 1/1 Running 0 29m router-default-5869df5488-lxtp7 1/1 Running 0 29m melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress-operator patch ingresscontroller/default --type=merge --patch='{"spec":{"routeSelector":{"matchLabels":{"type":"shard"}}}}'; ingresscontroller.operator.openshift.io/default patched melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress get pods NAME READY STATUS RESTARTS AGE router-default-5869df5488-54k74 1/1 Running 0 30m router-default-5869df5488-lxtp7 1/1 Terminating 0 30m router-default-69684fbfff-cmc4c 1/1 Running 0 5s router-default-69684fbfff-xmzks 1/1 Running 0 5s melvinjoseph@mjoseph-mac Downloads % oc get route -A NAMESPACE NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD openshift-authentication oauth-openshift oauth-openshift.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com oauth-openshift 6443 passthrough/Redirect None openshift-console console console-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com console https reencrypt/Redirect None openshift-console downloads downloads-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com downloads http edge/Redirect None openshift-ingress-canary canary canary-openshift-ingress-canary.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ingress-canary 8080 edge/Redirect None openshift-monitoring alertmanager-main alertmanager-main-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api alertmanager-main web reencrypt/Redirect None openshift-monitoring prometheus-k8s prometheus-k8s-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api prometheus-k8s web reencrypt/Redirect None openshift-monitoring prometheus-k8s-federate prometheus-k8s-federate-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /federate prometheus-k8s web reencrypt/Redirect None openshift-monitoring thanos-querier thanos-querier-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api thanos-querier web reencrypt/Redirect None melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress get pods NAME READY STATUS RESTARTS AGE router-default-69684fbfff-cmc4c 1/1 Running 0 94s router-default-69684fbfff-xmzks 1/1 Running 0 94s melvinjoseph@mjoseph-mac Downloads % oc get route -A NAMESPACE NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD openshift-authentication oauth-openshift oauth-openshift.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com oauth-openshift 6443 passthrough/Redirect None openshift-console console console-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com console https reencrypt/Redirect None openshift-console downloads downloads-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com downloads http edge/Redirect None openshift-ingress-canary canary canary-openshift-ingress-canary.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ingress-canary 8080 edge/Redirect None openshift-monitoring alertmanager-main alertmanager-main-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api alertmanager-main web reencrypt/Redirect None openshift-monitoring prometheus-k8s prometheus-k8s-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api prometheus-k8s web reencrypt/Redirect None openshift-monitoring prometheus-k8s-federate prometheus-k8s-federate-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /federate prometheus-k8s web reencrypt/Redirect None openshift-monitoring thanos-querier thanos-querier-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api thanos-querier web reencrypt/Redirect None melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-ingress-canary canary | jq '.status' {} melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-authentication oauth-openshift | jq '.status' {} melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-console console | jq '.status' {} melvinjoseph@mjoseph-mac Downloads % oc get route -A NAMESPACE NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD openshift-authentication oauth-openshift oauth-openshift.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com oauth-openshift 6443 passthrough/Redirect None openshift-console console console-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com console https reencrypt/Redirect None openshift-console downloads downloads-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com downloads http edge/Redirect None openshift-ingress-canary canary canary-openshift-ingress-canary.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ingress-canary 8080 edge/Redirect None openshift-monitoring alertmanager-main alertmanager-main-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api alertmanager-main web reencrypt/Redirect None openshift-monitoring prometheus-k8s prometheus-k8s-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api prometheus-k8s web reencrypt/Redirect None openshift-monitoring prometheus-k8s-federate prometheus-k8s-federate-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /federate prometheus-k8s web reencrypt/Redirect None openshift-monitoring thanos-querier thanos-querier-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api thanos-querier web reencrypt/Redirect None melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress-operator patch ingresscontroller/default --type=merge --patch='{"spec":{"routeSelector":{"matchLabels":{"type":null}}}}'; ingresscontroller.operator.openshift.io/default patched melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress get pods NAME READY STATUS RESTARTS AGE router-default-6876749bfd-62zmg 0/1 Running 0 4s router-default-6876749bfd-7ds42 0/1 Running 0 4s router-default-69684fbfff-cmc4c 1/1 Running 0 3m17s router-default-69684fbfff-xmzks 1/1 Terminating 0 3m17s melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-console console | jq '.status' { "ingress": [ { "conditions": [ { "lastTransitionTime": "2022-07-22T06:16:15Z", "status": "True", "type": "Admitted" } ], "host": "console-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerCanonicalHostname": "router-default.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerName": "default", "wildcardPolicy": "None" } ] } melvinjoseph@mjoseph-mac Downloads % oc get route -n openshift-ingress-canary -o yaml apiVersion: v1 items: - apiVersion: route.openshift.io/v1 kind: Route metadata: annotations: haproxy.router.openshift.io/balance: roundrobin openshift.io/host.generated: "true" creationTimestamp: "2022-07-22T05:47:42Z" labels: ingress.openshift.io/canary: canary_controller name: canary namespace: openshift-ingress-canary ownerReferences: - apiVersion: apps/v1 controller: true kind: daemonset name: ingress-canary uid: 061f05b5-9ff6-482e-a852-d3d95f8484e3 resourceVersion: "34424" uid: 6976e421-5449-4e1e-b9c1-63b07201a18d spec: host: canary-openshift-ingress-canary.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com port: targetPort: 8080 tls: insecureEdgeTerminationPolicy: Redirect termination: edge to: kind: Service name: ingress-canary weight: 100 wildcardPolicy: None status: ingress: - conditions: - lastTransitionTime: "2022-07-22T06:16:15Z" status: "True" type: Admitted host: canary-openshift-ingress-canary.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com routerCanonicalHostname: router-default.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com routerName: default wildcardPolicy: None kind: List metadata: resourceVersion: "" selfLink: "" melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress get pods NAME READY STATUS RESTARTS AGE router-default-6876749bfd-62zmg 1/1 Running 0 37s router-default-6876749bfd-7ds42 1/1 Running 0 37s router-default-69684fbfff-cmc4c 1/1 Terminating 0 3m50s router-default-69684fbfff-xmzks 0/1 Terminating 0 3m50s melvinjoseph@mjoseph-mac Downloads % oc get route -A NAMESPACE NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD openshift-authentication oauth-openshift oauth-openshift.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com oauth-openshift 6443 passthrough/Redirect None openshift-console console console-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com console https reencrypt/Redirect None openshift-console downloads downloads-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com downloads http edge/Redirect None openshift-ingress-canary canary canary-openshift-ingress-canary.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ingress-canary 8080 edge/Redirect None openshift-monitoring alertmanager-main alertmanager-main-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api alertmanager-main web reencrypt/Redirect None openshift-monitoring prometheus-k8s prometheus-k8s-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api prometheus-k8s web reencrypt/Redirect None openshift-monitoring prometheus-k8s-federate prometheus-k8s-federate-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /federate prometheus-k8s web reencrypt/Redirect None openshift-monitoring thanos-querier thanos-querier-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api thanos-querier web reencrypt/Redirect None melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % oc get route -A NAMESPACE NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD openshift-authentication oauth-openshift oauth-openshift.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com oauth-openshift 6443 passthrough/Redirect None openshift-console console console-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com console https reencrypt/Redirect None openshift-console downloads downloads-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com downloads http edge/Redirect None openshift-ingress-canary canary canary-openshift-ingress-canary.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ingress-canary 8080 edge/Redirect None openshift-monitoring alertmanager-main alertmanager-main-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api alertmanager-main web reencrypt/Redirect None openshift-monitoring prometheus-k8s prometheus-k8s-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api prometheus-k8s web reencrypt/Redirect None openshift-monitoring prometheus-k8s-federate prometheus-k8s-federate-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /federate prometheus-k8s web reencrypt/Redirect None openshift-monitoring thanos-querier thanos-querier-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api thanos-querier web reencrypt/Redirect None melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress get pods NAME READY STATUS RESTARTS AGE router-default-6876749bfd-62zmg 1/1 Running 0 102s router-default-6876749bfd-7ds42 1/1 Running 0 102s melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-ingress-canary canary | jq '.status' { "ingress": [ { "conditions": [ { "lastTransitionTime": "2022-07-22T06:16:15Z", "status": "True", "type": "Admitted" } ], "host": "canary-openshift-ingress-canary.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerCanonicalHostname": "router-default.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerName": "default", "wildcardPolicy": "None" } ] } melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % oc create -f shard.yaml 'ingresscontroller.operator.openshift.io/sharded created melvinjoseph@mjoseph-mac Downloads % 'oc -n openshift-ingress get pods quote> melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress get pods NAME READY STATUS RESTARTS AGE router-default-6876749bfd-62zmg 1/1 Running 0 3m18s router-default-6876749bfd-7ds42 1/1 Running 0 3m18s router-sharded-7575575587-gl6gr 1/1 Running 0 21s router-sharded-7575575587-nvfzg 1/1 Running 0 21s melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-authentication oauth-openshift | jq '.status' { "ingress": [ { "conditions": [ { "lastTransitionTime": "2022-07-22T06:16:15Z", "status": "True", "type": "Admitted" } ], "host": "oauth-openshift.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerCanonicalHostname": "router-default.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerName": "default", "wildcardPolicy": "None" }, { "conditions": [ { "lastTransitionTime": "2022-07-22T06:19:13Z", "status": "True", "type": "Admitted" } ], "host": "oauth-openshift.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerCanonicalHostname": "router-sharded.shard.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerName": "sharded", "wildcardPolicy": "None" } ] } melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-ingress-canary canary | jq '.status' { "ingress": [ { "conditions": [ { "lastTransitionTime": "2022-07-22T06:16:15Z", "status": "True", "type": "Admitted" } ], "host": "canary-openshift-ingress-canary.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerCanonicalHostname": "router-default.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerName": "default", "wildcardPolicy": "None" }, { "conditions": [ { "lastTransitionTime": "2022-07-22T06:19:13Z", "status": "True", "type": "Admitted" } ], "host": "canary-openshift-ingress-canary.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerCanonicalHostname": "router-sharded.shard.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerName": "sharded", "wildcardPolicy": "None" } ] } melvinjoseph@mjoseph-mac Downloads % oc delete -n openshift-ingress-operator ingresscontroller sharded ingresscontroller.operator.openshift.io "sharded" deleted melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress get pods NAME READY STATUS RESTARTS AGE router-default-6876749bfd-62zmg 1/1 Running 0 5m39s router-default-6876749bfd-7ds42 1/1 Running 0 5m39s melvinjoseph@mjoseph-mac Downloads % oc get route -A NAMESPACE NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD openshift-authentication oauth-openshift oauth-openshift.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com oauth-openshift 6443 passthrough/Redirect None openshift-console console console-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com console https reencrypt/Redirect None openshift-console downloads downloads-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com downloads http edge/Redirect None openshift-ingress-canary canary canary-openshift-ingress-canary.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ingress-canary 8080 edge/Redirect None openshift-monitoring alertmanager-main alertmanager-main-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api alertmanager-main web reencrypt/Redirect None openshift-monitoring prometheus-k8s prometheus-k8s-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api prometheus-k8s web reencrypt/Redirect None openshift-monitoring prometheus-k8s-federate prometheus-k8s-federate-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /federate prometheus-k8s web reencrypt/Redirect None openshift-monitoring thanos-querier thanos-querier-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api thanos-querier web reencrypt/Redirect None melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-ingress-canary canary | jq '.status' { "ingress": [ { "conditions": [ { "lastTransitionTime": "2022-07-22T06:16:15Z", "status": "True", "type": "Admitted" } ], "host": "canary-openshift-ingress-canary.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerCanonicalHostname": "router-default.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerName": "default", "wildcardPolicy": "None" } ] } melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-authentication oauth-openshift | jq '.status' { "ingress": [ { "conditions": [ { "lastTransitionTime": "2022-07-22T06:16:15Z", "status": "True", "type": "Admitted" } ], "host": "oauth-openshift.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerCanonicalHostname": "router-default.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerName": "default", "wildcardPolicy": "None" } ] } melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % cat <<'EOF' | oc apply -f - apiVersion: operator.openshift.io/v1 kind: IngressController metadata: name: sharded namespace: openshift-ingress-operator spec: domain: shard.ipi-cluster.example.com nodePlacement: nodeSelector: matchLabels: node-role.kubernetes.io/worker: "" EOF ingresscontroller.operator.openshift.io/sharded created melvinjoseph@mjoseph-mac Downloads % oc get routes -A NAMESPACE NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD openshift-authentication oauth-openshift oauth-openshift.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more oauth-openshift 6443 passthrough/Redirect None openshift-console console console-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more console https reencrypt/Redirect None openshift-console downloads downloads-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more downloads http edge/Redirect None openshift-ingress-canary canary canary-openshift-ingress-canary.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more ingress-canary 8080 edge/Redirect None openshift-monitoring alertmanager-main alertmanager-main-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more /api alertmanager-main web reencrypt/Redirect None openshift-monitoring prometheus-k8s prometheus-k8s-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more /api prometheus-k8s web reencrypt/Redirect None openshift-monitoring prometheus-k8s-federate prometheus-k8s-federate-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more /federate prometheus-k8s web reencrypt/Redirect None openshift-monitoring thanos-querier thanos-querier-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more /api thanos-querier web reencrypt/Redirect None melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress get pods NAME READY STATUS RESTARTS AGE router-default-6876749bfd-62zmg 1/1 Running 0 6m23s router-default-6876749bfd-7ds42 1/1 Running 0 6m23s router-sharded-5756cd444c-8t4cg 1/1 Running 0 15s router-sharded-5756cd444c-gf5v8 1/1 Running 0 15s melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-authentication oauth-openshift | jq '.status' { "ingress": [ { "conditions": [ { "lastTransitionTime": "2022-07-22T06:16:15Z", "status": "True", "type": "Admitted" } ], "host": "oauth-openshift.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerCanonicalHostname": "router-default.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerName": "default", "wildcardPolicy": "None" }, { "conditions": [ { "lastTransitionTime": "2022-07-22T06:22:23Z", "status": "True", "type": "Admitted" } ], "host": "oauth-openshift.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerCanonicalHostname": "router-sharded.shard.ipi-cluster.example.com", "routerName": "sharded", "wildcardPolicy": "None" } ] } melvinjoseph@mjoseph-mac Downloads % oc scale ingresscontroller -n openshift-ingress-operator --replicas 1 sharded ingresscontroller.operator.openshift.io/sharded scaled melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-authentication oauth-openshift | jq '.status' { "ingress": [ { "conditions": [ { "lastTransitionTime": "2022-07-22T06:16:15Z", "status": "True", "type": "Admitted" } ], "host": "oauth-openshift.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerCanonicalHostname": "router-default.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerName": "default", "wildcardPolicy": "None" }, { "conditions": [ { "lastTransitionTime": "2022-07-22T06:22:23Z", "status": "True", "type": "Admitted" } ], "host": "oauth-openshift.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerCanonicalHostname": "router-sharded.shard.ipi-cluster.example.com", "routerName": "sharded", "wildcardPolicy": "None" } ] } melvinjoseph@mjoseph-mac Downloads % oc get routes -A NAMESPACE NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD openshift-authentication oauth-openshift oauth-openshift.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more oauth-openshift 6443 passthrough/Redirect None openshift-console console console-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more console https reencrypt/Redirect None openshift-console downloads downloads-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more downloads http edge/Redirect None openshift-ingress-canary canary canary-openshift-ingress-canary.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more ingress-canary 8080 edge/Redirect None openshift-monitoring alertmanager-main alertmanager-main-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more /api alertmanager-main web reencrypt/Redirect None openshift-monitoring prometheus-k8s prometheus-k8s-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more /api prometheus-k8s web reencrypt/Redirect None openshift-monitoring prometheus-k8s-federate prometheus-k8s-federate-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more /federate prometheus-k8s web reencrypt/Redirect None openshift-monitoring thanos-querier thanos-querier-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more /api thanos-querier web reencrypt/Redirect None melvinjoseph@mjoseph-mac Downloads % oc scale ingresscontroller -n openshift-ingress-operator --replicas 1 default ingresscontroller.operator.openshift.io/default scaled melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % oc get routes -A NAMESPACE NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD openshift-authentication oauth-openshift oauth-openshift.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more oauth-openshift 6443 passthrough/Redirect None openshift-console console console-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more console https reencrypt/Redirect None openshift-console downloads downloads-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more downloads http edge/Redirect None openshift-ingress-canary canary canary-openshift-ingress-canary.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more ingress-canary 8080 edge/Redirect None openshift-monitoring alertmanager-main alertmanager-main-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more /api alertmanager-main web reencrypt/Redirect None openshift-monitoring prometheus-k8s prometheus-k8s-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more /api prometheus-k8s web reencrypt/Redirect None openshift-monitoring prometheus-k8s-federate prometheus-k8s-federate-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more /federate prometheus-k8s web reencrypt/Redirect None openshift-monitoring thanos-querier thanos-querier-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ... 1 more /api thanos-querier web reencrypt/Redirect None melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-authentication oauth-openshift | jq '.status' { "ingress": [ { "conditions": [ { "lastTransitionTime": "2022-07-22T06:16:15Z", "status": "True", "type": "Admitted" } ], "host": "oauth-openshift.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerCanonicalHostname": "router-default.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerName": "default", "wildcardPolicy": "None" }, { "conditions": [ { "lastTransitionTime": "2022-07-22T06:22:23Z", "status": "True", "type": "Admitted" } ], "host": "oauth-openshift.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerCanonicalHostname": "router-sharded.shard.ipi-cluster.example.com", "routerName": "sharded", "wildcardPolicy": "None" } ] } melvinjoseph@mjoseph-mac Downloads % oc delete -n openshift-ingress-operator ingresscontroller sharded ingresscontroller.operator.openshift.io "sharded" deleted melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-authentication oauth-openshift | jq '.status' { "ingress": [ { "conditions": [ { "lastTransitionTime": "2022-07-22T06:16:15Z", "status": "True", "type": "Admitted" } ], "host": "oauth-openshift.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerCanonicalHostname": "router-default.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerName": "default", "wildcardPolicy": "None" } ] } melvinjoseph@mjoseph-mac Downloads % oc get routes -A NAMESPACE NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD openshift-authentication oauth-openshift oauth-openshift.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com oauth-openshift 6443 passthrough/Redirect None openshift-console console console-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com console https reencrypt/Redirect None openshift-console downloads downloads-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com downloads http edge/Redirect None openshift-ingress-canary canary canary-openshift-ingress-canary.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ingress-canary 8080 edge/Redirect None openshift-monitoring alertmanager-main alertmanager-main-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api alertmanager-main web reencrypt/Redirect None openshift-monitoring prometheus-k8s prometheus-k8s-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api prometheus-k8s web reencrypt/Redirect None openshift-monitoring prometheus-k8s-federate prometheus-k8s-federate-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /federate prometheus-k8s web reencrypt/Redirect None openshift-monitoring thanos-querier thanos-querier-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api thanos-querier web reencrypt/Redirect None melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-ingress-canary canary | jq '.status' { "ingress": [ { "conditions": [ { "lastTransitionTime": "2022-07-22T06:16:15Z", "status": "True", "type": "Admitted" } ], "host": "canary-openshift-ingress-canary.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerCanonicalHostname": "router-default.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerName": "default", "wildcardPolicy": "None" } ] } melvinjoseph@mjoseph-mac Downloads % oc scale ingresscontroller -n openshift-ingress-operator --replicas 2 default ingresscontroller.operator.openshift.io/default scaled melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % melvinjoseph@mjoseph-mac Downloads % oc get route -o json -n openshift-ingress-canary canary | jq '.status' { "ingress": [ { "conditions": [ { "lastTransitionTime": "2022-07-22T06:16:15Z", "status": "True", "type": "Admitted" } ], "host": "canary-openshift-ingress-canary.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerCanonicalHostname": "router-default.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com", "routerName": "default", "wildcardPolicy": "None" } ] } melvinjoseph@mjoseph-mac Downloads % oc get routes -A NAMESPACE NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD openshift-authentication oauth-openshift oauth-openshift.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com oauth-openshift 6443 passthrough/Redirect None openshift-console console console-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com console https reencrypt/Redirect None openshift-console downloads downloads-openshift-console.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com downloads http edge/Redirect None openshift-ingress-canary canary canary-openshift-ingress-canary.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com ingress-canary 8080 edge/Redirect None openshift-monitoring alertmanager-main alertmanager-main-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api alertmanager-main web reencrypt/Redirect None openshift-monitoring prometheus-k8s prometheus-k8s-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api prometheus-k8s web reencrypt/Redirect None openshift-monitoring prometheus-k8s-federate prometheus-k8s-federate-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /federate prometheus-k8s web reencrypt/Redirect None openshift-monitoring thanos-querier thanos-querier-openshift-monitoring.apps.ci-ln-hhbncbk-72292.origin-ci-int-gce.dev.rhcloud.com /api thanos-querier web reencrypt/Redirect None melvinjoseph@mjoseph-mac Downloads % oc -n openshift-ingress get pods NAME READY STATUS RESTARTS AGE router-default-6876749bfd-62zmg 1/1 Running 0 9m4s router-default-6876749bfd-z6m6b 1/1 Running 0 18s melvinjoseph@mjoseph-mac Downloads %
the bug fix is merged to 4.12.0-0.nightly-2022-07-24-180529 build and hence marking this as Verified as the bot is unable to change status.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:7399