Description of problem: when all workloads including services, deployment and routes inside openshift-console namespace are removed, co/console is still reporting Available: True while console is totally disabled Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. set console managementState as Removed, all workloads in openshift-console namespace such as console deployment, service and route will be entirely removed $ oc get console.operator cluster -o json | jq .spec { "logLevel": "Normal", "managementState": "Removed", "operatorLogLevel": "Debug" } 2. Check co/console status $ while true;do oc get all -n openshift-console | grep console; oc get co | grep console; sleep 5; done No resources found in openshift-console namespace. console 4.9.0-0.nightly-2021-09-05-192114 True False False 27m No resources found in openshift-console namespace. console 4.9.0-0.nightly-2021-09-05-192114 True False False 27m No resources found in openshift-console namespace. console 4.9.0-0.nightly-2021-09-05-192114 True False False 27m No resources found in openshift-console namespace. console 4.9.0-0.nightly-2021-09-05-192114 True False False 27m No resources found in openshift-console namespace. console 4.9.0-0.nightly-2021-09-05-192114 True False False 28m No resources found in openshift-console namespace. console 4.9.0-0.nightly-2021-09-05-192114 True False False 28m No resources found in openshift-console namespace. console 4.9.0-0.nightly-2021-09-05-192114 True False False 28m No resources found in openshift-console namespace. console 4.9.0-0.nightly-2021-09-05-192114 True False False 28m No resources found in openshift-console namespace. console 4.9.0-0.nightly-2021-09-05-192114 True False False 28m No resources found in openshift-console namespace. console 4.9.0-0.nightly-2021-09-05-192114 True False False 28m No resources found in openshift-console namespace. console 4.9.0-0.nightly-2021-09-05-192114 True False False 28m No resources found in openshift-console namespace. console 4.9.0-0.nightly-2021-09-05-192114 True False False 28m No resources found in openshift-console namespace. console 4.9.0-0.nightly-2021-09-05-192114 True False False 29m No resources found in openshift-console namespace. console 4.9.0-0.nightly-2021-09-05-192114 True False False 29m No resources found in openshift-console namespace. console 4.9.0-0.nightly-2021-09-05-192114 True False False 29m Check console-operator logs: I0906 06:17:38.219270 1 controller.go:99] console is in a removed state: deleting ConsoleCliDownloads custom resources I0906 06:17:38.219732 1 controller.go:97] console-operator is in a removed state: deleting "downloads" service I0906 06:17:38.220376 1 controller.go:95] console is in an removed state: removing synced downloads deployment I0906 06:17:38.221685 1 operator.go:243] console has been removed. I0906 06:17:38.221701 1 operator.go:254] deleting console resources I0906 06:17:38.224760 1 controller.go:97] console-operator is in a removed state: deleting "console" service I0906 06:17:38.226293 1 controller.go:137] finished deleting ConsoleCliDownloads custom resources E0906 06:17:38.226724 1 base_controller.go:251] "ConsoleDownloadsDeploymentSyncController" controller failed to sync "key", err: deployments.apps "downloads" not found I0906 06:17:38.226884 1 reflector.go:381] github.com/openshift/client-go/oauth/informers/externalversions/factory.go:101: forcing resync I0906 06:17:38.226906 1 reflector.go:381] k8s.io/client-go/informers/factory.go:134: forcing resync I0906 06:17:38.226928 1 reflector.go:381] k8s.io/client-go/informers/factory.go:134: forcing resync I0906 06:17:38.226956 1 reflector.go:381] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: forcing resync I0906 06:17:38.230116 1 controller.go:99] console is in a removed state: deleting ConsoleCliDownloads custom resources I0906 06:17:38.230311 1 reflector.go:381] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: forcing resync I0906 06:17:38.237610 1 controller.go:137] finished deleting ConsoleCliDownloads custom resources I0906 06:17:38.243785 1 reflector.go:381] k8s.io/client-go/informers/factory.go:134: forcing resync I0906 06:17:38.254859 1 operator.go:276] finished deleting console resources I0906 06:17:38.254875 1 operator.go:229] finished syncing operator "cluster" (22.519µs) I0906 06:17:38.395944 1 request.go:600] Waited for 169.098818ms due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1/consoles/cluster I0906 06:17:38.399730 1 controller.go:95] console is in an removed state: removing synced downloads deployment E0906 06:17:38.402286 1 base_controller.go:251] "ConsoleDownloadsDeploymentSyncController" controller failed to sync "key", err: deployments.apps "downloads" not found I0906 06:17:38.596052 1 request.go:600] Waited for 366.520012ms due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1/consoles/cluster I0906 06:17:38.602627 1 controller.go:111] console-operator is in a removed state: deleting "console" route I0906 06:17:38.795633 1 request.go:600] Waited for 565.236754ms due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1/consoles/cluster I0906 06:17:38.800517 1 controller.go:97] console-operator is in a removed state: deleting "downloads" service I0906 06:17:38.995906 1 request.go:600] Waited for 763.809168ms due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1/consoles/cluster I0906 06:17:38.999645 1 controller.go:111] console-operator is in a removed state: deleting "downloads" route I0906 06:17:39.195848 1 request.go:600] Waited for 958.181465ms due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1/consoles/cluster I0906 06:17:39.199715 1 controller.go:97] console-operator is in a removed state: deleting "console" service I0906 06:17:39.395922 1 request.go:600] Waited for 1.140995178s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1/consoles/cluster I0906 06:17:39.395940 1 request.go:668] Waited for 1.140995178s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1/consoles/cluster I0906 06:17:39.407964 1 operator.go:181] started syncing operator "cluster" (2021-09-06 06:17:39.407955857 +0000 UTC m=+22870.770623255) I0906 06:17:39.421543 1 operator.go:243] console has been removed. I0906 06:17:39.421563 1 operator.go:254] deleting console resources I0906 06:17:39.445637 1 operator.go:276] finished deleting console resources I0906 06:17:39.445653 1 operator.go:229] finished syncing operator "cluster" (28.604µs) I0906 06:17:39.595886 1 request.go:600] Waited for 1.010927579s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1/consoles/cluster I0906 06:17:39.600289 1 controller.go:97] console-operator is in a removed state: skipping health checks I0906 06:17:39.795975 1 request.go:600] Waited for 1.176406105s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/operator.openshift.io/v1/consoles/cluster I0906 06:17:39.799882 1 controller.go:111] console-operator is in a removed state: deleting "console" route I0906 06:17:44.097611 1 controller.go:95] console is in an removed state: removing synced downloads deployment E0906 06:17:44.101325 1 base_controller.go:251] "ConsoleDownloadsDeploymentSyncController" controller failed to sync "key", err: deployments.apps "downloads" not found I0906 06:17:48.107616 1 httplog.go:89] "HTTP" verb="GET" URI="/healthz" latency="136.121µs" userAgent="kube-probe/1.22+" srcIP="10.130.0.1:43888" resp=200 I0906 06:17:48.111398 1 httplog.go:89] "HTTP" verb="GET" URI="/readyz" latency="4.380768ms" userAgent="kube-probe/1.22+" srcIP="10.130.0.1:43890" resp=200 I0906 06:17:55.227930 1 httplog.go:89] "HTTP" verb="GET" URI="/metrics" latency="5.034854ms" userAgent="Prometheus/2.29.2" srcIP="10.128.2.18:33640" resp=200 I0906 06:17:58.110608 1 httplog.go:89] "HTTP" verb="GET" URI="/healthz" latency="145.334µs" userAgent="kube-probe/1.22+" srcIP="10.130.0.1:44070" resp=200 I0906 06:17:58.110868 1 httplog.go:89] "HTTP" verb="GET" URI="/readyz" latency="2.160809ms" userAgent="kube-probe/1.22+" srcIP="10.130.0.1:44068" resp=200 I0906 06:18:07.860377 1 httplog.go:89] "HTTP" verb="GET" URI="/metrics" latency="7.130971ms" userAgent="Prometheus/2.29.2" srcIP="10.129.2.11:50168" resp=200 I0906 06:18:08.106778 1 httplog.go:89] "HTTP" verb="GET" URI="/readyz" latency="430.825µs" userAgent="kube-probe/1.22+" srcIP="10.130.0.1:44270" resp=200 I0906 06:18:08.106966 1 httplog.go:89] "HTTP" verb="GET" URI="/healthz" latency="111.263µs" userAgent="kube-probe/1.22+" srcIP="10.130.0.1:44268" resp=200 I0906 06:18:18.110641 1 httplog.go:89] "HTTP" verb="GET" URI="/healthz" latency="129.717µs" userAgent="kube-probe/1.22+" srcIP="10.130.0.1:44462" resp=200 I0906 06:18:18.112941 1 httplog.go:89] "HTTP" verb="GET" URI="/readyz" latency="4.288098ms" userAgent="kube-probe/1.22+" srcIP="10.130.0.1:44464" resp=200 I0906 06:18:25.232542 1 httplog.go:89] "HTTP" verb="GET" URI="/metrics" latency="9.188576ms" userAgent="Prometheus/2.29.2" srcIP="10.128.2.18:33640" resp=200 I0906 06:18:28.107569 1 httplog.go:89] "HTTP" verb="GET" URI="/healthz" latency="157.81µs" userAgent="kube-probe/1.22+" srcIP="10.130.0.1:44642" resp=200 I0906 06:18:28.108827 1 httplog.go:89] "HTTP" verb="GET" URI="/readyz" latency="2.33758ms" userAgent="kube-probe/1.22+" srcIP="10.130.0.1:44644" resp=200 I0906 06:18:37.864605 1 httplog.go:89] "HTTP" verb="GET" URI="/metrics" latency="11.181203ms" userAgent="Prometheus/2.29.2" srcIP="10.129.2.11:50168" resp=200 I0906 06:18:38.107179 1 httplog.go:89] "HTTP" verb="GET" URI="/healthz" latency="325.339µs" userAgent="kube-probe/1.22+" srcIP="10.130.0.1:44830" resp=200 I0906 06:18:38.107448 1 httplog.go:89] "HTTP" verb="GET" URI="/readyz" latency="97.67µs" userAgent="kube-probe/1.22+" srcIP="10.130.0.1:44832" resp=200 I0906 06:18:38.589726 1 controller.go:97] console-operator is in a removed state: skipping health checks Actual results: 2. co/console is reporting Available: True even console is disabled Expected results: 2. co/console should report Available: False when console is totally removed/disabled Additional info:
Console MUST report `Available: true` even when removed, otherwise the cluster will not be upgradeable. See Trevor's comment here: https://github.com/openshift/console-operator/pull/584#issuecomment-916228604 Available in this context means that "everything my component is configured to be doing, it's successfully doing."
Thank you everyone for looking at this ~