+++ This bug was initially created as a clone of Bug #1854978 +++ Description of problem: ServiceCatalog CRDs still exists after upgrading to 4.5. Version-Release number of selected component (if applicable): 4.4.11 - 4.5.0-rc.7 How reproducible: always Steps to Reproduce: 1, Enable the ServiceCatalog on 4.4.11. mac:~ jianzhang$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.11 True False 9m19s Cluster version is 4.4.11 mac:~ jianzhang$ oc get servicecatalogapiserver NAME AGE cluster 29m mac:~ jianzhang$ oc get clusterservicebroker NAME URL STATUS AGE template-service-broker https://apiserver.openshift-template-service-broker.svc:443/brokers/template.openshift.io Ready 5m 2, Disable the ServiceCatalog. mac:~ jianzhang$ oc patch servicecatalogapiserver cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge servicecatalogapiserver.operator.openshift.io/cluster patched mac:~ jianzhang$ oc patch servicecatalogcontrollermanager cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge servicecatalogcontrollermanager.operator.openshift.io/cluster patched mac:~ jianzhang$ oc get clusterservicebroker Error from server (NotFound): Unable to list "servicecatalog.k8s.io/v1beta1, Resource=clusterservicebrokers": the server could not find the requested resource (get clusterservicebrokers.servicecatalog.k8s.io) 3, Upgrade it to the 4.5.0-rc.7 without the `--force`. mac:~ jianzhang$ oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release@sha256:f284efc07cb0bfa06cb7260ce4dedff4dabb55002377a7bcc637b58730976a3f --allow-explicit-upgrade warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade to the update to preceed anyway Updating to release image quay.io/openshift-release-dev/ocp-release@sha256:f284efc07cb0bfa06cb7260ce4dedff4dabb55002377a7bcc637b58730976a3f mac:~ jianzhang$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.11 True True 41s Working towards 4.5.0-rc.7: 11% complete mac:~ jianzhang$ oc get job -A NAMESPACE NAME COMPLETIONS DURATION AGE openshift-cluster-version version--76gxn 1/1 7s 37m mac:~ jianzhang$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.0-rc.7 True False 44m Cluster version is 4.5.0-rc.7 4, Check if the ServiceCatalog removed. I can see the removed Jobs. mac:~ jianzhang$ oc project openshift-service-catalog-removed Now using project "openshift-service-catalog-removed" on server "https://api.qe-jiazha-09.qe.devcluster.openshift.com:6443". mac:~ jianzhang$ oc get job NAME COMPLETIONS DURATION AGE openshift-service-catalog-apiserver-remover 1/1 6s 66m openshift-service-catalog-controller-manager-remover 1/1 6s 66m 5, Check if ServiceCatalog CRDs still exists. Actual results: mac:~ jianzhang$ oc get servicecatalogapiserver No resources found mac:~ jianzhang$ oc get crd servicecatalogapiservers.operator.openshift.io NAME CREATED AT servicecatalogapiservers.operator.openshift.io 2020-07-08T11:47:09Z mac:~ jianzhang$ oc get crd servicecatalogcontrollermanagers.operator.openshift.io NAME CREATED AT servicecatalogcontrollermanagers.operator.openshift.io 2020-07-08T11:47:09Z Expected results: These ServiceCatalog CRDs should be removed since they're don't work in 4.5. Additional info: Here is a cluster for debugging: https://mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com/job/Launch%20Environment%20Flexy/101128/artifact/workdir/install-dir/auth/kubeconfig/*view*/ mac:~ jianzhang$ oc get pods NAME READY STATUS RESTARTS AGE openshift-service-catalog-apiserver-remover-8f6st 0/1 Completed 0 65m openshift-service-catalog-controller-manager-remover-977w6 0/1 Completed 0 64m mac:~ jianzhang$ oc logs openshift-service-catalog-apiserver-remover-8f6st time="2020-07-08T13:02:25Z" level=info msg="Starting openshift-service-catalog-apiserver-remover job" time="2020-07-08T13:02:25Z" level=info msg="ServiceCatalogAPIServer managementState is 'Removed'" time="2020-07-08T13:02:25Z" level=info msg="Removing target namespace openshift-service-catalog-apiserver-operator" time="2020-07-08T13:02:25Z" level=info msg="Removing the ServiceCatalogAPIServer CR" time="2020-07-08T13:02:25Z" level=info msg="ServiceCatalogAPIServer cr removed successfully." time="2020-07-08T13:02:25Z" level=info msg="Removing the service-catalog-apiserver clusteroperator" time="2020-07-08T13:02:25Z" level=info msg="Removing ClusterRoleBinding: openshift-service-catalog-apiserver-operator" time="2020-07-08T13:02:25Z" level=info msg="Removing ClusterRole: openshift-service-catalog-apiserver-operator" time="2020-07-08T13:02:25Z" level=info msg="The openshift-service-catalog-apiserver-remover job, has finished." mac:~ jianzhang$ oc logs openshift-service-catalog-controller-manager-remover-977w6 time="2020-07-08T13:02:34Z" level=info msg="Starting openshift-service-catalog-controller-manager-remover job" time="2020-07-08T13:02:34Z" level=info msg="ServiceCatalogControllerManager managementState is 'Removed'" time="2020-07-08T13:02:34Z" level=info msg="Removing target namespace openshift-service-catalog-controller-manager-operator" time="2020-07-08T13:02:34Z" level=info msg="Removing the ServiceCatalogControllerManager CR" time="2020-07-08T13:02:34Z" level=info msg="ServiceCatalogControllerManager cr removed successfully." time="2020-07-08T13:02:34Z" level=info msg="Removing the service-catalog-controller-manager clusteroperator" time="2020-07-08T13:02:34Z" level=info msg="Removing ClusterRoleBinding: openshift-service-catalog-controller-manager-operator" time="2020-07-08T13:02:34Z" level=info msg="Removing ClusterRole: openshift-service-catalog-controller-manager-operator" time="2020-07-08T13:02:34Z" level=info msg="The openshift-service-catalog-controller-manater-remover job has finished." --- Additional comment from Jian Zhang on 2020-07-09 07:52:56 UTC --- I also have a try with a cluster that no ServiceCatalog/any brokers installed, but still encounter this issue. As follows: 1, Install OCP 4.4.11, disable ServiceCatalog and no brokers installed: [root@preserve-olm-env client]# ./oc get clusterservicebroker No resources found [root@preserve-olm-env client]# ./oc patch servicecatalogapiserver cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge servicecatalogapiserver.operator.openshift.io/cluster patched [root@preserve-olm-env client]# ./oc patch servicecatalogcontrollermanager cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge servicecatalogcontrollermanager.operator.openshift.io/cluster patched [root@preserve-olm-env client]# ./oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.11 True False 13m Cluster version is 4.4.11 2, Upgrade it to the OCP 4.5.0-rc.7 [root@preserve-olm-env client]# ./oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release@sha256:f284efc07cb0bfa06cb7260ce4dedff4dabb55002377a7bcc637b58730976a3f Updating to release image registry.svc.ci.openshift.org/ocp/release@sha256:f284efc07cb0bfa06cb7260ce4dedff4dabb55002377a7bcc637b58730976a3f [root@preserve-olm-env client]# ./oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.11 True True 37s Working towards 4.5.0-rc.7: 11% complete 3, Check the ServiceCatalog CRDs, they are still exist. They should be removed on OCP 4.5. [root@preserve-olm-env client]# ./oc get servicecatalogapiserver No resources found [root@preserve-olm-env client]# ./oc get crd|grep catalog catalogsources.operators.coreos.com 2020-07-09T01:58:09Z servicecatalogapiservers.operator.openshift.io 2020-07-09T01:57:52Z servicecatalogcontrollermanagers.operator.openshift.io 2020-07-09T01:57:52Z [root@preserve-olm-env client]# ./oc get job -n openshift-service-catalog-removed NAME COMPLETIONS DURATION AGE openshift-service-catalog-apiserver-remover 1/1 6s 13m openshift-service-catalog-controller-manager-remover 1/1 6s 13m [root@preserve-olm-env client]# ./oc get pods -n openshift-service-catalog-removed NAME READY STATUS RESTARTS AGE openshift-service-catalog-apiserver-remover-4lgll 0/1 Completed 0 13m openshift-service-catalog-controller-manager-remover-rd2mv 0/1 Completed 0 13m [root@preserve-olm-env client]# ./oc logs openshift-service-catalog-apiserver-remover-4lgll -n openshift-service-catalog-removed time="2020-07-09T06:49:59Z" level=info msg="Starting openshift-service-catalog-apiserver-remover job" time="2020-07-09T06:49:59Z" level=info msg="ServiceCatalogAPIServer managementState is 'Removed'" time="2020-07-09T06:49:59Z" level=info msg="Removing target namespace openshift-service-catalog-apiserver-operator" time="2020-07-09T06:49:59Z" level=info msg="Removing the ServiceCatalogAPIServer CR" time="2020-07-09T06:49:59Z" level=info msg="ServiceCatalogAPIServer cr removed successfully." time="2020-07-09T06:49:59Z" level=info msg="Removing the service-catalog-apiserver clusteroperator" time="2020-07-09T06:49:59Z" level=info msg="Removing ClusterRoleBinding: openshift-service-catalog-apiserver-operator" time="2020-07-09T06:49:59Z" level=info msg="Removing ClusterRole: openshift-service-catalog-apiserver-operator" time="2020-07-09T06:49:59Z" level=info msg="The openshift-service-catalog-apiserver-remover job, has finished." [root@preserve-olm-env client]# ./oc logs openshift-service-catalog-controller-manager-remover -n openshift-service-catalog-removed Error from server (NotFound): pods "openshift-service-catalog-controller-manager-remover" not found [root@preserve-olm-env client]# ./oc logs openshift-service-catalog-controller-manager-remover-rd2mv -n openshift-service-catalog-removed time="2020-07-09T06:50:05Z" level=info msg="Starting openshift-service-catalog-controller-manager-remover job" time="2020-07-09T06:50:05Z" level=info msg="ServiceCatalogControllerManager managementState is 'Removed'" time="2020-07-09T06:50:05Z" level=info msg="Removing target namespace openshift-service-catalog-controller-manager-operator" time="2020-07-09T06:50:05Z" level=info msg="Removing the ServiceCatalogControllerManager CR" time="2020-07-09T06:50:05Z" level=info msg="ServiceCatalogControllerManager cr removed successfully." time="2020-07-09T06:50:05Z" level=info msg="Removing the service-catalog-controller-manager clusteroperator" time="2020-07-09T06:50:05Z" level=info msg="Removing ClusterRoleBinding: openshift-service-catalog-controller-manager-operator" time="2020-07-09T06:50:05Z" level=info msg="Removing ClusterRole: openshift-service-catalog-controller-manager-operator" time="2020-07-09T06:50:05Z" level=info msg="The openshift-service-catalog-controller-manater-remover job has finished."
Closing this for 4.6 as the jobs are no longer being built for 4.6