Description of problem: ServiceCatalog CRDs still exists after upgrading to 4.5. Version-Release number of selected component (if applicable): 4.4.11 - 4.5.0-rc.7 How reproducible: always Steps to Reproduce: 1, Enable the ServiceCatalog on 4.4.11. mac:~ jianzhang$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.11 True False 9m19s Cluster version is 4.4.11 mac:~ jianzhang$ oc get servicecatalogapiserver NAME AGE cluster 29m mac:~ jianzhang$ oc get clusterservicebroker NAME URL STATUS AGE template-service-broker https://apiserver.openshift-template-service-broker.svc:443/brokers/template.openshift.io Ready 5m 2, Disable the ServiceCatalog. mac:~ jianzhang$ oc patch servicecatalogapiserver cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge servicecatalogapiserver.operator.openshift.io/cluster patched mac:~ jianzhang$ oc patch servicecatalogcontrollermanager cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge servicecatalogcontrollermanager.operator.openshift.io/cluster patched mac:~ jianzhang$ oc get clusterservicebroker Error from server (NotFound): Unable to list "servicecatalog.k8s.io/v1beta1, Resource=clusterservicebrokers": the server could not find the requested resource (get clusterservicebrokers.servicecatalog.k8s.io) 3, Upgrade it to the 4.5.0-rc.7 without the `--force`. mac:~ jianzhang$ oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release@sha256:f284efc07cb0bfa06cb7260ce4dedff4dabb55002377a7bcc637b58730976a3f --allow-explicit-upgrade warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade to the update to preceed anyway Updating to release image quay.io/openshift-release-dev/ocp-release@sha256:f284efc07cb0bfa06cb7260ce4dedff4dabb55002377a7bcc637b58730976a3f mac:~ jianzhang$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.11 True True 41s Working towards 4.5.0-rc.7: 11% complete mac:~ jianzhang$ oc get job -A NAMESPACE NAME COMPLETIONS DURATION AGE openshift-cluster-version version--76gxn 1/1 7s 37m mac:~ jianzhang$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.0-rc.7 True False 44m Cluster version is 4.5.0-rc.7 4, Check if the ServiceCatalog removed. I can see the removed Jobs. mac:~ jianzhang$ oc project openshift-service-catalog-removed Now using project "openshift-service-catalog-removed" on server "https://api.qe-jiazha-09.qe.devcluster.openshift.com:6443". mac:~ jianzhang$ oc get job NAME COMPLETIONS DURATION AGE openshift-service-catalog-apiserver-remover 1/1 6s 66m openshift-service-catalog-controller-manager-remover 1/1 6s 66m 5, Check if ServiceCatalog CRDs still exists. Actual results: mac:~ jianzhang$ oc get servicecatalogapiserver No resources found mac:~ jianzhang$ oc get crd servicecatalogapiservers.operator.openshift.io NAME CREATED AT servicecatalogapiservers.operator.openshift.io 2020-07-08T11:47:09Z mac:~ jianzhang$ oc get crd servicecatalogcontrollermanagers.operator.openshift.io NAME CREATED AT servicecatalogcontrollermanagers.operator.openshift.io 2020-07-08T11:47:09Z Expected results: These ServiceCatalog CRDs should be removed since they're don't work in 4.5. Additional info: Here is a cluster for debugging: https://mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com/job/Launch%20Environment%20Flexy/101128/artifact/workdir/install-dir/auth/kubeconfig/*view*/ mac:~ jianzhang$ oc get pods NAME READY STATUS RESTARTS AGE openshift-service-catalog-apiserver-remover-8f6st 0/1 Completed 0 65m openshift-service-catalog-controller-manager-remover-977w6 0/1 Completed 0 64m mac:~ jianzhang$ oc logs openshift-service-catalog-apiserver-remover-8f6st time="2020-07-08T13:02:25Z" level=info msg="Starting openshift-service-catalog-apiserver-remover job" time="2020-07-08T13:02:25Z" level=info msg="ServiceCatalogAPIServer managementState is 'Removed'" time="2020-07-08T13:02:25Z" level=info msg="Removing target namespace openshift-service-catalog-apiserver-operator" time="2020-07-08T13:02:25Z" level=info msg="Removing the ServiceCatalogAPIServer CR" time="2020-07-08T13:02:25Z" level=info msg="ServiceCatalogAPIServer cr removed successfully." time="2020-07-08T13:02:25Z" level=info msg="Removing the service-catalog-apiserver clusteroperator" time="2020-07-08T13:02:25Z" level=info msg="Removing ClusterRoleBinding: openshift-service-catalog-apiserver-operator" time="2020-07-08T13:02:25Z" level=info msg="Removing ClusterRole: openshift-service-catalog-apiserver-operator" time="2020-07-08T13:02:25Z" level=info msg="The openshift-service-catalog-apiserver-remover job, has finished." mac:~ jianzhang$ oc logs openshift-service-catalog-controller-manager-remover-977w6 time="2020-07-08T13:02:34Z" level=info msg="Starting openshift-service-catalog-controller-manager-remover job" time="2020-07-08T13:02:34Z" level=info msg="ServiceCatalogControllerManager managementState is 'Removed'" time="2020-07-08T13:02:34Z" level=info msg="Removing target namespace openshift-service-catalog-controller-manager-operator" time="2020-07-08T13:02:34Z" level=info msg="Removing the ServiceCatalogControllerManager CR" time="2020-07-08T13:02:34Z" level=info msg="ServiceCatalogControllerManager cr removed successfully." time="2020-07-08T13:02:34Z" level=info msg="Removing the service-catalog-controller-manager clusteroperator" time="2020-07-08T13:02:34Z" level=info msg="Removing ClusterRoleBinding: openshift-service-catalog-controller-manager-operator" time="2020-07-08T13:02:34Z" level=info msg="Removing ClusterRole: openshift-service-catalog-controller-manager-operator" time="2020-07-08T13:02:34Z" level=info msg="The openshift-service-catalog-controller-manater-remover job has finished."
I also have a try with a cluster that no ServiceCatalog/any brokers installed, but still encounter this issue. As follows: 1, Install OCP 4.4.11, disable ServiceCatalog and no brokers installed: [root@preserve-olm-env client]# ./oc get clusterservicebroker No resources found [root@preserve-olm-env client]# ./oc patch servicecatalogapiserver cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge servicecatalogapiserver.operator.openshift.io/cluster patched [root@preserve-olm-env client]# ./oc patch servicecatalogcontrollermanager cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge servicecatalogcontrollermanager.operator.openshift.io/cluster patched [root@preserve-olm-env client]# ./oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.11 True False 13m Cluster version is 4.4.11 2, Upgrade it to the OCP 4.5.0-rc.7 [root@preserve-olm-env client]# ./oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release@sha256:f284efc07cb0bfa06cb7260ce4dedff4dabb55002377a7bcc637b58730976a3f Updating to release image registry.svc.ci.openshift.org/ocp/release@sha256:f284efc07cb0bfa06cb7260ce4dedff4dabb55002377a7bcc637b58730976a3f [root@preserve-olm-env client]# ./oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.11 True True 37s Working towards 4.5.0-rc.7: 11% complete 3, Check the ServiceCatalog CRDs, they are still exist. They should be removed on OCP 4.5. [root@preserve-olm-env client]# ./oc get servicecatalogapiserver No resources found [root@preserve-olm-env client]# ./oc get crd|grep catalog catalogsources.operators.coreos.com 2020-07-09T01:58:09Z servicecatalogapiservers.operator.openshift.io 2020-07-09T01:57:52Z servicecatalogcontrollermanagers.operator.openshift.io 2020-07-09T01:57:52Z [root@preserve-olm-env client]# ./oc get job -n openshift-service-catalog-removed NAME COMPLETIONS DURATION AGE openshift-service-catalog-apiserver-remover 1/1 6s 13m openshift-service-catalog-controller-manager-remover 1/1 6s 13m [root@preserve-olm-env client]# ./oc get pods -n openshift-service-catalog-removed NAME READY STATUS RESTARTS AGE openshift-service-catalog-apiserver-remover-4lgll 0/1 Completed 0 13m openshift-service-catalog-controller-manager-remover-rd2mv 0/1 Completed 0 13m [root@preserve-olm-env client]# ./oc logs openshift-service-catalog-apiserver-remover-4lgll -n openshift-service-catalog-removed time="2020-07-09T06:49:59Z" level=info msg="Starting openshift-service-catalog-apiserver-remover job" time="2020-07-09T06:49:59Z" level=info msg="ServiceCatalogAPIServer managementState is 'Removed'" time="2020-07-09T06:49:59Z" level=info msg="Removing target namespace openshift-service-catalog-apiserver-operator" time="2020-07-09T06:49:59Z" level=info msg="Removing the ServiceCatalogAPIServer CR" time="2020-07-09T06:49:59Z" level=info msg="ServiceCatalogAPIServer cr removed successfully." time="2020-07-09T06:49:59Z" level=info msg="Removing the service-catalog-apiserver clusteroperator" time="2020-07-09T06:49:59Z" level=info msg="Removing ClusterRoleBinding: openshift-service-catalog-apiserver-operator" time="2020-07-09T06:49:59Z" level=info msg="Removing ClusterRole: openshift-service-catalog-apiserver-operator" time="2020-07-09T06:49:59Z" level=info msg="The openshift-service-catalog-apiserver-remover job, has finished." [root@preserve-olm-env client]# ./oc logs openshift-service-catalog-controller-manager-remover -n openshift-service-catalog-removed Error from server (NotFound): pods "openshift-service-catalog-controller-manager-remover" not found [root@preserve-olm-env client]# ./oc logs openshift-service-catalog-controller-manager-remover-rd2mv -n openshift-service-catalog-removed time="2020-07-09T06:50:05Z" level=info msg="Starting openshift-service-catalog-controller-manager-remover job" time="2020-07-09T06:50:05Z" level=info msg="ServiceCatalogControllerManager managementState is 'Removed'" time="2020-07-09T06:50:05Z" level=info msg="Removing target namespace openshift-service-catalog-controller-manager-operator" time="2020-07-09T06:50:05Z" level=info msg="Removing the ServiceCatalogControllerManager CR" time="2020-07-09T06:50:05Z" level=info msg="ServiceCatalogControllerManager cr removed successfully." time="2020-07-09T06:50:05Z" level=info msg="Removing the service-catalog-controller-manager clusteroperator" time="2020-07-09T06:50:05Z" level=info msg="Removing ClusterRoleBinding: openshift-service-catalog-controller-manager-operator" time="2020-07-09T06:50:05Z" level=info msg="Removing ClusterRole: openshift-service-catalog-controller-manager-operator" time="2020-07-09T06:50:05Z" level=info msg="The openshift-service-catalog-controller-manater-remover job has finished."
1, Install the 4.4.21 cluster [root@preserve-olm-env data]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.21 True False 60m Cluster version is 4.4.21 2, Disable the ServiceCatalog. [root@preserve-olm-env data]# oc patch servicecatalogapiserver cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge servicecatalogapiserver.operator.openshift.io/cluster patched [root@preserve-olm-env data]# oc patch servicecatalogcontrollermanager cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge servicecatalogcontrollermanager.operator.openshift.io/cluster patched [root@preserve-olm-env data]# oc get clusterservicebroker Error from server (ServiceUnavailable): the server is currently unable to handle the request (get clusterservicebrokers.servicecatalog.k8s.io) 3, Upgrade it to the registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-09-12-063044 that contains the fixed PR. [root@preserve-olm-env data]# oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release@sha256:ab39c801730e87c903757c0d3e06dd0fcad81669e28e61eb3a82103db79d6d04 --allow-explicit-upgrade warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade to the update to proceed anyway Updating to release image registry.svc.ci.openshift.org/ocp/release@sha256:ab39c801730e87c903757c0d3e06dd0fcad81669e28e61eb3a82103db79d6d04 This new payload has NOT been sigined, we cannot upgrade it without `--force`, it's unavaliable for us to test it now. Change the status to MODIFIED. [root@preserve-olm-env data]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.21 True True 16s Unable to apply registry.svc.ci.openshift.org/ocp/release@sha256:ab39c801730e87c903757c0d3e06dd0fcad81669e28e61eb3a82103db79d6d04: the image may not be safe to use
1, Install the 4.4.22 cluster [root@preserve-olm-env data]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.22 True False 68m Cluster version is 4.4.22 [root@preserve-olm-env data]# oc get clusterservicebroker NAME URL STATUS AGE template-service-broker https://apiserver.openshift-template-service-broker.svc:443/brokers/template.openshift.io Ready 1h 2, Disable the ServiceCatalog. [root@preserve-olm-env data]# oc patch servicecatalogapiserver cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge servicecatalogapiserver.operator.openshift.io/cluster patched [root@preserve-olm-env data]# oc patch servicecatalogcontrollermanager cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge servicecatalogcontrollermanager.operator.openshift.io/cluster patched [root@preserve-olm-env data]# oc get clusterservicebroker Error from server (NotFound): Unable to list "servicecatalog.k8s.io/v1beta1, Resource=clusterservicebrokers": the server could not find the requested resource (get clusterservicebrokers.servicecatalog.k8s.io) 3, Upgrade it to 4.5.10 [root@preserve-olm-env data]# oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release@sha256:272a00b3dc91e1a8e45370969d4300f17bc89bc5cefb0f938f6c1b69333fe371 --allow-explicit-upgrade warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade to the update to proceed anyway Updating to release image quay.io/openshift-release-dev/ocp-release@sha256:272a00b3dc91e1a8e45370969d4300f17bc89bc5cefb0f938f6c1b69333fe371 [root@preserve-olm-env data]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.22 True True 51s Working towards 4.5.10: 15% complete [root@preserve-olm-env data]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.10 True False 3m22s Cluster version is 4.5.10 4, Check the ServiceCatalog CRDs if still exist. Failed to get them. LGTM, verify it. [root@preserve-olm-env data]# oc get servicecatalogapiserver error: the server doesn't have a resource type "servicecatalogapiserver" [root@preserve-olm-env data]# oc get servicecatalogcontrollermanager error: the server doesn't have a resource type "servicecatalogcontrollermanager"
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.5.13 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3760