Bug 1856978

Summary: CRDs still exist after upgraded to the 4.5.rc.7 from 4.4.11
Product: OpenShift Container Platform Reporter: Jesus M. Rodriguez <jesusr>
Component: Operator SDKAssignee: Joe Lanford <jlanford>
Status: CLOSED CURRENTRELEASE QA Contact: Fan Jia <jfan>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.5CC: adellape, aos-bugs, jiazha, marobrie
Target Milestone: ---Keywords: UpcomingSprint
Target Release: 4.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 1854978 Environment:
Last Closed: 2020-07-14 20:23:05 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1854978    

Description Jesus M. Rodriguez 2020-07-14 20:21:59 UTC 
+++ This bug was initially created as a clone of Bug #1854978 +++

Description of problem:
ServiceCatalog CRDs still exists after upgrading to 4.5.

Version-Release number of selected component (if applicable):
4.4.11 - 4.5.0-rc.7

How reproducible:
always

Steps to Reproduce:
1, Enable the ServiceCatalog on 4.4.11.
mac:~ jianzhang$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.11    True        False         9m19s   Cluster version is 4.4.11
mac:~ jianzhang$ oc get servicecatalogapiserver
NAME      AGE
cluster   29m
mac:~ jianzhang$ oc get clusterservicebroker
NAME                      URL                                                                                         STATUS   AGE
template-service-broker   https://apiserver.openshift-template-service-broker.svc:443/brokers/template.openshift.io   Ready    5m


2, Disable the ServiceCatalog.
mac:~ jianzhang$ oc patch servicecatalogapiserver cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge
servicecatalogapiserver.operator.openshift.io/cluster patched
mac:~ jianzhang$ oc patch servicecatalogcontrollermanager cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge
servicecatalogcontrollermanager.operator.openshift.io/cluster patched

mac:~ jianzhang$ oc get clusterservicebroker
Error from server (NotFound): Unable to list "servicecatalog.k8s.io/v1beta1, Resource=clusterservicebrokers": the server could not find the requested resource (get clusterservicebrokers.servicecatalog.k8s.io)

3, Upgrade it to the 4.5.0-rc.7 without the `--force`.
mac:~ jianzhang$ oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release@sha256:f284efc07cb0bfa06cb7260ce4dedff4dabb55002377a7bcc637b58730976a3f --allow-explicit-upgrade
warning: The requested upgrade image is not one of the available updates.  You have used --allow-explicit-upgrade to the update to preceed anyway
Updating to release image quay.io/openshift-release-dev/ocp-release@sha256:f284efc07cb0bfa06cb7260ce4dedff4dabb55002377a7bcc637b58730976a3f

mac:~ jianzhang$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.11    True        True          41s     Working towards 4.5.0-rc.7: 11% complete

mac:~ jianzhang$ oc get job -A
NAMESPACE                   NAME             COMPLETIONS   DURATION   AGE
openshift-cluster-version   version--76gxn   1/1           7s         37m

mac:~ jianzhang$ oc get clusterversion
NAME      VERSION      AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-rc.7   True        False         44m     Cluster version is 4.5.0-rc.7

4, Check if the ServiceCatalog removed. I can see the removed Jobs.
mac:~ jianzhang$ oc project openshift-service-catalog-removed
Now using project "openshift-service-catalog-removed" on server "https://api.qe-jiazha-09.qe.devcluster.openshift.com:6443".

mac:~ jianzhang$ oc get job
NAME                                                   COMPLETIONS   DURATION   AGE
openshift-service-catalog-apiserver-remover            1/1           6s         66m
openshift-service-catalog-controller-manager-remover   1/1           6s         66m

5, Check if ServiceCatalog CRDs still exists.

Actual results:
mac:~ jianzhang$ oc get servicecatalogapiserver
No resources found
mac:~ jianzhang$ oc get crd servicecatalogapiservers.operator.openshift.io
NAME                                             CREATED AT
servicecatalogapiservers.operator.openshift.io   2020-07-08T11:47:09Z
mac:~ jianzhang$ oc get crd servicecatalogcontrollermanagers.operator.openshift.io
NAME                                                     CREATED AT
servicecatalogcontrollermanagers.operator.openshift.io   2020-07-08T11:47:09Z


Expected results:
These ServiceCatalog CRDs should be removed since they're don't work in 4.5. 


Additional info:
Here is a cluster for debugging: https://mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com/job/Launch%20Environment%20Flexy/101128/artifact/workdir/install-dir/auth/kubeconfig/*view*/

mac:~ jianzhang$ oc get pods
NAME                                                         READY   STATUS      RESTARTS   AGE
openshift-service-catalog-apiserver-remover-8f6st            0/1     Completed   0          65m
openshift-service-catalog-controller-manager-remover-977w6   0/1     Completed   0          64m
mac:~ jianzhang$ oc logs openshift-service-catalog-apiserver-remover-8f6st
time="2020-07-08T13:02:25Z" level=info msg="Starting openshift-service-catalog-apiserver-remover job"
time="2020-07-08T13:02:25Z" level=info msg="ServiceCatalogAPIServer managementState is 'Removed'"
time="2020-07-08T13:02:25Z" level=info msg="Removing target namespace openshift-service-catalog-apiserver-operator"
time="2020-07-08T13:02:25Z" level=info msg="Removing the ServiceCatalogAPIServer CR"
time="2020-07-08T13:02:25Z" level=info msg="ServiceCatalogAPIServer cr removed successfully."
time="2020-07-08T13:02:25Z" level=info msg="Removing the service-catalog-apiserver clusteroperator"
time="2020-07-08T13:02:25Z" level=info msg="Removing ClusterRoleBinding: openshift-service-catalog-apiserver-operator"
time="2020-07-08T13:02:25Z" level=info msg="Removing ClusterRole: openshift-service-catalog-apiserver-operator"
time="2020-07-08T13:02:25Z" level=info msg="The openshift-service-catalog-apiserver-remover job, has finished."
mac:~ jianzhang$ oc logs openshift-service-catalog-controller-manager-remover-977w6
time="2020-07-08T13:02:34Z" level=info msg="Starting openshift-service-catalog-controller-manager-remover job"
time="2020-07-08T13:02:34Z" level=info msg="ServiceCatalogControllerManager managementState is 'Removed'"
time="2020-07-08T13:02:34Z" level=info msg="Removing target namespace openshift-service-catalog-controller-manager-operator"
time="2020-07-08T13:02:34Z" level=info msg="Removing the ServiceCatalogControllerManager CR"
time="2020-07-08T13:02:34Z" level=info msg="ServiceCatalogControllerManager cr removed successfully."
time="2020-07-08T13:02:34Z" level=info msg="Removing the service-catalog-controller-manager clusteroperator"
time="2020-07-08T13:02:34Z" level=info msg="Removing ClusterRoleBinding: openshift-service-catalog-controller-manager-operator"
time="2020-07-08T13:02:34Z" level=info msg="Removing ClusterRole: openshift-service-catalog-controller-manager-operator"
time="2020-07-08T13:02:34Z" level=info msg="The openshift-service-catalog-controller-manater-remover job has finished."

--- Additional comment from Jian Zhang on 2020-07-09 07:52:56 UTC ---

I also have a try with a cluster that no ServiceCatalog/any brokers installed, but still encounter this issue. As follows:

1, Install OCP 4.4.11, disable ServiceCatalog and no brokers installed:
[root@preserve-olm-env client]# ./oc get clusterservicebroker
No resources found
[root@preserve-olm-env client]# ./oc patch servicecatalogapiserver cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge
servicecatalogapiserver.operator.openshift.io/cluster patched
[root@preserve-olm-env client]# ./oc patch servicecatalogcontrollermanager cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge
servicecatalogcontrollermanager.operator.openshift.io/cluster patched
[root@preserve-olm-env client]# ./oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.11    True        False         13m     Cluster version is 4.4.11

2, Upgrade it to the OCP 4.5.0-rc.7
[root@preserve-olm-env client]# ./oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release@sha256:f284efc07cb0bfa06cb7260ce4dedff4dabb55002377a7bcc637b58730976a3f
Updating to release image registry.svc.ci.openshift.org/ocp/release@sha256:f284efc07cb0bfa06cb7260ce4dedff4dabb55002377a7bcc637b58730976a3f
[root@preserve-olm-env client]# ./oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.11    True        True          37s     Working towards 4.5.0-rc.7: 11% complete

3, Check the ServiceCatalog CRDs, they are still exist. They should be removed on OCP 4.5.
[root@preserve-olm-env client]# ./oc get servicecatalogapiserver
No resources found
[root@preserve-olm-env client]# ./oc get crd|grep catalog
catalogsources.operators.coreos.com                         2020-07-09T01:58:09Z
servicecatalogapiservers.operator.openshift.io              2020-07-09T01:57:52Z
servicecatalogcontrollermanagers.operator.openshift.io      2020-07-09T01:57:52Z
[root@preserve-olm-env client]# ./oc get job -n openshift-service-catalog-removed 
NAME                                                   COMPLETIONS   DURATION   AGE
openshift-service-catalog-apiserver-remover            1/1           6s         13m
openshift-service-catalog-controller-manager-remover   1/1           6s         13m
[root@preserve-olm-env client]# ./oc get pods -n openshift-service-catalog-removed 
NAME                                                         READY   STATUS      RESTARTS   AGE
openshift-service-catalog-apiserver-remover-4lgll            0/1     Completed   0          13m
openshift-service-catalog-controller-manager-remover-rd2mv   0/1     Completed   0          13m
[root@preserve-olm-env client]# ./oc logs openshift-service-catalog-apiserver-remover-4lgll  -n openshift-service-catalog-removed 
time="2020-07-09T06:49:59Z" level=info msg="Starting openshift-service-catalog-apiserver-remover job"
time="2020-07-09T06:49:59Z" level=info msg="ServiceCatalogAPIServer managementState is 'Removed'"
time="2020-07-09T06:49:59Z" level=info msg="Removing target namespace openshift-service-catalog-apiserver-operator"
time="2020-07-09T06:49:59Z" level=info msg="Removing the ServiceCatalogAPIServer CR"
time="2020-07-09T06:49:59Z" level=info msg="ServiceCatalogAPIServer cr removed successfully."
time="2020-07-09T06:49:59Z" level=info msg="Removing the service-catalog-apiserver clusteroperator"
time="2020-07-09T06:49:59Z" level=info msg="Removing ClusterRoleBinding: openshift-service-catalog-apiserver-operator"
time="2020-07-09T06:49:59Z" level=info msg="Removing ClusterRole: openshift-service-catalog-apiserver-operator"
time="2020-07-09T06:49:59Z" level=info msg="The openshift-service-catalog-apiserver-remover job, has finished."
[root@preserve-olm-env client]# ./oc logs openshift-service-catalog-controller-manager-remover   -n openshift-service-catalog-removed 
Error from server (NotFound): pods "openshift-service-catalog-controller-manager-remover" not found
[root@preserve-olm-env client]# ./oc logs openshift-service-catalog-controller-manager-remover-rd2mv   -n openshift-service-catalog-removed 
time="2020-07-09T06:50:05Z" level=info msg="Starting openshift-service-catalog-controller-manager-remover job"
time="2020-07-09T06:50:05Z" level=info msg="ServiceCatalogControllerManager managementState is 'Removed'"
time="2020-07-09T06:50:05Z" level=info msg="Removing target namespace openshift-service-catalog-controller-manager-operator"
time="2020-07-09T06:50:05Z" level=info msg="Removing the ServiceCatalogControllerManager CR"
time="2020-07-09T06:50:05Z" level=info msg="ServiceCatalogControllerManager cr removed successfully."
time="2020-07-09T06:50:05Z" level=info msg="Removing the service-catalog-controller-manager clusteroperator"
time="2020-07-09T06:50:05Z" level=info msg="Removing ClusterRoleBinding: openshift-service-catalog-controller-manager-operator"
time="2020-07-09T06:50:05Z" level=info msg="Removing ClusterRole: openshift-service-catalog-controller-manager-operator"
time="2020-07-09T06:50:05Z" level=info msg="The openshift-service-catalog-controller-manater-remover job has finished."
Comment 1 Jesus M. Rodriguez 2020-07-14 20:23:05 UTC 
Closing this for 4.6 as the jobs are no longer being built for 4.6