Bug 1854978 - CRDs still exist after upgraded to the 4.5.rc.7 from 4.4.11
Summary: CRDs still exist after upgraded to the 4.5.rc.7 from 4.4.11
Keywords:
Status: VERIFIED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Operator SDK
Version: 4.5
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.5.z
Assignee: Jesus M. Rodriguez
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On: 1856978
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-08 14:24 UTC by Jian Zhang
Modified: 2020-09-22 12:43 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1856978 (view as bug list)
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift cluster-svcat-apiserver-operator pull 91 None closed Bug 1854978: Remove servicecatalogapiserver CRD 2020-09-17 05:50:45 UTC
Github openshift cluster-svcat-controller-manager-operator pull 78 None closed Bug 1854978: Remove servicecatalogcontrollermanager CRD 2020-09-17 05:50:43 UTC

Description Jian Zhang 2020-07-08 14:24:58 UTC
Description of problem:
ServiceCatalog CRDs still exists after upgrading to 4.5.

Version-Release number of selected component (if applicable):
4.4.11 - 4.5.0-rc.7

How reproducible:
always

Steps to Reproduce:
1, Enable the ServiceCatalog on 4.4.11.
mac:~ jianzhang$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.11    True        False         9m19s   Cluster version is 4.4.11
mac:~ jianzhang$ oc get servicecatalogapiserver
NAME      AGE
cluster   29m
mac:~ jianzhang$ oc get clusterservicebroker
NAME                      URL                                                                                         STATUS   AGE
template-service-broker   https://apiserver.openshift-template-service-broker.svc:443/brokers/template.openshift.io   Ready    5m


2, Disable the ServiceCatalog.
mac:~ jianzhang$ oc patch servicecatalogapiserver cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge
servicecatalogapiserver.operator.openshift.io/cluster patched
mac:~ jianzhang$ oc patch servicecatalogcontrollermanager cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge
servicecatalogcontrollermanager.operator.openshift.io/cluster patched

mac:~ jianzhang$ oc get clusterservicebroker
Error from server (NotFound): Unable to list "servicecatalog.k8s.io/v1beta1, Resource=clusterservicebrokers": the server could not find the requested resource (get clusterservicebrokers.servicecatalog.k8s.io)

3, Upgrade it to the 4.5.0-rc.7 without the `--force`.
mac:~ jianzhang$ oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release@sha256:f284efc07cb0bfa06cb7260ce4dedff4dabb55002377a7bcc637b58730976a3f --allow-explicit-upgrade
warning: The requested upgrade image is not one of the available updates.  You have used --allow-explicit-upgrade to the update to preceed anyway
Updating to release image quay.io/openshift-release-dev/ocp-release@sha256:f284efc07cb0bfa06cb7260ce4dedff4dabb55002377a7bcc637b58730976a3f

mac:~ jianzhang$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.11    True        True          41s     Working towards 4.5.0-rc.7: 11% complete

mac:~ jianzhang$ oc get job -A
NAMESPACE                   NAME             COMPLETIONS   DURATION   AGE
openshift-cluster-version   version--76gxn   1/1           7s         37m

mac:~ jianzhang$ oc get clusterversion
NAME      VERSION      AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-rc.7   True        False         44m     Cluster version is 4.5.0-rc.7

4, Check if the ServiceCatalog removed. I can see the removed Jobs.
mac:~ jianzhang$ oc project openshift-service-catalog-removed
Now using project "openshift-service-catalog-removed" on server "https://api.qe-jiazha-09.qe.devcluster.openshift.com:6443".

mac:~ jianzhang$ oc get job
NAME                                                   COMPLETIONS   DURATION   AGE
openshift-service-catalog-apiserver-remover            1/1           6s         66m
openshift-service-catalog-controller-manager-remover   1/1           6s         66m

5, Check if ServiceCatalog CRDs still exists.

Actual results:
mac:~ jianzhang$ oc get servicecatalogapiserver
No resources found
mac:~ jianzhang$ oc get crd servicecatalogapiservers.operator.openshift.io
NAME                                             CREATED AT
servicecatalogapiservers.operator.openshift.io   2020-07-08T11:47:09Z
mac:~ jianzhang$ oc get crd servicecatalogcontrollermanagers.operator.openshift.io
NAME                                                     CREATED AT
servicecatalogcontrollermanagers.operator.openshift.io   2020-07-08T11:47:09Z


Expected results:
These ServiceCatalog CRDs should be removed since they're don't work in 4.5. 


Additional info:
Here is a cluster for debugging: https://mastern-jenkins-csb-openshift-qe.cloud.paas.psi.redhat.com/job/Launch%20Environment%20Flexy/101128/artifact/workdir/install-dir/auth/kubeconfig/*view*/

mac:~ jianzhang$ oc get pods
NAME                                                         READY   STATUS      RESTARTS   AGE
openshift-service-catalog-apiserver-remover-8f6st            0/1     Completed   0          65m
openshift-service-catalog-controller-manager-remover-977w6   0/1     Completed   0          64m
mac:~ jianzhang$ oc logs openshift-service-catalog-apiserver-remover-8f6st
time="2020-07-08T13:02:25Z" level=info msg="Starting openshift-service-catalog-apiserver-remover job"
time="2020-07-08T13:02:25Z" level=info msg="ServiceCatalogAPIServer managementState is 'Removed'"
time="2020-07-08T13:02:25Z" level=info msg="Removing target namespace openshift-service-catalog-apiserver-operator"
time="2020-07-08T13:02:25Z" level=info msg="Removing the ServiceCatalogAPIServer CR"
time="2020-07-08T13:02:25Z" level=info msg="ServiceCatalogAPIServer cr removed successfully."
time="2020-07-08T13:02:25Z" level=info msg="Removing the service-catalog-apiserver clusteroperator"
time="2020-07-08T13:02:25Z" level=info msg="Removing ClusterRoleBinding: openshift-service-catalog-apiserver-operator"
time="2020-07-08T13:02:25Z" level=info msg="Removing ClusterRole: openshift-service-catalog-apiserver-operator"
time="2020-07-08T13:02:25Z" level=info msg="The openshift-service-catalog-apiserver-remover job, has finished."
mac:~ jianzhang$ oc logs openshift-service-catalog-controller-manager-remover-977w6
time="2020-07-08T13:02:34Z" level=info msg="Starting openshift-service-catalog-controller-manager-remover job"
time="2020-07-08T13:02:34Z" level=info msg="ServiceCatalogControllerManager managementState is 'Removed'"
time="2020-07-08T13:02:34Z" level=info msg="Removing target namespace openshift-service-catalog-controller-manager-operator"
time="2020-07-08T13:02:34Z" level=info msg="Removing the ServiceCatalogControllerManager CR"
time="2020-07-08T13:02:34Z" level=info msg="ServiceCatalogControllerManager cr removed successfully."
time="2020-07-08T13:02:34Z" level=info msg="Removing the service-catalog-controller-manager clusteroperator"
time="2020-07-08T13:02:34Z" level=info msg="Removing ClusterRoleBinding: openshift-service-catalog-controller-manager-operator"
time="2020-07-08T13:02:34Z" level=info msg="Removing ClusterRole: openshift-service-catalog-controller-manager-operator"
time="2020-07-08T13:02:34Z" level=info msg="The openshift-service-catalog-controller-manater-remover job has finished."

Comment 1 Jian Zhang 2020-07-09 07:52:56 UTC
I also have a try with a cluster that no ServiceCatalog/any brokers installed, but still encounter this issue. As follows:

1, Install OCP 4.4.11, disable ServiceCatalog and no brokers installed:
[root@preserve-olm-env client]# ./oc get clusterservicebroker
No resources found
[root@preserve-olm-env client]# ./oc patch servicecatalogapiserver cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge
servicecatalogapiserver.operator.openshift.io/cluster patched
[root@preserve-olm-env client]# ./oc patch servicecatalogcontrollermanager cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge
servicecatalogcontrollermanager.operator.openshift.io/cluster patched
[root@preserve-olm-env client]# ./oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.11    True        False         13m     Cluster version is 4.4.11

2, Upgrade it to the OCP 4.5.0-rc.7
[root@preserve-olm-env client]# ./oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release@sha256:f284efc07cb0bfa06cb7260ce4dedff4dabb55002377a7bcc637b58730976a3f
Updating to release image registry.svc.ci.openshift.org/ocp/release@sha256:f284efc07cb0bfa06cb7260ce4dedff4dabb55002377a7bcc637b58730976a3f
[root@preserve-olm-env client]# ./oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.11    True        True          37s     Working towards 4.5.0-rc.7: 11% complete

3, Check the ServiceCatalog CRDs, they are still exist. They should be removed on OCP 4.5.
[root@preserve-olm-env client]# ./oc get servicecatalogapiserver
No resources found
[root@preserve-olm-env client]# ./oc get crd|grep catalog
catalogsources.operators.coreos.com                         2020-07-09T01:58:09Z
servicecatalogapiservers.operator.openshift.io              2020-07-09T01:57:52Z
servicecatalogcontrollermanagers.operator.openshift.io      2020-07-09T01:57:52Z
[root@preserve-olm-env client]# ./oc get job -n openshift-service-catalog-removed 
NAME                                                   COMPLETIONS   DURATION   AGE
openshift-service-catalog-apiserver-remover            1/1           6s         13m
openshift-service-catalog-controller-manager-remover   1/1           6s         13m
[root@preserve-olm-env client]# ./oc get pods -n openshift-service-catalog-removed 
NAME                                                         READY   STATUS      RESTARTS   AGE
openshift-service-catalog-apiserver-remover-4lgll            0/1     Completed   0          13m
openshift-service-catalog-controller-manager-remover-rd2mv   0/1     Completed   0          13m
[root@preserve-olm-env client]# ./oc logs openshift-service-catalog-apiserver-remover-4lgll  -n openshift-service-catalog-removed 
time="2020-07-09T06:49:59Z" level=info msg="Starting openshift-service-catalog-apiserver-remover job"
time="2020-07-09T06:49:59Z" level=info msg="ServiceCatalogAPIServer managementState is 'Removed'"
time="2020-07-09T06:49:59Z" level=info msg="Removing target namespace openshift-service-catalog-apiserver-operator"
time="2020-07-09T06:49:59Z" level=info msg="Removing the ServiceCatalogAPIServer CR"
time="2020-07-09T06:49:59Z" level=info msg="ServiceCatalogAPIServer cr removed successfully."
time="2020-07-09T06:49:59Z" level=info msg="Removing the service-catalog-apiserver clusteroperator"
time="2020-07-09T06:49:59Z" level=info msg="Removing ClusterRoleBinding: openshift-service-catalog-apiserver-operator"
time="2020-07-09T06:49:59Z" level=info msg="Removing ClusterRole: openshift-service-catalog-apiserver-operator"
time="2020-07-09T06:49:59Z" level=info msg="The openshift-service-catalog-apiserver-remover job, has finished."
[root@preserve-olm-env client]# ./oc logs openshift-service-catalog-controller-manager-remover   -n openshift-service-catalog-removed 
Error from server (NotFound): pods "openshift-service-catalog-controller-manager-remover" not found
[root@preserve-olm-env client]# ./oc logs openshift-service-catalog-controller-manager-remover-rd2mv   -n openshift-service-catalog-removed 
time="2020-07-09T06:50:05Z" level=info msg="Starting openshift-service-catalog-controller-manager-remover job"
time="2020-07-09T06:50:05Z" level=info msg="ServiceCatalogControllerManager managementState is 'Removed'"
time="2020-07-09T06:50:05Z" level=info msg="Removing target namespace openshift-service-catalog-controller-manager-operator"
time="2020-07-09T06:50:05Z" level=info msg="Removing the ServiceCatalogControllerManager CR"
time="2020-07-09T06:50:05Z" level=info msg="ServiceCatalogControllerManager cr removed successfully."
time="2020-07-09T06:50:05Z" level=info msg="Removing the service-catalog-controller-manager clusteroperator"
time="2020-07-09T06:50:05Z" level=info msg="Removing ClusterRoleBinding: openshift-service-catalog-controller-manager-operator"
time="2020-07-09T06:50:05Z" level=info msg="Removing ClusterRole: openshift-service-catalog-controller-manager-operator"
time="2020-07-09T06:50:05Z" level=info msg="The openshift-service-catalog-controller-manater-remover job has finished."

Comment 6 Jian Zhang 2020-09-14 03:47:10 UTC
1, Install the 4.4.21 cluster
[root@preserve-olm-env data]# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.21    True        False         60m     Cluster version is 4.4.21
2, Disable the ServiceCatalog.
[root@preserve-olm-env data]# oc patch servicecatalogapiserver cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge
servicecatalogapiserver.operator.openshift.io/cluster patched
[root@preserve-olm-env data]# oc patch servicecatalogcontrollermanager cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge
servicecatalogcontrollermanager.operator.openshift.io/cluster patched

[root@preserve-olm-env data]# oc get clusterservicebroker
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get clusterservicebrokers.servicecatalog.k8s.io)

3, Upgrade it to the registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-09-12-063044 that contains the fixed PR.
[root@preserve-olm-env data]# oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release@sha256:ab39c801730e87c903757c0d3e06dd0fcad81669e28e61eb3a82103db79d6d04 --allow-explicit-upgrade
warning: The requested upgrade image is not one of the available updates.  You have used --allow-explicit-upgrade to the update to proceed anyway
Updating to release image registry.svc.ci.openshift.org/ocp/release@sha256:ab39c801730e87c903757c0d3e06dd0fcad81669e28e61eb3a82103db79d6d04

This new payload has NOT been sigined, we cannot upgrade it without `--force`, it's unavaliable for us to test it now. Change the status to MODIFIED.
[root@preserve-olm-env data]# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.21    True        True          16s     Unable to apply registry.svc.ci.openshift.org/ocp/release@sha256:ab39c801730e87c903757c0d3e06dd0fcad81669e28e61eb3a82103db79d6d04: the image may not be safe to use

Comment 8 Jian Zhang 2020-09-17 08:30:39 UTC
1, Install the 4.4.22 cluster

[root@preserve-olm-env data]# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.22    True        False         68m     Cluster version is 4.4.22

[root@preserve-olm-env data]# oc get clusterservicebroker
NAME                      URL                                                                                         STATUS   AGE
template-service-broker   https://apiserver.openshift-template-service-broker.svc:443/brokers/template.openshift.io   Ready    1h

2, Disable the ServiceCatalog.

[root@preserve-olm-env data]# oc patch servicecatalogapiserver cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge
servicecatalogapiserver.operator.openshift.io/cluster patched
[root@preserve-olm-env data]# oc patch servicecatalogcontrollermanager cluster -p '{"spec":{"managementState":"Removed"}}' --type=merge
servicecatalogcontrollermanager.operator.openshift.io/cluster patched

[root@preserve-olm-env data]# oc get clusterservicebroker
Error from server (NotFound): Unable to list "servicecatalog.k8s.io/v1beta1, Resource=clusterservicebrokers": the server could not find the requested resource (get clusterservicebrokers.servicecatalog.k8s.io)

3, Upgrade it to 4.5.10
[root@preserve-olm-env data]# oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release@sha256:272a00b3dc91e1a8e45370969d4300f17bc89bc5cefb0f938f6c1b69333fe371 --allow-explicit-upgrade
warning: The requested upgrade image is not one of the available updates.  You have used --allow-explicit-upgrade to the update to proceed anyway
Updating to release image quay.io/openshift-release-dev/ocp-release@sha256:272a00b3dc91e1a8e45370969d4300f17bc89bc5cefb0f938f6c1b69333fe371

[root@preserve-olm-env data]# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.22    True        True          51s     Working towards 4.5.10: 15% complete

[root@preserve-olm-env data]# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.10    True        False         3m22s   Cluster version is 4.5.10

4, Check the ServiceCatalog CRDs if still exist. Failed to get them. LGTM, verify it.
[root@preserve-olm-env data]# oc get servicecatalogapiserver
error: the server doesn't have a resource type "servicecatalogapiserver"
[root@preserve-olm-env data]# oc get servicecatalogcontrollermanager
error: the server doesn't have a resource type "servicecatalogcontrollermanager"


Note You need to log in before you can comment on or make changes to this bug.