Description of problem: After enabling servicecatalogapiservers, if it is disabled, it does not clean up it apiservice resource. Version-Release number of selected component (if applicable): 4.1.0-rc.0 How reproducible: 100% Steps to Reproduce: 1. Set servicecatalogapiservers/cluster to Managed 2. Set servicecatalogcontrollermanagers/cluster to Managed 3. Allow apiservice time to initialize 4. Set servicecatalogapiservers/cluster to Removed 5. Set servicecatalogcontrollermanagers/cluster to Removed 6. Run `oc api-resources` and note the error: "error: unable to retrieve the complete list of server APIs: servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request" Actual results: The missing apiservice prevents the deletion of the namespaces in the cluster since the controller permanently awaits a vote from the offline apiservice. Expected results: Disabling the apiservice should remove the corresponding apiservice resource. Additional info: A workaround exists: oc delete apiservice v1beta1.servicecatalog.k8s.io
We discussed and general agreement between Shawn, Jesus, Jessica and myself is that this should be fixed immediately in 4.1.z and perhaps a tech note for 4.1 GA - - if you remove Service Catalog this way, you need to `oc delete apiservice v1beta1.servicecatalog.k8s.io`
I ran into something similar (but not exact) in a CRC VM: https://github.com/code-ready/crc/issues/268 I do not know if this is relevant to this bugzilla issue, but it seems similar.
John, From https://github.com/code-ready/crc/issues/268 description, seems like you didn't enable Service Catalogs. And, according to the workaround `oc delete apiservice v1beta1.metrics.k8s.io`, it's nothing with Service Catalog.
John, I guess you wanted is here: https://bugzilla.redhat.com/show_bug.cgi?id=1625194
PR for script to delete the owner references https://github.com/openshift/cluster-svcat-apiserver-operator/pull/59
*** Bug 1731869 has been marked as a duplicate of this bug. ***
Fixed in 4.2 by PR https://github.com/openshift/cluster-svcat-apiserver-operator/pull/62
The latest svcat-service-catalog-operator version is: io.openshift.build.commit.url=https://github.com/openshift/cluster-svcat-apiserver-operator/commit/926d8cbf2a4710d4aba6a286b3ee0880b3fe9762 The fixed PR hasn't been merged in. Change status to MODIFIED.
PR merged early this morning, https://github.com/openshift/cluster-svcat-apiserver-operator/pull/62 should be on the next build.
Hi, Jesus I found the version of the svcat-apiserver-operator is 926d8cbf2a4710d4aba6a286b3ee0880b3fe9762, which megered on August 6th. But, the current Cluster version is 4.2.0-0.nightly-2019-09-16-175247, anything wrong in the image building CI? Change status to POST since no fixed image for test. mac:~ jianzhang$ oc image info quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1cb4f9e8c48afa2239aa22bb513ee15129fde64fe9675b37f129367990bf5f10 Name: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1cb4f9e8c48afa2239aa22bb513ee15129fde64fe9675b37f129367990bf5f10 Media Type: application/vnd.docker.distribution.manifest.v2+json Created: 8d ago Image Size: 110.5MB in 5 layers Layers: 76.24MB sha256:7b1c937e0f6794db2535be6e4cb6d60a0b668ef78c2576611a3fb9c97a95ccdf 1.613kB sha256:bff3b73cbcc496de1de4ea51df88b7249169d0b6eb7d677169eaf90b8a92240e 3.502MB sha256:25020f838d5d1225b418ac90db9ab86ed1653f2b2f483e4e0e86d0ba3cbe076c 8.225MB sha256:285ffcda1054905e16a92d333fe946ebd84b01354601f256df10e252c48ed4e3 22.53MB sha256:97e7153b6b52ababbd79395c604782b464e6cdc0e2246ca5de11046f2f20930b OS: linux Arch: amd64 Command: /bin/bash User: 0 Environment: SOURCE_GIT_COMMIT=926d8cbf2a4710d4aba6a286b3ee0880b3fe9762 SOURCE_DATE_EPOCH=1565069143 BUILD_VERSION=v4.2.0 SOURCE_GIT_URL=https://github.com/openshift/cluster-svcat-apiserver-operator SOURCE_GIT_TAG=926d8cbf BUILD_RELEASE=201909081401 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=oci Labels: License=GPLv2+ architecture=x86_64 authoritative-source-url=registry.access.redhat.com build-date=2019-09-08T20:51:40.114473 com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com com.redhat.component=ose-cluster-svcat-apiserver-operator-container com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements description=This is the base image from which all OpenShift Container Platform images inherit. distribution-scope=public io.k8s.description=This is the base image from which all OpenShift Container Platform images inherit. io.k8s.display-name=OpenShift Container Platform RHEL 7 Base io.openshift.build.commit.id=926d8cbf2a4710d4aba6a286b3ee0880b3fe9762 io.openshift.build.commit.url=https://github.com/openshift/cluster-svcat-apiserver-operator/commit/926d8cbf2a4710d4aba6a286b3ee0880b3fe9762 io.openshift.build.source-location=https://github.com/openshift/cluster-svcat-apiserver-operator io.openshift.release.operator=true io.openshift.tags=openshift,base maintainer=Red Hat, Inc. name=openshift/ose-cluster-svcat-apiserver-operator release=201909081401 summary=Provides the latest release of the Red Hat Universal Base Image 7. url=https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-cluster-svcat-apiserver-operator/images/v4.2.0-201909081401 vcs-ref=d5f64704726931abcb4665ec087a6e1386204eb3 vcs-type=git vendor=Red Hat, Inc. version=v4.2.0
Disabling the apiservice should removed v1beta1.servicecatalog.k8s.io apiservice as expected. Marking as VERIFIED Cluster Version: 4.2.0-0.nightly-2019-09-12-114308 "io.openshift.build.commit.url": "https://github.com/openshift/cluster-svcat-controller-manager-operator/commit/b57aa59ceb99ad2493db2f288ae8f084ca4549c5" Steps used to validate: 1. Set servicecatalogapiservers/cluster to Managed oc patch ServiceCatalogAPIServer cluster -p '{"spec":{"managementState": "Managed"}}' --type=merge 2. Set servicecatalogcontrollermanagers/cluster to Managed oc patch ServiceCatalogControllerManager cluster -p '{"spec":{"managementState": "Managed"}}' --type=merge 3. Allow apiservice time to initialize 4. Check if apiservice is created: oc get apiservice v1beta1.servicecatalog.k8s.io NAME SERVICE AVAILABLE AGE v1beta1.servicecatalog.k8s.io openshift-service-catalog-apiserver/api True 6m22s 5. Set servicecatalogapiservers/cluster to Removed oc patch ServiceCatalogAPIServer cluster -p '{"spec":{"managementState": "Removed"}}' --type=merge 6. Set servicecatalogcontrollermanagers/cluster to Removed oc patch ServiceCatalogControllerManager cluster -p '{"spec":{"managementState": "Removed"}}' --type=merge 7. Run `oc api-resources` oc api-resources NAME SHORTNAMES APIGROUP NAMESPACED KIND bindings true Binding componentstatuses cs false ComponentStatus configmaps cm true ConfigMap endpoints ep true Endpoints events ev true Event limitranges limits true LimitRange namespaces ns false Namespace nodes no false Node persistentvolumeclaims pvc true PersistentVolumeClaim persistentvolumes pv false PersistentVolume pods po true Pod podtemplates true PodTemplate replicationcontrollers rc true ReplicationController resourcequotas quota true ResourceQuota secrets true Secret serviceaccounts sa true ServiceAccount services svc true Service mutatingwebhookconfigurations admissionregistration.k8s.io false MutatingWebhookConfiguration validatingwebhookconfigurations admissionregistration.k8s.io false ValidatingWebhookConfiguration customresourcedefinitions crd,crds apiextensions.k8s.io false CustomResourceDefinition apiservices apiregistration.k8s.io false APIService controllerrevisions apps true ControllerRevision daemonsets ds apps true DaemonSet deployments deploy apps true Deployment replicasets rs apps true ReplicaSet statefulsets sts apps true StatefulSet deploymentconfigs dc apps.openshift.io true DeploymentConfig tokenreviews authentication.k8s.io false TokenReview localsubjectaccessreviews authorization.k8s.io true LocalSubjectAccessReview selfsubjectaccessreviews authorization.k8s.io false SelfSubjectAccessReview selfsubjectrulesreviews authorization.k8s.io false SelfSubjectRulesReview subjectaccessreviews authorization.k8s.io false SubjectAccessReview clusterrolebindings authorization.openshift.io false ClusterRoleBinding clusterroles authorization.openshift.io false ClusterRole localresourceaccessreviews authorization.openshift.io true LocalResourceAccessReview localsubjectaccessreviews authorization.openshift.io true LocalSubjectAccessReview resourceaccessreviews authorization.openshift.io false ResourceAccessReview rolebindingrestrictions authorization.openshift.io true RoleBindingRestriction rolebindings authorization.openshift.io true RoleBinding roles authorization.openshift.io true Role selfsubjectrulesreviews authorization.openshift.io true SelfSubjectRulesReview subjectaccessreviews authorization.openshift.io false SubjectAccessReview subjectrulesreviews authorization.openshift.io true SubjectRulesReview bundlebindings automationbroker.io true BundleBinding bundleinstances automationbroker.io true BundleInstance bundles automationbroker.io true Bundle horizontalpodautoscalers hpa autoscaling true HorizontalPodAutoscaler clusterautoscalers autoscaling.openshift.io false ClusterAutoscaler machineautoscalers autoscaling.openshift.io true MachineAutoscaler cronjobs cj batch true CronJob jobs batch true Job buildconfigs bc build.openshift.io true BuildConfig builds build.openshift.io true Build certificatesigningrequests csr certificates.k8s.io false CertificateSigningRequest credentialsrequests cloudcredential.openshift.io true CredentialsRequest apiservers config.openshift.io false APIServer authentications config.openshift.io false Authentication builds config.openshift.io false Build clusteroperators co config.openshift.io false ClusterOperator clusterversions config.openshift.io false ClusterVersion consoles config.openshift.io false Console dnses config.openshift.io false DNS featuregates config.openshift.io false FeatureGate images config.openshift.io false Image infrastructures config.openshift.io false Infrastructure ingresses config.openshift.io false Ingress networks config.openshift.io false Network oauths config.openshift.io false OAuth operatorhubs config.openshift.io false OperatorHub projects config.openshift.io false Project proxies config.openshift.io false Proxy schedulers config.openshift.io false Scheduler consoleclidownloads console.openshift.io false ConsoleCLIDownload consoleexternalloglinks console.openshift.io false ConsoleExternalLogLink consolelinks console.openshift.io false ConsoleLink consolenotifications console.openshift.io false ConsoleNotification leases coordination.k8s.io true Lease etcdbackups etcd.database.coreos.com true EtcdBackup etcdclusters etcdclus,etcd etcd.database.coreos.com true EtcdCluster etcdrestores etcd.database.coreos.com true EtcdRestore events ev events.k8s.io true Event daemonsets ds extensions true DaemonSet deployments deploy extensions true Deployment ingresses ing extensions true Ingress networkpolicies netpol extensions true NetworkPolicy podsecuritypolicies psp extensions false PodSecurityPolicy replicasets rs extensions true ReplicaSet machinedisruptionbudgets mdb,mdbs healthchecking.openshift.io true MachineDisruptionBudget machinehealthchecks mhc,mhcs healthchecking.openshift.io true MachineHealthCheck images image.openshift.io false Image imagesignatures image.openshift.io false ImageSignature imagestreamimages isimage image.openshift.io true ImageStreamImage imagestreamimports image.openshift.io true ImageStreamImport imagestreammappings image.openshift.io true ImageStreamMapping imagestreams is image.openshift.io true ImageStream imagestreamtags istag image.openshift.io true ImageStreamTag configs imageregistry.operator.openshift.io false Config dnsrecords ingress.operator.openshift.io true DNSRecord network-attachment-definitions net-attach-def k8s.cni.cncf.io true NetworkAttachmentDefinition machines machine.openshift.io true Machine machinesets machine.openshift.io true MachineSet containerruntimeconfigs ctrcfg machineconfiguration.openshift.io false ContainerRuntimeConfig controllerconfigs machineconfiguration.openshift.io false ControllerConfig kubeletconfigs machineconfiguration.openshift.io false KubeletConfig machineconfigpools mcp machineconfiguration.openshift.io false MachineConfigPool machineconfigs mc machineconfiguration.openshift.io false MachineConfig mcoconfigs machineconfiguration.openshift.io true MCOConfig baremetalhosts bmh,bmhost metal3.io true BareMetalHost nodes metrics.k8s.io false NodeMetrics pods metrics.k8s.io true PodMetrics alertmanagers monitoring.coreos.com true Alertmanager podmonitors monitoring.coreos.com true PodMonitor prometheuses monitoring.coreos.com true Prometheus prometheusrules monitoring.coreos.com true PrometheusRule servicemonitors monitoring.coreos.com true ServiceMonitor clusternetworks network.openshift.io false ClusterNetwork egressnetworkpolicies network.openshift.io true EgressNetworkPolicy hostsubnets network.openshift.io false HostSubnet netnamespaces network.openshift.io false NetNamespace ingresses ing networking.k8s.io true Ingress networkpolicies netpol networking.k8s.io true NetworkPolicy runtimeclasses node.k8s.io false RuntimeClass oauthaccesstokens oauth.openshift.io false OAuthAccessToken oauthauthorizetokens oauth.openshift.io false OAuthAuthorizeToken oauthclientauthorizations oauth.openshift.io false OAuthClientAuthorization oauthclients oauth.openshift.io false OAuthClient authentications operator.openshift.io false Authentication consoles operator.openshift.io false Console dnses operator.openshift.io false DNS imagecontentsourcepolicies operator.openshift.io false ImageContentSourcePolicy ingresscontrollers operator.openshift.io true IngressController kubeapiservers operator.openshift.io false KubeAPIServer kubecontrollermanagers operator.openshift.io false KubeControllerManager kubeschedulers operator.openshift.io false KubeScheduler networks operator.openshift.io false Network openshiftapiservers operator.openshift.io false OpenShiftAPIServer openshiftcontrollermanagers operator.openshift.io false OpenShiftControllerManager servicecas operator.openshift.io false ServiceCA servicecatalogapiservers operator.openshift.io false ServiceCatalogAPIServer servicecatalogcontrollermanagers operator.openshift.io false ServiceCatalogControllerManager catalogsourceconfigs csc operators.coreos.com true CatalogSourceConfig catalogsources catsrc operators.coreos.com true CatalogSource clusterserviceversions csv,csvs operators.coreos.com true ClusterServiceVersion installplans ip operators.coreos.com true InstallPlan operatorgroups og operators.coreos.com true OperatorGroup operatorsources opsrc operators.coreos.com true OperatorSource subscriptions sub,subs operators.coreos.com true Subscription automationbrokers osb.openshift.io true AutomationBroker templateservicebrokers osb.openshift.io true TemplateServiceBroker packagemanifests packages.operators.coreos.com true PackageManifest poddisruptionbudgets pdb policy true PodDisruptionBudget podsecuritypolicies psp policy false PodSecurityPolicy projectrequests project.openshift.io false ProjectRequest projects project.openshift.io false Project appliedclusterresourcequotas quota.openshift.io true AppliedClusterResourceQuota clusterresourcequotas clusterquota quota.openshift.io false ClusterResourceQuota clusterrolebindings rbac.authorization.k8s.io false ClusterRoleBinding clusterroles rbac.authorization.k8s.io false ClusterRole rolebindings rbac.authorization.k8s.io true RoleBinding roles rbac.authorization.k8s.io true Role routes route.openshift.io true Route configs samples.operator.openshift.io false Config priorityclasses pc scheduling.k8s.io false PriorityClass podsecuritypolicyreviews security.openshift.io true PodSecurityPolicyReview podsecuritypolicyselfsubjectreviews security.openshift.io true PodSecurityPolicySelfSubjectReview podsecuritypolicysubjectreviews security.openshift.io true PodSecurityPolicySubjectReview rangeallocations security.openshift.io false RangeAllocation securitycontextconstraints scc security.openshift.io false SecurityContextConstraints csidrivers storage.k8s.io false CSIDriver csinodes storage.k8s.io false CSINode storageclasses sc storage.k8s.io false StorageClass volumeattachments storage.k8s.io false VolumeAttachment brokertemplateinstances template.openshift.io false BrokerTemplateInstance processedtemplates template.openshift.io true Template templateinstances template.openshift.io true TemplateInstance templates template.openshift.io true Template tuneds tuned.openshift.io true Tuned groups user.openshift.io false Group identities user.openshift.io false Identity useridentitymappings user.openshift.io false UserIdentityMapping users user.openshift.io false User 9) Check if Service catalog apiservice is removed as expected. oc get apiservice v1beta1.servicecatalog.k8s.io Error from server (NotFound): apiservices.apiregistration.k8s.io "v1beta1.servicecatalog.k8s.io" not found
Just correcting the version used to verify this issue: Cluster Version: 4.2.0-0.nightly-2019-09-18-114152 cluster-svcat-apiserver-operator git commit: https://github.com/openshift/cluster-svcat-controller-manager-operator/commit/9ef3f62c4b3d7b824c9b0b86c2081d694faff5ea
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922