Description of problem: Unable to deprecate/remove a version from CRD if that version was previously storage version. Version-Release number of selected component (if applicable): How reproducible: Deploy an etcd operator using etcdclusters CRD with v1beta2 version as serving and storage version. Next, upgrade the operator with etcdclusters CRD to add a new version v1beta3 as serving and storage version and switching v1beta2 to non-serving and no longer storage version (this happens by default as only one version is marked as storage version). Then, upgrade operator again with etcdclusters CRD with only v1beta3 version and remove v1beta2 version completely. The upgrade process will fail as it complains v1beta2 is still in storedVersion in CRD's status. Steps to Reproduce: 1. Deploy an etcd operator using etcdclusters CRD with v1beta2 version as serving and storage version 2. Upgrade the operator with etcdclusters CRD to add a new version v1beta3 as serving and storage version and switching v1beta2 to non-serving and no longer storage version 3. Upgrade operator again with etcdclusters CRD with only v1beta3 version and remove v1beta2 version completely Actual results: The upgrade process will fail as it complains v1beta2 is still in storedVersion in CRD's status. Expected results: The CRD should be upgraded properly as OLM should gracefully handle the deprecated version in storedVersions. Additional info:
Pending PR: https://github.com/operator-framework/operator-lifecycle-manager/pull/983
Current the latest payload is 4.2.0-0.nightly-2019-09-10-181551, which the fixed PR hasn't merged in. mac:~ jianzhang$ oc exec catalog-operator-5544569dd7-lggqq -- olm --version OLM version: 0.11.0 git commit: 5a5389cb8d831e79acade535d947d4ad4a5c40a7 Change status to MODIFIED.
Verify failed. Cluster Version:4.2.0-0.nightly-2019-09-13-113908 OLM version: 0.11.0 git commit: 2760276f78584ab485d448fa53abda8025fe5cb0 Steps used to reproduce: 1) Create the ConfigMap and the Catalog Source with etcd package with v1beta2 version as serving and storage version (https://github.com/bandrade/v3-testfiles/blob/v4.1/olm/configmap/configmap_etcd-v7.yaml#L667-L669) oc create -f https://raw.githubusercontent.com/bandrade/v3-testfiles/v4.1/olm/configmap/configmap_etcd-v7.yaml -n openshift-marketplace oc create -f https://raw.githubusercontent.com/bandrade/v3-testfiles/v4.1/olm/catalogsource/catalogsource.yaml -n openshift-marketplace 2) Create namespace test-operators oc create ns test-operators 3) Create the Operator Group oc create -f - <<EOF apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: test-operators-og namespace: test-operators spec: targetNamespaces: - test-operators EOF 4) Create the subscription, as below: oc create -f https://raw.githubusercontent.com/bandrade/v3-testfiles/v4.1/olm/subscription/test.yaml -n test-operators 5) The csv should be created successfully. oc get csv NAME DISPLAY VERSION REPLACES PHASE etcdoperator.v0.9.2 etcd 0.9.2 Succeeded 6) Check CRD: oc get crd etcdclusters.etcd.database.coreos.com -o yaml apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: creationTimestamp: "2019-09-14T00:55:31Z" generation: 1 name: etcdclusters.etcd.database.coreos.com resourceVersion: "156385" selfLink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/etcdclusters.etcd.database.coreos.com uid: 59d6b121-d68a-11e9-a45d-06e91e4192c2 spec: conversion: strategy: None group: etcd.database.coreos.com names: kind: EtcdCluster listKind: EtcdClusterList plural: etcdclusters shortNames: - etcdclus - etcd singular: etcdcluster scope: Namespaced validation: openAPIV3Schema: properties: spec: properties: size: description: Size of the etcd cluster type: integer required: - size type: object type: object version: v1beta2 versions: - name: v1beta2 served: true storage: true status: acceptedNames: kind: EtcdCluster listKind: EtcdClusterList plural: etcdclusters shortNames: - etcdclus - etcd singular: etcdcluster conditions: - lastTransitionTime: "2019-09-14T00:55:31Z" message: no conflicts found reason: NoConflicts status: "True" type: NamesAccepted - lastTransitionTime: null message: the initial names have been accepted reason: InitialNamesAccepted status: "True" type: Established storedVersions: - v1beta2 7) Create an EtcdCluster oc create -f - <<EOF apiVersion: etcd.database.coreos.com/v1beta2 kind: EtcdCluster metadata: name: example namespace: test-operators spec: size: 3 version: 3.2.13 EOF 8) Ensure that a CRD version addition is allowed (Adding v1beta3 was served and storage version and switching v1beta2 to non-serving and no longer storage version https://github.com/bandrade/v3-testfiles/blob/v4.1/olm/configmap/configmap_etcd-v8.yaml#L587-L591) Apply the following object that adds v1beta3 on EtcdCluster CRD. oc apply -f https://raw.githubusercontent.com/bandrade/v3-testfiles/v4.1/olm/configmap/configmap_etcd-v8.yaml -n openshift-marketplace 7) Edit this subscription object to target channel: beta oc get csv -n test-operators NAME DISPLAY VERSION REPLACES PHASE etcdoperator.v0.9.4 etcd 0.9.4 etcdoperator.v0.9.2 Succeeded 8) Check EtcdCluster api version: oc get etcdcluster example -o yaml -n test-operators apiVersion: etcd.database.coreos.com/v1beta3 kind: EtcdCluster metadata: creationTimestamp: "2019-09-14T00:58:46Z" generation: 6 name: example namespace: test-operators resourceVersion: "158394" selfLink: /apis/etcd.database.coreos.com/v1beta3/namespaces/test-operators/etcdclusters/example uid: cdf8a566-d68a-11e9-b75b-0acc3ecdcaa0 9) Check etcdcluster CRD oc get crd etcdclusters.etcd.database.coreos.com -o yaml apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: creationTimestamp: "2019-09-14T00:55:31Z" generation: 2 name: etcdclusters.etcd.database.coreos.com resourceVersion: "162373" selfLink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/etcdclusters.etcd.database.coreos.com uid: 59d6b121-d68a-11e9-a45d-06e91e4192c2 spec: conversion: strategy: None group: etcd.database.coreos.com names: kind: EtcdCluster listKind: EtcdClusterList plural: etcdclusters shortNames: - etcdclus - etcd singular: etcdcluster scope: Namespaced validation: openAPIV3Schema: properties: size: description: Size of the etcd cluster type: integer type: object version: v1beta3 versions: - name: v1beta3 served: true storage: true - name: v1beta2 served: false storage: false status: acceptedNames: kind: EtcdCluster listKind: EtcdClusterList plural: etcdclusters shortNames: - etcdclus - etcd singular: etcdcluster conditions: - lastTransitionTime: "2019-09-14T00:55:31Z" message: no conflicts found reason: NoConflicts status: "True" type: NamesAccepted - lastTransitionTime: null message: the initial names have been accepted reason: InitialNamesAccepted status: "True" type: Established storedVersions: - v1beta2 - v1beta3 10) Upgrade operator again with etcdclusters CRD with only v1beta3 version and remove v1beta2 version completely https://github.com/bandrade/v3-testfiles/blob/v4.1/olm/configmap/configmap_etcd-v9.yaml#L1517-L1521 Apply the following object that adds v1beta3 on EtcdCluster CRD. oc apply -f https://raw.githubusercontent.com/bandrade/v3-testfiles/v4.1/olm/configmap/configmap_etcd-v9.yaml -n openshift-marketplace After this update the CatalogSource starts crashing due deletion of v1beta2: oc logs -f installed-community-global-operators-cdlzr -n openshift-marketplace time="2019-09-14T01:20:33Z" level=info msg="Using in-cluster kube client config" time="2019-09-14T01:20:33Z" level=info msg="loading CRDs" configmap=installed-community-global-operators ns=openshift-marketplace time="2019-09-14T01:20:33Z" level=info msg="loading Bundles" configmap=installed-community-global-operators ns=openshift-marketplace time="2019-09-14T01:20:33Z" level=info msg="loading Packages" configmap=installed-community-global-operators ns=openshift-marketplace time="2019-09-14T01:20:33Z" level=fatal msg="permissive mode disabled" configMapName=installed-community-global-operators configMapNamespace=openshift-marketplace error="error loading manifests from configmap: couldn't find owned CRD in crd list etcd.database.coreos.com/v1beta2/EtcdCluster (etcdclusters): %!s(<nil>)" port=50051 Considering that it was not possible deprecate v1beta2, moving it back to ASSIGNED.
To test deprecating CRDs and removing them from the system, you need to use the new operator registry bundle format to leverage nested directories for each version. Please retest using that format. https://github.com/operator-framework/operator-registry#manifest-format
After changing the method to operator-registry, v1beta2 was deprecated as expected, marking as VERIFIED. Cluster Version: 4.2.0-0.nightly-2019-09-17-001320 OLM version: 0.11.0 git commit: ceebad14a2f8121266c3b5c9630428270d5d784b operator-registry structure: operator-registry/manifests/etcd/ ├── 0.6.1 │ ├── etcdcluster.crd.yaml │ └── etcdoperator.clusterserviceversion.yaml ├── 0.9.0 │ ├── etcdbackup.crd.yaml │ ├── etcdcluster.crd.yaml │ ├── etcdoperator.v0.9.0.clusterserviceversion.yaml │ └── etcdrestore.crd.yaml ├── 0.9.2 │ ├── etcdbackup.crd.yaml │ ├── etcdcluster.crd.yaml │ ├── etcdoperator.v0.9.2.clusterserviceversion.yaml │ └── etcdrestore.crd.yaml ├── 0.9.4 │ ├── etcdbackup.crd.yaml │ ├── etcdcluster.crd.yaml │ ├── etcdoperator.v0.9.4.clusterserviceversion.yaml │ └── etcdrestore.crd.yaml └── etcd.package.yaml Package details: cat operator-registry/manifests/etcd/etcd.package.yaml packageName: etcd channels: - name: alpha currentCSV: etcdoperator.v0.9.0 - name: beta currentCSV: etcdoperator.v0.9.2 - name: rc currentCSV: etcdoperator.v0.9.4 defaultChannel: alpha CRD Details: cat operator-registry/manifests/etcd/0.9.0/etcdcluster.crd.yaml apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: etcdclusters.etcd.database.coreos.com spec: group: etcd.database.coreos.com scope: Namespaced names: plural: etcdclusters singular: etcdcluster kind: EtcdCluster listKind: EtcdClusterList shortNames: - etcdclus - etcd version: v1beta2 versions: - name: v1beta2 served: true storage: true cat operator-registry/manifests/etcd/0.9.2/etcdcluster.crd.yaml apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: etcdclusters.etcd.database.coreos.com spec: group: etcd.database.coreos.com scope: Namespaced names: plural: etcdclusters singular: etcdcluster kind: EtcdCluster listKind: EtcdClusterList shortNames: - etcdclus - etcd version: v1beta3 versions: - name: v1beta3 served: true storage: true - name: v1beta2 served: false storage: false cat operator-registry/manifests/etcd/0.9.4/etcdcluster.crd.yaml apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: etcdclusters.etcd.database.coreos.com spec: group: etcd.database.coreos.com scope: Namespaced names: plural: etcdclusters singular: etcdcluster kind: EtcdCluster listKind: EtcdClusterList shortNames: - etcdclus - etcd version: v1beta3 versions: - name: v1beta3 served: true storage: true Build the openshift-registry image: docker build -f upstream-example.Dockerfile -t quay.io/bandrade/etcd-operator:bug-1746270 . Create an OperatorSource pointing to this appregistry repo oc apply -f - <<EOF apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: name: etcd-bug-operator namespace: openshift-marketplace spec: sourceType: grpc image: quay.io/bandrade/etcd-operator:bug-1746270-1 displayName: ETCD Bug Operators publisher: bandrade EOF Create the OperatorGroup on test-operators namespace: oc apply -f - <<EOF apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: test-operators-og namespace: test-operators spec: targetNamespaces: - test-operators EOF Create a subscription with a package that exists on this : oc apply -f - <<EOF apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: etcd-bug namespace: test-operators spec: source: etcd-bug-operator sourceNamespace: openshift-marketplace name: etcd startingCSV: etcdoperator.v0.9.0 channel: alpha installPlanApproval: Automatic EOF The csv should be created successfully. oc get csv -n test-operators NAME DISPLAY VERSION REPLACES PHASE etcdoperator.v0.9.0 etcd 0.9.0 Succeeded Check etcdcluster CRD oc get crd etcdclusters.etcd.database.coreos.com -o yaml apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: creationTimestamp: "2019-09-17T17:08:39Z" generation: 1 name: etcdclusters.etcd.database.coreos.com resourceVersion: "55693" selfLink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/etcdclusters.etcd.database.coreos.com uid: cb5f88f2-d96d-11e9-b608-0a1e1889ecf4 spec: conversion: strategy: None group: etcd.database.coreos.com names: kind: EtcdCluster listKind: EtcdClusterList plural: etcdclusters shortNames: - etcdclus - etcd singular: etcdcluster scope: Namespaced version: v1beta2 versions: - name: v1beta2 served: true storage: true status: acceptedNames: kind: EtcdCluster listKind: EtcdClusterList plural: etcdclusters shortNames: - etcdclus - etcd singular: etcdcluster conditions: - lastTransitionTime: "2019-09-17T17:08:39Z" message: no conflicts found reason: NoConflicts status: "True" type: NamesAccepted - lastTransitionTime: null message: the initial names have been accepted reason: InitialNamesAccepted status: "True" type: Established storedVersions: - v1beta2 Edit this subscription object to target channel: beta oc get csv -n test-operators NAME DISPLAY VERSION REPLACES PHASE etcdoperator.v0.9.2 etcd 0.9.2 etcdoperator.v0.9.0 Succeed Check etcdcluster CRD oc get crd etcdclusters.etcd.database.coreos.com -o yaml apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: creationTimestamp: "2019-09-17T17:08:39Z" generation: 2 name: etcdclusters.etcd.database.coreos.com resourceVersion: "57091" selfLink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/etcdclusters.etcd.database.coreos.com uid: cb5f88f2-d96d-11e9-b608-0a1e1889ecf4 spec: conversion: strategy: None group: etcd.database.coreos.com names: kind: EtcdCluster listKind: EtcdClusterList plural: etcdclusters shortNames: - etcdclus - etcd singular: etcdcluster scope: Namespaced version: v1beta3 versions: - name: v1beta3 served: true storage: true - name: v1beta2 served: false storage: false status: acceptedNames: kind: EtcdCluster listKind: EtcdClusterList plural: etcdclusters shortNames: - etcdclus - etcd singular: etcdcluster conditions: - lastTransitionTime: "2019-09-17T17:08:39Z" message: no conflicts found reason: NoConflicts status: "True" type: NamesAccepted - lastTransitionTime: null message: the initial names have been accepted reason: InitialNamesAccepted status: "True" type: Established storedVersions: - v1beta2 - v1beta3 Edit this subscription object to target channel: rc oc get csv -n test-operators NAME DISPLAY VERSION REPLACES PHASE etcdoperator.v0.9.4 etcd 0.9.4 etcdoperator.v0.9.2 Succeeded oc get crd etcdclusters.etcd.database.coreos.com -o yaml apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: creationTimestamp: "2019-09-17T17:08:39Z" generation: 3 name: etcdclusters.etcd.database.coreos.com resourceVersion: "58331" selfLink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/etcdclusters.etcd.database.coreos.com uid: cb5f88f2-d96d-11e9-b608-0a1e1889ecf4 spec: conversion: strategy: None group: etcd.database.coreos.com names: kind: EtcdCluster listKind: EtcdClusterList plural: etcdclusters shortNames: - etcdclus - etcd singular: etcdcluster scope: Namespaced version: v1beta3 versions: - name: v1beta3 served: true storage: true status: acceptedNames: kind: EtcdCluster listKind: EtcdClusterList plural: etcdclusters shortNames: - etcdclus - etcd singular: etcdcluster conditions: - lastTransitionTime: "2019-09-17T17:08:39Z" message: no conflicts found reason: NoConflicts status: "True" type: NamesAccepted - lastTransitionTime: null message: the initial names have been accepted reason: InitialNamesAccepted status: "True" type: Established storedVersions: - v1beta3
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922