Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1746270

Summary: Unable to deprecate/remove a version from CRD if that version was previously storage version
Product: OpenShift Container Platform Reporter: Vu Dinh <vdinh>
Component: OLMAssignee: Vu Dinh <vdinh>
OLM sub component: OLM QA Contact: Bruno Andrade <bandrade>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: bandrade, chuo, dageoffr, ecordell, jfan, scolange
Version: 4.2.0   
Target Milestone: ---   
Target Release: 4.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-16 06:38:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vu Dinh 2019-08-28 06:55:51 UTC 
Description of problem:

Unable to deprecate/remove a version from CRD if that version was previously storage version.


Version-Release number of selected component (if applicable):


How reproducible:

Deploy an etcd operator using etcdclusters CRD with v1beta2 version as serving and storage version. Next, upgrade the operator with etcdclusters CRD to add a new version v1beta3 as serving and storage version and switching v1beta2 to non-serving and no longer storage version (this happens by default as only one version is marked as storage version). Then, upgrade operator again with etcdclusters CRD with only v1beta3 version and remove v1beta2 version completely. The upgrade process will fail as it complains v1beta2 is still in storedVersion in CRD's status.

Steps to Reproduce:
1. Deploy an etcd operator using etcdclusters CRD with v1beta2 version as serving and storage version
2. Upgrade the operator with etcdclusters CRD to add a new version v1beta3 as serving and storage version and switching v1beta2 to non-serving and no longer storage version
3. Upgrade operator again with etcdclusters CRD with only v1beta3 version and remove v1beta2 version completely

Actual results: The upgrade process will fail as it complains v1beta2 is still in storedVersion in CRD's status.

Expected results: The CRD should be upgraded properly as OLM should gracefully handle the deprecated version in storedVersions.

Additional info:
Comment 1 Vu Dinh 2019-08-28 06:58:04 UTC 
Pending PR: https://github.com/operator-framework/operator-lifecycle-manager/pull/983
Comment 3 Jian Zhang 2019-09-11 03:12:57 UTC 
Current the latest payload is 4.2.0-0.nightly-2019-09-10-181551, which the fixed PR hasn't merged in.
mac:~ jianzhang$ oc exec catalog-operator-5544569dd7-lggqq  -- olm --version
OLM version: 0.11.0
git commit: 5a5389cb8d831e79acade535d947d4ad4a5c40a7

Change status to MODIFIED.
Comment 5 Bruno Andrade 2019-09-14 01:43:51 UTC 
Verify failed.

Cluster Version:4.2.0-0.nightly-2019-09-13-113908
OLM version: 0.11.0
git commit: 2760276f78584ab485d448fa53abda8025fe5cb0

Steps used to reproduce:

1) Create the ConfigMap and the Catalog Source with etcd package with v1beta2 version as serving and storage version (https://github.com/bandrade/v3-testfiles/blob/v4.1/olm/configmap/configmap_etcd-v7.yaml#L667-L669)

oc create -f https://raw.githubusercontent.com/bandrade/v3-testfiles/v4.1/olm/configmap/configmap_etcd-v7.yaml -n openshift-marketplace
oc create -f https://raw.githubusercontent.com/bandrade/v3-testfiles/v4.1/olm/catalogsource/catalogsource.yaml -n openshift-marketplace

2) Create namespace test-operators
oc create ns test-operators

3) Create the Operator Group

oc create -f - <<EOF
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: test-operators-og
  namespace: test-operators
spec:
  targetNamespaces:
  - test-operators
EOF

4) Create the subscription, as below:
oc create -f https://raw.githubusercontent.com/bandrade/v3-testfiles/v4.1/olm/subscription/test.yaml -n test-operators

5) The csv should be created successfully.

oc get csv
NAME                  DISPLAY   VERSION   REPLACES   PHASE
etcdoperator.v0.9.2   etcd      0.9.2                Succeeded

6) Check CRD:
oc get crd etcdclusters.etcd.database.coreos.com -o yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  creationTimestamp: "2019-09-14T00:55:31Z"
  generation: 1
  name: etcdclusters.etcd.database.coreos.com
  resourceVersion: "156385"
  selfLink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/etcdclusters.etcd.database.coreos.com
  uid: 59d6b121-d68a-11e9-a45d-06e91e4192c2
spec:
  conversion:
    strategy: None
  group: etcd.database.coreos.com
  names:
    kind: EtcdCluster
    listKind: EtcdClusterList
    plural: etcdclusters
    shortNames:
    - etcdclus
    - etcd
    singular: etcdcluster
  scope: Namespaced
  validation:
    openAPIV3Schema:
      properties:
        spec:
          properties:
            size:
              description: Size of the etcd cluster
              type: integer
          required:
          - size
          type: object
      type: object
  version: v1beta2
  versions:
  - name: v1beta2
    served: true
    storage: true
status:
  acceptedNames:
    kind: EtcdCluster
    listKind: EtcdClusterList
    plural: etcdclusters
    shortNames:
    - etcdclus
    - etcd
    singular: etcdcluster
  conditions:
  - lastTransitionTime: "2019-09-14T00:55:31Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: null
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  storedVersions:
  - v1beta2


7) Create an EtcdCluster

oc create -f - <<EOF
apiVersion: etcd.database.coreos.com/v1beta2
kind: EtcdCluster
metadata:
  name: example
  namespace: test-operators
spec:
  size: 3
  version: 3.2.13
EOF

8) Ensure that a CRD version addition is allowed (Adding v1beta3 was served and storage version and switching v1beta2 to non-serving and no longer storage version https://github.com/bandrade/v3-testfiles/blob/v4.1/olm/configmap/configmap_etcd-v8.yaml#L587-L591)

Apply the following object that adds v1beta3 on EtcdCluster CRD.

oc apply -f https://raw.githubusercontent.com/bandrade/v3-testfiles/v4.1/olm/configmap/configmap_etcd-v8.yaml -n openshift-marketplace

7) Edit this subscription object to target channel: beta

oc get csv -n test-operators
NAME                  DISPLAY   VERSION   REPLACES              PHASE
etcdoperator.v0.9.4   etcd      0.9.4     etcdoperator.v0.9.2   Succeeded


8) Check EtcdCluster api version:
oc get etcdcluster example -o yaml -n test-operators
apiVersion: etcd.database.coreos.com/v1beta3
kind: EtcdCluster
metadata:
  creationTimestamp: "2019-09-14T00:58:46Z"
  generation: 6
  name: example
  namespace: test-operators
  resourceVersion: "158394"
  selfLink: /apis/etcd.database.coreos.com/v1beta3/namespaces/test-operators/etcdclusters/example
  uid: cdf8a566-d68a-11e9-b75b-0acc3ecdcaa0

9) Check etcdcluster CRD

oc get crd etcdclusters.etcd.database.coreos.com -o yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  creationTimestamp: "2019-09-14T00:55:31Z"
  generation: 2
  name: etcdclusters.etcd.database.coreos.com
  resourceVersion: "162373"
  selfLink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/etcdclusters.etcd.database.coreos.com
  uid: 59d6b121-d68a-11e9-a45d-06e91e4192c2
spec:
  conversion:
    strategy: None
  group: etcd.database.coreos.com
  names:
    kind: EtcdCluster
    listKind: EtcdClusterList
    plural: etcdclusters
    shortNames:
    - etcdclus
    - etcd
    singular: etcdcluster
  scope: Namespaced
  validation:
    openAPIV3Schema:
      properties:
        size:
          description: Size of the etcd cluster
          type: integer
      type: object
  version: v1beta3
  versions:
  - name: v1beta3
    served: true
    storage: true
  - name: v1beta2
    served: false
    storage: false
status:
  acceptedNames:
    kind: EtcdCluster
    listKind: EtcdClusterList
    plural: etcdclusters
    shortNames:
    - etcdclus
    - etcd
    singular: etcdcluster
  conditions:
  - lastTransitionTime: "2019-09-14T00:55:31Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: null
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  storedVersions:
  - v1beta2
  - v1beta3

10) Upgrade operator again with etcdclusters CRD with only v1beta3 version and remove v1beta2 version completely https://github.com/bandrade/v3-testfiles/blob/v4.1/olm/configmap/configmap_etcd-v9.yaml#L1517-L1521

Apply the following object that adds v1beta3 on EtcdCluster CRD.
oc apply -f https://raw.githubusercontent.com/bandrade/v3-testfiles/v4.1/olm/configmap/configmap_etcd-v9.yaml -n openshift-marketplace

After this update the CatalogSource starts crashing due deletion of v1beta2:

oc logs -f installed-community-global-operators-cdlzr -n openshift-marketplace
time="2019-09-14T01:20:33Z" level=info msg="Using in-cluster kube client config"
time="2019-09-14T01:20:33Z" level=info msg="loading CRDs" configmap=installed-community-global-operators ns=openshift-marketplace
time="2019-09-14T01:20:33Z" level=info msg="loading Bundles" configmap=installed-community-global-operators ns=openshift-marketplace
time="2019-09-14T01:20:33Z" level=info msg="loading Packages" configmap=installed-community-global-operators ns=openshift-marketplace
time="2019-09-14T01:20:33Z" level=fatal msg="permissive mode disabled" configMapName=installed-community-global-operators configMapNamespace=openshift-marketplace error="error loading manifests from configmap: couldn't find owned CRD in crd list etcd.database.coreos.com/v1beta2/EtcdCluster (etcdclusters): %!s(<nil>)" port=50051


Considering that it was not possible deprecate v1beta2, moving it back to ASSIGNED.
Comment 6 Dan Geoffroy 2019-09-17 13:02:45 UTC 
To test deprecating CRDs and removing them from the system, you need to use the new operator registry bundle format to leverage nested directories for each version.  Please retest using that format.  https://github.com/operator-framework/operator-registry#manifest-format
Comment 7 Bruno Andrade 2019-09-17 17:21:44 UTC 
After changing the method to operator-registry, v1beta2 was deprecated as expected, marking as VERIFIED.

Cluster Version: 4.2.0-0.nightly-2019-09-17-001320
OLM version: 0.11.0
git commit: ceebad14a2f8121266c3b5c9630428270d5d784b

operator-registry structure:
 operator-registry/manifests/etcd/

├── 0.6.1
│   ├── etcdcluster.crd.yaml
│   └── etcdoperator.clusterserviceversion.yaml
├── 0.9.0
│   ├── etcdbackup.crd.yaml
│   ├── etcdcluster.crd.yaml
│   ├── etcdoperator.v0.9.0.clusterserviceversion.yaml
│   └── etcdrestore.crd.yaml
├── 0.9.2
│   ├── etcdbackup.crd.yaml
│   ├── etcdcluster.crd.yaml
│   ├── etcdoperator.v0.9.2.clusterserviceversion.yaml
│   └── etcdrestore.crd.yaml
├── 0.9.4
│   ├── etcdbackup.crd.yaml
│   ├── etcdcluster.crd.yaml
│   ├── etcdoperator.v0.9.4.clusterserviceversion.yaml
│   └── etcdrestore.crd.yaml
└── etcd.package.yaml


Package details:
cat operator-registry/manifests/etcd/etcd.package.yaml 
packageName: etcd
channels:
- name: alpha
  currentCSV: etcdoperator.v0.9.0
- name: beta
  currentCSV: etcdoperator.v0.9.2
- name: rc
  currentCSV: etcdoperator.v0.9.4
defaultChannel: alpha

CRD Details:

cat operator-registry/manifests/etcd/0.9.0/etcdcluster.crd.yaml 
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: etcdclusters.etcd.database.coreos.com
spec:
  group: etcd.database.coreos.com
  scope: Namespaced
  names:
    plural: etcdclusters
    singular: etcdcluster
    kind: EtcdCluster
    listKind: EtcdClusterList
    shortNames:
      - etcdclus
      - etcd
  version: v1beta2
  versions:
  - name: v1beta2
    served: true
    storage: true

cat operator-registry/manifests/etcd/0.9.2/etcdcluster.crd.yaml 
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: etcdclusters.etcd.database.coreos.com
spec:
  group: etcd.database.coreos.com
  scope: Namespaced
  names:
    plural: etcdclusters
    singular: etcdcluster
    kind: EtcdCluster
    listKind: EtcdClusterList
    shortNames:
      - etcdclus
      - etcd
  version: v1beta3
  versions:
  - name: v1beta3
    served: true
    storage: true
  - name: v1beta2
    served: false
    storage: false

cat operator-registry/manifests/etcd/0.9.4/etcdcluster.crd.yaml 
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: etcdclusters.etcd.database.coreos.com
spec:
  group: etcd.database.coreos.com
  scope: Namespaced
  names:
    plural: etcdclusters
    singular: etcdcluster
    kind: EtcdCluster
    listKind: EtcdClusterList
    shortNames:
      - etcdclus
      - etcd
  version: v1beta3
  versions:
  - name: v1beta3
    served: true
    storage: true

Build the openshift-registry image:
docker build -f upstream-example.Dockerfile -t quay.io/bandrade/etcd-operator:bug-1746270 . 

Create an OperatorSource pointing to this appregistry repo

oc apply -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: etcd-bug-operator
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/bandrade/etcd-operator:bug-1746270-1
  displayName: ETCD Bug Operators
  publisher: bandrade
EOF


Create the OperatorGroup on test-operators namespace:

oc apply -f - <<EOF
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: test-operators-og
  namespace: test-operators
spec:
  targetNamespaces:
  - test-operators
EOF


Create a subscription with a package that exists on this :

oc apply -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd-bug
  namespace: test-operators
spec:
  source: etcd-bug-operator
  sourceNamespace: openshift-marketplace
  name: etcd
  startingCSV: etcdoperator.v0.9.0
  channel: alpha
  installPlanApproval: Automatic
EOF

The csv should be created successfully.

oc get csv -n test-operators
NAME                  DISPLAY   VERSION   REPLACES   PHASE
etcdoperator.v0.9.0   etcd      0.9.0                Succeeded

Check etcdcluster CRD

oc get crd etcdclusters.etcd.database.coreos.com -o yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  creationTimestamp: "2019-09-17T17:08:39Z"
  generation: 1
  name: etcdclusters.etcd.database.coreos.com
  resourceVersion: "55693"
  selfLink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/etcdclusters.etcd.database.coreos.com
  uid: cb5f88f2-d96d-11e9-b608-0a1e1889ecf4
spec:
  conversion:
    strategy: None
  group: etcd.database.coreos.com
  names:
    kind: EtcdCluster
    listKind: EtcdClusterList
    plural: etcdclusters
    shortNames:
    - etcdclus
    - etcd
    singular: etcdcluster
  scope: Namespaced
  version: v1beta2
  versions:
  - name: v1beta2
    served: true
    storage: true
status:
  acceptedNames:
    kind: EtcdCluster
    listKind: EtcdClusterList
    plural: etcdclusters
    shortNames:
    - etcdclus
    - etcd
    singular: etcdcluster
  conditions:
  - lastTransitionTime: "2019-09-17T17:08:39Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: null
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  storedVersions:
  - v1beta2

Edit this subscription object to target channel: beta

oc get csv -n test-operators
NAME                  DISPLAY   VERSION   REPLACES              PHASE
etcdoperator.v0.9.2   etcd      0.9.2     etcdoperator.v0.9.0   Succeed

Check etcdcluster CRD
oc get crd etcdclusters.etcd.database.coreos.com -o yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  creationTimestamp: "2019-09-17T17:08:39Z"
  generation: 2
  name: etcdclusters.etcd.database.coreos.com
  resourceVersion: "57091"
  selfLink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/etcdclusters.etcd.database.coreos.com
  uid: cb5f88f2-d96d-11e9-b608-0a1e1889ecf4
spec:
  conversion:
    strategy: None
  group: etcd.database.coreos.com
  names:
    kind: EtcdCluster
    listKind: EtcdClusterList
    plural: etcdclusters
    shortNames:
    - etcdclus
    - etcd
    singular: etcdcluster
  scope: Namespaced
  version: v1beta3
  versions:
  - name: v1beta3
    served: true
    storage: true
  - name: v1beta2
    served: false
    storage: false
status:
  acceptedNames:
    kind: EtcdCluster
    listKind: EtcdClusterList
    plural: etcdclusters
    shortNames:
    - etcdclus
    - etcd
    singular: etcdcluster
  conditions:
  - lastTransitionTime: "2019-09-17T17:08:39Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: null
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  storedVersions:
  - v1beta2
  - v1beta3

Edit this subscription object to target channel: rc
oc get csv -n test-operators
NAME                  DISPLAY   VERSION   REPLACES              PHASE
etcdoperator.v0.9.4   etcd      0.9.4     etcdoperator.v0.9.2   Succeeded

oc get crd etcdclusters.etcd.database.coreos.com -o yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  creationTimestamp: "2019-09-17T17:08:39Z"
  generation: 3
  name: etcdclusters.etcd.database.coreos.com
  resourceVersion: "58331"
  selfLink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/etcdclusters.etcd.database.coreos.com
  uid: cb5f88f2-d96d-11e9-b608-0a1e1889ecf4
spec:
  conversion:
    strategy: None
  group: etcd.database.coreos.com
  names:
    kind: EtcdCluster
    listKind: EtcdClusterList
    plural: etcdclusters
    shortNames:
    - etcdclus
    - etcd
    singular: etcdcluster
  scope: Namespaced
  version: v1beta3
  versions:
  - name: v1beta3
    served: true
    storage: true
status:
  acceptedNames:
    kind: EtcdCluster
    listKind: EtcdClusterList
    plural: etcdclusters
    shortNames:
    - etcdclus
    - etcd
    singular: etcdcluster
  conditions:
  - lastTransitionTime: "2019-09-17T17:08:39Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: null
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  storedVersions:
  - v1beta3
Comment 8 errata-xmlrpc 2019-10-16 06:38:11 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922