Bug 1907792 - The `overrides` of the OperatorCondition cannot block the operator upgrade
Summary: The `overrides` of the OperatorCondition cannot block the operator upgrade
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.7
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.7.0
Assignee: Evan Cordell
QA Contact: Jian Zhang
URL:
Whiteboard:
: 1909640 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-15 09:10 UTC by Jian Zhang
Modified: 2021-02-24 15:44 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-24 15:43:55 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github operator-framework operator-lifecycle-manager pull 1911 0 None closed Bug 1907792: Change the OperatorUpgradeable condition to Upgradeable 2021-01-13 06:51:35 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:44:37 UTC

Description Jian Zhang 2020-12-15 09:10:22 UTC
Description of problem:
The `overrides` of the OperatorCondition cannot block the operator upgrade.

Version-Release number of selected component (if applicable):
[root@preserve-olm-env data]# oc -n openshift-operator-lifecycle-manager exec catalog-operator-bb89cfd76-jbdqf -- olm --version
OLM version: 0.17.0
git commit: 4b66803055a8ab611447c33ed86e755ad39cb313

How reproducible:
always

Steps to Reproduce:
1. Create a CatalogSource to consume the image: quay.io/olmqe/etcd-index:0.9.2-oc
[root@preserve-olm-env data]# cat cs-poll.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: oc-test
spec:
  displayName: Jian Test
  sourceType: grpc
  image: quay.io/olmqe/etcd-index:0.9.2-oc
  updateStrategy:
    registryPoll:
      interval: 10m

2. Subscribe the etcdoperator.v0.9.2 with automatic upgrade.
[root@preserve-olm-env data]# cat sub-new.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd
  namespace: default
spec:
  channel: "4.6"
  installPlanApproval: Automatic
  name: etcd
  source: oc-test
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.2

[root@preserve-olm-env data]# oc create -f  sub-new.yaml 
subscription.operators.coreos.com/etcd created

[root@preserve-olm-env data]# oc get sub -A
NAMESPACE   NAME   PACKAGE   SOURCE    CHANNEL
default     etcd   etcd      oc-test   4.6
[root@preserve-olm-env data]# oc get ip
NAME            CSV                   APPROVAL    APPROVED
install-bgrtm   etcdoperator.v0.9.2   Automatic   true
[root@preserve-olm-env data]# oc get csv
NAME                  DISPLAY   VERSION   REPLACES   PHASE
etcdoperator.v0.9.2   etcd      0.9.2                Succeeded

[root@preserve-olm-env data]# oc get operatorcondition
NAME                  AGE
etcdoperator.v0.9.2   9m

3. Modify the OperatorCondition etcdoperator.v0.9.2, add the `overrrides` field:
Set the `Upgradeable` to `False`.

[root@preserve-olm-env data]# oc get operatorcondition etcdoperator.v0.9.2 -o yaml
apiVersion: operators.coreos.com/v1
kind: OperatorCondition
...
spec:
  deployments:
  - etcd-operator
  overrides:
  - lastTransitionTime: "2020-12-15T23:15:55Z"
    message: Disbale the upgrade.
    reason: upgradeIsNotSafe
    status: "False"
    type: Upgradeable
  serviceAccounts:
  - etcd-operator

4, Add the etcd 0.9.4 bundle to this index image: quay.io/olmqe/etcd-index:0.9.2-oc
[root@preserve-olm-env etcd]# opm index add -u docker -b quay.io/olmqe/etcd-bundle:0.9.4 -f quay.io/olmqe/etcd-index:0.9.2-oc -t quay.io/olmqe/etcd-index:0.9.2-oc --mode semver
INFO[0000] building the index                            bundles="[quay.io/olmqe/etcd-bundle:0.9.4]"
...
INFO[0002] [docker build -f index.Dockerfile923284778 -t quay.io/olmqe/etcd-index:0.9.2-oc .]  bundles="[quay.io/olmqe/etcd-bundle:0.9.4]"
[root@preserve-olm-env etcd]# docker push quay.io/olmqe/etcd-index:0.9.2-oc
...

5, Wait for the CatalogSource upgrade.
[root@preserve-olm-env data]# oc get pods -w
NAME                                                              READY   STATUS      RESTARTS   AGE
8cc7f4add666d4b2bd624f5b2ebc5f0dc940bb77327e333b208b8453f377p5p   0/1     Completed   0          9m58s
...
oc-test-6bp9r                                                     0/1     Pending             0          0s
community-operators-rzh9z                                         0/1     Pending             0          0s
oc-test-6bp9r                                                     0/1     Pending             0          0s
community-operators-rzh9z                                         0/1     Pending             0          0s
oc-test-6bp9r                                                     0/1     ContainerCreating   0          0s
community-operators-rzh9z                                         0/1     ContainerCreating   0          0s
community-operators-rzh9z                                         0/1     ContainerCreating   0          2s
oc-test-6bp9r                                                     0/1     ContainerCreating   0          2s

6, Check if the installed 0.9.2 etcd operator upgrade.


Actual results:
The etcdoperatorv0.9.2 was upgraded to etcdoperator.v0.9.4, but it should not.

[root@preserve-olm-env data]# oc get csv -n default
NAME                  DISPLAY   VERSION   REPLACES              PHASE
etcdoperator.v0.9.4   etcd      0.9.4     etcdoperator.v0.9.2   Succeeded
[root@preserve-olm-env data]# oc get operatorcondition -n default
NAME                  AGE
etcdoperator.v0.9.4   110s
[root@preserve-olm-env data]# oc get operatorcondition -n default etcdoperator.v0.9.4 -o yaml
apiVersion: operators.coreos.com/v1
kind: OperatorCondition
...
spec:
  deployments:
  - etcd-operator
  serviceAccounts:
  - etcd-operator

Expected results:
The etcdoperatorv0.9.2 cannot be upgraded to 0.9.4 since the `Upgradeable` of OperatorCondition is false.

Additional info:

Comment 1 Nick Hale 2020-12-21 18:26:50 UTC
*** Bug 1909640 has been marked as a duplicate of this bug. ***

Comment 2 Vu Dinh 2020-12-21 18:29:58 UTC
The issue here is we use "OperatorUpgradeable" as the status. That is a mistake. We are correcting this to be "Upgradeable" as it is specified in the enhancement.

Comment 6 Jian Zhang 2021-01-06 01:58:05 UTC
Cluster version is 4.7.0-0.nightly-2021-01-04-215816
[root@preserve-olm-env data]# oc  -n openshift-operator-lifecycle-manager exec catalog-operator-5b4b6587c4-2qgfb -- olm --version
OLM version: 0.17.0
git commit: 69dc076d610b033d3964ed40c4ac27ece3424851


1, Install the etcdoperator v0.9.2 with Manual approval
[root@preserve-olm-env data]# cat sub-etcd-cluster.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd
  namespace: openshift-operators
spec:
  channel: clusterwide-alpha
  installPlanApproval: Manual
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.2-clusterwide
[root@preserve-olm-env data]# oc create -f sub-etcd-cluster.yaml 
subscription.operators.coreos.com/etcd created

2, Approve the v0.9.2 installation.
[root@preserve-olm-env data]# oc get ip
NAME            CSV                               APPROVAL   APPROVED
install-2b8z7   etcdoperator.v0.9.4-clusterwide   Manual     false
install-tcjbg   etcdoperator.v0.9.2-clusterwide   Manual     true
[root@preserve-olm-env data]# oc get csv
NAME                              DISPLAY   VERSION             REPLACES              PHASE
etcdoperator.v0.9.2-clusterwide   etcd      0.9.2-clusterwide   etcdoperator.v0.9.0   Installing
[root@preserve-olm-env data]# oc get operatorcondition
NAME                              AGE
etcdoperator.v0.9.2-clusterwide   38s

3, Add the `overrides` field to this OperatorCondition etcdoperator.v0.9.2-clusterwide.
...
spec:
  deployments:
  - etcd-operator
  serviceAccounts:
  - etcd-operator
  overrides:
  - message: Disbale the upgrade.
    reason: upgradeIsNotSafe
    status: "False"
    type: Upgradeable

[root@preserve-olm-env data]# oc edit operatorcondition etcdoperator.v0.9.2-clusterwide
error: operatorconditions.operators.coreos.com "etcdoperator.v0.9.2-clusterwide" is invalid
operatorcondition.operators.coreos.com/etcdoperator.v0.9.2-clusterwide edited

4, Approve the v0.9.4 installation.
[root@preserve-olm-env data]# oc get ip
NAME            CSV                               APPROVAL   APPROVED
install-2b8z7   etcdoperator.v0.9.4-clusterwide   Manual     true
install-tcjbg   etcdoperator.v0.9.2-clusterwide   Manual     true
[root@preserve-olm-env data]# oc get csv
NAME                              DISPLAY   VERSION             REPLACES                          PHASE
etcdoperator.v0.9.2-clusterwide   etcd      0.9.2-clusterwide   etcdoperator.v0.9.0               Replacing
etcdoperator.v0.9.4-clusterwide   etcd      0.9.4-clusterwide   etcdoperator.v0.9.2-clusterwide   Pending

As you can see, the v0.9.4 CSV in `Pending` status.
[root@preserve-olm-env data]# oc describe csv etcdoperator.v0.9.4-clusterwide
Name:         etcdoperator.v0.9.4-clusterwide
...
Events:
  Type    Reason                           Age        From                        Message
  ----    ------                           ----       ----                        -------
  Normal  RequirementsUnknown              <invalid>  operator-lifecycle-manager  requirements not yet checked
  Normal  OperatorConditionNotUpgradeable  <invalid>  operator-lifecycle-manager  operator is not upgradeable: The operator is not upgradeable: Disbale the upgrade.

5, Change the Upgradable status of the etcdoperator.v0.9.2-clusterwide to "True" from the "False".

[root@preserve-olm-env data]# oc get operatorcondition
NAME                              AGE
etcdoperator.v0.9.2-clusterwide   13m
etcdoperator.v0.9.4-clusterwide   10m

[root@preserve-olm-env data]# oc edit operatorcondition etcdoperator.v0.9.2-clusterwide 
error: operatorconditions.operators.coreos.com "etcdoperator.v0.9.2-clusterwide" is invalid
operatorcondition.operators.coreos.com/etcdoperator.v0.9.2-clusterwide edited

[root@preserve-olm-env data]# oc get operatorcondition
NAME                              AGE
etcdoperator.v0.9.4-clusterwide   11m
[root@preserve-olm-env data]# oc get csv
NAME                              DISPLAY   VERSION             REPLACES                          PHASE
etcdoperator.v0.9.4-clusterwide   etcd      0.9.4-clusterwide   etcdoperator.v0.9.2-clusterwide   Succeeded
[root@preserve-olm-env data]# 
[root@preserve-olm-env data]# oc get pods
NAME                             READY   STATUS        RESTARTS   AGE
etcd-operator-5b8f9c4fbc-jhn5d   3/3     Running       0          30s
etcd-operator-869664477d-t57vn   0/3     Terminating   0          15m

The v0.9.2 can be upgraded to v0.9.4 successfully, looks good to me, verify it.

Comment 9 errata-xmlrpc 2021-02-24 15:43:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.