Bug 1862407 - Upgrade to new OCS version is not starting cause of issue on OLM side
Summary: Upgrade to new OCS version is not starting cause of issue on OLM side
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.4
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.5.z
Assignee: Evan Cordell
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On: 1861690
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-31 11:26 UTC by Evan Cordell
Modified: 2020-08-17 20:06 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1861690
Environment:
Last Closed: 2020-08-17 20:05:57 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:3330 0 None None None 2020-08-17 20:06:19 UTC

Comment 5 Jian Zhang 2020-08-07 04:03:37 UTC
The fixed PR is: https://github.com/operator-framework/operator-lifecycle-manager/pull/1689
Cluster version is 4.5.0-0.nightly-2020-08-06-215703
[root@preserve-olm-env data]# oc exec catalog-operator-9bc6764d5-xm2gt -- olm --version
OLM version: 0.15.1
git commit: 56976b6ce26de6b0685784a28f7947e37c0870c9

1, I tried to install the OCS 4.4.1 version, but it failed. Seems like there is only v4.4.2 version for the OCS operator.
[root@preserve-olm-env data]# cat sub-ocs-41.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: ocs-operator
  namespace: openshift-storage
spec:
  channel: stable-4.4
  installPlanApproval: Automatic
  name: ocs-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  startingCSV: ocs-operator.v4.4.1

[root@preserve-olm-env data]# oc get sub 
NAME           PACKAGE        SOURCE             CHANNEL
ocs-operator   ocs-operator   redhat-operators   stable-4.4
[root@preserve-olm-env data]# oc get ip
No resources found in openshift-storage namespace.


E0807 03:56:44.306965       1 queueinformer_operator.go:290] sync "openshift-storage" failed: {ocs-operator stable-4.4 ocs-operator.v4.4.1 {redhat-operators openshift-marketplace}} not found: rpc error: code = Unknown desc = no entry found for ocs-operator stable-4.4 ocs-operator.v4.4.1

2, Install the OCS 4.4.2 instead of 4.4.1.

[root@preserve-olm-env data]# oc get sub
NAME           PACKAGE        SOURCE             CHANNEL
ocs-operator   ocs-operator   redhat-operators   stable-4.4
[root@preserve-olm-env data]# oc get ip
NAME            CSV                   APPROVAL    APPROVED
install-4tmdt   ocs-operator.v4.4.2   Automatic   true
[root@preserve-olm-env data]# oc get csv
NAME                                           DISPLAY                       VERSION                 REPLACES   PHASE
elasticsearch-operator.4.5.0-202008061550.p0   Elasticsearch Operator        4.5.0-202008061550.p0              Succeeded
ocs-operator.v4.4.2                            OpenShift Container Storage   4.4.2                              Succeeded

3, Add custom catalog to internal 4.4.2 build.

[root@preserve-olm-env data]# cat cs-ocs.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ocs-catalogsource
  namespace: openshift-marketplace
  labels:
      ocs-operator-internal: "true"
spec:
  displayName: Openshift Container Storage
  icon:
    base64data: ""
    mediatype: ""
  image: quay.io/rhceph-dev/ocs-olm-operator:4.4.2-510.ci
  publisher: Red Hat
  sourceType: grpc
[root@preserve-olm-env data]# oc create -f cs-ocs.yaml
catalogsource.operators.coreos.com/ocs-catalogsource created

4, Change subscription source to point to custom catalog.
[root@preserve-olm-env data]# oc get catalogsource -n openshift-marketplace
NAME                  DISPLAY                       TYPE   PUBLISHER   AGE
...
ocs-catalogsource     Openshift Container Storage   grpc   Red Hat     16m

[root@preserve-olm-env data]# oc get pods -n openshift-marketplace
NAME                                    READY   STATUS              RESTARTS   AGE
...
ocs-catalogsource-q85rj                 0/1     Running             0          2m18s

[root@preserve-olm-env data]# oc edit sub ocs-operator
subscription.operators.coreos.com/ocs-operator edited

[root@preserve-olm-env data]# oc get sub ocs-operator -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
...
spec:
  channel: stable-4.4
  installPlanApproval: Automatic
  name: ocs-operator
  source: ocs-catalogsource
  sourceNamespace: openshift-marketplace
  startingCSV: ocs-operator.v4.4.2-510.ci

5, Check if upgrade. No any updates.
[root@preserve-olm-env data]# oc get sub
NAME           PACKAGE        SOURCE              CHANNEL
ocs-operator   ocs-operator   ocs-catalogsource   stable-4.4
[root@preserve-olm-env data]# oc get ip
NAME            CSV                   APPROVAL    APPROVED
install-4tmdt   ocs-operator.v4.4.2   Automatic   true
[root@preserve-olm-env data]# oc get csv
NAME                                           DISPLAY                       VERSION                 REPLACES   PHASE
elasticsearch-operator.4.5.0-202008061550.p0   Elasticsearch Operator        4.5.0-202008061550.p0              Succeeded
ocs-operator.v4.4.2                            OpenShift Container Storage   4.4.2                              Succeeded

@petr I guess the v4.4.2 doesn't have the upgrade path for 4.4.2-510.ci. Could you help share how to install the 4.4.1? Or which version the v4.4.2 upgrade for?

Comment 7 Jian Zhang 2020-08-07 08:29:14 UTC
Hi Petr,

Many thanks for your detailed explanation! Test steps as follows:

1, Custome the 4.4.1 Catalogsource
[root@preserve-olm-env data]# oc get pods
NAME                                    READY   STATUS    RESTARTS   AGE
...
ocs-catalogsource-8jqps                 1/1     Running   0          35s

2, Subscribe the 4.4.1 ocs-operator.
[root@preserve-olm-env data]# cat sub-ocs-41.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: ocs-operator-4.4.1
  namespace: openshift-storage
spec:
  channel: stable-4.4
  installPlanApproval: Automatic
  name: ocs-operator
  source: ocs-catalogsource
  sourceNamespace: openshift-marketplace
  startingCSV: ocs-operator.v4.4.1-476.ci

[root@preserve-olm-env data]# oc get sub -n openshift-storage  
NAME                                                                      PACKAGE                   SOURCE                CHANNEL
awss3-operator-registry-alpha-community-operators-openshift-marketplace   awss3-operator-registry   community-operators   alpha
lib-bucket-provisioner-alpha-community-operators-openshift-marketplace    lib-bucket-provisioner    community-operators   alpha
ocs-operator-4.4.1                                                        ocs-operator              ocs-catalogsource     stable-4.4

[root@preserve-olm-env data]# oc get ip -n openshift-storage  
NAME            CSV                          APPROVAL    APPROVED
install-bxpnb   ocs-operator.v4.4.1-476.ci   Automatic   true
install-zj68b   awss3operator.1.0.1          Automatic   true
[root@preserve-olm-env data]# oc get csv -n openshift-storage  
NAME                                           DISPLAY                       VERSION                 REPLACES              PHASE
awss3operator.1.0.1                            AWS S3 Operator               1.0.1                   awss3operator.1.0.0   Succeeded
elasticsearch-operator.4.5.0-202008061550.p0   Elasticsearch Operator        4.5.0-202008061550.p0                         Succeeded
lib-bucket-provisioner.v1.0.0                  lib-bucket-provisioner        1.0.0                                         Succeeded
ocs-operator.v4.4.1-476.ci                     OpenShift Container Storage   4.4.1-476.ci                                  Succeeded

3, Edit the subscription object and upgrade it to the 4.4.2.
[root@preserve-olm-env data]# oc edit sub ocs-operator-4.4.1  -n openshift-storage
subscription.operators.coreos.com/ocs-operator-4.4.1 edited

[root@preserve-olm-env data]# oc get sub ocs-operator-4.4.1  -n openshift-storage -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
...
spec:
  channel: stable-4.4
  installPlanApproval: Automatic
  name: ocs-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  startingCSV: ocs-operator.v4.4.2
...

[root@preserve-olm-env data]# oc get ip -n openshift-storage
NAME            CSV                          APPROVAL    APPROVED
install-bxpnb   ocs-operator.v4.4.1-476.ci   Automatic   true
install-xbk6w   ocs-operator.v4.4.2          Automatic   true
install-zj68b   awss3operator.1.0.1          Automatic   true

[root@preserve-olm-env data]# oc get csv -n openshift-storage
NAME                                           DISPLAY                       VERSION                 REPLACES                        PHASE
awss3operator.1.0.1                            AWS S3 Operator               1.0.1                   awss3operator.1.0.0             Succeeded
elasticsearch-operator.4.5.0-202008061550.p0   Elasticsearch Operator        4.5.0-202008061550.p0                                   Succeeded
lib-bucket-provisioner.v2.0.0                  lib-bucket-provisioner        2.0.0                   lib-bucket-provisioner.v1.0.0   Succeeded
ocs-operator.v4.4.2                            OpenShift Container Storage   4.4.2                   ocs-operator.v4.4.1-476.ci      Succeeded
[root@preserve-olm-env data]# oc get pods -n openshift-storage
NAME                                      READY   STATUS    RESTARTS   AGE
aws-s3-provisioner-67fff7c5d-gcwjp        1/1     Running   0          15m
lib-bucket-provisioner-6586f6bdfc-pw728   1/1     Running   0          10m
noobaa-operator-699d559957-88fsd          1/1     Running   0          10m
ocs-operator-7f6c9db7c6-wt2dw             1/1     Running   0          10m
rook-ceph-operator-7b96b847-pbwv2         1/1     Running   0          10m

It can be upgrade successufuly, LGTM, verify it.

Comment 9 errata-xmlrpc 2020-08-17 20:05:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.5.6 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3330


Note You need to log in before you can comment on or make changes to this bug.