Bug 1861690 - Upgrade to new OCS version is not starting cause of issue on OLM side
Summary: Upgrade to new OCS version is not starting cause of issue on OLM side
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.4
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.6.0
Assignee: Evan Cordell
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On: 1862852
Blocks: 1862407
TreeView+ depends on / blocked
 
Reported: 2020-07-29 09:39 UTC by Petr Balogh
Modified: 2020-08-10 07:34 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1862407 (view as bug list)
Environment:
Last Closed: 2020-08-06 01:39:33 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github operator-framework operator-lifecycle-manager pull 1689 0 None closed [release-4.5] Bug 1862407: fix(resolver): be smarter about the way downgrades are attempted 2021-02-02 13:39:03 UTC

Comment 5 Jian Zhang 2020-08-03 03:41:35 UTC
Cluster version is 4.6.0-0.nightly-2020-08-02-091622

1, Install OCS 4.4.1, failed.

mac:~ jianzhang$ oc get sub -n openshift-storage 
NAME           PACKAGE        SOURCE             CHANNEL
ocs-operator   ocs-operator   redhat-operators   stable-4.4
mac:~ jianzhang$ oc get ip -n openshift-storage 
No resources found in openshift-storage namespace.

E0803 02:34:50.343893       1 queueinformer_operator.go:290] sync "openshift-storage" failed: [found duplicate entries for ocs-operator.v4.4.1 in {community-operators openshift-marketplace}, found duplicate entries for ocs-operator.v4.2.3 in {community-operators openshift-marketplace}]
I0803 02:34:50.344014       1 event.go:278] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"openshift-storage", UID:"90e24d18-c76a-43af-b27d-70f9218fe171", APIVersion:"v1", ResourceVersion:"75129", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' [found duplicate entries for ocs-operator.v4.4.1 in {community-operators openshift-marketplace}, found duplicate entries for ocs-operator.v4.2.3 in {community-operators openshift-marketplace}]

Report a bug for this issue: https://bugzilla.redhat.com/show_bug.cgi?id=1862852

Comment 6 Evan Cordell 2020-08-03 17:45:42 UTC
We just identified and fixed that issue: https://github.com/operator-framework/operator-lifecycle-manager/pull/1702

Now that that has merged, we should be able to verify OCS installation in 4.6

Comment 7 Jian Zhang 2020-08-04 10:06:33 UTC
Hi Evan, 

Yes, we can install it on the OCP cluster, as follows.
Cluster version is 4.6.0-0.nightly-2020-08-04-035157
[root@preserve-olm-env data]# oc exec catalog-operator-746bcb45b5-bptgn -- olm --version
OLM version: 0.16.0
git commit: 83e1f768792f1b2d5f3b34c9d945c73394c873f6

[root@preserve-olm-env data]# oc get sub
NAME                                                                     PACKAGE                  SOURCE                 CHANNEL
lib-bucket-provisioner-alpha-community-operators-openshift-marketplace   lib-bucket-provisioner   community-operators    alpha
ocs-operator                                                             ocs-operator             redhat-operators-old   stable-4.4
[root@preserve-olm-env data]# oc get ip
NAME            CSV                   APPROVAL    APPROVED
install-2gjnf   ocs-operator.v4.4.1   Automatic   true
[root@preserve-olm-env data]# oc get csv
NAME                            DISPLAY                       VERSION   REPLACES   PHASE
lib-bucket-provisioner.v1.0.0   lib-bucket-provisioner        1.0.0                Succeeded
ocs-operator.v4.4.1             OpenShift Container Storage   4.4.1                Succeeded
[root@preserve-olm-env data]# oc get pods
NAME                                      READY   STATUS    RESTARTS   AGE
lib-bucket-provisioner-5dddd8c448-sdvw8   1/1     Running   0          2m33s
noobaa-operator-85c55d6dd5-fdxkp          1/1     Running   0          2m28s
ocs-operator-5d85b68f9b-4qbbz             1/1     Running   0          2m28s
rook-ceph-operator-5f5444857c-8mrbc       1/1     Running   0          2m28s

> 3. Add custom catalog to internal 4.4.2 build

@Petr Could you help share the custom CatalogSource with me so that I can upgrade it to the internal 4.4.2? Thanks!

Comment 10 Vu Dinh 2020-08-04 20:57:12 UTC
Hi,

I checked the cluster and I don't see the lib-bucket-provisioner v2 available in `openshift-storage` namespace. That v2 is needed for the upgrade process. It should be available in community catalog source.

Vu

Comment 11 Jian Zhang 2020-08-05 01:36:34 UTC
Hi, Vu

Thanks for your information! 

> It should be available in community catalog source.

I guess no. It from the RedHat operators, and I can only get the released version: 1.0.0, as follows.

@petr Do you happen to know how to get the CatalogSource that contains the v2 version lib-bucket-provisioner? Thanks!

[root@preserve-olm-env data]# oc get packagemanifest lib-bucket-provisioner -o yaml
apiVersion: packages.operators.coreos.com/v1
kind: PackageManifest
metadata:
  creationTimestamp: "2020-08-04T09:22:16Z"
  labels:
    catalog: community-operators
    catalog-namespace: openshift-marketplace
    operatorframework.io/arch.amd64: supported
    operatorframework.io/os.linux: supported
    provider: Red Hat
    provider-url: ""
  name: lib-bucket-provisioner
  namespace: openshift-storage
  selfLink: /apis/packages.operators.coreos.com/v1/namespaces/openshift-storage/packagemanifests/lib-bucket-provisioner
spec: {}
status:
  catalogSource: community-operators
  catalogSourceDisplayName: Community Operators
  catalogSourceNamespace: openshift-marketplace
  catalogSourcePublisher: Red Hat
  channels:
  - currentCSV: lib-bucket-provisioner.v1.0.0
    currentCSVDesc:
      annotations:
        alm-examples: |-
          [
            {
              "apiVersion": "objectbucket.io/v1alpha1",
              "kind": "ObjectBucketClaim",
              "metadata": {
                "name": "my-obc",
                "namespace": "my-app"
              },
              "spec": {
                "storageClassName": "object-bucket-class",
                "generateBucketName": "my-obc",
                "SSL": false,
                "versioned": false,
                "additionalConfig": {}
              }
            },
            {
              "apiVersion": "objectbucket.io/v1alpha1",
              "kind": "ObjectBucket",
              "metadata": {
                "name": "my-obc"
              },
              "spec": {
                "storageClassName": "object-bucket-class",
                "reclaimPolicy": "Delete",
                "claimRef": {
                  "name": "my-obc",
                  "namespace": "my-app"
                },
                "endpoint": {
                  "bucketHost": "xxx",
                  "bucketPort": 80,
                  "bucketName": "my-obc-1234-5678-1234-5678",
                  "region": "xxx",
                  "subRegion": "xxx",
                  "ssl": false,
                  "additionalConfig": {}
                },
                "additionalState": {}
              }
            }
          ]
        capabilities: Basic Install
        categories: Storage,Big Data
        certified: "false"
        containerImage: kubernetes/pause
        createdAt: "2014-07-19 07:02:32.267701"
        description: Library for the dynamic provisioning of object store buckets
          to be used by object store providers.
        repository: https://github.com/kube-object-storage/lib-bucket-provisioner
        support: Red Hat
      apiservicedefinitions: {}
      customresourcedefinitions:
        owned:
        - description: Claim a bucket just like claiming a PV. Automate you app bucket
            provisioning by creating OBC with your app deployment. A secret and configmap
            (name=claim) will be created with access details for the app pods.
          displayName: ObjectBucketClaim
          kind: ObjectBucketClaim
          name: objectbucketclaims.objectbucket.io
          version: v1alpha1
        - description: Used under-the-hood. Created per ObjectBucketClaim and keeps
            provisioning information
          displayName: ObjectBucket
          kind: ObjectBucket
          name: objectbuckets.objectbucket.io
          version: v1alpha1
      description: "### CRD-only Operator\n\nThis operator package is **CRD-only**
        and the operator is a no-op operator.\n\nInstead, bucket provisioners using
        this library are using these CRD's and using CSV [required-crds](https://github.com/operator-framework/operator-lifecycle-manager/blob/master/doc/design/building-your-csv.md#required-crds)
        them so that OLM can install it as a dependency.\n\n### Generic Bucket Provisioning\n\nKubernetes
        natively supports dynamic provisioning for many types of file and block storage,
        but lacks support for object bucket provisioning. \n\nThis repo is a placeholder
        for an object store bucket provisioning library, very similar to the Kubernetes
        [sig-storage-lib-external-provisioner](https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner/blob/master/controller/controller.go)
        library.\n\n### Known Provisioners\n- https://github.com/noobaa/noobaa-operator
        (NooBaa)\n- https://github.com/rook/rook (Rook-Ceph)\n- https://github.com/yard-turkey/aws-s3-provisioner
        (AWS-S3)\n"
      displayName: lib-bucket-provisioner
      installModes:
      - supported: true
        type: OwnNamespace
      - supported: true
        type: SingleNamespace
      - supported: true
        type: MultiNamespace
      - supported: true
        type: AllNamespaces
      keywords:
      - kubernetes
      - openshift
      - object
      - bucket
      - storage
      - cloud
      - s3
      links:
      - name: Github
        url: https://github.com/kube-object-storage/lib-bucket-provisioner
      maintainers:
      - email: jcope@redhat.com
        name: Jon Cope
      - email: jvance@redhat.com
        name: Jeff Vance
      - email: gmargali@redhat.com
        name: Guy Margalit
      - email: dzaken@redhat.com
        name: Danny Zaken
      - email: nbecker@redhat.com
        name: Nimrod Becker
      maturity: alpha
      provider:
        name: Red Hat
      relatedImages:
      - kubernetes/pause
      version: 1.0.0
    name: alpha
  defaultChannel: alpha
  packageName: lib-bucket-provisioner
  provider:
    name: Red Hat

Comment 12 Jian Zhang 2020-08-05 04:03:30 UTC
Ah, I find the reason. In OCP 4.6, as default, we use the CatalogSource instead of the OperatorSource. So, I guess there hasn't no lib-bucket operator in the Community CatalogSource yet. I create an OperatorSource for community operators. I can find it now.

1, Disable the community CatalogSource.
[root@preserve-olm-env data]# cat operatorhub.yaml
apiVersion: config.openshift.io/v1
kind: OperatorHub
metadata:
  name: cluster
spec:
  disableAllDefaultSources: false
  sources: [
    {
      name: "community-operators",
      disabled: true,
    }
  ]
[root@preserve-olm-env data]# oc apply -f operatorhub.yaml 
Warning: oc apply should be used on resource created by either oc create --save-config or oc apply
operatorhub.config.openshift.io/cluster configured

2, Create an OperatorSource for the community operators.
[root@preserve-olm-env data]# cat operatorsource-community.yaml
apiVersion: operators.coreos.com/v1
kind: OperatorSource
metadata:
  name: community-operators
  namespace: openshift-marketplace
spec:
  authorizationToken: {}
  displayName: Community QE Operators
  endpoint: https://quay.io/cnr
  publisher: QE
  registryNamespace: community-operators
  type: appregistry
[root@preserve-olm-env data]# oc create -f operatorsource-community.yaml
operatorsource.operators.coreos.com/community-operators created

[root@preserve-olm-env data]# oc get pods
NAME                                    READY   STATUS        RESTARTS   AGE
certified-operators-8qcmg               1/1     Running       0          14h
certified-operators-lrd85               1/1     Running       0          16m
community-operators-5f86684c7b-dpv7p    0/1     Terminating   0          4s
community-operators-6896b7f88d-97xvp    0/1     Pending       0          1s
community-operators-68bc549459-5kqph    0/1     Pending       0          0s
community-operators-6b7df9b8f9-pkg9c    0/1     Terminating   0          2s
community-operators-6d457ff7cb-dhfkx    0/1     Terminating   0          2s
community-operators-7b76d9cfdb-55nwk    0/1     Terminating   0          5s
community-operators-cb4dff97f-dm9xf     0/1     Terminating   0          5s

But, the communit pods in the creating-Terminating endless loop. Report a bug for this issue: https://bugzilla.redhat.com/show_bug.cgi?id=1866169
I think I shouldn't use the default catalogsource name: community-operators

Change another way to verify it.

1, Create an OpertorSource object called "community-operators-old" for community operators.
[root@preserve-olm-env data]# cat operatorsource-community.yaml
apiVersion: operators.coreos.com/v1
kind: OperatorSource
metadata:
  name: community-operators-old
  namespace: openshift-marketplace
spec:
  authorizationToken: {}
  displayName: Community QE Operators
  endpoint: https://quay.io/cnr
  publisher: QE
  registryNamespace: community-operators
  type: appregistry

[root@preserve-olm-env data]# oc get catalogsource -n openshift-marketplace
NAME                      DISPLAY                       TYPE   PUBLISHER   AGE
certified-operators       Certified Operators           grpc   Red Hat     18h
community-operators-old   Community QE Operators        grpc   QE          6m37s
...

[root@preserve-olm-env data]# oc create -f operatorsource-community.yaml
operatorsource.operators.coreos.com/community-operators-old created

[root@preserve-olm-env data]# oc get packagemanifest |grep lib
lib-bucket-provisioner                  Community QE Operators        4m13s

2, Modify this lib-bucket-provisioner subscription to use the new CatalogSource: community-operators-old

[root@preserve-olm-env data]# oc edit sub lib-bucket-provisioner
subscription.operators.coreos.com/lib-bucket-provisioner edited

3, Waiting for the automated upgrade.

[root@preserve-olm-env data]# oc get sub
NAME                                                                             PACKAGE                   SOURCE                    CHANNEL
awss3-operator-registry-original-community-operators-old-openshift-marketplace   awss3-operator-registry   community-operators-old   original
lib-bucket-provisioner                                                           lib-bucket-provisioner    community-operators-old   alpha
ocs-operator                                                                     ocs-operator              ocs-catalogsource         stable-4.4

[root@preserve-olm-env data]# oc get ip
NAME            CSV                             APPROVAL    APPROVED
install-hr2cc   awss3operator.1.0.0             Automatic   true
install-plmz5   lib-bucket-provisioner.v1.0.0   Automatic   true
install-s6mbk   ocs-operator.v4.4.1             Automatic   true

[root@preserve-olm-env data]# oc get csv
NAME                            DISPLAY                       VERSION   REPLACES                        PHASE
awss3operator.1.0.0             AWS S3 Operator               1.0.0                                     Succeeded
lib-bucket-provisioner.v2.0.0   lib-bucket-provisioner        2.0.0     lib-bucket-provisioner.v1.0.0   Succeeded
ocs-operator.v4.4.1             OpenShift Container Storage   4.4.1                                     Succeeded

The lib-bucket-provisioner has been updated to 2.0.0 auttomaically, but the ocs-operator still in 4.4.1 version.

Comment 16 Petr Balogh 2020-08-10 07:34:27 UTC
Sure, removing need info from me.


Note You need to log in before you can comment on or make changes to this bug.