Bug 1861690

Cluster version is 4.6.0-0.nightly-2020-08-02-091622

1, Install OCS 4.4.1, failed.

mac:~ jianzhang$ oc get sub -n openshift-storage 
NAME           PACKAGE        SOURCE             CHANNEL
ocs-operator   ocs-operator   redhat-operators   stable-4.4
mac:~ jianzhang$ oc get ip -n openshift-storage 
No resources found in openshift-storage namespace.

E0803 02:34:50.343893       1 queueinformer_operator.go:290] sync "openshift-storage" failed: [found duplicate entries for ocs-operator.v4.4.1 in {community-operators openshift-marketplace}, found duplicate entries for ocs-operator.v4.2.3 in {community-operators openshift-marketplace}]
I0803 02:34:50.344014       1 event.go:278] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"openshift-storage", UID:"90e24d18-c76a-43af-b27d-70f9218fe171", APIVersion:"v1", ResourceVersion:"75129", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' [found duplicate entries for ocs-operator.v4.4.1 in {community-operators openshift-marketplace}, found duplicate entries for ocs-operator.v4.2.3 in {community-operators openshift-marketplace}]

Report a bug for this issue: https://bugzilla.redhat.com/show_bug.cgi?id=1862852

We just identified and fixed that issue: https://github.com/operator-framework/operator-lifecycle-manager/pull/1702

Now that that has merged, we should be able to verify OCS installation in 4.6

Hi Evan, 

Yes, we can install it on the OCP cluster, as follows.
Cluster version is 4.6.0-0.nightly-2020-08-04-035157
[root@preserve-olm-env data]# oc exec catalog-operator-746bcb45b5-bptgn -- olm --version
OLM version: 0.16.0
git commit: 83e1f768792f1b2d5f3b34c9d945c73394c873f6

[root@preserve-olm-env data]# oc get sub
NAME                                                                     PACKAGE                  SOURCE                 CHANNEL
lib-bucket-provisioner-alpha-community-operators-openshift-marketplace   lib-bucket-provisioner   community-operators    alpha
ocs-operator                                                             ocs-operator             redhat-operators-old   stable-4.4
[root@preserve-olm-env data]# oc get ip
NAME            CSV                   APPROVAL    APPROVED
install-2gjnf   ocs-operator.v4.4.1   Automatic   true
[root@preserve-olm-env data]# oc get csv
NAME                            DISPLAY                       VERSION   REPLACES   PHASE
lib-bucket-provisioner.v1.0.0   lib-bucket-provisioner        1.0.0                Succeeded
ocs-operator.v4.4.1             OpenShift Container Storage   4.4.1                Succeeded
[root@preserve-olm-env data]# oc get pods
NAME                                      READY   STATUS    RESTARTS   AGE
lib-bucket-provisioner-5dddd8c448-sdvw8   1/1     Running   0          2m33s
noobaa-operator-85c55d6dd5-fdxkp          1/1     Running   0          2m28s
ocs-operator-5d85b68f9b-4qbbz             1/1     Running   0          2m28s
rook-ceph-operator-5f5444857c-8mrbc       1/1     Running   0          2m28s

> 3. Add custom catalog to internal 4.4.2 build

@Petr Could you help share the custom CatalogSource with me so that I can upgrade it to the internal 4.4.2? Thanks!

Hi,

I checked the cluster and I don't see the lib-bucket-provisioner v2 available in `openshift-storage` namespace. That v2 is needed for the upgrade process. It should be available in community catalog source.

Vu

Hi, Vu

Thanks for your information! 

> It should be available in community catalog source.

I guess no. It from the RedHat operators, and I can only get the released version: 1.0.0, as follows.

@petr Do you happen to know how to get the CatalogSource that contains the v2 version lib-bucket-provisioner? Thanks!

[root@preserve-olm-env data]# oc get packagemanifest lib-bucket-provisioner -o yaml
apiVersion: packages.operators.coreos.com/v1
kind: PackageManifest
metadata:
  creationTimestamp: "2020-08-04T09:22:16Z"
  labels:
    catalog: community-operators
    catalog-namespace: openshift-marketplace
    operatorframework.io/arch.amd64: supported
    operatorframework.io/os.linux: supported
    provider: Red Hat
    provider-url: ""
  name: lib-bucket-provisioner
  namespace: openshift-storage
  selfLink: /apis/packages.operators.coreos.com/v1/namespaces/openshift-storage/packagemanifests/lib-bucket-provisioner
spec: {}
status:
  catalogSource: community-operators
  catalogSourceDisplayName: Community Operators
  catalogSourceNamespace: openshift-marketplace
  catalogSourcePublisher: Red Hat
  channels:
  - currentCSV: lib-bucket-provisioner.v1.0.0
    currentCSVDesc:
      annotations:
        alm-examples: |-
          [
            {
              "apiVersion": "objectbucket.io/v1alpha1",
              "kind": "ObjectBucketClaim",
              "metadata": {
                "name": "my-obc",
                "namespace": "my-app"
              },
              "spec": {
                "storageClassName": "object-bucket-class",
                "generateBucketName": "my-obc",
                "SSL": false,
                "versioned": false,
                "additionalConfig": {}
              }
            },
            {
              "apiVersion": "objectbucket.io/v1alpha1",
              "kind": "ObjectBucket",
              "metadata": {
                "name": "my-obc"
              },
              "spec": {
                "storageClassName": "object-bucket-class",
                "reclaimPolicy": "Delete",
                "claimRef": {
                  "name": "my-obc",
                  "namespace": "my-app"
                },
                "endpoint": {
                  "bucketHost": "xxx",
                  "bucketPort": 80,
                  "bucketName": "my-obc-1234-5678-1234-5678",
                  "region": "xxx",
                  "subRegion": "xxx",
                  "ssl": false,
                  "additionalConfig": {}
                },
                "additionalState": {}
              }
            }
          ]
        capabilities: Basic Install
        categories: Storage,Big Data
        certified: "false"
        containerImage: kubernetes/pause
        createdAt: "2014-07-19 07:02:32.267701"
        description: Library for the dynamic provisioning of object store buckets
          to be used by object store providers.
        repository: https://github.com/kube-object-storage/lib-bucket-provisioner
        support: Red Hat
      apiservicedefinitions: {}
      customresourcedefinitions:
        owned:
        - description: Claim a bucket just like claiming a PV. Automate you app bucket
            provisioning by creating OBC with your app deployment. A secret and configmap
            (name=claim) will be created with access details for the app pods.
          displayName: ObjectBucketClaim
          kind: ObjectBucketClaim
          name: objectbucketclaims.objectbucket.io
          version: v1alpha1
        - description: Used under-the-hood. Created per ObjectBucketClaim and keeps
            provisioning information
          displayName: ObjectBucket
          kind: ObjectBucket
          name: objectbuckets.objectbucket.io
          version: v1alpha1
      description: "### CRD-only Operator\n\nThis operator package is **CRD-only**
        and the operator is a no-op operator.\n\nInstead, bucket provisioners using
        this library are using these CRD's and using CSV [required-crds](https://github.com/operator-framework/operator-lifecycle-manager/blob/master/doc/design/building-your-csv.md#required-crds)
        them so that OLM can install it as a dependency.\n\n### Generic Bucket Provisioning\n\nKubernetes
        natively supports dynamic provisioning for many types of file and block storage,
        but lacks support for object bucket provisioning. \n\nThis repo is a placeholder
        for an object store bucket provisioning library, very similar to the Kubernetes
        [sig-storage-lib-external-provisioner](https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner/blob/master/controller/controller.go)
        library.\n\n### Known Provisioners\n- https://github.com/noobaa/noobaa-operator
        (NooBaa)\n- https://github.com/rook/rook (Rook-Ceph)\n- https://github.com/yard-turkey/aws-s3-provisioner
        (AWS-S3)\n"
      displayName: lib-bucket-provisioner
      installModes:
      - supported: true
        type: OwnNamespace
      - supported: true
        type: SingleNamespace
      - supported: true
        type: MultiNamespace
      - supported: true
        type: AllNamespaces
      keywords:
      - kubernetes
      - openshift
      - object
      - bucket
      - storage
      - cloud
      - s3
      links:
      - name: Github
        url: https://github.com/kube-object-storage/lib-bucket-provisioner
      maintainers:
      - email: jcope
        name: Jon Cope
      - email: jvance
        name: Jeff Vance
      - email: gmargali
        name: Guy Margalit
      - email: dzaken
        name: Danny Zaken
      - email: nbecker
        name: Nimrod Becker
      maturity: alpha
      provider:
        name: Red Hat
      relatedImages:
      - kubernetes/pause
      version: 1.0.0
    name: alpha
  defaultChannel: alpha
  packageName: lib-bucket-provisioner
  provider:
    name: Red Hat

Ah, I find the reason. In OCP 4.6, as default, we use the CatalogSource instead of the OperatorSource. So, I guess there hasn't no lib-bucket operator in the Community CatalogSource yet. I create an OperatorSource for community operators. I can find it now.

1, Disable the community CatalogSource.
[root@preserve-olm-env data]# cat operatorhub.yaml
apiVersion: config.openshift.io/v1
kind: OperatorHub
metadata:
  name: cluster
spec:
  disableAllDefaultSources: false
  sources: [
    {
      name: "community-operators",
      disabled: true,
    }
  ]
[root@preserve-olm-env data]# oc apply -f operatorhub.yaml 
Warning: oc apply should be used on resource created by either oc create --save-config or oc apply
operatorhub.config.openshift.io/cluster configured

2, Create an OperatorSource for the community operators.
[root@preserve-olm-env data]# cat operatorsource-community.yaml
apiVersion: operators.coreos.com/v1
kind: OperatorSource
metadata:
  name: community-operators
  namespace: openshift-marketplace
spec:
  authorizationToken: {}
  displayName: Community QE Operators
  endpoint: https://quay.io/cnr
  publisher: QE
  registryNamespace: community-operators
  type: appregistry
[root@preserve-olm-env data]# oc create -f operatorsource-community.yaml
operatorsource.operators.coreos.com/community-operators created

[root@preserve-olm-env data]# oc get pods
NAME                                    READY   STATUS        RESTARTS   AGE
certified-operators-8qcmg               1/1     Running       0          14h
certified-operators-lrd85               1/1     Running       0          16m
community-operators-5f86684c7b-dpv7p    0/1     Terminating   0          4s
community-operators-6896b7f88d-97xvp    0/1     Pending       0          1s
community-operators-68bc549459-5kqph    0/1     Pending       0          0s
community-operators-6b7df9b8f9-pkg9c    0/1     Terminating   0          2s
community-operators-6d457ff7cb-dhfkx    0/1     Terminating   0          2s
community-operators-7b76d9cfdb-55nwk    0/1     Terminating   0          5s
community-operators-cb4dff97f-dm9xf     0/1     Terminating   0          5s

But, the communit pods in the creating-Terminating endless loop. Report a bug for this issue: https://bugzilla.redhat.com/show_bug.cgi?id=1866169
I think I shouldn't use the default catalogsource name: community-operators

Change another way to verify it.

1, Create an OpertorSource object called "community-operators-old" for community operators.
[root@preserve-olm-env data]# cat operatorsource-community.yaml
apiVersion: operators.coreos.com/v1
kind: OperatorSource
metadata:
  name: community-operators-old
  namespace: openshift-marketplace
spec:
  authorizationToken: {}
  displayName: Community QE Operators
  endpoint: https://quay.io/cnr
  publisher: QE
  registryNamespace: community-operators
  type: appregistry

[root@preserve-olm-env data]# oc get catalogsource -n openshift-marketplace
NAME                      DISPLAY                       TYPE   PUBLISHER   AGE
certified-operators       Certified Operators           grpc   Red Hat     18h
community-operators-old   Community QE Operators        grpc   QE          6m37s
...

[root@preserve-olm-env data]# oc create -f operatorsource-community.yaml
operatorsource.operators.coreos.com/community-operators-old created

[root@preserve-olm-env data]# oc get packagemanifest |grep lib
lib-bucket-provisioner                  Community QE Operators        4m13s

2, Modify this lib-bucket-provisioner subscription to use the new CatalogSource: community-operators-old

[root@preserve-olm-env data]# oc edit sub lib-bucket-provisioner
subscription.operators.coreos.com/lib-bucket-provisioner edited

3, Waiting for the automated upgrade.

[root@preserve-olm-env data]# oc get sub
NAME                                                                             PACKAGE                   SOURCE                    CHANNEL
awss3-operator-registry-original-community-operators-old-openshift-marketplace   awss3-operator-registry   community-operators-old   original
lib-bucket-provisioner                                                           lib-bucket-provisioner    community-operators-old   alpha
ocs-operator                                                                     ocs-operator              ocs-catalogsource         stable-4.4

[root@preserve-olm-env data]# oc get ip
NAME            CSV                             APPROVAL    APPROVED
install-hr2cc   awss3operator.1.0.0             Automatic   true
install-plmz5   lib-bucket-provisioner.v1.0.0   Automatic   true
install-s6mbk   ocs-operator.v4.4.1             Automatic   true

[root@preserve-olm-env data]# oc get csv
NAME                            DISPLAY                       VERSION   REPLACES                        PHASE
awss3operator.1.0.0             AWS S3 Operator               1.0.0                                     Succeeded
lib-bucket-provisioner.v2.0.0   lib-bucket-provisioner        2.0.0     lib-bucket-provisioner.v1.0.0   Succeeded
ocs-operator.v4.4.1             OpenShift Container Storage   4.4.1                                     Succeeded

The lib-bucket-provisioner has been updated to 2.0.0 auttomaically, but the ocs-operator still in 4.4.1 version.

Sure, removing need info from me.