Bug 1671928 - Cannot specify the version of an operator when installing via the OLM
Summary: Cannot specify the version of an operator when installing via the OLM
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.1.0
Assignee: Evan Cordell
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On: 1684049
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-02-02 06:50 UTC by Jian Zhang
Modified: 2019-06-04 10:43 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-04 10:42:31 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 None None None 2019-06-04 10:43:52 UTC

Description Jian Zhang 2019-02-02 06:50:11 UTC
Description of problem:
I'd like to install the Etcd-operator v0.6.1, but there are two problems:
1, I didn't find any entry for selecting the version of the operators on the Web console.
2, In the backend, for etcd-operator, there are three versions:
etcdoperator.v0.9.2, etcdoperator.v0.9.0, etcdoperator.v0.6.1. I want to use the v0.6.1, so I create a subscription with manual approve, but no corresponding csv generated.

Version-Release number of selected component (if applicable):
OLM: 
 io.openshift.build.commit.id=1f312481ae3641eea471abb792f9b056206e4cf4

How reproducible:
always

Steps to Reproduce:
1. Install the OCP 4.0
2. Create a subscription, like below:
[core@ip-10-0-37-169 ~]$ cat etcdoperator.v0.6.1-sub.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd-v0.6.1
  namespace: openshift-operators
spec:
  channel: alpha
  name: etcd
  installPlanApproval: Manual
  source: installed-community-global-operators
  sourceNamespace: openshift-operators
  startingCSV: etcdoperator.v0.6.1

Actual results:
No pods of etcd operator v0.6.1 created. No csv generated.
And, the CSV of the "installplan" display "etcdoperator.v0.9.2"! I think it's wrong.

[core@ip-10-0-37-169 ~]$ oc get sub
NAME          PACKAGE   SOURCE                                 CHANNEL
etcd-v0.6.1   etcd      installed-community-global-operators   alpha
[core@ip-10-0-37-169 ~]$ oc get installplan
NAME            CSV                   SOURCE   APPROVAL   APPROVED
install-l9f92   etcdoperator.v0.9.2            Manual     false
[core@ip-10-0-37-169 ~]$ oc get csv
No resources found.

Expected results:
1, should display the entry of selecting the version of an operator on Web console. 
2, The etcd operator v0.6.1 pods should be generated successfully.

Additional info:

Comment 1 ge liu 2019-02-02 09:18:18 UTC
@jiazha, Have you ever create RBAC? if you run etcd-operator in command line, it require to create RBAD at first, if create operator in web console, RBAC seems needn't be setup by manual.

Comment 3 Jian Zhang 2019-02-12 09:32:13 UTC
Sudha,

> is this still a defect?
Yes, I think so.

ge,

> Have you ever create RBAC?
What do you mean? You mean I have to create the RBAC policy of the etcdoperator.v0.6.1 manually?
As far as I know, the OLM will install the RBAC policies automatically when installing operators.
Anyway, for this bug, etcd-operator has three versions, but only the etcdoperator.v0.9.2 can be installed successfully via the OLM.
Or am I missing something?

Comment 5 ge liu 2019-02-12 10:39:14 UTC
    @jiazha, I tried  etcdoperator.v0.6.1 on 4.0 env successfully, of course, we need to create RBAC at first.

    1. Create RBAC at first, Create RBAC for etcdoperator, it's not special for any one etcd operator version(0.6.1, 0.9.0,0.9.2)

    ./lgtest  ==>this is script I wrote for myself
    Creating role with ROLE_NAME=etcd-operator, NAMESPACE=default
    clusterrole.rbac.authorization.k8s.io/etcd-operator created
    Creating role binding with ROLE_NAME=etcd-operator, ROLE_BINDING_NAME=etcd-operator, NAMESPACE=default
    clusterrolebinding.rbac.authorization.k8s.io/etcd-operator created


    2.$ oc create -f subscription-etcd-operator.yaml 
    subscription.operators.coreos.com/etcd-v0.6.1 created


    $ cat subscription-etcd-operator.yaml  ==>this is what you wanted
    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: etcd-v0.6.1
      namespace: default
    spec:
      channel: alpha
      name: etcd
      installPlanApproval: Manual
      source: installed-community-global-operators
      sourceNamespace: default
      startingCSV: etcdoperator.v0.6.1

    $ oc get sub
    NAME          PACKAGE   SOURCE                                 CHANNEL
    etcd-v0.6.1   etcd      installed-community-global-operators   alpha
    [core@ip-10-0-xxx lgtest]$ oc get csv
    NAME                    DISPLAY           VERSION   REPLACES   PHASE
    clusterlogging.v0.0.1   Cluster Logging   0.0.1                Succeeded
    svcat.v0.1.34           Service Catalog   0.1.34               Succeeded
    [core@ip-10-xxx lgtest]$ oc get installplan
    No resources found.

    3.$ oc create -f deployment.yaml
    deployment.extensions/etcd-operator created

    4.$ oc create -f example-etcd-cluster.yaml
    etcdcluster.etcd.database.coreos.com/example-etcd-cluster created

    [core@ip-10-0-xxx lgtest]$ oc get pods
    NAME                             READY   STATUS    RESTARTS   AGE
    etcd-operator-77c788c44b-8648w   1/1     Running   0          103s
    example-etcd-cluster-0000        1/1     Running   0          56s
    example-etcd-cluster-0001        1/1     Running   0          40s
    example-etcd-cluster-0002        1/1     Running   0          19s


    $ oc describe pods etcd-operator-77c788c44b-8648w|grep etcd-operator
    Name:               etcd-operator-77c788c44b-8648w
    Labels:             name=etcd-operator
    Controlled By:      ReplicaSet/etcd-operator-77c788c44b
      etcd-operator:
        Image:         quay.io/coreos/etcd-operator:v0.6.1
        Image ID:      quay.io/coreos/etcd-operator@sha256:bd944a211eaf8f31da5e6d69e8541e7cada8f16a9f7a5a570b22478997819943
          etcd-operator
          MY_POD_NAME:       etcd-operator-77c788c44b-8648w (v1:metadata.name)
      Normal  Scheduled  36s   default-scheduler                                  Successfully assigned default/etcd-operator-77c788c44b-8648w to ip-10-0-130-9.us-east-2.compute.internal
      Normal  Pulling    29s   kubelet, ip-10-0-xxx.us-east-2.compute.internal  pulling image "quay.io/coreos/etcd-operator:v0.6.1"
      Normal  Pulled     25s   kubelet, ip-10-0-xxx.us-east-2.compute.internal  Successfully pulled image "quay.io/coreos/etcd-operator:v0.6.1"
    [core@ip-10-0-xxx lgtest]$ oc create -f example-etcd-cluster.yaml

Comment 6 ge liu 2019-02-12 10:43:44 UTC
@sponnaga i have a question is why we need to config RBAC at command line mode, by we needn't do it on web console??
and is there detailed steps for enable etcd-operator and etcd-cluster in web console, we run it on ocp 3.11, and it's too complex in 4.0 NextGen, could you or Evan help to explain the details to me? thx

Comment 8 Jian Zhang 2019-02-13 07:48:50 UTC
Ge,

Thank you for your information! But, actually, it was failed per comment 5.
>     $ oc get sub
    NAME          PACKAGE   SOURCE                                 CHANNEL
    etcd-v0.6.1   etcd      installed-community-global-operators   alpha
    [core@ip-10-0-xxx lgtest]$ oc get csv
    NAME                    DISPLAY           VERSION   REPLACES   PHASE
    clusterlogging.v0.0.1   Cluster Logging   0.0.1                Succeeded
    svcat.v0.1.34           Service Catalog   0.1.34               Succeeded
    [core@ip-10-xxx lgtest]$ oc get installplan
    No resources found.

For step 2 in comment 5, no installplan/csv objects of the etcd-operator generated. It was failed.

>     3.$ oc create -f deployment.yaml
    deployment.extensions/etcd-operator created

For step 3 in comment 5, you were creating the etcd-operator by using the deployment YAML, not the OLM.

Comment 10 Nick Hale 2019-02-27 15:13:08 UTC
Ge, Jian, 

You shouldn't need to create RBAC for things installed via an OLM Subscription as long as the resolved CSV specifies its requirements properly.

A Subscription is meant to keep an operator installation up-to-date in accordance with the latest CSV associated with the specified package and channel in the CatalogSource targeted by the Subscription. Creating a manual Subscription with a startingCSV will let you generate successive manual approval InstallPlans with CSVs leading up to the latest for the CatalogSource/package/channel. You could generate a CatalogSource that has a channel with the latest CSV pointing to the version of the CSV you want. This would let you generate an automatic Subscription - and when the CatalogSource is updated point to a newer CSV in that package/channel, your installation will be upgrade.

> No pods of etcd operator v0.6.1 created. No csv generated.

This should not be the expected behavior of a Subscription set to manual approval. You should expect an InstallPlan set to manual approval to be generated. This will not create any operator resources until it is "approved". 

> And, the CSV of the "installplan" display "etcdoperator.v0.9.2"! I think it's wrong.

We merged a fix to respect starting CSV: https://github.com/operator-framework/operator-lifecycle-manager/pull/676

but you also need to make sure the startingCSV has a replaces chain to the latest (v0.9.2) in the CatalogSource you're using.

Comment 11 Jian Zhang 2019-02-28 06:46:55 UTC
Nick,

> You shouldn't need to create RBAC for things installed via an OLM Subscription as long as the resolved CSV specifies its requirements properly.

Yes, thanks. Ge wasn't familiar with the OLM, I have told him this in comment 8.

> This should not be the expected behavior of a Subscription set to manual approval. You should expect an InstallPlan set to manual approval to be generated. This will not create any operator resources until it is "approved". 

Yes, I know. In previous, we provide three versions for the etcd-operator in the ConfigMap. But, the user can only use the latest version(v0.9.2).
And, we didn't provide the entry to select the version of the operator in Web console.
And, even if we specify the low version(e.g. etcdoperator.v0.6.1) of the operator in the subscription YAML, still the latest version(etcdoperator.v0.9.2) installed after the manual approved.
So, here the problem is that although we provide multi-versions for the operator in the source, the user can only use the latest version.

Comment 12 Evan Cordell 2019-02-28 16:15:46 UTC
> And, even if we specify the low version(e.g. etcdoperator.v0.6.1) of the operator in the subscription YAML, still the latest version(etcdoperator.v0.9.2) installed after the manual approved.
So, here the problem is that although we provide multi-versions for the operator in the source, the user can only use the latest version.

Users can specify the _starting_ operator in the subscription. But the subscription's job is to update from that to newer versions.

Can you please try the following:

- Create a subscription for etcd, with starting CSV set to 0.6.1 and approval mode set to manual.
- Show the installplans that are created in the namespace

This will allow us to verify if there is a bug. We have an e2e test for this case: https://github.com/operator-framework/operator-lifecycle-manager/blob/4cd7c2e65e875241313b38e30b77249ac22e6066/test/e2e/subscription_e2e_test.go#L501

Comment 13 Jian Zhang 2019-03-01 13:52:08 UTC
Evan,

> Can you please try the following:
> - Create a subscription for etcd, with starting CSV set to 0.6.1 and approval mode set to manual.
> - Show the installplans that are created in the namespace

Sure, I have tried to create the CatalogSourceConfig and v0.6.1 subscription objects, but no InstallPlan generated and got below errors:
[jzhang@dhcp-140-18 multitenant]$ oc logs catalog-operator-686c4d88f6-kxcq8 |grep etcd
time="2019-03-01T06:25:09Z" level=info msg="building connection to registry" currentSource="{etcd-jian openshift-operators}" id=OOPci source=etcd-jian
time="2019-03-01T06:25:09Z" level=info msg="client hasn't yet become healthy, attempt a health check" currentSource="{etcd-jian openshift-operators}" id=OOPci source=etcd-jian
time="2019-03-01T06:25:12Z" level=info msg="building connection to registry" currentSource="{etcd-jian openshift-operators}" id=PiLLn source=etcd-jian
...

[jzhang@dhcp-140-18 multitenant]$ oc get sub
NAME          PACKAGE   SOURCE      CHANNEL
etcd-v0.6.1   etcd      etcd-jian   alpha
[jzhang@dhcp-140-18 multitenant]$ oc get ip
No resources found.

[jzhang@dhcp-140-18 multitenant]$ oc create -f csc.yaml 
catalogsourceconfig.marketplace.redhat.com/etcd-jian created
[jzhang@dhcp-140-18 multitenant]$ oc create -f etcd-v0.6-sub.yaml 
subscription.operators.coreos.com/etcd-v0.6.1 created

[jzhang@dhcp-140-18 multitenant]$ cat csc.yaml 
apiVersion: marketplace.redhat.com/v1alpha1
kind: CatalogSourceConfig
metadata:
  name: etcd-jian
  namespace: openshift-marketplace
spec:
  targetNamespace: openshift-operators 
  packages: etcd

[jzhang@dhcp-140-18 multitenant]$ cat etcd-v0.6-sub.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd-v0.6.1
  namespace: openshift-operators
spec:
  channel: alpha
  name: etcd 
  source: etcd-jian
  startingCSV: etcdoperator.v0.6.1
  installPlanApproval: Manual

So, how to specify the version of the etcd-operator for now? Cluster version is 4.0.0-0.nightly-2019-02-28-054829

Comment 14 Nick Hale 2019-03-01 14:14:45 UTC
Jian,

Try adding "sourceNamespace: openshift-operators: to the Subscription spec.

Comment 15 Jian Zhang 2019-03-04 10:03:43 UTC
Nick,

Thanks! I will have a try.

Comment 16 Jian Zhang 2019-03-04 10:49:47 UTC
Nick,

> - Show the installplans that are created in the namespace

As below:
mac:aws-ocp jianzhang$ oc get sub
NAME              PACKAGE           SOURCE                                    CHANNEL
camel-k           camel-k           installed-community-openshift-operators   alpha
cluster-logging   cluster-logging   installed-community-openshift-operators   preview
descheduler       descheduler       installed-community-openshift-operators   alpha
etcd-v0.6.1       etcd              etcd-jian                                 alpha
federationv2      federationv2      installed-community-openshift-operators   alpha

mac:aws-ocp jianzhang$ oc get installplan
NAME            CSV                     SOURCE    APPROVAL    APPROVED
install-b4pq6   descheduler.v0.0.1                Automatic   true
install-bs5gp   federationv2.v0.0.2               Automatic   true
install-kl99t   descheduler.v0.0.1                Automatic   true
install-rzpz7   descheduler.v0.0.1                Automatic   true
install-rzvmn   clusterlogging.v0.0.1             Manual      false

Too many csv in this installPlan, I paste part of it. And, as you can see, the InstallPlan still to resolve the etcdoperator.v0.9.2.
mac:aws-ocp jianzhang$ oc get installplan install-rzvmn -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: InstallPlan
metadata:
  creationTimestamp: 2019-03-04T10:40:30Z
  generateName: install-
  generation: 1
  name: install-rzvmn
  namespace: openshift-operators
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: Subscription
    name: etcd-v0.6.1
    uid: ea46b90f-3e69-11e9-b998-06210c98f03c
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: Subscription
    name: descheduler
    uid: 3bc7a12e-3e49-11e9-a743-06210c98f03c
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: Subscription
    name: cluster-logging
    uid: ff3dac8c-3e4e-11e9-b12e-020ce83edd9a
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: Subscription
    name: federationv2
    uid: 2e887707-3e49-11e9-a743-06210c98f03c
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: Subscription
    name: camel-k
    uid: 1211b534-3e58-11e9-b1ae-06210c98f03c
  resourceVersion: "184864"
  selfLink: /apis/operators.coreos.com/v1alpha1/namespaces/openshift-operators/installplans/install-rzvmn
  uid: ee31bfd6-3e69-11e9-b998-06210c98f03c
spec:
  approval: Manual
  approved: false
  clusterServiceVersionNames:
  - clusterlogging.v0.0.1
  - federationv2.v0.0.2
  - camel-k-operator.v0.2.0
  - etcdoperator.v0.9.2
  - descheduler.v0.0.1
  source: ""
  sourceNamespace: ""
status:
  catalogSources:
  - installed-community-openshift-operators
  - etcd-jian
  phase: RequiresApproval
  plan:
  - resolving: clusterlogging.v0.0.1
    resource:
      group: operators.coreos.com
      kind: ClusterServiceVersion
      manifest: |
...

Comment 17 Evan Cordell 2019-03-11 18:01:22 UTC
I couldn't reproduce this exactly, but I did see that when specifying a startingCSV, multiple installplans are created (one for each version from startngCSV to current). Now that we can reproduce we will work on a fix.

Comment 19 Evan Cordell 2019-03-29 18:04:00 UTC
This issue has been fixed, and I have a PR with an e2e test verifying the behavior: https://github.com/operator-framework/operator-lifecycle-manager/pull/794. The PR is not necessary to verify that this is fixed, though.

Comment 21 Jian Zhang 2019-04-03 08:32:22 UTC
1, Create the CatalogSourceConfig in the openshift-marketplace project. Like below:
[jzhang@dhcp-140-18 ocp43]$ cat csc.yaml 
apiVersion: operators.coreos.com/v1
kind: CatalogSourceConfig
metadata:
  name: etcd-jian
  namespace: openshift-marketplace
spec:
  targetNamespace: openshift-operators 
  packages: etcd
[jzhang@dhcp-140-18 ocp43]$ oc create -f csc.yaml 
catalogsourceconfig.operators.coreos.com/etcd-jian created

2, Create etcd subscription to specify the original version: "etcdoperator.v0.9.2-clusterwide", like below:
[jzhang@dhcp-140-18 ocp43]$ cat sub.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcdoperator.v0.9.2-clusterwide
  namespace: openshift-operators
spec:
  channel: clusterwide-alpha
  name: etcd 
  source: etcd-jian
  startingCSV: etcdoperator.v0.9.2-clusterwide
  installPlanApproval: Manual
  sourceNamespace: openshift-operators
[jzhang@dhcp-140-18 ocp43]$ oc create -f sub.yaml 
subscription.operators.coreos.com/etcdoperator.v0.9.2-clusterwide created

3, Check the csv in the "openshift-operators" project. Like below:
[jzhang@dhcp-140-18 ocp43]$ oc get sub
NAME                              PACKAGE   SOURCE      CHANNEL
etcdoperator.v0.9.2-clusterwide   etcd      etcd-jian   clusterwide-alpha
[jzhang@dhcp-140-18 ocp43]$ oc get ip
NAME            CSV                               SOURCE   APPROVAL   APPROVED
install-sllm2   etcdoperator.v0.9.2-clusterwide            Manual     false
[jzhang@dhcp-140-18 ocp43]$ oc get csv
No resources found.

4, Approve it and check it again.
[jzhang@dhcp-140-18 ocp43]$ oc get ip (The user cannot find the Approval button to approve it on the Web console)
NAME            CSV                               SOURCE   APPROVAL   APPROVED
install-rcjv2   etcdoperator.v0.9.4-clusterwide            Manual     false
install-sllm2   etcdoperator.v0.9.2-clusterwide            Manual     true
[jzhang@dhcp-140-18 ocp43]$ oc get csv
NAME                              DISPLAY   VERSION             REPLACES              PHASE
etcdoperator.v0.9.2-clusterwide   etcd      0.9.2-clusterwide   etcdoperator.v0.9.0   Succeeded
[jzhang@dhcp-140-18 ocp43]$ oc get pods
NAME                            READY   STATUS    RESTARTS   AGE
etcd-operator-6b57ff7bd-cf5ps   3/3     Running   0          7m40s

5, Approve the etcdoperator.v0.9.4-clusterwide and check it again.
[jzhang@dhcp-140-18 ocp43]$ oc get csv
NAME                              DISPLAY   VERSION             REPLACES                          PHASE
etcdoperator.v0.9.2-clusterwide   etcd      0.9.2-clusterwide   etcdoperator.v0.9.0               Replacing
etcdoperator.v0.9.4-clusterwide   etcd      0.9.4-clusterwide   etcdoperator.v0.9.2-clusterwide   Installing

[jzhang@dhcp-140-18 ocp43]$ oc get pods
NAME                            READY   STATUS    RESTARTS   AGE
etcd-operator-7d86d767c-pq7gj   3/3     Running   0          55s
[jzhang@dhcp-140-18 ocp43]$ oc get csv
NAME                              DISPLAY   VERSION             REPLACES                          PHASE
etcdoperator.v0.9.4-clusterwide   etcd      0.9.4-clusterwide   etcdoperator.v0.9.2-clusterwide   Succeeded

LGTM, the upgrade works well on the back end. But, there are two issues remain:
1) the user cannot select the version of the operator on the Web console.
2) the Web console doesn't provide an entry to let the users approve the InstallPlan.

Verify failed. OLM version:
               io.openshift.build.commit.id=9ba3512c5406b62179968e2432b284e9a30c321e
               io.openshift.build.commit.url=https://github.com/operator-framework/operator-lifecycle-manager/commit/9ba3512c5406b62179968e2432b284e9a30c321e

Comment 22 Evan Cordell 2019-04-08 18:00:40 UTC
The marketplace UI intentionally hides this option. It is available via the OLM UI.

The InstallPlan approval should be here: https://github.com/openshift/console/pull/1343

Comment 23 Jian Zhang 2019-04-10 01:35:25 UTC
Evan,

Thanks! But, where is the entry for the user to choose the operator's version on the Web console?

Comment 24 Evan Cordell 2019-04-12 17:37:19 UTC
That's not currently an option via the UI

Comment 25 Jian Zhang 2019-04-19 02:14:17 UTC
Evan

Thanks for your information. But, now, it's difficult for the end user to use a specific version operator.
I highly suggest we add this option on the Web console.

Comment 26 Jian Zhang 2019-04-24 05:51:03 UTC
Evan,

I'm sorry, but it would great if we can provide this option on the Web console from the users perspective.
If we cannot do it in 4.1, we can move it to 4.2.

Comment 27 Stephen Cuppett 2019-04-29 18:43:17 UTC
Marking FIXED. UI feature request will be created in JIRA.

Comment 29 Jian Zhang 2019-04-30 03:15:37 UTC
Hi Stephen,

Thanks! Verify this bug since we have a UI feature request in JIRA.

Cluster version is 4.1.0-0.nightly-2019-04-28-064010
OLM version: io.openshift.build.commit.id=49ca4c57934ed4b0c974ce9bc3af354d5fc7146b

Comment 31 errata-xmlrpc 2019-06-04 10:42:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.