Bug 1671928

@jiazha, Have you ever create RBAC? if you run etcd-operator in command line, it require to create RBAD at first, if create operator in web console, RBAC seems needn't be setup by manual.

Sudha,

> is this still a defect?
Yes, I think so.

ge,

> Have you ever create RBAC?
What do you mean? You mean I have to create the RBAC policy of the etcdoperator.v0.6.1 manually?
As far as I know, the OLM will install the RBAC policies automatically when installing operators.
Anyway, for this bug, etcd-operator has three versions, but only the etcdoperator.v0.9.2 can be installed successfully via the OLM.
Or am I missing something?

    @jiazha, I tried  etcdoperator.v0.6.1 on 4.0 env successfully, of course, we need to create RBAC at first.

    1. Create RBAC at first, Create RBAC for etcdoperator, it's not special for any one etcd operator version(0.6.1, 0.9.0,0.9.2)

    ./lgtest  ==>this is script I wrote for myself
    Creating role with ROLE_NAME=etcd-operator, NAMESPACE=default
    clusterrole.rbac.authorization.k8s.io/etcd-operator created
    Creating role binding with ROLE_NAME=etcd-operator, ROLE_BINDING_NAME=etcd-operator, NAMESPACE=default
    clusterrolebinding.rbac.authorization.k8s.io/etcd-operator created


    2.$ oc create -f subscription-etcd-operator.yaml 
    subscription.operators.coreos.com/etcd-v0.6.1 created


    $ cat subscription-etcd-operator.yaml  ==>this is what you wanted
    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: etcd-v0.6.1
      namespace: default
    spec:
      channel: alpha
      name: etcd
      installPlanApproval: Manual
      source: installed-community-global-operators
      sourceNamespace: default
      startingCSV: etcdoperator.v0.6.1

    $ oc get sub
    NAME          PACKAGE   SOURCE                                 CHANNEL
    etcd-v0.6.1   etcd      installed-community-global-operators   alpha
    [core@ip-10-0-xxx lgtest]$ oc get csv
    NAME                    DISPLAY           VERSION   REPLACES   PHASE
    clusterlogging.v0.0.1   Cluster Logging   0.0.1                Succeeded
    svcat.v0.1.34           Service Catalog   0.1.34               Succeeded
    [core@ip-10-xxx lgtest]$ oc get installplan
    No resources found.

    3.$ oc create -f deployment.yaml
    deployment.extensions/etcd-operator created

    4.$ oc create -f example-etcd-cluster.yaml
    etcdcluster.etcd.database.coreos.com/example-etcd-cluster created

    [core@ip-10-0-xxx lgtest]$ oc get pods
    NAME                             READY   STATUS    RESTARTS   AGE
    etcd-operator-77c788c44b-8648w   1/1     Running   0          103s
    example-etcd-cluster-0000        1/1     Running   0          56s
    example-etcd-cluster-0001        1/1     Running   0          40s
    example-etcd-cluster-0002        1/1     Running   0          19s


    $ oc describe pods etcd-operator-77c788c44b-8648w|grep etcd-operator
    Name:               etcd-operator-77c788c44b-8648w
    Labels:             name=etcd-operator
    Controlled By:      ReplicaSet/etcd-operator-77c788c44b
      etcd-operator:
        Image:         quay.io/coreos/etcd-operator:v0.6.1
        Image ID:      quay.io/coreos/etcd-operator@sha256:bd944a211eaf8f31da5e6d69e8541e7cada8f16a9f7a5a570b22478997819943
          etcd-operator
          MY_POD_NAME:       etcd-operator-77c788c44b-8648w (v1:metadata.name)
      Normal  Scheduled  36s   default-scheduler                                  Successfully assigned default/etcd-operator-77c788c44b-8648w to ip-10-0-130-9.us-east-2.compute.internal
      Normal  Pulling    29s   kubelet, ip-10-0-xxx.us-east-2.compute.internal  pulling image "quay.io/coreos/etcd-operator:v0.6.1"
      Normal  Pulled     25s   kubelet, ip-10-0-xxx.us-east-2.compute.internal  Successfully pulled image "quay.io/coreos/etcd-operator:v0.6.1"
    [core@ip-10-0-xxx lgtest]$ oc create -f example-etcd-cluster.yaml

@sponnaga i have a question is why we need to config RBAC at command line mode, by we needn't do it on web console??
and is there detailed steps for enable etcd-operator and etcd-cluster in web console, we run it on ocp 3.11, and it's too complex in 4.0 NextGen, could you or Evan help to explain the details to me? thx

Ge,

Thank you for your information! But, actually, it was failed per comment 5.
>     $ oc get sub
    NAME          PACKAGE   SOURCE                                 CHANNEL
    etcd-v0.6.1   etcd      installed-community-global-operators   alpha
    [core@ip-10-0-xxx lgtest]$ oc get csv
    NAME                    DISPLAY           VERSION   REPLACES   PHASE
    clusterlogging.v0.0.1   Cluster Logging   0.0.1                Succeeded
    svcat.v0.1.34           Service Catalog   0.1.34               Succeeded
    [core@ip-10-xxx lgtest]$ oc get installplan
    No resources found.

For step 2 in comment 5, no installplan/csv objects of the etcd-operator generated. It was failed.

>     3.$ oc create -f deployment.yaml
    deployment.extensions/etcd-operator created

For step 3 in comment 5, you were creating the etcd-operator by using the deployment YAML, not the OLM.

Ge, Jian, 

You shouldn't need to create RBAC for things installed via an OLM Subscription as long as the resolved CSV specifies its requirements properly.

A Subscription is meant to keep an operator installation up-to-date in accordance with the latest CSV associated with the specified package and channel in the CatalogSource targeted by the Subscription. Creating a manual Subscription with a startingCSV will let you generate successive manual approval InstallPlans with CSVs leading up to the latest for the CatalogSource/package/channel. You could generate a CatalogSource that has a channel with the latest CSV pointing to the version of the CSV you want. This would let you generate an automatic Subscription - and when the CatalogSource is updated point to a newer CSV in that package/channel, your installation will be upgrade.

> No pods of etcd operator v0.6.1 created. No csv generated.

This should not be the expected behavior of a Subscription set to manual approval. You should expect an InstallPlan set to manual approval to be generated. This will not create any operator resources until it is "approved". 

> And, the CSV of the "installplan" display "etcdoperator.v0.9.2"! I think it's wrong.

We merged a fix to respect starting CSV: https://github.com/operator-framework/operator-lifecycle-manager/pull/676

but you also need to make sure the startingCSV has a replaces chain to the latest (v0.9.2) in the CatalogSource you're using.

Nick,

> You shouldn't need to create RBAC for things installed via an OLM Subscription as long as the resolved CSV specifies its requirements properly.

Yes, thanks. Ge wasn't familiar with the OLM, I have told him this in comment 8.

> This should not be the expected behavior of a Subscription set to manual approval. You should expect an InstallPlan set to manual approval to be generated. This will not create any operator resources until it is "approved". 

Yes, I know. In previous, we provide three versions for the etcd-operator in the ConfigMap. But, the user can only use the latest version(v0.9.2).
And, we didn't provide the entry to select the version of the operator in Web console.
And, even if we specify the low version(e.g. etcdoperator.v0.6.1) of the operator in the subscription YAML, still the latest version(etcdoperator.v0.9.2) installed after the manual approved.
So, here the problem is that although we provide multi-versions for the operator in the source, the user can only use the latest version.

> And, even if we specify the low version(e.g. etcdoperator.v0.6.1) of the operator in the subscription YAML, still the latest version(etcdoperator.v0.9.2) installed after the manual approved.
So, here the problem is that although we provide multi-versions for the operator in the source, the user can only use the latest version.

Users can specify the _starting_ operator in the subscription. But the subscription's job is to update from that to newer versions.

Can you please try the following:

- Create a subscription for etcd, with starting CSV set to 0.6.1 and approval mode set to manual.
- Show the installplans that are created in the namespace

This will allow us to verify if there is a bug. We have an e2e test for this case: https://github.com/operator-framework/operator-lifecycle-manager/blob/4cd7c2e65e875241313b38e30b77249ac22e6066/test/e2e/subscription_e2e_test.go#L501

Evan,

> Can you please try the following:
> - Create a subscription for etcd, with starting CSV set to 0.6.1 and approval mode set to manual.
> - Show the installplans that are created in the namespace

Sure, I have tried to create the CatalogSourceConfig and v0.6.1 subscription objects, but no InstallPlan generated and got below errors:
[jzhang@dhcp-140-18 multitenant]$ oc logs catalog-operator-686c4d88f6-kxcq8 |grep etcd
time="2019-03-01T06:25:09Z" level=info msg="building connection to registry" currentSource="{etcd-jian openshift-operators}" id=OOPci source=etcd-jian
time="2019-03-01T06:25:09Z" level=info msg="client hasn't yet become healthy, attempt a health check" currentSource="{etcd-jian openshift-operators}" id=OOPci source=etcd-jian
time="2019-03-01T06:25:12Z" level=info msg="building connection to registry" currentSource="{etcd-jian openshift-operators}" id=PiLLn source=etcd-jian
...

[jzhang@dhcp-140-18 multitenant]$ oc get sub
NAME          PACKAGE   SOURCE      CHANNEL
etcd-v0.6.1   etcd      etcd-jian   alpha
[jzhang@dhcp-140-18 multitenant]$ oc get ip
No resources found.

[jzhang@dhcp-140-18 multitenant]$ oc create -f csc.yaml 
catalogsourceconfig.marketplace.redhat.com/etcd-jian created
[jzhang@dhcp-140-18 multitenant]$ oc create -f etcd-v0.6-sub.yaml 
subscription.operators.coreos.com/etcd-v0.6.1 created

[jzhang@dhcp-140-18 multitenant]$ cat csc.yaml 
apiVersion: marketplace.redhat.com/v1alpha1
kind: CatalogSourceConfig
metadata:
  name: etcd-jian
  namespace: openshift-marketplace
spec:
  targetNamespace: openshift-operators 
  packages: etcd

[jzhang@dhcp-140-18 multitenant]$ cat etcd-v0.6-sub.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd-v0.6.1
  namespace: openshift-operators
spec:
  channel: alpha
  name: etcd 
  source: etcd-jian
  startingCSV: etcdoperator.v0.6.1
  installPlanApproval: Manual

So, how to specify the version of the etcd-operator for now? Cluster version is 4.0.0-0.nightly-2019-02-28-054829

Jian,

Try adding "sourceNamespace: openshift-operators: to the Subscription spec.

Nick,

Thanks! I will have a try.

Nick,

> - Show the installplans that are created in the namespace

As below:
mac:aws-ocp jianzhang$ oc get sub
NAME              PACKAGE           SOURCE                                    CHANNEL
camel-k           camel-k           installed-community-openshift-operators   alpha
cluster-logging   cluster-logging   installed-community-openshift-operators   preview
descheduler       descheduler       installed-community-openshift-operators   alpha
etcd-v0.6.1       etcd              etcd-jian                                 alpha
federationv2      federationv2      installed-community-openshift-operators   alpha

mac:aws-ocp jianzhang$ oc get installplan
NAME            CSV                     SOURCE    APPROVAL    APPROVED
install-b4pq6   descheduler.v0.0.1                Automatic   true
install-bs5gp   federationv2.v0.0.2               Automatic   true
install-kl99t   descheduler.v0.0.1                Automatic   true
install-rzpz7   descheduler.v0.0.1                Automatic   true
install-rzvmn   clusterlogging.v0.0.1             Manual      false

Too many csv in this installPlan, I paste part of it. And, as you can see, the InstallPlan still to resolve the etcdoperator.v0.9.2.
mac:aws-ocp jianzhang$ oc get installplan install-rzvmn -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: InstallPlan
metadata:
  creationTimestamp: 2019-03-04T10:40:30Z
  generateName: install-
  generation: 1
  name: install-rzvmn
  namespace: openshift-operators
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: Subscription
    name: etcd-v0.6.1
    uid: ea46b90f-3e69-11e9-b998-06210c98f03c
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: Subscription
    name: descheduler
    uid: 3bc7a12e-3e49-11e9-a743-06210c98f03c
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: Subscription
    name: cluster-logging
    uid: ff3dac8c-3e4e-11e9-b12e-020ce83edd9a
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: Subscription
    name: federationv2
    uid: 2e887707-3e49-11e9-a743-06210c98f03c
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: Subscription
    name: camel-k
    uid: 1211b534-3e58-11e9-b1ae-06210c98f03c
  resourceVersion: "184864"
  selfLink: /apis/operators.coreos.com/v1alpha1/namespaces/openshift-operators/installplans/install-rzvmn
  uid: ee31bfd6-3e69-11e9-b998-06210c98f03c
spec:
  approval: Manual
  approved: false
  clusterServiceVersionNames:
  - clusterlogging.v0.0.1
  - federationv2.v0.0.2
  - camel-k-operator.v0.2.0
  - etcdoperator.v0.9.2
  - descheduler.v0.0.1
  source: ""
  sourceNamespace: ""
status:
  catalogSources:
  - installed-community-openshift-operators
  - etcd-jian
  phase: RequiresApproval
  plan:
  - resolving: clusterlogging.v0.0.1
    resource:
      group: operators.coreos.com
      kind: ClusterServiceVersion
      manifest: |
...

I couldn't reproduce this exactly, but I did see that when specifying a startingCSV, multiple installplans are created (one for each version from startngCSV to current). Now that we can reproduce we will work on a fix.

This issue has been fixed, and I have a PR with an e2e test verifying the behavior: https://github.com/operator-framework/operator-lifecycle-manager/pull/794. The PR is not necessary to verify that this is fixed, though.

1, Create the CatalogSourceConfig in the openshift-marketplace project. Like below:
[jzhang@dhcp-140-18 ocp43]$ cat csc.yaml 
apiVersion: operators.coreos.com/v1
kind: CatalogSourceConfig
metadata:
  name: etcd-jian
  namespace: openshift-marketplace
spec:
  targetNamespace: openshift-operators 
  packages: etcd
[jzhang@dhcp-140-18 ocp43]$ oc create -f csc.yaml 
catalogsourceconfig.operators.coreos.com/etcd-jian created

2, Create etcd subscription to specify the original version: "etcdoperator.v0.9.2-clusterwide", like below:
[jzhang@dhcp-140-18 ocp43]$ cat sub.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcdoperator.v0.9.2-clusterwide
  namespace: openshift-operators
spec:
  channel: clusterwide-alpha
  name: etcd 
  source: etcd-jian
  startingCSV: etcdoperator.v0.9.2-clusterwide
  installPlanApproval: Manual
  sourceNamespace: openshift-operators
[jzhang@dhcp-140-18 ocp43]$ oc create -f sub.yaml 
subscription.operators.coreos.com/etcdoperator.v0.9.2-clusterwide created

3, Check the csv in the "openshift-operators" project. Like below:
[jzhang@dhcp-140-18 ocp43]$ oc get sub
NAME                              PACKAGE   SOURCE      CHANNEL
etcdoperator.v0.9.2-clusterwide   etcd      etcd-jian   clusterwide-alpha
[jzhang@dhcp-140-18 ocp43]$ oc get ip
NAME            CSV                               SOURCE   APPROVAL   APPROVED
install-sllm2   etcdoperator.v0.9.2-clusterwide            Manual     false
[jzhang@dhcp-140-18 ocp43]$ oc get csv
No resources found.

4, Approve it and check it again.
[jzhang@dhcp-140-18 ocp43]$ oc get ip (The user cannot find the Approval button to approve it on the Web console)
NAME            CSV                               SOURCE   APPROVAL   APPROVED
install-rcjv2   etcdoperator.v0.9.4-clusterwide            Manual     false
install-sllm2   etcdoperator.v0.9.2-clusterwide            Manual     true
[jzhang@dhcp-140-18 ocp43]$ oc get csv
NAME                              DISPLAY   VERSION             REPLACES              PHASE
etcdoperator.v0.9.2-clusterwide   etcd      0.9.2-clusterwide   etcdoperator.v0.9.0   Succeeded
[jzhang@dhcp-140-18 ocp43]$ oc get pods
NAME                            READY   STATUS    RESTARTS   AGE
etcd-operator-6b57ff7bd-cf5ps   3/3     Running   0          7m40s

5, Approve the etcdoperator.v0.9.4-clusterwide and check it again.
[jzhang@dhcp-140-18 ocp43]$ oc get csv
NAME                              DISPLAY   VERSION             REPLACES                          PHASE
etcdoperator.v0.9.2-clusterwide   etcd      0.9.2-clusterwide   etcdoperator.v0.9.0               Replacing
etcdoperator.v0.9.4-clusterwide   etcd      0.9.4-clusterwide   etcdoperator.v0.9.2-clusterwide   Installing

[jzhang@dhcp-140-18 ocp43]$ oc get pods
NAME                            READY   STATUS    RESTARTS   AGE
etcd-operator-7d86d767c-pq7gj   3/3     Running   0          55s
[jzhang@dhcp-140-18 ocp43]$ oc get csv
NAME                              DISPLAY   VERSION             REPLACES                          PHASE
etcdoperator.v0.9.4-clusterwide   etcd      0.9.4-clusterwide   etcdoperator.v0.9.2-clusterwide   Succeeded

LGTM, the upgrade works well on the back end. But, there are two issues remain:
1) the user cannot select the version of the operator on the Web console.
2) the Web console doesn't provide an entry to let the users approve the InstallPlan.

Verify failed. OLM version:
               io.openshift.build.commit.id=9ba3512c5406b62179968e2432b284e9a30c321e
               io.openshift.build.commit.url=https://github.com/operator-framework/operator-lifecycle-manager/commit/9ba3512c5406b62179968e2432b284e9a30c321e

The marketplace UI intentionally hides this option. It is available via the OLM UI.

The InstallPlan approval should be here: https://github.com/openshift/console/pull/1343

Evan,

Thanks! But, where is the entry for the user to choose the operator's version on the Web console?

That's not currently an option via the UI

Evan

Thanks for your information. But, now, it's difficult for the end user to use a specific version operator.
I highly suggest we add this option on the Web console.

Evan,

I'm sorry, but it would great if we can provide this option on the Web console from the users perspective.
If we cannot do it in 4.1, we can move it to 4.2.

Marking FIXED. UI feature request will be created in JIRA.

Hi Stephen,

Thanks! Verify this bug since we have a UI feature request in JIRA.

Cluster version is 4.1.0-0.nightly-2019-04-28-064010
OLM version: io.openshift.build.commit.id=49ca4c57934ed4b0c974ce9bc3af354d5fc7146b

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758