Bug 1655026 - Install the AMQ failed via the OLM
Summary: Install the AMQ failed via the OLM
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.1.0
Assignee: Evan Cordell
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-30 11:14 UTC by Jian Zhang
Modified: 2019-06-04 10:41 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-04 10:41:04 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 None None None 2019-06-04 10:41:10 UTC

Description Jian Zhang 2018-11-30 11:14:09 UTC
Description of problem:
The errors from the AMQ pods:
2018-11-30 11:00:24 WARN  WatchConnectionManager:185 - Exec Failure: HTTP 403, Status: 403 - kafkamirrormakers.kafka.strimzi.io is forbidden: User "system:serviceaccount:amq:strimzi-cluster-operator" cannot watch kafkamirrormakers.kafka.strimzi.io in the namespace "amq": no RBAC policy matched

Version-Release number of selected component (if applicable):
[core@jian-master-0 ~]$ oc exec olm-operator-796dc97869-ddgq5 -- olm -version
OLM version: 0.8.0
git commit: bb46d55

How reproducible:
always

Steps to Reproduce:
1. Install the AMQ via the subscription. Like below:
[core@jian-master-0 ~]$ cat amq-streams.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  namespace: amq
  generateName: amq-
spec:
  source: rh-operators
  name: amq-streams
  startingCSV: amqstreams.v1.0.0.beta
  channel: preview

2. Check the status of the AMQ.

Actual results:
[core@jian-master-0 ~]$ oc get pods
NAME                                        READY     STATUS             RESTARTS   AGE
strimzi-cluster-operator-79db8d9b4c-5lc58   0/1       CrashLoopBackOff   18         1h
[core@jian-master-0 ~]$ oc get csv
NAME                     DISPLAY       VERSION      REPLACES   PHASE
amqstreams.v1.0.0.beta   AMQ Streams   1.0.0-Beta              Installing

[core@jian-master-0 ~]$ oc logs -f strimzi-cluster-operator-79db8d9b4c-5lc58
...
2018-11-30 11:00:24 INFO  ClusterOperator:86 - Starting ClusterOperator for namespace amq
2018-11-30 11:00:24 INFO  ClusterOperator:93 - Started operator for Kafka kind
2018-11-30 11:00:24 WARN  WatchConnectionManager:185 - Exec Failure: HTTP 403, Status: 403 - kafkamirrormakers.kafka.strimzi.io is forbidden: User "system:serviceaccount:amq:strimzi-cluster-operator" cannot watch kafkamirrormakers.kafka.strimzi.io in the namespace "amq": no RBAC policy matched

Expected results:
The AMQ Streams can be installed successfully via the OLM.

Additional info:
I checked the aggregate-olm-edit clusterrole and found it didn't include these verbs: 
  - get
  - list
  - patch
  - watch

I tried to edit the "aggregate-olm-edit" to add the above verbs, but the "aggregate-olm-edit" will revert to the original even if modified successfully. So, what's the correct approach to modifying the "aggregate-olm-edit"?
 
[core@jian-master-0 ~]$ oc get clusterrole  aggregate-olm-edit -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "false"
  creationTimestamp: 2018-11-28T09:48:53Z
  labels:
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
  name: aggregate-olm-edit
  resourceVersion: "2739377"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/aggregate-olm-edit
  uid: d0fdc055-f2f2-11e8-b24e-92c2a8062742
rules:
- apiGroups:
  - operators.coreos.com
  resources:
  - clusterserviceversions
  - catalogsources
  - installplans
  - subscriptions
  - packagemanifests
  verbs:
  - create
  - update
  - patch
  - delete

Comment 1 Evan Cordell 2018-11-30 13:56:52 UTC
It looks like the AMQ CSV does not list out all of the permissions that it needs.

The error is for `kafkamirrormakers.kafka.strimzi.io` which I don't see listed as a required permission at all:

https://github.com/operator-framework/operator-lifecycle-manager/blob/master/deploy/chart/catalog_resources/rh-operators/amq-streams.v1.0.0.clusterserviceversion.yaml#L88

This will require a fix in the CSV

Comment 3 Jian Zhang 2019-01-21 03:23:43 UTC
Evan,

Could you help post the fixed PR in here? Change status to "ASSIGNED" first.

Comment 4 Evan Cordell 2019-02-25 16:57:07 UTC
This is not an OLM bug, this is an issue with Strimzi. Please file with that team. We can assist them with their CSV if they have questions.

aggregate-olm-edit is not related to Strimzi permissions, it is just the role that grants edit permission to OLM types.

Comment 5 Jian Zhang 2019-02-27 05:59:46 UTC
It works well now. Verify it. Details as below:

Install it on Web console: "Catalog"->"Operator Hub"->"AMQ Streams". Select the "default" namespace.

[jzhang@dhcp-140-18 ~]$ oc get pods -n default
NAME                                        READY   STATUS    RESTARTS   AGE
strimzi-cluster-operator-54868b4588-qp4p5   1/1     Running   0          8m18s
[jzhang@dhcp-140-18 ~]$ oc get sub -n default
NAME          PACKAGE       SOURCE                     CHANNEL
amq-streams   amq-streams   installed-redhat-default   final
[jzhang@dhcp-140-18 ~]$ oc get csv -n default
NAME                DISPLAY       VERSION   REPLACES   PHASE
amqstreams.v1.0.0   AMQ Streams   1.0.0                Succeeded

[jzhang@dhcp-140-18 ~]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-02-26-125216   True        False         105m    Cluster version is 4.0.0-0.nightly-2019-02-26-125216

OLM image info:
               io.openshift.build.commit.id=1ac9ace63034ea90f31e5ad33435d3f5476c7d85
               io.openshift.build.commit.url=https://github.com/operator-framework/operator-lifecycle-manager/commit/1ac9ace63034ea90f31e5ad33435d3f5476c7d85
               io.openshift.build.source-location=https://github.com/operator-framework/operator-lifecycle-manager

Comment 7 Caden Marchese 2019-05-13 16:57:29 UTC
A customer has noted the same issue with OCP 3.11:

2019-05-08 07:08:39 WARN WatchConnectionManager:185 - Exec Failure: HTTP 403, Status: 403 - kafkamirrormakers.kafka.strimzi.io is forbidden: User "system:serviceaccount:amq-streams:strimzi-cluster-operator" cannot watch kafkamirrormakers.kafka.strimzi.io in the namespace "amq-streams": no RBAC policy matched

They were able to fix it with the following steps:

1) Manually install cluster  operator as mentioned in documentation https://access.redhat.com/documentation/en-us/red_hat_amq/7.2/html/using_amq_streams_on_openshift_container_platform/assembly-operators-str
2) Use Operator Framework to install AMQ stream via subscription

Comment 9 errata-xmlrpc 2019-06-04 10:41:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.