Bug 2108475 - Unable to upgrade file integrity operator using catalog source
Summary: Unable to upgrade file integrity operator using catalog source
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: File Integrity Operator
Version: 4.10
Hardware: ppc64le
OS: Linux
high
high
Target Milestone: ---
: 4.12.0
Assignee: Matt Rogers
QA Contact: xiyuan
Jeana Routh
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-07-19 07:29 UTC by Varad Ahirwadkar
Modified: 2022-12-22 21:56 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
* Previously, the File Integrity Operator daemon used the `ClusterRoles` parameter instead of the `Roles` parameter for a recent permission change. As a result, OLM could not upgrade the Operator. With this release, the Operator daemon reverts to using the `Roles` parameter and upgrades from older versions to version 0.1.29 are successful. (link:https://bugzilla.redhat.com/show_bug.cgi?id=2108475[*BZ#2108475*])
Clone Of:
Environment:
Last Closed: 2022-08-02 08:17:03 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift file-integrity-operator pull 267 0 None open Change daemon RBAC back to Role 2022-07-19 17:38:42 UTC
Red Hat Product Errata RHBA-2022:5538 0 None None None 2022-08-02 08:17:09 UTC

Description Varad Ahirwadkar 2022-07-19 07:29:26 UTC
Description of problem:
Unable to upgrade file integrity operator from 0.1.27 to 0.1.28 using catalog source. Old CSV is stuck in Replacing and the new one is in Pending state.

Version-Release number of selected component (if applicable):
OCP 4.10.15 release and File Integrity operator v.0.1.27

How reproducible:
Every time.

Steps to Reproduce:
1. Install File Integrity Operator
2. Create a new catalog source and update the subscription

Actual results:
Upgradation does not complete

# oc get catsrc -A | grep file
openshift-marketplace   file-integrity-operator       file-integrity-operator        grpc   grpc                                           69m
openshift-marketplace   file-integrity-operator-new   File Integrity Operator        grpc   github.com/openshift/file-integrity-operator   55m

# oc get sub
NAME                      PACKAGE                   SOURCE                        CHANNEL
file-integrity-operator   file-integrity-operator   file-integrity-operator-new   release-0.1

# oc get ip
NAME            CSV                               APPROVAL    APPROVED
install-5r6bw   file-integrity-operator.v0.1.28   Automatic   true
install-v9dfn   file-integrity-operator.v0.1.27   Automatic   true

# oc get pod
NAME                                       READY   STATUS    RESTARTS      AGE
file-integrity-operator-766dc4dbcc-r9r5b   1/1     Running   1 (68m ago)   68m

# oc get csv
NAME                              DISPLAY                   VERSION   REPLACES                          PHASE
file-integrity-operator.v0.1.27   File Integrity Operator   0.1.27                                      Replacing
file-integrity-operator.v0.1.28   File Integrity Operator   0.1.28    file-integrity-operator.v0.1.27   Pending

# oc describe csv file-integrity-operator.v0.1.28
[...]
  Requirement Status:
    Group:    apiextensions.k8s.io
    Kind:     CustomResourceDefinition
    Message:  CRD is present and Established condition is true
    Name:     fileintegrities.fileintegrity.openshift.io
    Status:   Present
    Uuid:     c5ea5d73-ddd2-48fa-90b4-6c8282a3fd2a
    Version:  v1
    Group:    apiextensions.k8s.io
    Kind:     CustomResourceDefinition
    Message:  CRD installed alongside other CSV(s): file-integrity-operator.v0.1.27
    Name:     fileintegritynodestatuses.fileintegrity.openshift.io
    Status:   PresentNotSatisfied
    Version:  v1
    Group:
    Kind:     ServiceAccount
    Message:  Service account is owned by another ClusterServiceVersion
    Name:     file-integrity-operator
    Status:   PresentNotSatisfied
    Version:  v1
Events:
  Type    Reason               Age                From                        Message
  ----    ------               ----               ----                        -------
  Normal  RequirementsUnknown  27m                operator-lifecycle-manager  requirements not yet checked
  Normal  RequirementsNotMet   27m (x3 over 27m)  operator-lifecycle-manager  one or more requirements couldn't be found



Expected results:
File Integrity Operator should be upgraded successfully.

Additional info:
Must gather logs: https://drive.google.com/file/d/1AV6GZTNTITYWnJMIRlfgLX0CajaVRi5b/view?usp=sharing

Comment 1 Varad Ahirwadkar 2022-07-19 14:18:25 UTC
Performed same scenario on OCP 4.11.0-rc.2 release on ppc64le architecture. Seen same issue as mentioned above.

# oc version
Client Version: 4.11.0-rc.2
Kustomize Version: v4.5.4
Server Version: 4.11.0-rc.2
Kubernetes Version: v1.24.0+9546431

# oc get csv
NAME                              DISPLAY                   VERSION   REPLACES                          PHASE
file-integrity-operator.v0.1.27   File Integrity Operator   0.1.27                                      Replacing
file-integrity-operator.v0.1.28   File Integrity Operator   0.1.28    file-integrity-operator.v0.1.27   Pending

Must gather logs: https://drive.google.com/file/d/1otp-g1pXjWyqagWddSpFFR9Qz75_LVXd/view?usp=sharing

Comment 2 Matt Rogers 2022-07-19 17:37:10 UTC
We changed a role to a clusterrole previously, and that seems to be not allowed with OLM.

    - apiVersion: operators.coreos.com/v1alpha1
      conditions:
      - lastTransitionTime: "2022-07-18T17:16:51Z"
        message: all available catalogsources are healthy
        reason: AllCatalogSourcesHealthy
        status: "False"
        type: CatalogSourcesUnhealthy
      - message: 'constraints not satisfiable: @existing/openshift-file-integrity//file-integrity-operator.v0.1.28
          and @existing/openshift-file-integrity//file-integrity-operator.v0.1.27
          originate from package file-integrity-operator, subscription file-integrity-operator
          requires @existing/openshift-file-integrity//file-integrity-operator.v0.1.28,
          subscription file-integrity-operator exists, clusterserviceversion file-integrity-operator.v0.1.27
          exists and is not referenced by a subscription'
        reason: ConstraintsNotSatisfiable
        status: "True"
        type: ResolutionFailed
      - lastTransitionTime: "2022-07-18T17:13:30Z"
        message: 'error updating rolebinding file-integrity-daemon: RoleBinding.rbac.authorization.k8s.io
          "file-integrity-daemon" is invalid: roleRef: Invalid value: rbac.RoleRef{APIGroup:"rbac.authorization.k8s.io",
          Kind:"ClusterRole", Name:"file-integrity-daemon"}: cannot change roleRef'
        reason: InstallComponentFailed
        status: "True"
        type: InstallPlanFailed
      kind: Subscription
      name: file-integrity-operator
      namespace: openshift-file-integrity
    - apiVersion: operators.coreos.com/v1alpha1
      conditions:
      - lastTransitionTime: "2022-07-18T17:13:17Z"
        lastUpdateTime: "2022-07-18T17:13:17Z"
        message: 'error updating rolebinding file-integrity-daemon: RoleBinding.rbac.authorization.k8s.io
          "file-integrity-daemon" is invalid: roleRef: Invalid value: rbac.RoleRef{APIGroup:"rbac.authorization.k8s.io",
          Kind:"ClusterRole", Name:"file-integrity-daemon"}: cannot change roleRef'
        reason: InstallComponentFailed
        status: "False"
        type: Installed
      kind: InstallPlan
      name: install-5r6bw
      namespace: openshift-file-integrity

Comment 3 Matt Rogers 2022-07-19 17:38:43 UTC
Note that this would also affect other arches not just ppc

Comment 7 Varad Ahirwadkar 2022-07-20 13:43:53 UTC
I tried upgrade scenario with new build but facing the same issue
Upgrade scenarios v0.1.28 to v0.1.29
Cluster version: 4.11.0-rc.2 on Power

# oc get catsrc/file-integrity-operator-old -nopenshift-marketplace  -o=jsonpath={.spec.image} -nopenshift-marketplace
brew.registry.redhat.io/rh-osbs/iib:274277

# oc get pods
NAME                                      READY   STATUS    RESTARTS      AGE
file-integrity-operator-f4fcf556d-wc6lm   1/1     Running   1 (40s ago)   50s


### Using new the catalog source to upgrade
# oc get catsrc/file-integrity-operator  -nopenshift-marketplace  -o=jsonpath={.spec.image} -nopenshift-marketplace
brew.registry.redhat.io/rh-osbs/iib:276833

# oc patch subscriptions file-integrity-operator -p '{"spec":{"source":"file-integrity-operator"}}' --type='merge'
subscription.operators.coreos.com/file-integrity-operator patched

# oc get sub
NAME                      PACKAGE                   SOURCE                    CHANNEL
file-integrity-operator   file-integrity-operator   file-integrity-operator   release-0.1

# oc get csv -w
NAME                              DISPLAY                   VERSION   REPLACES                          PHASE
file-integrity-operator.v0.1.28   File Integrity Operator   0.1.28                                      Replacing
file-integrity-operator.v0.1.29   File Integrity Operator   0.1.29    file-integrity-operator.v0.1.28   Pending

Must gather logs: https://drive.google.com/file/d/10HjFBDOtqxZCkTZ7gdDtYgcfVAVs5I1h/view?usp=sharing

Comment 8 Lance Bragstad 2022-07-20 14:40:12 UTC
Installing 0.1.28 and upgrading to 0.1.29 will cause the same issue since the initial version uses a different RBAC role type than the one being upgraded to. Upgrading from 0.1.24 to 0.1.28 will fail because OLM can't change Role to ClusterRole. Conversely, upgrading from 0.1.28 to 0.1.29 will fail because OLM can't change ClusterRole to Role.

However, it should be possible to upgrade past 0.1.28 without issue (e.g., 0.1.24 -> 0.1.29).

Comment 9 Matt Rogers 2022-07-21 17:15:39 UTC
Fixing this bug led us to the upgrade issue - we tracking that in https://bugzilla.redhat.com/show_bug.cgi?id=2109153

Comment 10 xiyuan 2022-07-28 06:36:11 UTC
Verification pass with payload 4.12.0-0.nightly-2022-07-27-133042 and file-integrity-operator.v0.1.30:
1. install file-integrity-operator.v0.1.27:
2. Upgrade to file-integrity-operator.v0.1.30:
$ oc get ip
NAME            CSV                               APPROVAL    APPROVED
install-jdj7k   file-integrity-operator.v0.1.27   Automatic   true
install-r2c9d   file-integrity-operator.v0.1.30   Automatic   true
$ oc get csv -w
NAME                              DISPLAY                            VERSION   REPLACES                          PHASE
elasticsearch-operator.v5.5.0     OpenShift Elasticsearch Operator   5.5.0                                       Succeeded
file-integrity-operator.v0.1.30   File Integrity Operator            0.1.30    file-integrity-operator.v0.1.27   Succeeded
^C$ oc get fileintegrity example-fileintegrity -o=jsonpath={.status}
{"phase":"Initializing"}
$ oc get fileintegrity example-fileintegrity -o=jsonpath={.status}
{"phase":"Active"}
$ oc get pod
NAME                                       READY   STATUS    RESTARTS   AGE
aide-example-fileintegrity-bvkdw           1/1     Running   0          8m
aide-example-fileintegrity-f8z69           1/1     Running   0          8m
aide-example-fileintegrity-fb9sb           1/1     Running   0          8m
aide-example-fileintegrity-l7xn5           1/1     Running   0          8m
aide-example-fileintegrity-lb4gs           1/1     Running   0          8m
aide-example-fileintegrity-x5trn           1/1     Running   0          8m
file-integrity-operator-79bbc8dc86-tvvw7   1/1     Running   0          9m

Comment 13 errata-xmlrpc 2022-08-02 08:17:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift File Integrity Operator bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:5538


Note You need to log in before you can comment on or make changes to this bug.