Bug 2108475
| Summary: | Unable to upgrade file integrity operator using catalog source | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Varad Ahirwadkar <vahirwad> |
| Component: | File Integrity Operator | Assignee: | Matt Rogers <mrogers> |
| Status: | CLOSED ERRATA | QA Contact: | xiyuan |
| Severity: | high | Docs Contact: | Jeana Routh <jrouth> |
| Priority: | high | ||
| Version: | 4.10 | CC: | jhrozek, lbragsta, wenshen |
| Target Milestone: | --- | ||
| Target Release: | 4.12.0 | ||
| Hardware: | ppc64le | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
* Previously, the File Integrity Operator daemon used the `ClusterRoles` parameter instead of the `Roles` parameter for a recent permission change. As a result, OLM could not upgrade the Operator. With this release, the Operator daemon reverts to using the `Roles` parameter and upgrades from older versions to version 0.1.29 are successful.
(link:https://bugzilla.redhat.com/show_bug.cgi?id=2108475[*BZ#2108475*])
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-08-02 08:17:03 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Performed same scenario on OCP 4.11.0-rc.2 release on ppc64le architecture. Seen same issue as mentioned above. # oc version Client Version: 4.11.0-rc.2 Kustomize Version: v4.5.4 Server Version: 4.11.0-rc.2 Kubernetes Version: v1.24.0+9546431 # oc get csv NAME DISPLAY VERSION REPLACES PHASE file-integrity-operator.v0.1.27 File Integrity Operator 0.1.27 Replacing file-integrity-operator.v0.1.28 File Integrity Operator 0.1.28 file-integrity-operator.v0.1.27 Pending Must gather logs: https://drive.google.com/file/d/1otp-g1pXjWyqagWddSpFFR9Qz75_LVXd/view?usp=sharing We changed a role to a clusterrole previously, and that seems to be not allowed with OLM.
- apiVersion: operators.coreos.com/v1alpha1
conditions:
- lastTransitionTime: "2022-07-18T17:16:51Z"
message: all available catalogsources are healthy
reason: AllCatalogSourcesHealthy
status: "False"
type: CatalogSourcesUnhealthy
- message: 'constraints not satisfiable: @existing/openshift-file-integrity//file-integrity-operator.v0.1.28
and @existing/openshift-file-integrity//file-integrity-operator.v0.1.27
originate from package file-integrity-operator, subscription file-integrity-operator
requires @existing/openshift-file-integrity//file-integrity-operator.v0.1.28,
subscription file-integrity-operator exists, clusterserviceversion file-integrity-operator.v0.1.27
exists and is not referenced by a subscription'
reason: ConstraintsNotSatisfiable
status: "True"
type: ResolutionFailed
- lastTransitionTime: "2022-07-18T17:13:30Z"
message: 'error updating rolebinding file-integrity-daemon: RoleBinding.rbac.authorization.k8s.io
"file-integrity-daemon" is invalid: roleRef: Invalid value: rbac.RoleRef{APIGroup:"rbac.authorization.k8s.io",
Kind:"ClusterRole", Name:"file-integrity-daemon"}: cannot change roleRef'
reason: InstallComponentFailed
status: "True"
type: InstallPlanFailed
kind: Subscription
name: file-integrity-operator
namespace: openshift-file-integrity
- apiVersion: operators.coreos.com/v1alpha1
conditions:
- lastTransitionTime: "2022-07-18T17:13:17Z"
lastUpdateTime: "2022-07-18T17:13:17Z"
message: 'error updating rolebinding file-integrity-daemon: RoleBinding.rbac.authorization.k8s.io
"file-integrity-daemon" is invalid: roleRef: Invalid value: rbac.RoleRef{APIGroup:"rbac.authorization.k8s.io",
Kind:"ClusterRole", Name:"file-integrity-daemon"}: cannot change roleRef'
reason: InstallComponentFailed
status: "False"
type: Installed
kind: InstallPlan
name: install-5r6bw
namespace: openshift-file-integrity
Note that this would also affect other arches not just ppc I tried upgrade scenario with new build but facing the same issue
Upgrade scenarios v0.1.28 to v0.1.29
Cluster version: 4.11.0-rc.2 on Power
# oc get catsrc/file-integrity-operator-old -nopenshift-marketplace -o=jsonpath={.spec.image} -nopenshift-marketplace
brew.registry.redhat.io/rh-osbs/iib:274277
# oc get pods
NAME READY STATUS RESTARTS AGE
file-integrity-operator-f4fcf556d-wc6lm 1/1 Running 1 (40s ago) 50s
### Using new the catalog source to upgrade
# oc get catsrc/file-integrity-operator -nopenshift-marketplace -o=jsonpath={.spec.image} -nopenshift-marketplace
brew.registry.redhat.io/rh-osbs/iib:276833
# oc patch subscriptions file-integrity-operator -p '{"spec":{"source":"file-integrity-operator"}}' --type='merge'
subscription.operators.coreos.com/file-integrity-operator patched
# oc get sub
NAME PACKAGE SOURCE CHANNEL
file-integrity-operator file-integrity-operator file-integrity-operator release-0.1
# oc get csv -w
NAME DISPLAY VERSION REPLACES PHASE
file-integrity-operator.v0.1.28 File Integrity Operator 0.1.28 Replacing
file-integrity-operator.v0.1.29 File Integrity Operator 0.1.29 file-integrity-operator.v0.1.28 Pending
Must gather logs: https://drive.google.com/file/d/10HjFBDOtqxZCkTZ7gdDtYgcfVAVs5I1h/view?usp=sharing
Installing 0.1.28 and upgrading to 0.1.29 will cause the same issue since the initial version uses a different RBAC role type than the one being upgraded to. Upgrading from 0.1.24 to 0.1.28 will fail because OLM can't change Role to ClusterRole. Conversely, upgrading from 0.1.28 to 0.1.29 will fail because OLM can't change ClusterRole to Role. However, it should be possible to upgrade past 0.1.28 without issue (e.g., 0.1.24 -> 0.1.29). Fixing this bug led us to the upgrade issue - we tracking that in https://bugzilla.redhat.com/show_bug.cgi?id=2109153 Verification pass with payload 4.12.0-0.nightly-2022-07-27-133042 and file-integrity-operator.v0.1.30:
1. install file-integrity-operator.v0.1.27:
2. Upgrade to file-integrity-operator.v0.1.30:
$ oc get ip
NAME CSV APPROVAL APPROVED
install-jdj7k file-integrity-operator.v0.1.27 Automatic true
install-r2c9d file-integrity-operator.v0.1.30 Automatic true
$ oc get csv -w
NAME DISPLAY VERSION REPLACES PHASE
elasticsearch-operator.v5.5.0 OpenShift Elasticsearch Operator 5.5.0 Succeeded
file-integrity-operator.v0.1.30 File Integrity Operator 0.1.30 file-integrity-operator.v0.1.27 Succeeded
^C$ oc get fileintegrity example-fileintegrity -o=jsonpath={.status}
{"phase":"Initializing"}
$ oc get fileintegrity example-fileintegrity -o=jsonpath={.status}
{"phase":"Active"}
$ oc get pod
NAME READY STATUS RESTARTS AGE
aide-example-fileintegrity-bvkdw 1/1 Running 0 8m
aide-example-fileintegrity-f8z69 1/1 Running 0 8m
aide-example-fileintegrity-fb9sb 1/1 Running 0 8m
aide-example-fileintegrity-l7xn5 1/1 Running 0 8m
aide-example-fileintegrity-lb4gs 1/1 Running 0 8m
aide-example-fileintegrity-x5trn 1/1 Running 0 8m
file-integrity-operator-79bbc8dc86-tvvw7 1/1 Running 0 9m
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift File Integrity Operator bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:5538 |
Description of problem: Unable to upgrade file integrity operator from 0.1.27 to 0.1.28 using catalog source. Old CSV is stuck in Replacing and the new one is in Pending state. Version-Release number of selected component (if applicable): OCP 4.10.15 release and File Integrity operator v.0.1.27 How reproducible: Every time. Steps to Reproduce: 1. Install File Integrity Operator 2. Create a new catalog source and update the subscription Actual results: Upgradation does not complete # oc get catsrc -A | grep file openshift-marketplace file-integrity-operator file-integrity-operator grpc grpc 69m openshift-marketplace file-integrity-operator-new File Integrity Operator grpc github.com/openshift/file-integrity-operator 55m # oc get sub NAME PACKAGE SOURCE CHANNEL file-integrity-operator file-integrity-operator file-integrity-operator-new release-0.1 # oc get ip NAME CSV APPROVAL APPROVED install-5r6bw file-integrity-operator.v0.1.28 Automatic true install-v9dfn file-integrity-operator.v0.1.27 Automatic true # oc get pod NAME READY STATUS RESTARTS AGE file-integrity-operator-766dc4dbcc-r9r5b 1/1 Running 1 (68m ago) 68m # oc get csv NAME DISPLAY VERSION REPLACES PHASE file-integrity-operator.v0.1.27 File Integrity Operator 0.1.27 Replacing file-integrity-operator.v0.1.28 File Integrity Operator 0.1.28 file-integrity-operator.v0.1.27 Pending # oc describe csv file-integrity-operator.v0.1.28 [...] Requirement Status: Group: apiextensions.k8s.io Kind: CustomResourceDefinition Message: CRD is present and Established condition is true Name: fileintegrities.fileintegrity.openshift.io Status: Present Uuid: c5ea5d73-ddd2-48fa-90b4-6c8282a3fd2a Version: v1 Group: apiextensions.k8s.io Kind: CustomResourceDefinition Message: CRD installed alongside other CSV(s): file-integrity-operator.v0.1.27 Name: fileintegritynodestatuses.fileintegrity.openshift.io Status: PresentNotSatisfied Version: v1 Group: Kind: ServiceAccount Message: Service account is owned by another ClusterServiceVersion Name: file-integrity-operator Status: PresentNotSatisfied Version: v1 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal RequirementsUnknown 27m operator-lifecycle-manager requirements not yet checked Normal RequirementsNotMet 27m (x3 over 27m) operator-lifecycle-manager one or more requirements couldn't be found Expected results: File Integrity Operator should be upgraded successfully. Additional info: Must gather logs: https://drive.google.com/file/d/1AV6GZTNTITYWnJMIRlfgLX0CajaVRi5b/view?usp=sharing