Description of problem: Unable to upgrade file integrity operator from 0.1.27 to 0.1.28 using catalog source. Old CSV is stuck in Replacing and the new one is in Pending state. Version-Release number of selected component (if applicable): OCP 4.10.15 release and File Integrity operator v.0.1.27 How reproducible: Every time. Steps to Reproduce: 1. Install File Integrity Operator 2. Create a new catalog source and update the subscription Actual results: Upgradation does not complete # oc get catsrc -A | grep file openshift-marketplace file-integrity-operator file-integrity-operator grpc grpc 69m openshift-marketplace file-integrity-operator-new File Integrity Operator grpc github.com/openshift/file-integrity-operator 55m # oc get sub NAME PACKAGE SOURCE CHANNEL file-integrity-operator file-integrity-operator file-integrity-operator-new release-0.1 # oc get ip NAME CSV APPROVAL APPROVED install-5r6bw file-integrity-operator.v0.1.28 Automatic true install-v9dfn file-integrity-operator.v0.1.27 Automatic true # oc get pod NAME READY STATUS RESTARTS AGE file-integrity-operator-766dc4dbcc-r9r5b 1/1 Running 1 (68m ago) 68m # oc get csv NAME DISPLAY VERSION REPLACES PHASE file-integrity-operator.v0.1.27 File Integrity Operator 0.1.27 Replacing file-integrity-operator.v0.1.28 File Integrity Operator 0.1.28 file-integrity-operator.v0.1.27 Pending # oc describe csv file-integrity-operator.v0.1.28 [...] Requirement Status: Group: apiextensions.k8s.io Kind: CustomResourceDefinition Message: CRD is present and Established condition is true Name: fileintegrities.fileintegrity.openshift.io Status: Present Uuid: c5ea5d73-ddd2-48fa-90b4-6c8282a3fd2a Version: v1 Group: apiextensions.k8s.io Kind: CustomResourceDefinition Message: CRD installed alongside other CSV(s): file-integrity-operator.v0.1.27 Name: fileintegritynodestatuses.fileintegrity.openshift.io Status: PresentNotSatisfied Version: v1 Group: Kind: ServiceAccount Message: Service account is owned by another ClusterServiceVersion Name: file-integrity-operator Status: PresentNotSatisfied Version: v1 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal RequirementsUnknown 27m operator-lifecycle-manager requirements not yet checked Normal RequirementsNotMet 27m (x3 over 27m) operator-lifecycle-manager one or more requirements couldn't be found Expected results: File Integrity Operator should be upgraded successfully. Additional info: Must gather logs: https://drive.google.com/file/d/1AV6GZTNTITYWnJMIRlfgLX0CajaVRi5b/view?usp=sharing
Performed same scenario on OCP 4.11.0-rc.2 release on ppc64le architecture. Seen same issue as mentioned above. # oc version Client Version: 4.11.0-rc.2 Kustomize Version: v4.5.4 Server Version: 4.11.0-rc.2 Kubernetes Version: v1.24.0+9546431 # oc get csv NAME DISPLAY VERSION REPLACES PHASE file-integrity-operator.v0.1.27 File Integrity Operator 0.1.27 Replacing file-integrity-operator.v0.1.28 File Integrity Operator 0.1.28 file-integrity-operator.v0.1.27 Pending Must gather logs: https://drive.google.com/file/d/1otp-g1pXjWyqagWddSpFFR9Qz75_LVXd/view?usp=sharing
We changed a role to a clusterrole previously, and that seems to be not allowed with OLM. - apiVersion: operators.coreos.com/v1alpha1 conditions: - lastTransitionTime: "2022-07-18T17:16:51Z" message: all available catalogsources are healthy reason: AllCatalogSourcesHealthy status: "False" type: CatalogSourcesUnhealthy - message: 'constraints not satisfiable: @existing/openshift-file-integrity//file-integrity-operator.v0.1.28 and @existing/openshift-file-integrity//file-integrity-operator.v0.1.27 originate from package file-integrity-operator, subscription file-integrity-operator requires @existing/openshift-file-integrity//file-integrity-operator.v0.1.28, subscription file-integrity-operator exists, clusterserviceversion file-integrity-operator.v0.1.27 exists and is not referenced by a subscription' reason: ConstraintsNotSatisfiable status: "True" type: ResolutionFailed - lastTransitionTime: "2022-07-18T17:13:30Z" message: 'error updating rolebinding file-integrity-daemon: RoleBinding.rbac.authorization.k8s.io "file-integrity-daemon" is invalid: roleRef: Invalid value: rbac.RoleRef{APIGroup:"rbac.authorization.k8s.io", Kind:"ClusterRole", Name:"file-integrity-daemon"}: cannot change roleRef' reason: InstallComponentFailed status: "True" type: InstallPlanFailed kind: Subscription name: file-integrity-operator namespace: openshift-file-integrity - apiVersion: operators.coreos.com/v1alpha1 conditions: - lastTransitionTime: "2022-07-18T17:13:17Z" lastUpdateTime: "2022-07-18T17:13:17Z" message: 'error updating rolebinding file-integrity-daemon: RoleBinding.rbac.authorization.k8s.io "file-integrity-daemon" is invalid: roleRef: Invalid value: rbac.RoleRef{APIGroup:"rbac.authorization.k8s.io", Kind:"ClusterRole", Name:"file-integrity-daemon"}: cannot change roleRef' reason: InstallComponentFailed status: "False" type: Installed kind: InstallPlan name: install-5r6bw namespace: openshift-file-integrity
Note that this would also affect other arches not just ppc
I tried upgrade scenario with new build but facing the same issue Upgrade scenarios v0.1.28 to v0.1.29 Cluster version: 4.11.0-rc.2 on Power # oc get catsrc/file-integrity-operator-old -nopenshift-marketplace -o=jsonpath={.spec.image} -nopenshift-marketplace brew.registry.redhat.io/rh-osbs/iib:274277 # oc get pods NAME READY STATUS RESTARTS AGE file-integrity-operator-f4fcf556d-wc6lm 1/1 Running 1 (40s ago) 50s ### Using new the catalog source to upgrade # oc get catsrc/file-integrity-operator -nopenshift-marketplace -o=jsonpath={.spec.image} -nopenshift-marketplace brew.registry.redhat.io/rh-osbs/iib:276833 # oc patch subscriptions file-integrity-operator -p '{"spec":{"source":"file-integrity-operator"}}' --type='merge' subscription.operators.coreos.com/file-integrity-operator patched # oc get sub NAME PACKAGE SOURCE CHANNEL file-integrity-operator file-integrity-operator file-integrity-operator release-0.1 # oc get csv -w NAME DISPLAY VERSION REPLACES PHASE file-integrity-operator.v0.1.28 File Integrity Operator 0.1.28 Replacing file-integrity-operator.v0.1.29 File Integrity Operator 0.1.29 file-integrity-operator.v0.1.28 Pending Must gather logs: https://drive.google.com/file/d/10HjFBDOtqxZCkTZ7gdDtYgcfVAVs5I1h/view?usp=sharing
Installing 0.1.28 and upgrading to 0.1.29 will cause the same issue since the initial version uses a different RBAC role type than the one being upgraded to. Upgrading from 0.1.24 to 0.1.28 will fail because OLM can't change Role to ClusterRole. Conversely, upgrading from 0.1.28 to 0.1.29 will fail because OLM can't change ClusterRole to Role. However, it should be possible to upgrade past 0.1.28 without issue (e.g., 0.1.24 -> 0.1.29).
Fixing this bug led us to the upgrade issue - we tracking that in https://bugzilla.redhat.com/show_bug.cgi?id=2109153
Verification pass with payload 4.12.0-0.nightly-2022-07-27-133042 and file-integrity-operator.v0.1.30: 1. install file-integrity-operator.v0.1.27: 2. Upgrade to file-integrity-operator.v0.1.30: $ oc get ip NAME CSV APPROVAL APPROVED install-jdj7k file-integrity-operator.v0.1.27 Automatic true install-r2c9d file-integrity-operator.v0.1.30 Automatic true $ oc get csv -w NAME DISPLAY VERSION REPLACES PHASE elasticsearch-operator.v5.5.0 OpenShift Elasticsearch Operator 5.5.0 Succeeded file-integrity-operator.v0.1.30 File Integrity Operator 0.1.30 file-integrity-operator.v0.1.27 Succeeded ^C$ oc get fileintegrity example-fileintegrity -o=jsonpath={.status} {"phase":"Initializing"} $ oc get fileintegrity example-fileintegrity -o=jsonpath={.status} {"phase":"Active"} $ oc get pod NAME READY STATUS RESTARTS AGE aide-example-fileintegrity-bvkdw 1/1 Running 0 8m aide-example-fileintegrity-f8z69 1/1 Running 0 8m aide-example-fileintegrity-fb9sb 1/1 Running 0 8m aide-example-fileintegrity-l7xn5 1/1 Running 0 8m aide-example-fileintegrity-lb4gs 1/1 Running 0 8m aide-example-fileintegrity-x5trn 1/1 Running 0 8m file-integrity-operator-79bbc8dc86-tvvw7 1/1 Running 0 9m
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift File Integrity Operator bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:5538