Description of the problem: Upgrading from ACM 2.4 operator to ACM 2.5 operator has failed due to InstallPlan not being generated for the Subscription. This bug is basically caused by a different OCP bug: #2058417 - Subscription has olm.generated-by annotation but missing InstallPlanRef and installplan Release version: ACM 2.5 Operator snapshot version: # Source image path: brew.registry.redhat.io/rh-osbs/iib:215251 # Target image path: image-registry.openshift-image-registry.svc:5000/ocm/acm-operator-bundle-index:v2.5.0 OCP version: 4.8.0 ACM base version (to be upgraded): 2.4.2 Browser Info: N/A Steps to reproduce: 1. Install ACM 2.4 2. Follow this guide to upgrade to ACM 2.5.0: https://github.com/stolostron/release/blob/master/doc/how-to-orchestrate-operator-upgrade.md Actual results: https://qe-jenkins-csb-skynet.apps.ocp4.prod.psi.redhat.com/job/ACM-Upgrade-2.5.0-Submariner-0.12.0-AWS-AWS-GCP/45/Test-Report/ # Import operator image: $ oc import-image image-registry.openshift-image-registry.svc:5000/ocm/acm-operator-bundle-index:v2.5.0 --from=brew.registry.redhat.io/rh-osbs/iib:215251 -n ocm --confirm # Create catalog source: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: name: advanced-cluster-management-release-2-5-catalog namespace: ocm spec: sourceType: grpc image: image-registry.openshift-image-registry.svc:5000/ocm/acm-operator-bundle-index:v2.5.0 displayName: acm-operator-bundle-release-2.5 Catalog Source publisher: Red Hat Partner (Test) updateStrategy: registryPoll: interval: 5m # Verify that the package manifests now includes "acm-operator-bundle-release-2.5 Catalog Source": $ oc get packagemanifests -n ocm # Verify CSV of 2.5.0 is listed: $ oc get packagemanifests -n ocm advanced-cluster-management -o json | jq -r '(.status.channels[].currentCSVDesc.version)' # Create operator group: apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: my-ocm-operators-group namespace: ocm spec: targetNamespaces: - ocm # Create new Subscription: apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: advanced-cluster-management-release-2.5-subscription namespace: ocm spec: channel: release-2.5 installPlanApproval: Manual name: advanced-cluster-management source: advanced-cluster-management-release-2-5-catalog sourceNamespace: ocm starting_csv: advanced-cluster-management.v2.5.0 # List all subscription (should see both of ACM 2.4 and ACM 2.5 now): $ oc get sub -n ocm --ignore-not-found NAME PACKAGE SOURCE CHANNEL advanced-cluster-management-release-2.4-subscription advanced-cluster-management advanced-cluster-management-release-2-4-catalog release-2.4 advanced-cluster-management-release-2.5-subscription advanced-cluster-management advanced-cluster-management-release-2-5-catalog release-2.5 # Watch if the new Install Plan for ACM 2.5 was added: oc get installplan -n ocm -o yaml # But since the InstallPlan was not generated for ACM 2.5 Subscription, the CSV for ACM 2.5.0 was not created either: $ oc get csv -n ocm --ignore-not-found NAME DISPLAY VERSION REPLACES PHASE advanced-cluster-management.v2.4.2 Advanced Cluster Management for Kubernetes 2.4.2 advanced-cluster-management.v2.4.1 Succeeded # Looking at the subscription, we see that InstallPlanRef is missing: $ oc describe subs/advanced-cluster-management-release-2.5-subscription -n ocm Name: advanced-cluster-management-release-2.5-subscription Namespace: ocm Labels: operators.coreos.com/advanced-cluster-management.ocm= Annotations: <none> API Version: operators.coreos.com/v1alpha1 Kind: Subscription Metadata: Creation Timestamp: 2022-04-18T08:29:32Z Generation: 1 Managed Fields: API Version: operators.coreos.com/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:catalogHealth: f:conditions: f:lastUpdated: Manager: catalog Operation: Update Time: 2022-04-18T08:29:32Z API Version: operators.coreos.com/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubectl.kubernetes.io/last-applied-configuration: f:spec: .: f:channel: f:installPlanApproval: f:name: f:source: f:sourceNamespace: f:startingCSV: Manager: kubectl-client-side-apply Operation: Update Time: 2022-04-18T08:29:32Z API Version: operators.coreos.com/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:labels: .: f:operators.coreos.com/advanced-cluster-management.ocm: Manager: olm Operation: Update Time: 2022-04-18T08:29:32Z Resource Version: 231141 UID: 054cbffb-a129-482c-9060-ac999f534106 Spec: Channel: release-2.5 Install Plan Approval: Manual Name: advanced-cluster-management Source: advanced-cluster-management-release-2-5-catalog Source Namespace: ocm Starting CSV: advanced-cluster-management.v2.5.0 Status: Catalog Health: Catalog Source Ref: API Version: operators.coreos.com/v1alpha1 Kind: CatalogSource Name: advanced-cluster-management-release-2-4-catalog Namespace: ocm Resource Version: 225655 UID: 43047e03-f7a5-4849-b4ed-4677babf7c34 Healthy: true Last Updated: 2022-04-18T08:29:32Z Catalog Source Ref: API Version: operators.coreos.com/v1alpha1 Kind: CatalogSource Name: advanced-cluster-management-release-2-5-catalog Namespace: ocm Resource Version: 230064 UID: bafc2627-2049-446e-a66d-7fd8f8f918d6 Healthy: true Last Updated: 2022-04-18T08:29:32Z Conditions: Last Transition Time: 2022-04-18T08:29:32Z Message: all available catalogsources are healthy Reason: AllCatalogSourcesHealthy Status: False Type: CatalogSourcesUnhealthy Last Updated: 2022-04-18T08:29:32Z Events: <none> Expected results: The Subscription for ACM 2.5 should initiate a new InstallPlan, that should Upgrade the Operator bundle from ACM 2.4.2 to ACM 2.5.0 Additional info: A workaround as described in BZ 2058417: Delete ACM 2.4 CSV before creating the new Subscription for ACM 2.5.0
G2Bsync 1108934073 comment JakobGray Mon, 25 Apr 2022 19:06:19 UTC G2Bsync Since this is a recognized issue with Openshift I don't think there is anything we can do to fix it and will need to wait for a patch from the Openshift team
When upgrading an operator you should not create a new subscription. If you wish to update for 2.4 to 2.5 you should update the existing subscription's channel from release-2.4 to release-2.5. In manual mode you may also need to remove the starting csv field and approve an install plan.
Hi @nmanos , in reading Jakob's comment above, is this a valid test? If it is valid, would it still be a High severity (Sev-1)?
When you tried the upgrade by updating existing subscription channel were you using one catalogsource or two different sources like in your original post? ``` NAME PACKAGE SOURCE CHANNEL advanced-cluster-management-release-2.4-subscription advanced-cluster-management advanced-cluster-management-release-2-4-catalog release-2.4 advanced-cluster-management-release-2.5-subscription advanced-cluster-management advanced-cluster-management-release-2-5-catalog release-2.5 ``` I learned from some QE people that some of the upgrade tooling being used by QE was not using the same catalogsource, but that has since been corrected. I've also heard that people have encountered issues before when upgrading while not at the latest z-stream version of a channel. When you are upgrading from 2.4.2 is that the latest version available in the 2.4 channel? If not you should update to whatever is current before upgrading to 2.5. I am not yet able to replicate this issue using 2.5.0-DOWNSTREAM-2022-05-13-16-47-16 from acm-d/acm-custom-registry.
At the begging I tried to delete existing subscription and catalog source (of ACM 2.4), and then create a new subscription and catalog source - with the same names as before, but for ACM 2.5. This did not work. On another test scenario (from scratch) I tried to keep the existing subscription + catalog source (of ACM 2.4), and create a new subscription + catalog source for ACM 2.5. This did not work either. Since it's a manual install plan, it will not auto-update to latest minor version (e.g 2.4.2 --> 2.4.3), as far as I understand. In addition, I did not see any instructions/requirements for the user to do a minor upgrade, before attempting a major upgrade (to 2.5)
The dependency on bug 2058417 (Subscription has olm.generated-by annotation but missing InstallPlanRef and installplan) is still relevant. To workaround this issue - Delete previous Subscription and CSV from the ACM namespace ("ocm"): $ oc delete subs --all -n ocm --wait subscription.operators.coreos.com "old-operator-subscription" deleted $ oc delete csv --all -n ocm --wait clusterserviceversion.operators.coreos.com "old-operator.v2.4.2" deleted Once the previous CSV and Subscription (of ACM 2.4.2) were deleted, the new Subscription (of ACM 2.5.0) initiates a new InstallPlan, that upgrades the Operator bundle from ACM 2.4.2 to ACM 2.5.0.
I have not encountered others with this issue, nor have I been able to replicate it myself. The bug you reference has since been closed. I'd like to close this as well if it's not still happening.