Bug 2314636 - MCG Standalone upgrade is broken
Summary: MCG Standalone upgrade is broken
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: Multi-Cloud Object Gateway
Version: 4.16
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ODF 4.17.0
Assignee: Malay Kumar parida
QA Contact: Uday kurundwade
URL:
Whiteboard:
Depends On:
Blocks: 2281703
TreeView+ depends on / blocked
 
Reported: 2024-09-25 09:55 UTC by Uday kurundwade
Modified: 2025-02-28 04:25 UTC (History)
7 users (show)

Fixed In Version: 4.17.0-123
Doc Type: Bug Fix
Doc Text:
.MCG standalone upgrade working as expected Previously, a bug caused NooBaa pods to have incorrect affinity settings, leaving them stuck in the pending state. This fix ensures that any previously incorrect affinity settings on the NooBaa pods are cleared. Affinity is now only applied when the proper conditions are met, preventing the issue from recurring after the upgrade. After upgrading to the fixed version, the pending NooBaa pods won't automatically restart. To finalize the upgrade, manually delete the old pending pods. The new pods will then start with the correct affinity settings, allowing them to run successfully.
Clone Of:
Environment:
Last Closed: 2024-10-30 14:35:59 UTC
Embargoed:
ukurundw: needinfo-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github noobaa noobaa-operator pull 1446 0 None Merged NOOBAA_LOG_COLOR - Add check env var is not null 2024-10-01 15:12:25 UTC
Github noobaa noobaa-operator pull 1453/commits 0 None None None 2024-10-01 15:13:15 UTC
Github red-hat-storage ocs-operator pull 2852 0 None Merged Bug 2314636: [release-4.17] Clear the affinity on Noobaa if it was set previously to handle upgrades 2024-10-15 08:34:54 UTC
Red Hat Issue Tracker OCSBZM-9342 0 None None None 2024-10-08 10:41:48 UTC
Red Hat Product Errata RHSA-2024:8676 0 None None None 2024-10-30 14:36:07 UTC

Description Uday kurundwade 2024-09-25 09:55:05 UTC
Description of problem (please be detailed as possible and provide log
snippests):
When MCG standalone cluster is upgraded from ODF 4.16 to ODF 4.17, noobaa pods can not be scheduled and upgrade does not complete.

Version of all relevant components (if applicable):
Pre upgrade: 4.16.2
Post Upgrade: 4.17.0-106

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
N/A

If this is a regression, please provide more details to justify this:
yes

Steps to Reproduce:
1. Deploy OCS without adding `cluster.ocs.openshift.io/openshift-storage: ''` labels
2. Deploy Stand-alone MCG(PODs will be in pending state due to https://bugzilla.redhat.com/show_bug.cgi?id=2314432)
3. Remove nodeAffinity from noobaa CRD
4. Wait for all noobaa PODs to come into running state
5. Upgrade cluster to 4.17


Actual results:
Noobaa Operator is stuck in Installing mode and PODs are in CLBO mode

cephcsi-operator.v4.17.0-106.stable          CephCSI operator                   4.17.0-106.stable                                           Succeeded
mcg-operator.v4.17.0-106.stable              NooBaa Operator                    4.17.0-106.stable   mcg-operator.v4.16.2-rhodf              Installing
ocs-client-operator.v4.17.0-106.stable       OpenShift Data Foundation Client   4.17.0-106.stable   ocs-client-operator.v4.16.2-rhodf       Succeeded
ocs-operator.v4.17.0-106.stable              OpenShift Container Storage        4.17.0-106.stable   ocs-operator.v4.16.2-rhodf              Succeeded
odf-csi-addons-operator.v4.17.0-106.stable   CSI Addons                         4.17.0-106.stable   odf-csi-addons-operator.v4.16.2-rhodf   Succeeded
odf-operator.v4.17.0-106.stable              OpenShift Data Foundation          4.17.0-106.stable   odf-operator.v4.16.2-rhodf              Succeeded
odf-prometheus-operator.v4.17.0-106.stable   Prometheus Operator                4.17.0-106.stable   odf-prometheus-operator.v4.16.2-rhodf   Succeeded
recipe.v4.17.0-106.stable                    Recipe                             4.17.0-106.stable   recipe.v4.16.2-rhodf                    Succeeded
rook-ceph-operator.v4.17.0-106.stable        Rook-Ceph                          4.17.0-106.stable   rook-ceph-operator.v4.16.2-rhodf        Succeeded

➜  ~ oc get pods | grep noobaa
noobaa-core-0                                      2/2     Running            0             4m51s
noobaa-db-pg-0                                     1/1     Running            0             5m20s
noobaa-default-backing-store-noobaa-pod-be2e916f   0/1     CrashLoopBackOff   3 (51s ago)   13m
noobaa-endpoint-6cc7f54c5f-98zhp                   1/1     Running            0             5m21s
noobaa-operator-5d7745f8f6-k8vp4                   0/1     CrashLoopBackOff   4 (26s ago)   5m1s

Expected results:
Noobaa Operator should upgrade successfully and all POD should be in running state

Additional info:

Comment 6 Sunil Kumar Acharya 2024-10-08 13:17:11 UTC
Please update the RDT flag/text appropriately.

Comment 15 errata-xmlrpc 2024-10-30 14:35:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.17.0 Security, Enhancement, & Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:8676

Comment 16 Red Hat Bugzilla 2025-02-28 04:25:38 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days


Note You need to log in before you can comment on or make changes to this bug.