Bug 2278617 - [Backport to 4.15.z] [IBM Support] After upgrade to OCP/ODF 4.14 Stand alone MCG backing store getting rejected/not ready with ENOSUPT
Summary: [Backport to 4.15.z] [IBM Support] After upgrade to OCP/ODF 4.14 Stand alone ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: Multi-Cloud Object Gateway
Version: 4.14
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: ODF 4.15.3
Assignee: Nimrod Becker
QA Contact: Mahesh Shetty
URL:
Whiteboard:
Depends On: 2278389
Blocks: 2278619
TreeView+ depends on / blocked
 
Reported: 2024-05-02 12:31 UTC by Nimrod Becker
Modified: 2024-07-24 10:30 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, when trying to use persistent volume (PV) pool, `xattr` are used to save metadata of objects. However, updates to that metadata fails as Filesystem on PV does not support `xattr`. With this fix, there will be a failback if Filesystem does not support `xattr` and metadata is saved in a file.
Clone Of: 2278389
: 2278619 (view as bug list)
Environment:
Last Closed: 2024-06-11 16:41:49 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2024:3806 0 None None None 2024-06-11 16:41:57 UTC

Description Nimrod Becker 2024-05-02 12:31:14 UTC
+++ This bug was initially created as a clone of Bug #2278389 +++

Description of problem (please be detailed as possible and provide log
snippests):
Once the Fusion Data Foundation operator was upgraded to 4.14 we encountered an issue with the backingstore going into a rejected state with a reason of "no capacity", which is a generic error for noobaa not being able to talk to the storage for the backingstore.

During troubleshooting in today's session  with the customer we encountered an error reporting ENOTSUP (Error Not Supported) and it was due to the backing storage for the storage class (Netapp Trident) not supported xattrs that Noobaa now has a requirement in ODF 4.14 which was not present in ODF 4.12, this issue is documented here: https://github.com/noobaa/noobaa-core/issues/7843. This was preventing the backingstore from coming ready and communicating with the Netapp Trident Storage.

Version of all relevant components (if applicable):
OCP 4.14/ODF 4.14

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes, Backing store is down

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
4

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
Yes

If this is a regression, please provide more details to justify this:
Yes, This worked in ODF 4.12

Steps to Reproduce:
1. Deploy MCG in ODF 4.12 cluster and create backing store with storage provided that does not support xattr
2.Upgrade ODF to 4.14



Actual results:
Backing store enters rejected state

Expected results:
Backing store to stay phase:ready and serve IO

Additional info:
Details here: https://github.com/noobaa/noobaa-core/issues/7843

--- Additional comment from RHEL Program Management on 2024-05-01 20:07:01 UTC ---

This bug having no release flag set previously, is now set with release flag 'odf‑4.16.0' to '?', and so is being proposed to be fixed at the ODF 4.16.0 release. Note that the 3 Acks (pm_ack, devel_ack, qa_ack), if any previously set while release flag was missing, have now been reset since the Acks are to be set against a release flag.

--- Additional comment from RHEL Program Management on 2024-05-01 20:07:01 UTC ---

Since this bug has severity set to 'urgent', it is being proposed as a blocker for the currently set release flag. Please resolve ASAP.

--- Additional comment from RHEL Program Management on 2024-05-01 20:07:01 UTC ---

The 'Target Release' is not to be set manually at the Red Hat OpenShift Data Foundation product.

The 'Target Release' will be auto set appropriately, after the 3 Acks (pm,devel,qa) are set to "+" for a specific release flag and that release flag gets auto set to "+".

--- Additional comment from Mike Hackett on 2024-05-01 20:17:59 UTC ---

We have a Hotfix request coming in for Verizon once a fix is available for https://github.com/noobaa/noobaa-core/issues/7843.

- Issue Description
Nooba Backing Store entering rejected state ENOTSUP after upgrade from ODF 4.12 to ODF 4.14. This is due to change in ODF 4.14 in which Noobaa requires support for xattr from the storage for the backing store.

- Business Justification
Currently backingstore is down and VZ cannot go live to an end user. This is escalated issue within IBM currently up to Danny Mace.
- Case
TS016028639

- Bug
https://bugzilla.redhat.com/show_bug.cgi?id=2278389

- Product Version
ODF 4.14

- Upstream Issue
https://github.com/noobaa/noobaa-core/issues/7843.

Comment 13 errata-xmlrpc 2024-06-11 16:41:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Data Foundation 4.15.3 Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:3806


Note You need to log in before you can comment on or make changes to this bug.