Bug 2278617

Summary: [Backport to 4.15.z] [IBM Support] After upgrade to OCP/ODF 4.14 Stand alone MCG backing store getting rejected/not ready with ENOSUPT
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Nimrod Becker <nbecker>
Component: Multi-Cloud Object GatewayAssignee: Nimrod Becker <nbecker>
Status: CLOSED ERRATA QA Contact: Mahesh Shetty <mashetty>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.14CC: asriram, hnallurv, kbg, kramdoss, mhackett, odf-bz-bot, usrivast
Target Milestone: ---   
Target Release: ODF 4.15.3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, when trying to use persistent volume (PV) pool, `xattr` are used to save metadata of objects. However, updates to that metadata fails as Filesystem on PV does not support `xattr`. With this fix, there will be a failback if Filesystem does not support `xattr` and metadata is saved in a file.
Story Points: ---
Clone Of: 2278389
: 2278619 (view as bug list) Environment:
Last Closed: 2024-06-11 16:41:49 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2278389    
Bug Blocks: 2278619    

Description Nimrod Becker 2024-05-02 12:31:14 UTC
+++ This bug was initially created as a clone of Bug #2278389 +++

Description of problem (please be detailed as possible and provide log
snippests):
Once the Fusion Data Foundation operator was upgraded to 4.14 we encountered an issue with the backingstore going into a rejected state with a reason of "no capacity", which is a generic error for noobaa not being able to talk to the storage for the backingstore.

During troubleshooting in today's session  with the customer we encountered an error reporting ENOTSUP (Error Not Supported) and it was due to the backing storage for the storage class (Netapp Trident) not supported xattrs that Noobaa now has a requirement in ODF 4.14 which was not present in ODF 4.12, this issue is documented here: https://github.com/noobaa/noobaa-core/issues/7843. This was preventing the backingstore from coming ready and communicating with the Netapp Trident Storage.

Version of all relevant components (if applicable):
OCP 4.14/ODF 4.14

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes, Backing store is down

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
4

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
Yes

If this is a regression, please provide more details to justify this:
Yes, This worked in ODF 4.12

Steps to Reproduce:
1. Deploy MCG in ODF 4.12 cluster and create backing store with storage provided that does not support xattr
2.Upgrade ODF to 4.14



Actual results:
Backing store enters rejected state

Expected results:
Backing store to stay phase:ready and serve IO

Additional info:
Details here: https://github.com/noobaa/noobaa-core/issues/7843

--- Additional comment from RHEL Program Management on 2024-05-01 20:07:01 UTC ---

This bug having no release flag set previously, is now set with release flag 'odf‑4.16.0' to '?', and so is being proposed to be fixed at the ODF 4.16.0 release. Note that the 3 Acks (pm_ack, devel_ack, qa_ack), if any previously set while release flag was missing, have now been reset since the Acks are to be set against a release flag.

--- Additional comment from RHEL Program Management on 2024-05-01 20:07:01 UTC ---

Since this bug has severity set to 'urgent', it is being proposed as a blocker for the currently set release flag. Please resolve ASAP.

--- Additional comment from RHEL Program Management on 2024-05-01 20:07:01 UTC ---

The 'Target Release' is not to be set manually at the Red Hat OpenShift Data Foundation product.

The 'Target Release' will be auto set appropriately, after the 3 Acks (pm,devel,qa) are set to "+" for a specific release flag and that release flag gets auto set to "+".

--- Additional comment from Mike Hackett on 2024-05-01 20:17:59 UTC ---

We have a Hotfix request coming in for Verizon once a fix is available for https://github.com/noobaa/noobaa-core/issues/7843.

- Issue Description
Nooba Backing Store entering rejected state ENOTSUP after upgrade from ODF 4.12 to ODF 4.14. This is due to change in ODF 4.14 in which Noobaa requires support for xattr from the storage for the backing store.

- Business Justification
Currently backingstore is down and VZ cannot go live to an end user. This is escalated issue within IBM currently up to Danny Mace.
- Case
TS016028639

- Bug
https://bugzilla.redhat.com/show_bug.cgi?id=2278389

- Product Version
ODF 4.14

- Upstream Issue
https://github.com/noobaa/noobaa-core/issues/7843.

Comment 13 errata-xmlrpc 2024-06-11 16:41:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Data Foundation 4.15.3 Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:3806