Bug 2262323

Summary: [IBM Support] Both MDS in consistent CLBO due to OOM: !normal: cannot ephemeral random pin [inode seen
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Mike Hackett <mhackett>
Component: cephAssignee: Venky Shankar <vshankar>
ceph sub component: CephFS QA Contact: Elad <ebenahar>
Status: ASSIGNED --- Docs Contact:
Severity: high    
Priority: unspecified CC: akandath, assingh, bniver, gfarnum, gsitlani, muagarwa, pdhange, pdonnell, sheggodu, sostapov, sshreeka, vshankar, vumrao
Version: 4.12Flags: vshankar: needinfo? (assingh)
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mike Hackett 2024-02-01 20:04:51 UTC 
Description of problem (please be detailed as possible and provide log
snippests):
Both active and Primay MDS enter CLBO consistently due to OOM. We increased default memory from 8GiB to 16Gib and this did not resolve the issue. Memory consumption on the nodes is around 45% with 16GiB set for the MDS's.

We set debug on primary and secondary MDS to catch and issues during boot before crash and during boot of the MDS after OOM we can see the following messages in the standby MDS:

!normal: cannot ephemeral random pin [inode 0x1000136d2ca [2,head] /volumes/csi/csi-vol-fe6bdadf-2e7a-4684-b340-f7779c93c974/f07968b8-5ab2-4033-8cfe-10326680dbb3/content_repository/361/1706680874886-7132521 auth v30188 s=0 n(v0 1=1+0) (iversion lock) 0x563159fff080] 2024-02-01T18:11:34.288+0000 7f4baf916700 20 mds.0.cache.dir(0x100000091ae) _fetched pos 6794 marker 'i' dname '1706680874282-7131497 [2,head]

Working with prashant Dhange we reviewed mds_oft_prefetch_dirfrags which was set to default false.

We have gathered latest mustgather and debug logs from primary and standby MDS for review.

Need assistance determining cause for OOM and memory consumption and possible resolution. 

Version of all relevant components (if applicable):
ODF 4.12.2
RH Ceph 5.1z1

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes, Both MDS are down. Impact to customer applications consuming file based PVC's.

Is there any workaround available to the best of your knowledge?
No, we cannot get MDS to stay up.

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

4

Can this issue reproducible?
In customer environement, yes

Can this issue reproduce from the UI?
No

If this is a regression, please provide more details to justify this:
unaware

Steps to Reproduce:
Issue hit in Customer's cluster


Actual results:
MDS goes into OOM

Expected results:
MDS to not OOM

Additional info:

IBM Case number: TS015350570
Comment 25 Venky Shankar 2024-02-08 04:54:56 UTC 
@all, tracker https://tracker.ceph.com/issues/64348 to investigate a possible memleak in up:rejoin.

Will have someone work on it on prio. We should also be prepared with what debug logs to gather the next time some customer runs into this.