Bug 118503

Summary: inode slab cache grows to obscene sizes and doesn't shrink
Product: Red Hat Enterprise Linux 2.1 Reporter: Kurtis Rader <kdrader>
Component: kernelAssignee: Jason Baron <jbaron>
Status: CLOSED NOTABUG QA Contact: Brian Brock <bbrock>
Severity: medium Docs Contact:
Priority: high    
Version: 2.1CC: knoel, riel
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2005-02-21 15:56:36 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kurtis Rader 2004-03-17 04:50:07 UTC
On an ia32 system with lots of memory (e.g., 16GiB) but with little
high mem page cache pressure it is possible for the inode cache to
grow so large that it puts serious pressure on kernel low mem.

The system in question had been setup in the previous 24 hours and had
not been rebooted since installing the application software (Oracle
Application Suite and Oracle RDBMS). The inode cache is using over
350MB of memory:

CACHE    NAME           OBJSIZE  ALLOCATED     TOTAL  SLABS SSIZE
f7f59544 inode_cache        512     630760    631120  90160    4k

Of which 81% are unused:

crash> p inodes_stat
inodes_stat = $1 = {
  nr_inodes = 777101,
  nr_unused = 629526,
}                                                                    
           

The problem appears to be that prune_icache() isn't freeing unused
inodes.

I picked an inode at random which isn't in use: 0xc04e8100. The
i_data.nrpages value is 3. All the pages are on the clean_pages list.
The i_mapping.nrpages value is also 3 so all the pages are due to the
pages being in the page cache. Unless those pages are freed the inode
won't be freed. All of the other inodes I examined had similar
characteristics.

Looking at the prune_icache() code path I'm suspicious of the
heuristic for selecting pages and inodes to free. I wrote a small
program to put pressure on the page cache by simply allocating a
512MiB block and touching each page. I then started thirty instances.
That caused the inode cache to be shrunk to a more reasonable size
(but note that it's still ten times larger than it really should be):
           

CACHE    NAME           OBJSIZE  ALLOCATED     TOTAL  SLABS SSIZE
f7f59544 inode_cache        512      41849     62454   8922    4k

There is now plenty of free low mem and kswapd is no longer thrashing:

ZONE  NAME        FREE   ACTIVE  INACTIVE_DIRTY  INACTIVE_CLEAN
  0   DMA         1688       63              21               0
  1   Normal     98794     1046             651               0
  2   HighMem  3567514    48426          275700            4319


Version-Release number of selected component (if applicable):
kernel-2.4.9-e.37

How reproducible:
Didn't try

Steps to Reproduce:
At this time I have not reproduced the problem (I'm in the process of
negotiating access to a system with 16GiB of memory). I suspect that
simply reading a large number of files with no other page cache
allocations will grow the inode cache to fill all of kernel low
memory. As long as there remain plenty of free high mem pages any
activity that requires kernel low mem will cause the system to thrash.

Comment 1 Larry Woodman 2004-04-01 16:07:07 UTC
Can you please collect a few "AltSysrq M" outputs when kswapd is
"thrashing" and attach them to this bug?

Thanks, Larry


Comment 2 Larry Woodman 2004-05-11 18:19:18 UTC
I havent seen any more activity on this bug lately but prune_icache()
will not run ulness the system is out of memory.  Its called from
within kswapd as part of memory reclamation.  

Larry


Comment 3 Jason Baron 2005-02-21 15:56:36 UTC
I'm closing this as not a bug. If a reproducer is found, pleaes re-open.