Bug 117561 - Race condition in fs/dcache.c
Race condition in fs/dcache.c
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 3
Classification: Red Hat
Component: kernel (Show other bugs)
3.0
i686 Linux
medium Severity high
: ---
: ---
Assigned To: Jeff Layton
: Reopened
Depends On:
Blocks: 203310
  Show dependency treegraph
 
Reported: 2004-03-05 07:54 EST by Claus Rasmussen
Modified: 2014-06-18 03:34 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2006-11-03 16:12:37 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
updated patch (1.59 KB, patch)
2006-08-23 09:41 EDT, Jeff Layton
no flags Details | Diff
possible debug patch for nr_unused race (515 bytes, patch)
2007-01-30 13:25 EST, Jeff Layton
no flags Details | Diff

  None (edit)
Description Claus Rasmussen 2004-03-05 07:54:43 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1)
Gecko/20030225

Description of problem:
The nr_dentry entry in /proc/sys/fs/dentry-state decreases under heavy
load from certain programs and eventually becomes negative. The
decrease is generally continuous, but interupted by spikes from time
to time:

2004-03-05 07:00:42  -626  2424    45      0       0       0
2004-03-05 07:00:52   278  3263    45      0       0       0

and a little later

2004-03-05 07:03:07  2078  5019    45      0       0       0
2004-03-05 07:03:17 -3219  42      45      0       0       0

We've seen values down to -13.000.000 after three days at load 10-15.

'sar -v' also reports unbelievable values in its dentunusd column:
4294963724 - clearly a sign error.


Version-Release number of selected component (if applicable):
kernel-smp-2.4.21-9.0.1.EL

How reproducible:
Always

Steps to Reproduce:
Unfortunately this is only reproducable by running a propetary program
 of our own making. The program is multi-threaded and communicates
through sockets with a perl script. From time to time the program
flushes large amounts of data to disk. 

The problem seems only to show up  when we're running several
instances of the program simultaneously and bringing the machine to a
load of 10-15. But it is reproducable every time we try.


Actual Results:  The dentry-nr value in /proc/sys/fs/dentry-state goes
negative; load falls to 2-5 seemingly without any reason;
communication between the binary program and the perl scripts slows to
a crawl; netstat show all sockets to have the same inode number;
'netstat -p' looses the program information; 'sar -v' reports bogus
values for dentunusd and generally the machine is behaving oddly.

Note that we're not sure that all of the above symptoms are related to
 the dentry readings, but by inspecting the whole /proc hierachy this
was the only odd reading we could find.

Expected Results:  Contrary to all available documentation which says
that dentry-nr should be zero most of the time, the code in
fs/dcache.c shows that dentry_nr should have a positive value and
certainly not a negative one.

Additional info:

We've found a patch from Andrew Morton towards 2.5.47 that probably
addresses the problem we are seing. An exerpt from the accompanied
message reads:

    The patch also arranges (awkwardly) for all modifications of
    dentry_stat.nr_dentry to occur inside dcache_lock - it was racy.

The patch-set can be found here:
http://www.zip.com.au/~akpm/linux/patches/2.5/old/2.5.47.tar.gz (the 
inode-reclaim-balancing.patch patch)

The message in the patch-set does not include the excerpt above (we're
pretty sure its the same patch, however); instead it can be found
here: http://www.linuxhq.com/kernel/changelog/v2.5/48/ . Search for
'nr_dentry'. This seems to be the changelog for the kernel proper
between v2.5.47 and v2.5.48.
Comment 2 Claus Rasmussen 2004-12-21 22:22:01 EST
This is fixed in RHEL4 beta 1. All of the above symptoms have 
vanished completely. 
Comment 4 Ernie Petrides 2006-05-01 14:37:59 EDT
RHEL3 is now closed.
Comment 8 Jeff Layton 2006-08-23 09:41:08 EDT
Created attachment 134710 [details]
updated patch

This patch is a backport of this one from upstream:

http://lkml.org/lkml/2004/9/19/9

It looks rather innocuous, but we'll need to make a determination if it meets
the criticality threshold for 3.9.
Comment 9 RHEL Product and Program Management 2006-10-02 12:47:51 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.
Comment 16 Jeff Layton 2006-10-25 10:07:56 EDT
From what I can tell, nr_dentry isn't actually used for anything in the VM, so
the fact that it goes negative shouldnt have any impact on the box. It just
means that the nr_dentry counter is off when you look at it in /proc. The patch
is probably pretty harmless, but the fix seems to be cosmetic.
Comment 22 Ernie Petrides 2006-11-03 16:12:37 EST
Closing based on last comment.
Comment 24 Eric Sandeen 2007-01-30 13:14:05 EST
As an aside:

The issue in Bug #117561 is purely cosmetic; nr_dentry is only set, never read
(except in /proc output) in the kernel.

Bug #117400 is more worrisome since the vm makes decisions based on that number.
Comment 25 Eric Sandeen 2007-01-30 13:17:13 EST
*sigh* firefox tabs strike again, comment hit the wrong bug.
Comment 26 Jeff Layton 2007-01-30 13:25:13 EST
Created attachment 146939 [details]
possible debug patch for nr_unused race

Also as an aside...

The problem in 117400 we think is likely that something is calling dget_locked
without holding the dcache_lock. This patch is a possible way to check for
this, though it won't tell you if someone else is holding the dcache_lock when
this task calls it. Still it might be a good way to track down how it is
occurring.

Note You need to log in before you can comment on or make changes to this bug.