Bug 206282 - NFS hangs under heavy use
NFS hangs under heavy use
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
9
powerpc Linux
medium Severity medium
: ---
: ---
Assigned To: Steve Dickson
Brian Brock
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2006-09-13 10:11 EDT by IBM Bug Proxy
Modified: 2008-08-02 19:40 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-05-14 23:51:07 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
dmesg.txt (30.48 KB, text/plain)
2006-09-25 20:11 EDT, IBM Bug Proxy
no flags Details
messages (71.14 KB, text/plain)
2006-10-30 20:10 EST, IBM Bug Proxy
no flags Details
sysrq-m (1.32 KB, text/plain)
2006-10-31 11:46 EST, IBM Bug Proxy
no flags Details
purposed upstream patch (1.79 KB, patch)
2006-12-05 07:48 EST, Steve Dickson
no flags Details | Diff
the upstream patch seem to fix this problem which has been ported to the devel kernel. (1.79 KB, patch)
2006-12-05 10:37 EST, Steve Dickson
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
IBM Linux Technology Center 27121 None None None Never

  None (edit)
Description IBM Bug Proxy 2006-09-13 10:11:24 EDT
LTC Owner is: dmosby@us.ibm.com
LTC Originator is: jklewis@us.ibm.com


Problem description:
Running a test which uses dd to a network mounted filesystem causes apps on the
system to hang

       Provide output from "uname -a", if possible:
cell4 / # uname -a
Linux cell4.ltc.austin.ibm.com 2.6.18-rc2-SDK #3 SMP Thu Aug 3 10:42:33 CDT 2006
ppc64 ppc64 ppc64 GNU/Linux

Hardware Environment
    Machine type (p650, x235, SF2, etc.): Cell
    Cpu type (Power4, Power5, IA-64, etc.): Cell Broadband Engine, altivec supported
    
Is this reproducible? Yes.
    If so, how long does it (did it) take to reproduce it? About 10 minutes
    Describe the steps:

loop1:
Perform an NFS mount
run dd to it
umount
go to loop1

    
Is the system (not just the application) hung? Any app that needs memory is hung
or will hang
    If so, describe how you determined this: Trace
All of the hung apps have trace info similar to this:

kswapd0       S 0000000000000000     0   152     83           153    95 (L-TLB)
Call Trace:
[C00000003ED8B1F0] [C00000000003FF28] .find_busiest_group+0x21c/0x5e8 (unreliab)
[C00000003ED8B3C0] [C00000000000FBB4] .__switch_to+0xec/0x110
[C00000003ED8B450] [C00000000029D53C] .schedule+0x89c/0x9ec
[C00000003ED8B550] [D0000000001F84A4] .nfs_wait_bit_interruptible+0x30/0x48 [nf]
[C00000003ED8B5D0] [C00000000029E704] .__wait_on_bit+0xa0/0x114
[C00000003ED8B680] [C00000000029E828] .out_of_line_wait_on_bit+0xb0/0xe0
[C00000003ED8B7C0] [D0000000001F8434] .nfs_wait_on_request+0x78/0xb8 [nfs]
[C00000003ED8B870] [D0000000001FCB74] .nfs_wait_on_requests_locked+0x94/0x114 []
[C00000003ED8B920] [D0000000001FE7E0] .nfs_sync_inode_wait+0x78/0x24c [nfs]
[C00000003ED8B9F0] [D0000000001F210C] .nfs_release_page+0x2c/0x5c [nfs]
[C00000003ED8BA70] [C0000000000AE5A0] .try_to_release_page+0x70/0x98
[C00000003ED8BAF0] [C000000000085FA4] .shrink_zone+0xc48/0x1028
[C00000003ED8BD60] [C000000000086C84] .kswapd+0x35c/0x47c
[C00000003ED8BEE0] [C000000000062D08] .kthread+0x120/0x170
[C00000003ED8BF90] [C000000000023D54] .kernel_thread+0x4c/0x68

Did the system produce an OOPS message on the console? No.
   

Is the system sitting in a debugger right now? No, is running normally after
rebooting.
    

Additional information:
This was found while researching Bug #25021 on Cell.

Created an attachment (id=19898)
Trace information
Comment 1 Dave Jones 2006-09-17 00:18:39 EDT
That isn't a Fedora kernel.
There have also been numerous NFS fixes since 2.6.18rc2.

Please retry with the latest rawhide kernel, and reopen if reproducable.
Comment 2 IBM Bug Proxy 2006-09-17 01:10:57 EDT
----- Additional Comments From jklewis@us.ibm.com  2006-09-17 01:08 EDT -------
The 2.6.18-rc2 kernel that I'm running on my Cell blade is not a Fedora kernel?
What does that mean? I might be able to go up to 2.6.18-rc7, is that a Fedora
kernel? 
Comment 3 Steve Dickson 2006-09-22 06:19:08 EDT
Would it be possible to get a bzip2 binary tethereal network
trace of this hang something like: 
    tethereal -w /tmp/data.pcap host <server> ; bzip2 /tmp/data.pcap

also a SysRq-T system trace would be good as well.. 
Comment 4 IBM Bug Proxy 2006-09-25 17:21:07 EDT
----- Additional Comments From jklewis@us.ibm.com  2006-09-25 17:15 EDT -------
I am somewhat familiar with ethereal but have not heard of tethereal. If you can
tell me how/where to get it and any special installation instructions I will
give it a try. I noticed that SysRq doesn't seem to be active by default, tell
me where that is and I'll try that too. 
Comment 5 IBM Bug Proxy 2006-09-25 18:05:50 EDT
----- Additional Comments From dmosby@us.ibm.com (prefers email at k7fw@us.ibm.com)  2006-09-25 18:03 EDT -------
To enable SysRq:
  echo 1 > /proc/sys/kernel/sysrq

To trigger it:
  alt + SysRq + t
or
  echo t > /proc/sysrq-trigger

Details in:  /usr/src/linux/Documentation/sysrq.txt

See if /usr/sbin/tetheereal is present. 
Comment 6 IBM Bug Proxy 2006-09-25 20:11:18 EDT
Created attachment 137101 [details]
dmesg.txt
Comment 7 IBM Bug Proxy 2006-09-25 20:11:45 EDT
----- Additional Comments From jklewis@us.ibm.com  2006-09-25 20:08 EDT -------
 
dmesg after invoking system trace 
Comment 8 IBM Bug Proxy 2006-09-25 20:20:58 EDT
----- Additional Comments From jklewis@us.ibm.com  2006-09-25 20:16 EDT -------
Note: I am now running Fedora Core 6 Test 3 on this Cell blade. Kernel version
2.6.18 with latest patches.

The failure occurred on the first attempt to run the test. However, the system
is more usable than it was with FC5. 

I have attached the contents of dmesg after the system trace. 

This system does not have ethereal/tethereal/tetheereal installed. 
Comment 9 Steve Dickson 2006-09-27 09:22:13 EDT
tethereal is now a part of the wireshark rpm and
I strongly suggest you install it... but
in the meantime you probably have tcpdump installed
so please generate a binary trace with that, but with
the -s0 argument so the trace is meaningful


also what args are giving dd to create this
problem.. 

Comment 10 IBM Bug Proxy 2006-10-03 11:21:25 EDT
----- Additional Comments From jklewis@us.ibm.com  2006-10-03 11:19 EDT -------
I'll look into getting tethereal installed on my blade. I'll assume this has
already been tested on Cell.

The parms to dd are as follows:

Writing: "time dd if=/dev/zero of=tempofile count=1000 bs=1M"
Reading: "time dd if=tempofile of=/dev/null"

I have seen it fail on both reads and writes. 
Comment 11 David Woodhouse 2006-10-19 16:46:51 EDT
Please report precisely which Fedora kernel build you're using.
Comment 12 IBM Bug Proxy 2006-10-20 11:31:08 EDT
----- Additional Comments From jklewis@us.ibm.com  2006-10-20 11:29 EDT -------
I am running FC6 Test3. I went back to kernel 2.6.17-1.2630.fc6 #1 SMP. The test
will not even get past the first interation, with several "server not responding
messages" on the console. Note that with kernel 2.6.18-mm2 the test went much
further before hanging (no messages on console). Also note that under "normal"
NFS use I see no problems of any kind, under several different kernels. 
Comment 13 Steve Dickson 2006-10-20 12:14:36 EDT
Very recently there were some major VM issues that was
caused NFS not to flush data pages correctly which in turned
caused some very strange behavior... 

THe problems should be fixed in the 2.6.18 kernel... so please
update to see if the same behavior exists... 
Comment 14 IBM Bug Proxy 2006-10-20 16:16:28 EDT
----- Additional Comments From jklewis@us.ibm.com  2006-10-20 16:11 EDT -------
I have already been running this test with 2.6.18-mm2, is that late enough? I'll
go ahead install the very latest patches to see if it helps. 
Comment 15 IBM Bug Proxy 2006-10-23 15:51:00 EDT
----- Additional Comments From jklewis@us.ibm.com  2006-10-23 15:46 EDT -------
I installed 2.6.18.1 and the test ran to 167 iterations before hanging. I just
now compiled and tried to run 2.6.19-rc2 but the kernel got a panic on boot up.
I'll see if I can figure out what that's all about. 
Comment 16 Steve Dickson 2006-10-24 07:14:42 EDT
I'm not seeing this hang using two xen guest (2.6.18-1.2732.el5xen and
2.6.18-1.2798.fc6xen) but of course that environment is 120% different
than yours...

One thing I did notice from the dmesg in Comment #6 was the
dd is hung waiting to lock a page. So the question is, who has that
page locked... The only other process thats in the filesystem code is
syslogd which is waiting to sync an inode.

Of course this all could be a red herring.... but in other hangs, is there
a similar foot print? Meaning is NFS hung waiting for a page while
another process is lost somewhere in the filesystem code?
Comment 17 IBM Bug Proxy 2006-10-25 13:11:05 EDT
----- Additional Comments From jklewis@us.ibm.com  2006-10-25 13:06 EDT -------
The hang, or whatever it is, still occurred using 2.6.19-rc3. There were no
relevant messages in the logs.

I believe you are on the right track about this being a lock issue. I'll try
running the test without the lock parameter to see if that makes any difference. 
Comment 18 IBM Bug Proxy 2006-10-25 15:55:57 EDT
----- Additional Comments From jklewis@us.ibm.com  2006-10-25 15:54 EDT -------
It fails (hangs) even if I remove the lock parameter from the script. 
Comment 19 Steve Dickson 2006-10-27 08:00:23 EDT
Sorry for the delay... Theres been a bit of excitement around here
lately... ;-) 

Could you post a SysRq-T system trace of hang w/out the lock parameter?
Comment 20 IBM Bug Proxy 2006-10-27 11:01:07 EDT
----- Additional Comments From jklewis@us.ibm.com  2006-10-27 10:58 EDT -------
Yes, I can just imagine :)

I'm currently running another test but I'll try and get the SysRq trace on
Monday. Just for fun I ran this exact testcase between 2 Intel boxes both
running Fedora Core 5. I guess I expected (hoped?) that it would fail in that
scenario as well. It did not fail, and is still going. I'm not even sure how
long this test goes for, I have never seen it run this long before. 
Comment 21 IBM Bug Proxy 2006-10-30 20:10:58 EST
Created attachment 139789 [details]
messages
Comment 22 IBM Bug Proxy 2006-10-30 20:11:24 EST
----- Additional Comments From jklewis@us.ibm.com  2006-10-30 20:05 EDT -------
 
Trace running test without lock parm

Hung really quick this time. For what it's worth the test between the 2 Intel
boxes ran to completion (801 loops). 
Comment 23 Steve Dickson 2006-10-30 20:21:04 EST
hmm... the dd is stuck in the same place as before...

dd            D 000000000ff1a148     0 12162  12161                     (NOTLB)
Call Trace:
    .__switch_to+0xec/0x110
    .schedule+0x94c/0xad0
    .io_schedule+0x40/0x74
    .sync_page+0x7c/0x98
    .__wait_on_bit_lock+0x8c/0x110
    .__lock_page+0x70/0x90
    .do_generic_mapping_read+0x230/0x4cc
    .generic_file_aio_read+0x18c/0x234
    .nfs_file_read+0xb8/0xe4 [nfs]
    .do_sync_read+0xd8/0x138
    .vfs_read+0xd0/0x1b4
    .sys_read+0x4c/0x8c
    syscall_exit+0x0/0x40

waiting for a page...

Please post an SysRq-M which will dump the current memory usages
at the time of the hang.... 

Comment 24 IBM Bug Proxy 2006-10-31 11:46:06 EST
Created attachment 139872 [details]
sysrq-m
Comment 25 IBM Bug Proxy 2006-10-31 11:46:32 EST
----- Additional Comments From jklewis@us.ibm.com  2006-10-31 11:42 EDT -------
 
SysRq-M trace 
Comment 28 IBM Bug Proxy 2006-11-30 11:21:57 EST
----- Additional Comments From jwboyer@us.ibm.com (prefers email at jwboyer@linux.vnet.ibm.com)  2006-11-30 08:50 EDT -------
We encountered an identical backtrace on a different kernel version and spent
quite some time debugging it.  After trying several different kernel versions
including 2.6.18, we found that the patch found in this thread:

http://lkml.org/lkml/2006/11/4/113

appears to have fixed the issue.  Our tests have run for 2 days 15 hours so far.
 Again, this wasn't on a Fedora kernel however it seems to be a generic upstream
problem.  You might want to try the patch. 
Comment 29 IBM Bug Proxy 2006-11-30 11:51:04 EST
----- Additional Comments From linas@us.ibm.com (prefers email at linas@austin.ibm.com)  2006-11-30 11:45 EDT -------
Thanks for the link. Howevre, if you follow the discussion, you'll see
that the patch is rejected because it introduces a different bug.
A revised patch is at 

 http://lkml.org/lkml/2006/11/8/243

Niether patch is in linux-2.6.19-rc4-git3 yet. 
Comment 30 IBM Bug Proxy 2006-11-30 12:26:10 EST
------- Additional Comments From jwboyer@us.ibm.com (prefers email at jwboyer@linux.vnet.ibm.com)  2006-11-30 12:19 EDT -------
(In reply to comment #31)
> Thanks for the link. Howevre, if you follow the discussion, you'll see
> that the patch is rejected because it introduces a different bug.
> A revised patch is at 
> 
>  http://lkml.org/lkml/2006/11/8/243

Yes I'm aware of that.  That is actually the patch we have been testing.  Sorry
for the confusion. 
Comment 31 Steve Dickson 2006-12-05 07:48:11 EST
Created attachment 142831 [details]
purposed upstream patch 

Attached the devel kernel patch is the outcome of the discussion 
sited in Comment #29
Comment 32 Steve Dickson 2006-12-05 10:37:31 EST
Created attachment 142860 [details]
the upstream patch seem to fix this problem which has been ported to the devel kernel.
Comment 33 Steve Dickson 2007-03-09 09:26:06 EST
This seems to be fixed in later kernels... do you agree?
Comment 34 Bug Zapper 2008-05-13 22:21:19 EDT
Changing version to '9' as part of upcoming Fedora 9 GA.
More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 35 Jon Stanley 2008-05-14 23:51:07 EDT
Closing since this seems to have been fixed long ago.  Feel free to re-open if not.

Note You need to log in before you can comment on or make changes to this bug.