Bug 408861 - applications hang when using a loopback NFSV4 mount
applications hang when using a loopback NFSV4 mount
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel (Show other bugs)
5.1
x86_64 Linux
medium Severity medium
: ---
: ---
Assigned To: Steve Dickson
Martin Jenner
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2007-12-03 12:14 EST by Aime Le Rouzic
Modified: 2009-06-19 21:47 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-12-09 16:16:47 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Aime Le Rouzic 2007-12-03 12:14:19 EST
Description of problem:

Running several benchmarks(connectathon,iozone,fss_stress,fsx,ffsb)
simultaneously under a loopback NFSV4 mount (one machine used as client and
server), those benchmarks hang after one hour and half or two hours because the
mounted repertory no longer accessible.


Version-Release number of selected component (if applicable):
Linux version 2.6.18-53.el5 (brewbuilder@hs20-bc1-7.build.redhat.com) (gcc
version 4.1.2 20070626 (Red Hat 4.1.2-14)) #1 SMP Wed Oct 10 16:34:19 EDT 2007


How reproducible:


Run the connectathon benchmark in a while true loop:
while true
do
./runtests -a -t /mnt/nosec/cthon_nfs4_nfs1_gb
date
sleep 30
done


Run the iozone benchmark in a while true loop:
while true
do
./iozone -+q 30 -ace -r 64 -g 2G -i 0 -i 1 -i 2 -f /mnt/nosec/file1_nfs1_alto_gb
date
sleep 30
done


Run the fsx benchmark in a while true loop:
while true; do ./fsx-linux -N 50000 /mnt/nosec/fsx_nfs1_nfs4_gb; date; sleep 30;
done

Run the fssstress benchmark in a while true loop:
while true; do for p in 2 4 8 16 32 ; do for n in 1 2 4 8 16 32 64 128; do
./fsstress -d /mnt/nosec/fsstress_nfs1_nfs4_gb -p $p -n $n ; date; sleep 60;
done; done; done


Run the ffsb benchmark in a while true loop:

while true
do
../ffsb profile_stress_test
date
sleep 120
done


After one hour and half , benchmarks hang with the following in
/var/log/messages if  we do  echo t > /proc/sysrq-trigger

kernel 64a9d>] do_page_fault+0x4eb/0x81d
kernel:  [<ffffffff80060f29>] thread_return+0x0/0xeb
kernel:  [<ffffffff800b27ae>] audit_syscall_entry+0x14d/0x180
kernel:  [<ffffffff80049ff7>] sys_poll+0x2c/0x33
kernel:  [<ffffffff8005b28d>] tracesys+0xd5/0xe0
kernel:
kernel: rpc.statd     S 7fffffffffffffff     0  2379      1          2404  2350
(NOTLB)
kernel:  ffff81006de6da18 0000000000000086 ffff81007f616690 0000004400000000
kernel:  0000000000000008 ffff8100762b07a0 ffff81007723b100 0000002dfffd85e0
kernel:  000000000000352d ffff8100762b0988 0000000000000000 ffff81000000fc10
kernel: Call Trace:
kernel:  [<ffffffff8024135d>] inet_fill_ifaddr+0x1f2/0x27b
kernel:  [<ffffffff800617cd>] schedule_timeout+0x1e/0xad
kernel:  [<ffffffff800476f7>] add_wait_queue+0x24/0x34
kernel:  [<ffffffff8001334f>] tcp_poll+0x1c/0x12f
kernel:  [<ffffffff80011341>] do_select+0x3fe/0x462
kernel:  [<ffffffff8001e234>] __pollwait+0x0/0xe2
kernel:  [<ffffffff800884ac>] default_wake_function+0x0/0xe
last message repeated 2 times
kernell:  [<ffffffff8009b446>] autoremove_wake_function+0x0/0x2e
kernel:  [<ffffffff800532b4>] skb_queue_tail+0x17/0x3e
kernel:  [<ffffffff80012874>] sock_def_readable+0x10/0x5f
kernel:  [<ffffffff8025a870>] unix_dgram_sendmsg+0x43d/0x4cf
kernel:  [<ffffffff80052bd5>] sock_sendmsg+0xf3/0x110
kernel:  [<ffffffff8000e630>] link_path_walk+0xd3/0xe5
kernel:  [<ffffffff8009b446>] autoremove_wake_function+0x0/0x2e
kernel:  [<ffffffff800dd334>] core_sys_select+0x1bc/0x265
kernel:  [<ffffffff8000c1a5>] _atomic_dec_and_lock+0x39/0x57
kernel:  [<ffffffff8002c86b>] mntput_no_expire+0x19/0x89
kernel:  [<ffffffff80209578>] sys_sendto+0x11c/0x14f
kernel:  [<ffffffff80016004>] sys_select+0xc1/0x17c
kernel:  [<ffffffff8005b28d>] tracesys+0xd5/0xe0
kernel:
kernel:
kernel: rpc.idmapd    S 00002aaaaaece560     0  2404      1          2454  2379
(NOTLB)
kernel:  ffff81006df5de78 0000000000000086 0000000000000292 0000000000000003
kernel:  000000000000000a ffff810037fd77e0 ffff81007f9f4100 00000a054a8cd83e
kernel:  000000000000151d ffff810037fd79c8 ffff810000000001 ffff81007f9f4100
kernel: Call Trace:
kernel:  [<ffffffff80061839>] schedule_timeout+0x8a/0xad
kernel:  [<ffffffff80092ada>] process_timeout+0x0/0x5
kernel:  [<ffffffff800ebc37>] sys_epoll_wait+0x184/0x3f9
kernel:  [<ffffffff800884ac>] default_wake_function+0x0/0xe
kernel:  [<ffffffff8005b28d>] tracesys+0xd5/0xe0
kernel:
last message repeated 5 times
kernel:
kernel: rpciod/0      S ffff810074da00c0     0  2526     27          2527  1996
(L-TLB)
kernel:  ffff81006d01be70 0000000000000046 00000000fffffff5 ffff81006bef6b90
kernel:  000000000000000a ffff8100780d6080 ffff81007f232820 000004da29e49afa
kernel:  0000000000001e7c ffff8100780d6268 0000000000000000 0000000000000003
kernel: Call Trace:
kernel:  [<ffffffff8002dfa2>] __wake_up+0x38/0x4f
kernel:  [<ffffffff8833dc30>] :sunrpc:rpc_async_schedule+0x0/0x5
kernel:  [<ffffffff8004b2d1>] run_workqueue+0xba/0xe5
kernel:  [<ffffffff8009b283>] keventd_create_kthread+0x0/0x61
kernel:  [<ffffffff80047c09>] worker_thread+0x0/0x122
kernel:  [<ffffffff80047cc9>] worker_thread+0xc0/0x122
kernel:  [<ffffffff800884ac>] default_wake_function+0x0/0xe
kernel:  [<ffffffff8009b283>] keventd_create_kthread+0x0/0x61
kernel:  [<ffffffff8009b283>] keventd_create_kthread+0x0/0x61
kernel:  [<ffffffff800321d8>] kthread+0xfe/0x132
kernel:  [<ffffffff8005bfb1>] child_rip+0xa/0x11
kernel:  [<ffffffff8009b283>] keventd_create_kthread+0x0/0x61
kernel:  [<ffffffff800320da>] kthread+0x0/0x132
kernel:  [<ffffffff8005bfa7>] child_rip+0x0/0x11
kernel: rpciod/1      S ffffffff8009b283     0  2527     27          3130  2526
(L-TLB)
kernel:  ffff81006da9de70 0000000000000046 00000000fffffff5 ffff81006bef6128
kernel:  000000000000000a ffff8100755a30c0 ffff81007f9f4100 000009fcd6b10c51
kernel:  0000000000000a73 ffff8100755a32a8 0000000000000001 0000000000000003
kernel: Call Trace:
kernel:  [<ffffffff8009b283>] keventd_create_kthread+0x0/0x61
kernel:  [<ffffffff80047c09>] worker_thread+0x0/0x122
kernel:  [<ffffffff80047cc9>] worker_thread+0xc0/0x122
kernel:  [<ffffffff800884ac>] default_wake_function+0x0/0xe
kernel:  [<ffffffff8009b283>] keventd_create_kthread+0x0/0x61
kernel:  [<ffffffff8009b283>] keventd_create_kthread+0x0/0x61
kernel:  [<ffffffff800321d8>] kthread+0xfe/0x132
kernel:  [<ffffffff8005bfb1>] child_rip+0xa/0x11
kernel:  [<ffffffff8009b283>] keventd_create_kthread+0x0/0x61
kernel:  [<ffffffff800320da>] kthread+0x0/0x132
kernel:  [<ffffffff8005bfa7>] child_rip+0x0/0x11
kernel:
kernel:
kernel: lockd         S 7fffffffffffffff     0  3129      1          3131  3126
(L-TLB)
kernel:  ffff81007737be60 0000000000000046 0000000000000000 0000000000000021
kernel:  0000000000000001 ffff8100755a3820 ffffffff802dcae0 00000069dfc08918
kernel:  0000000000000c3f ffff8100755a3a08 ffff810000000000 ffff810075bb47a0
kernel: Call Trace:
kernel:  [<ffffffff800617cd>] schedule_timeout+0x1e/0xad
kernel:  [<ffffffff80062aa9>] _spin_lock_bh+0x9/0x14
kernel:  [<ffffffff800476f7>] add_wait_queue+0x24/0x34
kernel:  [<ffffffff88341e7e>] :sunrpc:svc_recv+0x306/0x4bc
kernel:  [<ffffffff800884ac>] default_wake_function+0x0/0xe
kernel:  [<ffffffff8837608b>] :lockd:lockd+0x0/0x275
kernel:  [<ffffffff8837608b>] :lockd:lockd+0x0/0x275
kernel:  [<ffffffff883761da>] :lockd:lockd+0x14f/0x275
kernel:  [<ffffffff8005bfb1>] child_rip+0xa/0x11
kernel:  [<ffffffff8837608b>] :lockd:lockd+0x0/0x275
kernel:  [<ffffffff8837608b>] :lockd:lockd+0x0/0x275
kernel:  [<ffffffff8005bfa7>] child_rip+0x0/0x11
kernel:
kernel:
kernel: nfsd4         S ffffffff8009b283     0  3130     27         13680  2527
(L-TLB)
kernel:  ffff810061eb7e70 0000000000000046 0000000000000286 ffffffff8001c40f
kernel:  000000000000000a ffff8100761b6080 ffffffff802dcae0 00000a0585a47ce6
kernel:  0000000000000af8 ffff8100761b6268 0000000000000000 0000000000000003
kernel: Call Trace:
kernel:  [<ffffffff8001c40f>] __mod_timer+0xb0/0xbe
kernel:  [<ffffffff8009b283>] keventd_create_kthread+0x0/0x61
kernel:  [<ffffffff80047c09>] worker_thread+0x0/0x122
kernel:  [<ffffffff80047cc9>] worker_thread+0xc0/0x122
kernel:  [<ffffffff800884ac>] default_wake_function+0x0/0xe
kernel:  [<ffffffff8009b283>] keventd_create_kthread+0x0/0x61
kernel:  [<ffffffff8009b283>] keventd_create_kthread+0x0/0x61
kernel:  [<ffffffff800321d8>] kthread+0xfe/0x132
kernel:  [<ffffffff8005bfb1>] child_rip+0xa/0x11
kernel:  [<ffffffff8009b283>] keventd_create_kthread+0x0/0x61
kernel:  [<ffffffff800320da>] kthread+0x0/0x132
kernel:  [<ffffffff8005bfa7>] child_rip+0x0/0x11
kernel:
kernel: nfsd          D ffff810067eea4c4     0  3131      1          3132  3129
(L-TLB)
kernel:  ffff810061f597a0 0000000000000046 ffff810075509e80 0000000800000001
kernel:  000000000000000a ffff8100765dc860 ffff81007807e820 000004b4574d695d
kernel:  00000000000256b8 ffff8100765dca48 0000000000000000 ffffffff800e0578
kernel: Call Trace:
kernel:  [<ffffffff800e0578>] inode_wait+0x0/0xd
kernel:  [<ffffffff800619c4>] __wait_on_bit+0x60/0x6f
kernel:  [<ffffffff800e0578>] inode_wait+0x0/0xd
kernel:  [<ffffffff80061be8>] __mutex_lock_slowpath+0x55/0x90
kernel:  [<ffffffff80061c32>] .text.lock.mutex+0xf/0x14
kernel:  [<ffffffff800bdf72>] generic_file_writev+0x37/0xa2
kernel:  [<ffffffff80017880>] do_sync_write+0x0/0x104
kernel:  [<ffffffff800d5eae>] do_readv_writev+0x176/0x295
kernel:  [<ffffffff80017880>] do_sync_write+0x0/0x104
kernel:  [<ffffffff8839a8a0>] :nfsd:nfsd_acceptable+0x0/0xd8
kernel:  [<ffffffff80052373>] tcp_cong_avoid+0xe/0x1d
kernel:  [<ffffffff80025003>] tcp_ack+0x1719/0x1871
kernel:  [<ffffffff88345c6a>] :sunrpc:sunrpc_cache_lookup+0x4b/0x127
kernel:  [<ffffffff8839b77d>] :nfsd:nfsd_vfs_write+0xf2/0x2e1
kernel:  [<ffffffff80095d48>] set_current_groups+0x159/0x164
kernel:  [<ffffffff8839bfa9>] :nfsd:nfsd_write+0x6d/0xd5
kernel:  [<ffffffff883a82e5>] :nfsd:nfsd4_proc_compound+0x149a/0x1655
kernel:  [<ffffffff80031648>] sock_common_recvmsg+0x2d/0x42
kernel:  [<ffffffff8002ffe2>] sock_recvmsg+0x101/0x120
kernel:  [<ffffffff8002b610>] local_bh_enable+0x90/0xa1
kernel:  [<ffffffff8009b446>] autoremove_wake_function+0x0/0x2e
kernel:  [<ffffffff80033dd3>] ip_queue_xmit+0x3fb/0x450
kernel:  [<ffffffff80044cf9>] try_to_wake_up+0x407/0x418
kernel:  [<ffffffff8003d03b>] lock_timer_base+0x1b/0x3c
kernel:  [<ffffffff8001c40f>] __mod_timer+0xb0/0xbe
kernel:  [<ffffffff80209ce9>] kernel_recvmsg+0x3b/0x4d
kernel:  [<ffffffff8833fd96>] :sunrpc:svc_recvfrom+0x89/0xbf
kernel:  [<ffffffff88340000>] :sunrpc:svc_sock_enqueue+0x234/0x247
kernel:  [<ffffffff88341afa>] :sunrpc:svc_tcp_recvfrom+0x7a2/0x820
kernel:  [<ffffffff883425be>] :sunrpc:ip_map_match+0x19/0x30
kernel:  [<ffffffff800c35e8>] zone_statistics+0x3e/0x6d
kernel:  [<ffffffff88345c6a>] :sunrpc:sunrpc_cache_lookup+0x4b/0x127
kernel:  [<ffffffff883aa44f>] :nfsd:nfs4svc_decode_compoundargs+0x362/0xe54
kernel:  [<ffffffff883981c4>] :nfsd:nfsd_dispatch+0xd7/0x198
kernel:  [<ffffffff8833f514>] :sunrpc:svc_process+0x44d/0x70b
kernel:  [<ffffffff800625bf>] __down_read+0x12/0x92
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff883986fb>] :nfsd:nfsd+0x1ae/0x2db
kernel:  [<ffffffff8005bfb1>] child_rip+0xa/0x11
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff8005bfa7>] child_rip+0x0/0x11
kernel:
kernel: nfsd          S ffffffff88451fb5     0  3132      1          3133  3131
(L-TLB)
kernel:  ffff810061ce7980 0000000000000046 0000000000000007 ffffffff80025003
kernel:  0000000000000009 ffff81007f2367a0 ffff81007f9f4100 000004b0472a57c3
kernel:  0000000000005e64 ffff81007f236988 0000000000000001 ffff81a00fd00000
kernel: Call Trace:
kernel:  [<ffffffff80025003>] tcp_ack+0x1719/0x1871
kernel:  [<ffffffff88451fb5>] :nfs:nfs_wait_bit_interruptible+0x0/0x28
kernel:  [<ffffffff88451fd7>] :nfs:nfs_wait_bit_interruptible+0x22/0x28
kernel:  [<ffffffff800619a4>] __wait_on_bit+0x40/0x6f
kernel:  [<ffffffff88451fb5>] :nfs:nfs_wait_bit_interruptible+0x0/0x28
kernel:  [<ffffffff80061a3f>] out_of_line_wait_on_bit+0x6c/0x78
kernel:  [<ffffffff8009b474>] wake_bit_function+0x0/0x23
kernel:  [<ffffffff88451f9b>] :nfs:nfs_wait_on_request+0x56/0x70
kernel:  [<ffffffff884556d6>] :nfs:nfs_wait_on_requests_locked+0x70/0xca
kernel:  [<ffffffff8845684e>] :nfs:nfs_sync_inode_wait+0x60/0x1db
kernel:  [<ffffffff8844ce8c>] :nfs:nfs_release_page+0x2c/0x4d
kernel:  [<ffffffff800c2489>] shrink_inactive_list+0x4d8/0x7e7
kernel:  [<ffffffff800c1a53>] isolate_lru_pages+0x98/0xbf
kernel:  [<ffffffff80045d41>] __pagevec_release+0x19/0x22
kernel:  [<ffffffff800c1e90>] shrink_active_list+0x416/0x426
kernel:  [<ffffffff800129ad>] shrink_zone+0xe3/0x108
kernel:  [<ffffffff800c2e5f>] try_to_free_pages+0x163/0x275
kernel:  [<ffffffff8000eff7>] __alloc_pages+0x1b1/0x2b4
kernel:  [<ffffffff88341c7c>] :sunrpc:svc_recv+0x104/0x4bc
kernel:  [<ffffffff800884ac>] default_wake_function+0x0/0xe
kernel:  [<ffffffff800625bf>] __down_read+0x12/0x92
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff8005bfa7>] child_rip+0x0/0x11
kernel:
kernel: nfsd          D ffff810067eea4c4     0  3133      1          3134  3132
(L-TLB)
kernel:  ffff810061f957a0 0000000000000046 ffffffff8009b283 ffffffff80060f29
kernel:  0000000000000009 ffff81007800c0c0 ffff8100787ba7a0 000004b1ec639b49
kernel:  000000000000b7ad ffff81007800c2a8 ffffffff00000001 ffff81000a37e400
kernel: Call Trace:
kernel:  [<ffffffff8009b283>] keventd_create_kthread+0x0/0x61
kernel:  [<ffffffff80060f29>] thread_return+0x0/0xeb
kernel:  [<ffffffff80087d09>] __activate_task+0x27/0x39
kernel:  [<ffffffff80044cf9>] try_to_wake_up+0x407/0x418
kernel:  [<ffffffff80061be8>] __mutex_lock_slowpath+0x55/0x90
kernel:  [<ffffffff80061c32>] .text.lock.mutex+0xf/0x14
kernel:  [<ffffffff800bdf72>] generic_file_writev+0x37/0xa2
kernel:  [<ffffffff80017880>] do_sync_write+0x0/0x104
kernel:  [<ffffffff800d5eae>] do_readv_writev+0x176/0x295
kernel:  [<ffffffff80017880>] do_sync_write+0x0/0x104
kernel:  [<ffffffff8839a8a0>] :nfsd:nfsd_acceptable+0x0/0xd8
kernel:  [<ffffffff80052373>] tcp_cong_avoid+0xe/0x1d
kernel:  [<ffffffff80025003>] tcp_ack+0x1719/0x1871
kernel:  [<ffffffff88345c6a>] :sunrpc:sunrpc_cache_lookup+0x4b/0x127
kernel:  [<ffffffff8839b77d>] :nfsd:nfsd_vfs_write+0xf2/0x2e1
kernel:  [<ffffffff80095d48>] set_current_groups+0x159/0x164
kernel:  [<ffffffff8839bfa9>] :nfsd:nfsd_write+0x6d/0xd5
kernel:  [<ffffffff883a82e5>] :nfsd:nfsd4_proc_compound+0x149a/0x1655
kernel:  [<ffffffff80031648>] sock_common_recvmsg+0x2d/0x42
kernel:  [<ffffffff8002ffe2>] sock_recvmsg+0x101/0x120
kernel:  [<ffffffff8002b610>] local_bh_enable+0x90/0xa1
kernel:  [<ffffffff8009b446>] autoremove_wake_function+0x0/0x2e
kernel:  [<ffffffff80033dd3>] ip_queue_xmit+0x3fb/0x450
kernel:  [<ffffffff8001c40f>] __mod_timer+0xb0/0xbe
kernel:  [<ffffffff8003d03b>] lock_timer_base+0x1b/0x3c
kernel:  [<ffffffff8001c40f>] __mod_timer+0xb0/0xbe
kernel:  [<ffffffff80209ce9>] kernel_recvmsg+0x3b/0x4d
kernel:  [<ffffffff8833fd96>] :sunrpc:svc_recvfrom+0x89/0xbf
kernel:  [<ffffffff88340000>] :sunrpc:svc_sock_enqueue+0x234/0x247
kernel:  [<ffffffff88341afa>] :sunrpc:svc_tcp_recvfrom+0x7a2/0x820
kernel:  [<ffffffff883425be>] :sunrpc:ip_map_match+0x19/0x30
kernel:  [<ffffffff800c35e8>] zone_statistics+0x3e/0x6d
kernel:  [<ffffffff88345c6a>] :sunrpc:sunrpc_cache_lookup+0x4b/0x127
kernel:  [<ffffffff883aa44f>] :nfsd:nfs4svc_decode_compoundargs+0x362/0xe54
kernel:  [<ffffffff883981c4>] :nfsd:nfsd_dispatch+0xd7/0x198
kernel:  [<ffffffff8833f514>] :sunrpc:svc_process+0x44d/0x70b
kernel:  [<ffffffff800625bf>] __down_read+0x12/0x92
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff883986fb>] :nfsd:nfsd+0x1ae/0x2db
kernel:  [<ffffffff8005bfb1>] child_rip+0xa/0x11
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff8005bfa7>] child_rip+0x0/0x11
kernel:
kernel: nfsd          D ffff810067eea4c4     0  3134      1          3135  3133
(L-TLB)
kernel:  ffff810061f5f7a0 0000000000000046 ffff810055bb14f0 ffff81001d482670
kernel:  0000000000000009 ffff81007807e820 ffff810075497100 000004b4574f58ee
kernel:  000000000000bae1 ffff81007807ea08 0000000300000000 ffff81007e4b9ed8
kernel: Call Trace:
kernel:  [<ffffffff80061be8>] __mutex_lock_slowpath+0x55/0x90
kernel:  [<ffffffff80061c32>] .text.lock.mutex+0xf/0x14
kernel:  [<ffffffff800bdf72>] generic_file_writev+0x37/0xa2
kernel:  [<ffffffff80017880>] do_sync_write+0x0/0x104
kernel:  [<ffffffff800d5eae>] do_readv_writev+0x176/0x295
kernel:  [<ffffffff80017880>] do_sync_write+0x0/0x104
kernel:  [<ffffffff8839a8a0>] :nfsd:nfsd_acceptable+0x0/0xd8
kernel:  [<ffffffff80052373>] tcp_cong_avoid+0xe/0x1d
kernel:  [<ffffffff80025003>] tcp_ack+0x1719/0x1871
kernel:  [<ffffffff88345c6a>] :sunrpc:sunrpc_cache_lookup+0x4b/0x127
kernel:  [<ffffffff8839b77d>] :nfsd:nfsd_vfs_write+0xf2/0x2e1
kernel:  [<ffffffff80095d48>] set_current_groups+0x159/0x164
kernel:  [<ffffffff8839bfa9>] :nfsd:nfsd_write+0x6d/0xd5
kernel:  [<ffffffff883a82e5>] :nfsd:nfsd4_proc_compound+0x149a/0x1655
kernel:  [<ffffffff80031648>] sock_common_recvmsg+0x2d/0x42
kernel:  [<ffffffff8002ffe2>] sock_recvmsg+0x101/0x120
kernel:  [<ffffffff8002b610>] local_bh_enable+0x90/0xa1
ernel:  [<ffffffff8009b446>] autoremove_wake_function+0x0/0x2e
kernel:  [<ffffffff80033dd3>] ip_queue_xmit+0x3fb/0x450
kernel:  [<ffffffff80044cf9>] try_to_wake_up+0x407/0x418
kernel:  [<ffffffff80021b63>] tcp_transmit_skb+0x64a/0x682
kernel:  [<ffffffff80209ce9>] kernel_recvmsg+0x3b/0x4d
kernel:  [<ffffffff8833fd96>] :sunrpc:svc_recvfrom+0x89/0xbf
kernel:  [<ffffffff88340000>] :sunrpc:svc_sock_enqueue+0x234/0x247
kernel:  [<ffffffff88341afa>] :sunrpc:svc_tcp_recvfrom+0x7a2/0x820
kernel:  [<ffffffff883425be>] :sunrpc:ip_map_match+0x19/0x30
kernel:  [<ffffffff800c35e8>] zone_statistics+0x3e/0x6d
kernel:  [<ffffffff88345c6a>] :sunrpc:sunrpc_cache_lookup+0x4b/0x127
kernel:  [<ffffffff883aa44f>] :nfsd:nfs4svc_decode_compoundargs+0x362/0xe54
kernel:  [<ffffffff883981c4>] :nfsd:nfsd_dispatch+0xd7/0x198
kernel:  [<ffffffff8833f514>] :sunrpc:svc_process+0x44d/0x70b
kernel:  [<ffffffff800625bf>] __down_read+0x12/0x92
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff883986fb>] :nfsd:nfsd+0x1ae/0x2db
kernel:  [<ffffffff8005bfb1>] child_rip+0xa/0x11
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff8005bfa7>] child_rip+0x0/0x11
kernel:
kernel: nfsd          S ffffffff88451fb5     0  3135      1          3136  3134
(L-TLB)
kernel:  ffff810061e5b030 0000000000000046 000000000000bf18 ffff810061fc12b0
kernel:  0000000000000008 ffff810061fc10c0 ffffffff802dcae0 000004b04729aed0
kernel:  000000000000e747 ffff810061fc12a8 0000000000000000 ffff81000bb1d310
kernel: Call Trace:
kernel:  [<ffffffff88451fb5>] :nfs:nfs_wait_bit_interruptible+0x0/0x28
kernel:  [<ffffffff88451fd7>] :nfs:nfs_wait_bit_interruptible+0x22/0x28
kernel:  [<ffffffff800619a4>] __wait_on_bit+0x40/0x6f
kernel:  [<ffffffff88451fb5>] :nfs:nfs_wait_bit_interruptible+0x0/0x28
kernel:  [<ffffffff80061a3f>] out_of_line_wait_on_bit+0x6c/0x78
kernel:  [<ffffffff8009b474>] wake_bit_function+0x0/0x23
kernel:  [<ffffffff88451f9b>] :nfs:nfs_wait_on_request+0x56/0x70
kernel:  [<ffffffff884556d6>] :nfs:nfs_wait_on_requests_locked+0x70/0xca
kernel:  [<ffffffff8845684e>] :nfs:nfs_sync_inode_wait+0x60/0x1db
kernel:  [<ffffffff8844ce8c>] :nfs:nfs_release_page+0x2c/0x4d
kernel:  [<ffffffff800c2489>] shrink_inactive_list+0x4d8/0x7e7
kernel:  [<ffffffff800c1a53>] isolate_lru_pages+0x98/0xbf
kernel:  [<ffffffff80045d41>] __pagevec_release+0x19/0x22
kernel:  [<ffffffff800c1e90>] shrink_active_list+0x416/0x426
kernel:  [<ffffffff800129ad>] shrink_zone+0xe3/0x108
kernel:  [<ffffffff800c2e5f>] try_to_free_pages+0x163/0x275
kernel:  [<ffffffff8000eff7>] __alloc_pages+0x1b1/0x2b4
kernel:  [<ffffffff8000f8fd>] generic_file_buffered_write+0x19d/0x6d8
kernel:  [<ffffffff883ac68f>] :nfsd:nfsd4_encode_fattr+0x13a2/0x1440
kernel:  [<ffffffff8000ddc8>] current_fs_time+0x3b/0x40
kernel:  [<ffffffff80015db5>] __generic_file_aio_write_nolock+0x36c/0x3b8
kernel:  [<ffffffff80060f29>] thread_return+0x0/0xeb
kernel:  [<ffffffff800733d3>] smp_send_reschedule+0x4e/0x53
kernel:  [<ffffffff800bdf22>] __generic_file_write_nolock+0x8f/0xa8
kernel:  [<ffffffff8009b446>] autoremove_wake_function+0x0/0x2e
kernel:  [<ffffffff80061bf0>] __mutex_lock_slowpath+0x5d/0x90
kernel:  [<ffffffff80061a94>] mutex_lock+0xd/0x1d
kernel:  [<ffffffff800bdf83>] generic_file_writev+0x48/0xa2
kernel:  [<ffffffff80017880>] do_sync_write+0x0/0x104
kernel:  [<ffffffff800d5eae>] do_readv_writev+0x176/0x295
kernel:  [<ffffffff80017880>] do_sync_write+0x0/0x104
kernel:  [<ffffffff8839a8a0>] :nfsd:nfsd_acceptable+0x0/0xd8
kernel:  [<ffffffff80052373>] tcp_cong_avoid+0xe/0x1d
kernel:  [<ffffffff80025003>] tcp_ack+0x1719/0x1871
kernel:  [<ffffffff88345c6a>] :sunrpc:sunrpc_cache_lookup+0x4b/0x127
kernel:  [<ffffffff8839b77d>] :nfsd:nfsd_vfs_write+0xf2/0x2e1
kernel:  [<ffffffff80095d48>] set_current_groups+0x159/0x164
kernel:  [<ffffffff8839bfa9>] :nfsd:nfsd_write+0x6d/0xd5
kernel:  [<ffffffff883a82e5>] :nfsd:nfsd4_proc_compound+0x149a/0x1655
kernel:  [<ffffffff80031648>] sock_common_recvmsg+0x2d/0x42
kernel:  [<ffffffff8002ffe2>] sock_recvmsg+0x101/0x120
kernel:  [<ffffffff8002b610>] local_bh_enable+0x90/0xa1
kernel:  [<ffffffff8009b446>] autoremove_wake_function+0x0/0x2e
kernel:  [<ffffffff80033dd3>] ip_queue_xmit+0x3fb/0x450
kernel:  [<ffffffff80044cf9>] try_to_wake_up+0x407/0x418
kernel:  [<ffffffff80021b63>] tcp_transmit_skb+0x64a/0x682
kernel:  [<ffffffff80209ce9>] kernel_recvmsg+0x3b/0x4d
kernel:  [<ffffffff8833fd96>] :sunrpc:svc_recvfrom+0x89/0xbf
kernel:  [<ffffffff88340000>] :sunrpc:svc_sock_enqueue+0x234/0x247
kernel:  [<ffffffff88341afa>] :sunrpc:svc_tcp_recvfrom+0x7a2/0x820
kernel:  [<ffffffff883425be>] :sunrpc:ip_map_match+0x19/0x30
kernel:  [<ffffffff800c35e8>] zone_statistics+0x3e/0x6d
kernel:  [<ffffffff88345c6a>] :sunrpc:sunrpc_cache_lookup+0x4b/0x127
kernel:  [<ffffffff883aa44f>] :nfsd:nfs4svc_decode_compoundargs+0x362/0xe54
kernel:  [<ffffffff883981c4>] :nfsd:nfsd_dispatch+0xd7/0x198
kernel:  [<ffffffff8833f514>] :sunrpc:svc_process+0x44d/0x70b
kernel:  [<ffffffff800625bf>] __down_read+0x12/0x92
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff883986fb>] :nfsd:nfsd+0x1ae/0x2db
kernel:  [<ffffffff8005bfb1>] child_rip+0xa/0x11
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff8005bfa7>] child_rip+0x0/0x11
kernel:
kernel: nfsd          D ffff810067eea4c4     0  3136      1          3137  3135
(L-TLB)
kernel:  ffff810061c657a0 0000000000000046 ffffffff8009b283 ffffffff80060f29
kernel:  0000000000000009 ffff81006c0cb080 ffff81007800c0c0 000004b18aaea968
kernel:  000000000000c907 ffff81006c0cb268 ffffffff00000001 ffff81000a37e400
kernel: Call Trace:
kernel:  [<ffffffff8009b283>] keventd_create_kthread+0x0/0x61
kernel:  [<ffffffff80060f29>] thread_return+0x0/0xeb
kernel:  [<ffffffff80087d09>] __activate_task+0x27/0x39
kernel:  [<ffffffff80044cf9>] try_to_wake_up+0x407/0x418
kernel:  [<ffffffff80061be8>] __mutex_lock_slowpath+0x55/0x90
kernel:  [<ffffffff80061c32>] .text.lock.mutex+0xf/0x14
kernel:  [<ffffffff800bdf72>] generic_file_writev+0x37/0xa2
kernel:  [<ffffffff80017880>] do_sync_write+0x0/0x104
kernel:  [<ffffffff800d5eae>] do_readv_writev+0x176/0x295
kernel:  [<ffffffff80017880>] do_sync_write+0x0/0x104
kernel:  [<ffffffff8839a8a0>] :nfsd:nfsd_acceptable+0x0/0xd8
kernel:  [<ffffffff80052373>] tcp_cong_avoid+0xe/0x1d
kernel:  [<ffffffff80025003>] tcp_ack+0x1719/0x1871
kernel:  [<ffffffff88345c6a>] :sunrpc:sunrpc_cache_lookup+0x4b/0x127
kernel:  [<ffffffff8839b77d>] :nfsd:nfsd_vfs_write+0xf2/0x2e1
kernel:  [<ffffffff80095d48>] set_current_groups+0x159/0x164
kernel:  [<ffffffff8839bfa9>] :nfsd:nfsd_write+0x6d/0xd5
kernel:  [<ffffffff883a82e5>] :nfsd:nfsd4_proc_compound+0x149a/0x1655
kernel:  [<ffffffff80031648>] sock_common_recvmsg+0x2d/0x42
kernel:  [<ffffffff8002ffe2>] sock_recvmsg+0x101/0x120
kernel:  [<ffffffff8002b610>] local_bh_enable+0x90/0xa1
kernel:  [<ffffffff8009b446>] autoremove_wake_function+0x0/0x2e
kernel:  [<ffffffff80033dd3>] ip_queue_xmit+0x3fb/0x450
kernel:  [<ffffffff80044cf9>] try_to_wake_up+0x407/0x418
kernel:  [<ffffffff88340f5d>] :sunrpc:svc_tcp_data_ready+0x40/0x6c
kernel:  [<ffffffff8003d03b>] lock_timer_base+0x1b/0x3c
kernel:  [<ffffffff8001c40f>] __mod_timer+0xb0/0xbe
kernel:  [<ffffffff80209ce9>] kernel_recvmsg+0x3b/0x4d
kernel:  [<ffffffff8833fd96>] :sunrpc:svc_recvfrom+0x89/0xbf
kernel:  [<ffffffff88340000>] :sunrpc:svc_sock_enqueue+0x234/0x247
kernel:  [<ffffffff88341afa>] :sunrpc:svc_tcp_recvfrom+0x7a2/0x820
kernel:  [<ffffffff883425be>] :sunrpc:ip_map_match+0x19/0x30
kernel:  [<ffffffff800c35e8>] zone_statistics+0x3e/0x6d
kernel:  [<ffffffff88345c6a>] :sunrpc:sunrpc_cache_lookup+0x4b/0x127
kernel:  [<ffffffff883aa44f>] :nfsd:nfs4svc_decode_compoundargs+0x362/0xe54
kernel:  [<ffffffff883981c4>] :nfsd:nfsd_dispatch+0xd7/0x198
kernel:  [<ffffffff8833f514>] :sunrpc:svc_process+0x44d/0x70b
kernel:  [<ffffffff800625bf>] __down_read+0x12/0x92
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff883986fb>] :nfsd:nfsd+0x1ae/0x2db
kernel:  [<ffffffff8005bfb1>] child_rip+0xa/0x11
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff8005bfa7>] child_rip+0x0/0x11
kernel:
kernel:
kernel: nfsd          D ffff810061c87948     0  3137      1          3138  3136
(L-TLB)
kernel:  ffff810061c877a0 0000000000000046 ffff81005c33d000 ffff81007d264b70
kernel:  0000000000000009 ffff8100787ba7a0 ffff81007f9f4100 000004b1ec65fb98
kernel:  0000000000005161 ffff8100787ba988 ffff810000000001 ffffffff800bdf22
kernel: Call Trace:
kernel:  [<ffffffff800bdf22>] __generic_file_write_nolock+0x8f/0xa8
kernel:  [<ffffffff80061be8>] __mutex_lock_slowpath+0x55/0x90
kernel:  [<ffffffff80061c32>] .text.lock.mutex+0xf/0x14
kernel:  [<ffffffff800bdf72>] generic_file_writev+0x37/0xa2
kernel:  [<ffffffff80017880>] do_sync_write+0x0/0x104
kernel:  [<ffffffff800d5eae>] do_readv_writev+0x176/0x295
kernel:  [<ffffffff80017880>] do_sync_write+0x0/0x104
kernel:  [<ffffffff8839a8a0>] :nfsd:nfsd_acceptable+0x0/0xd8
kernel:  [<ffffffff80052373>] tcp_cong_avoid+0xe/0x1d
kernel:  [<ffffffff80025003>] tcp_ack+0x1719/0x1871
kernel:  [<ffffffff88345c6a>] :sunrpc:sunrpc_cache_lookup+0x4b/0x127
kernel:  [<ffffffff8839b77d>] :nfsd:nfsd_vfs_write+0xf2/0x2e1
kernel:  [<ffffffff80095d48>] set_current_groups+0x159/0x164
kernel:  [<ffffffff8839bfa9>] :nfsd:nfsd_write+0x6d/0xd5
kernel:  [<ffffffff883a82e5>] :nfsd:nfsd4_proc_compound+0x149a/0x1655
kernel:  [<ffffffff80031648>] sock_common_recvmsg+0x2d/0x42
kernel:  [<ffffffff8002ffe2>] sock_recvmsg+0x101/0x120
kernel:  [<ffffffff8002b610>] local_bh_enable+0x90/0xa1
kernel:  [<ffffffff8009b446>] autoremove_wake_function+0x0/0x2e
kernel:  [<ffffffff80033dd3>] ip_queue_xmit+0x3fb/0x450
kernel:  [<ffffffff80044cf9>] try_to_wake_up+0x407/0x418
kernel:  [<ffffffff8003d03b>] lock_timer_base+0x1b/0x3c
kernel:  [<ffffffff8001c40f>] __mod_timer+0xb0/0xbe
kernel:  [<ffffffff80209ce9>] kernel_recvmsg+0x3b/0x4d
kernel:  [<ffffffff8833fd96>] :sunrpc:svc_recvfrom+0x89/0xbf
kernel:  [<ffffffff88340000>] :sunrpc:svc_sock_enqueue+0x234/0x247
kernel:  [<ffffffff88341afa>] :sunrpc:svc_tcp_recvfrom+0x7a2/0x820
kernel:  [<ffffffff883425be>] :sunrpc:ip_map_match+0x19/0x30
kernel:  [<ffffffff800c35e8>] zone_statistics+0x3e/0x6d
kernel:  [<ffffffff88345c6a>] :sunrpc:sunrpc_cache_lookup+0x4b/0x127
kernel:  [<ffffffff883aa44f>] :nfsd:nfs4svc_decode_compoundargs+0x362/0xe54
kernel:  [<ffffffff883981c4>] :nfsd:nfsd_dispatch+0xd7/0x198
kernel:  [<ffffffff8833f514>] :sunrpc:svc_process+0x44d/0x70b
kernel:  [<ffffffff800625bf>] __down_read+0x12/0x92
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff883986fb>] :nfsd:nfsd+0x1ae/0x2db
kernel:  [<ffffffff8005bfb1>] child_rip+0xa/0x11
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff8005bfa7>] child_rip+0x0/0x11
kernel:
kernel: nfsd          D ffff810067eea4c4     0  3138      1          3141  3137
(L-TLB)
kernel:  ffff810061cf17a0 0000000000000046 ffffffff8833dc30 ffffffff80060f29
kernel:  0000000000000009 ffff8100777bd100 ffff81006c0cb080 000004b09932bc07
kernel:  000000000000b68f ffff8100777bd2e8 ffffffff00000001 ffff81000a37e400
kernel: Call Trace:
kernel:  [<ffffffff8833dc30>] :sunrpc:rpc_async_schedule+0x0/0x5
kernel:  [<ffffffff80060f29>] thread_return+0x0/0xeb
kernel:  [<ffffffff80087d09>] __activate_task+0x27/0x39
kernel:  [<ffffffff80044cf9>] try_to_wake_up+0x407/0x418
kernel:  [<ffffffff80061be8>] __mutex_lock_slowpath+0x55/0x90
kernel:  [<ffffffff80061c32>] .text.lock.mutex+0xf/0x14
kernel:  [<ffffffff800bdf72>] generic_file_writev+0x37/0xa2
kernel:  [<ffffffff80017880>] do_sync_write+0x0/0x104
kernel:  [<ffffffff800d5eae>] do_readv_writev+0x176/0x295
kernel:  [<ffffffff80017880>] do_sync_write+0x0/0x104
kernel:  [<ffffffff8839a8a0>] :nfsd:nfsd_acceptable+0x0/0xd8
kernel:  [<ffffffff88345c6a>] :sunrpc:sunrpc_cache_lookup+0x4b/0x127
kernel:  [<ffffffff8839b77d>] :nfsd:nfsd_vfs_write+0xf2/0x2e1
kernel:  [<ffffffff80095d48>] set_current_groups+0x159/0x164
kernel:  [<ffffffff8839bfa9>] :nfsd:nfsd_write+0x6d/0xd5
kernel:  [<ffffffff883a82e5>] :nfsd:nfsd4_proc_compound+0x149a/0x1655
kernel:  [<ffffffff80031648>] sock_common_recvmsg+0x2d/0x42
kernel:  [<ffffffff801b1b56>] loopback_xmit+0xa7/0xad
kernel:  [<ffffffff8002ffe2>] sock_recvmsg+0x101/0x120
kernel:  [<ffffffff8009b446>] autoremove_wake_function+0x0/0x2e
kernel:  [<ffffffff80052373>] tcp_cong_avoid+0xe/0x1d
kernel:  [<ffffffff80025003>] tcp_ack+0x1719/0x1871
kernel:  [<ffffffff8001b75a>] tcp_rcv_established+0x86c/0x917
kernel:  [<ffffffff80209ce9>] kernel_recvmsg+0x3b/0x4d
kernel:  [<ffffffff8833fd96>] :sunrpc:svc_recvfrom+0x89/0xbf
kernel:  [<ffffffff88340000>] :sunrpc:svc_sock_enqueue+0x234/0x247
kernel:  [<ffffffff88341afa>] :sunrpc:svc_tcp_recvfrom+0x7a2/0x820
kernel:  [<ffffffff883425be>] :sunrpc:ip_map_match+0x19/0x30
kernel:  [<ffffffff800c35e8>] zone_statistics+0x3e/0x6d
kernel:  [<ffffffff88345c6a>] :sunrpc:sunrpc_cache_lookup+0x4b/0x127
kernel:  [<ffffffff883aa44f>] :nfsd:nfs4svc_decode_compoundargs+0x362/0xe54
kernel:  [<ffffffff883981c4>] :nfsd:nfsd_dispatch+0xd7/0x198
kernel:  [<ffffffff8833f514>] :sunrpc:svc_process+0x44d/0x70b
kernel:  [<ffffffff800625bf>] __down_read+0x12/0x92
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff883986fb>] :nfsd:nfsd+0x1ae/0x2db
kernel:  [<ffffffff8005bfb1>] child_rip+0xa/0x11
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff8839854d>] :nfsd:nfsd+0x0/0x2db
kernel:  [<ffffffff8005bfa7>] child_rip+0x0/0x11
kernel:
kernel: rpc.mountd    S 7fffffffffffffff     0  3141      1          3169  3138
(NOTLB)
kernel:  ffff810061dbba18 0000000000000082 0000000000000000 000000448021486d
kernel:  0000000000000009 ffff810076247860 ffff810050346040 000005b4aaa177b8
kernel:  00000000000073ff ffff810076247a48 ffff810000000000 ffff810061dbbac8
kernel: Call Trace:
kernel:  [<ffffffff8804e0f7>] :ext3:ext3_get_blocks_handle+0xc6/0x9f1
kernel:  [<ffffffff800617cd>] schedule_timeout+0x1e/0xad
kernel:  [<ffffffff800476f7>] add_wait_queue+0x24/0x34
kernel:  [<ffffffff8001334f>] tcp_poll+0x1c/0x12f
kernel:  [<ffffffff80011341>] do_select+0x3fe/0x462
kernel:  [<ffffffff8001e234>] __pollwait+0x0/0xe2
kernel:  [<ffffffff800884ac>] default_wake_function+0x0/0xe
last message repeated 4 times
kernel:  [<ffffffff8009b446>] autoremove_wake_function+0x0/0x2e
kernel:  [<ffffffff8009b283>] keventd_create_kthread+0x0/0x61
kernel:  [<ffffffff880322e8>] :jbd:__journal_file_buffer+0x13e/0x243
kernel:  [<ffffffff880322e8>] :jbd:__journal_file_buffer+0x13e/0x243
kernel:  [<ffffffff800dd334>] core_sys_select+0x1bc/0x265
kernel:  [<ffffffff88033e55>] :jbd:journal_dirty_metadata+0x1b5/0x1e3
kernel:  [<ffffffff880327ac>] :jbd:journal_stop+0x1f3/0x1ff
kernel:  [<ffffffff80016004>] sys_select+0xc1/0x17c
kernel:  [<ffffffff8005b28d>] tracesys+0xd5/0xe0
kernel:
kernel: nfsv4-svc     S 7fffffffffffffff     0  3169      1                3141
(L-TLB)
kernel:  ffff810061eb1e60 0000000000000046 0000000000000000 ffffffff80062c51
kernel:  0000000000000004 ffff810037f61100 ffff8100772880c0 00000079f975b25d
kernel:  0000000000000dd2 ffff810037f612e8 0000000000000000 ffff810037f61100
kernel: Call Trace:
kernel:  [<ffffffff80062c51>] __reacquire_kernel_lock+0x2c/0x45
kernel:  [<ffffffff80044cf9>] try_to_wake_up+0x407/0x418
kernel:  [<ffffffff800617cd>] schedule_timeout+0x1e/0xad
kernel:  [<ffffffff80062aa9>] _spin_lock_bh+0x9/0x14
kernel:  [<ffffffff800476f7>] add_wait_queue+0x24/0x34
kernel:  [<ffffffff88341e7e>] :sunrpc:svc_recv+0x306/0x4bc
kernel:  [<ffffffff800884ac>] default_wake_function+0x0/0xe
kernel:  [<ffffffff88469dc5>] :nfs:nfs_callback_svc+0x0/0x15a
kernel:  [<ffffffff88469dc5>] :nfs:nfs_callback_svc+0x0/0x15a
kernel:  [<ffffffff88469e85>] :nfs:nfs_callback_svc+0xc0/0x15a
kernel:  [<ffffffff8005bfb1>] child_rip+0xa/0x11
kernel:  [<ffffffff88469dc5>] :nfs:nfs_callback_svc+0x0/0x15a
kernel:  [<ffffffff88469dc5>] :nfs:nfs_callback_svc+0x0/0x15a
kernel:  [<ffffffff8005bfa7>] child_rip+0x0/0x11
kernel:
kernel:
kernel: iozone        S ffffffff80060765     0  4084   3645                    
(NOTLB)
kernel:  ffff810060da3d48 0000000000000082 ffff81006d235100 ffffffff800172d6
kernel:  000000000000000a ffff8100608c9820 ffff810061fc10c0 000004b025c545fa
kernel:  0000000000002153 ffff8100608c9a08 ffff810000000000 ffff81003dc5c7e0
kernel: Call Trace:
kernel:  [<ffffffff800172d6>] cache_grow+0x356/0x395
kernel:  [<ffffffff88451fb5>] :nfs:nfs_wait_bit_interruptible+0x0/0x28
kernel:  [<ffffffff88451fd7>] :nfs:nfs_wait_bit_interruptible+0x22/0x28
kernel:  [<ffffffff800619a4>] __wait_on_bit+0x40/0x6f
kernel:  [<ffffffff88451fb5>] :nfs:nfs_wait_bit_interruptible+0x0/0x28
kernel:  [<ffffffff80061a3f>] out_of_line_wait_on_bit+0x6c/0x78
kernel:  [<ffffffff8009b474>] wake_bit_function+0x0/0x23
kernel:  [<ffffffff88451f9b>] :nfs:nfs_wait_on_request+0x56/0x70
kernel:  [<ffffffff884556d6>] :nfs:nfs_wait_on_requests_locked+0x70/0xca
kernel:  [<ffffffff88455a82>] :nfs:nfs_flush_inode+0x5c/0x6f
kernel:  [<ffffffff8845744e>] :nfs:nfs_writepages+0xe6/0x13e
kernel:  [<ffffffff80059036>] do_writepages+0x20/0x2f
kernel:  [<ffffffff8004d593>] __filemap_fdatawrite_range+0x51/0x5b
kernel:  [<ffffffff8004de5b>] do_fsync+0x2f/0xa3
kernel:  [<ffffffff800d644e>] __do_fsync+0x23/0x36
kernel:  [<ffffffff8005b28d>] tracesys+0xd5/0xe0
kernel:
kernel: ffsb          S 7fffffffffffffff     0 16092   5048         16138      
(NOTLB)
kernel:  ffff81001d48fd48 0000000000000082 0000000000000001 0000000000000000
kernel:  0000000000000009 ffff8100765c8040 ffff810070e24820 000004850a30bcb8
kernel:  0000000000000c09 ffff8100765c8228 0000000000000001 0000000000000001
kernel: Call Trace:
kernel:  [<ffffffff8002dfa2>] __wake_up+0x38/0x4f
kernel:  [<ffffffff800617cd>] schedule_timeout+0x1e/0xad
kernel:  [<ffffffff800476f7>] add_wait_queue+0x24/0x34
kernel:  [<ffffffff8003d7a1>] do_futex+0x1da/0xbc7
kernel:  [<ffffffff800884ac>] default_wake_function+0x0/0xe
kernel:  [<ffffffff80060f29>] thread_return+0x0/0xeb
kernel:  [<ffffffff80039d8a>] tty_ldisc_deref+0x68/0x7b
kernel:  [<ffffffff8009e935>] sys_futex+0x101/0x123
kernel:  [<ffffffff8005b28d>] tracesys+0xd5/0xe0
kernel:
kernel: ffsb          S ffff810070e24820     0 16138   5048         16139 16092
(NOTLB)
kernel:  ffff81002503fd48 0000000000000082 0000000300000000 0000000000000096
kernel:  0000000000000007 ffff810070e24820 ffffffff802dcae0 000004cadc527330
kernel:  00000000000013ef ffff810070e24a08 ffff810000000000 ffffffff8008692a
kernel: Call Trace:
kernel:  [<ffffffff8008692a>] __wake_up_common+0x3e/0x68
kernel:  [<ffffffff800617cd>] schedule_timeout+0x1e/0xad
kernel:  [<ffffffff800476f7>] add_wait_queue+0x24/0x34
kernel:  [<ffffffff8003d7a1>] do_futex+0x1da/0xbc7
kernel:  [<ffffffff80087ccd>] enqueue_task+0x41/0x56
kernel:  [<ffffffff800884ac>] default_wake_function+0x0/0xe
kernel:  [<ffffffff8833dc30>] :sunrpc:rpc_async_schedule+0x0/0x5
kernel:  [<ffffffff80060f29>] thread_return+0x0/0xeb
kernel:  [<ffffffff8009e935>] sys_futex+0x101/0x123
kernel:  [<ffffffff8005b28d>] tracesys+0xd5/0xe0
kernel:
kernel: ffsb          D ffff81006df081c0     0 16139   5048         16140 16138
(NOTLB)
kernel:  ffff810072ca7c58 0000000000000082 ffff810072ca7bf8 ffff810072ca7d18
kernel:  0000000000000009 ffff810070e240c0 ffff81007f2367a0 000004b02705c7bf
kernel:  0000000000005e5b ffff810070e242a8 0000000000000001 000000003b376ed0
kernel: Call Trace:
kernel:  [<ffffffff884574a6>] :nfs:nfs_writepage+0x0/0x283
kernel:  [<ffffffff884561c7>] :nfs:nfs_wait_on_write_congestion+0x172/0x18f
kernel:  [<ffffffff8009b446>] autoremove_wake_function+0x0/0x2e
kernel:  [<ffffffff884573ce>] :nfs:nfs_writepages+0x66/0x13e
kernel:  [<ffffffff80059036>] do_writepages+0x20/0x2f
kernel:  [<ffffffff8004d593>] __filemap_fdatawrite_range+0x51/0x5b
kernel:  [<ffffffff800bdd7d>] filemap_write_and_wait+0x17/0x32
kernel:  [<ffffffff8844ede6>] :nfs:nfs_sync_mapping+0x25/0x45
kernel:  [<ffffffff8844f5c1>] :nfs:nfs_revalidate_mapping+0xa4/0x1b4
kernel:  [<ffffffff8844d632>] :nfs:nfs_file_read+0x45/0x97
kernel:  [<ffffffff8000c927>] do_sync_read+0xc7/0x104
kernel:  [<ffffffff8009b446>] autoremove_wake_function+0x0/0x2e
kernel:  [<ffffffff8000b270>] vfs_read+0xcb/0x171
kernel:  [<ffffffff800114f7>] sys_read+0x45/0x6e
kernel:  [<ffffffff8005b28d>] tracesys+0xd5/0xe0
kernel: fsstress      S 00000000fffffff6     0 16621   4109 16623              
(NOTLB)
kernel:  ffff810047995ee8 0000000000000086 ffff810047995e68 ffff810037ff19c0
kernel:  0000000000000009 ffff81004ad7a080 ffff8100772880c0 00000495c048acc0
kernel:  0000000000002b39 ffff81004ad7a268 ffff810000000000 ffffffff800fa2ce
kernel: Call Trace:
kernel:  [<ffffffff800fa2ce>] proc_flush_task+0x4e/0x1f6
kernel:  [<ffffffff8008be59>] sched_exit+0x9f/0xaa
kernel:  [<ffffffff80027dea>] do_wait+0x936/0xa0a
kernel:  [<ffffffff800884ac>] default_wake_function+0x0/0xe
kernel:  [<ffffffff8005b28d>] tracesys+0xd5/0xe0
kernel:
kernel: fsstress      S ffffffff8833d37e     0 16623  16621         16627      
(NOTLB)
kernel:  ffff81001fadda58 0000000000000082 0000000000000246 ffffffff8001bf35
kernel:  0000000000000007 ffff8100606d00c0 ffff81007f9f4100 000004b475ad244a
kernel:  0000000000003525 ffff8100606d02a8 ffff810000000001 ffff81007f616670
kernel: Call Trace:
kernel:  [<ffffffff8001bf35>] cfq_set_request+0x246/0x389
kernel:  [<ffffffff8833d37e>] :sunrpc:rpc_wait_bit_interruptible+0x0/0x28
kernel:  [<ffffffff8833d3a0>] :sunrpc:rpc_wait_bit_interruptible+0x22/0x28
kernel:  [<ffffffff800619a4>] __wait_on_bit+0x40/0x6f
kernel:  [<ffffffff8833d37e>] :sunrpc:rpc_wait_bit_interruptible+0x0/0x28
kernel:  [<ffffffff80061a3f>] out_of_line_wait_on_bit+0x6c/0x78
kernel:  [<ffffffff8009b474>] wake_bit_function+0x0/0x23
kernel:  [<ffffffff8833daf2>] :sunrpc:__rpc_execute+0x110/0x24e
kernel:  [<ffffffff88338596>] :sunrpc:rpc_call_sync+0x7e/0xa8
kernel:  [<ffffffff8845c36e>] :nfs:_nfs4_proc_access+0xb4/0xe9
kernel:  [<ffffffff8845d1a2>] :nfs:nfs4_proc_access+0x21/0x47
kernel:  [<ffffffff8844b27f>] :nfs:nfs_permission+0x136/0x1cb
kernel:  [<ffffffff8000d41d>] permission+0x81/0xc8
kernel:  [<ffffffff8000965b>] __link_path_walk+0x173/0xf42
kernel:  [<ffffffff8833d37e>] :sunrpc:rpc_wait_bit_interruptible+0x0/0x28
kernel:  [<ffffffff80060f29>] thread_return+0x0/0xeb
ernel:  [<ffffffff8000e5b9>] link_path_walk+0x5c/0xe5
kernel:  [<ffffffff8000c7e8>] do_path_lookup+0x270/0x2e8
kernel:  [<ffffffff800232cf>] __path_lookup_intent_open+0x56/0x97
kernel:  [<ffffffff8001a7ae>] open_namei+0xb9/0x6fd
kernel:  [<ffffffff80047dee>] wait_on_page_writeback_range+0xc3/0x10e
kernel:  [<ffffffff800270ce>] do_filp_open+0x1c/0x38
kernel:  [<ffffffff80019523>] do_sys_open+0x44/0xbe
kernel:  [<ffffffff8005b28d>] tracesys+0xd5/0xe0
 kernel:
kernel: fsstress      S ffffffff8833d37e     0 16627  16621         16628 16623
(NOTLB)
kernel:  ffff8100577abb28 0000000000000082 ffff81007fb54030 0000000000000000
kernel:  0000000000000007 ffff81005e71d860 ffff81007f9f4100 000004b476b0c9c0
kernel:  0000000000001e45 ffff81005e71da48 0000000000000001 0000000100000002
kernel: Call Trace:
kernel:  [<ffffffff8833d37e>] :sunrpc:rpc_wait_bit_interruptible+0x0/0x28
kernel:  [<ffffffff8833d3a0>] :sunrpc:rpc_wait_bit_interruptible+0x22/0x28
kernel:  [<ffffffff800619a4>] __wait_on_bit+0x40/0x6f
kernel:  [<ffffffff8833d37e>] :sunrpc:rpc_wait_bit_interruptible+0x0/0x28
kernel:  [<ffffffff80061a3f>] out_of_line_wait_on_bit+0x6c/0x78
kernel:  [<ffffffff8009b474>] wake_bit_function+0x0/0x23
kernel:  [<ffffffff8833daf2>] :sunrpc:__rpc_execute+0x110/0x24e
kernel:  [<ffffffff88338596>] :sunrpc:rpc_call_sync+0x7e/0xa8
kernel:  [<ffffffff8833e9aa>] :sunrpc:rpcauth_lookup_credcache+0x12e/0x24c
kernel:  [<ffffffff8845c36e>] :nfs:_nfs4_proc_access+0xb4/0xe9
kernel:  [<ffffffff8844ae5c>] :nfs:nfs_access_free_entry+0x15/0x1f
kernel:  [<ffffffff8845d1a2>] :nfs:nfs4_proc_access+0x21/0x47
kernel:  [<ffffffff8844b27f>] :nfs:nfs_permission+0x136/0x1cb
kernel:  [<ffffffff8000d41d>] permission+0x81/0xc8
kernel:
kernel:
kernel: fsx-linux     S ffffffff8833d37e     0 17658   4583                    
(NOTLB)
kernel:  ffff81006f777bf8 0000000000000082 ffff81006f777bc8 ffffffff80051deb
kernel:  000000000000000a ffff810076df0080 ffffffff802dcae0 000004b1ec65836b
kernel:  0000000000005a3c ffff810076df0268 0000000000000000 ffffffff80062aa9
kernel: Call Trace:
kernel:  [<ffffffff80051deb>] tcp_init_tso_segs+0x2b/0x3a
kernel:  [<ffffffff80062aa9>] _spin_lock_bh+0x9/0x14
kernel:  [<ffffffff8833d37e>] :sunrpc:rpc_wait_bit_interruptible+0x0/0x28
kernel:  [<ffffffff8833d3a0>] :sunrpc:rpc_wait_bit_interruptible+0x22/0x28
kernel:  [<ffffffff800619a4>] __wait_on_bit+0x40/0x6f
kernel:  [<ffffffff8833d37e>] :sunrpc:rpc_wait_bit_interruptible+0x0/0x28
kernel:  [<ffffffff80061a3f>] out_of_line_wait_on_bit+0x6c/0x78
kernel:  [<ffffffff8009b474>] wake_bit_function+0x0/0x23
kernel:  [<ffffffff8833daf2>] :sunrpc:__rpc_execute+0x110/0x24e
kernel:  [<ffffffff88338596>] :sunrpc:rpc_call_sync+0x7e/0xa8
kernel:  [<ffffffff8845d353>] :nfs:nfs4_proc_getattr+0x83/0xa2
kernel:  [<ffffffff8844e89f>] :nfs:__nfs_revalidate_inode+0x14a/0x294
kernel:  [<ffffffff88451fa8>] :nfs:nfs_wait_on_request+0x63/0x70
kernel:  [<ffffffff88455715>] :nfs:nfs_wait_on_requests_locked+0xaf/0xca
kernel:  [<ffffffff88456871>] :nfs:nfs_sync_inode_wait+0x83/0x1db
kernel:  [<ffffffff8844f4fa>] :nfs:nfs_getattr+0x80/0xa3
kernel:  [<ffffffff80021c2d>] vfs_fstat+0x29/0x3a
kernel:  [<ffffffff8002a6b9>] sys_newfstat+0x11/0x29
kernel:  [<ffffffff8005b229>] tracesys+0x71/0xe0
kernel:  [<ffffffff8005b28d>] tracesys+0xd5/0xe0
kernel:




Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 2 Steve Dickson 2007-12-08 07:50:52 EST
This looks to me that it could be memory contention problem which
is a k
Comment 3 RHEL Product and Program Management 2008-07-25 13:06:54 EDT
This request was evaluated by Red Hat Product Management for
inclusion, but this component is not scheduled to be updated in
the current Red Hat Enterprise Linux release. If you would like
this request to be reviewed for the next minor release, ask your
support representative to set the next rhel-x.y flag to "?".
Comment 4 Ludek Smid 2008-07-25 17:53:50 EDT
Unfortunately the previous automated notification about the
non-inclusion of this request in Red Hat Enterprise Linux 5.3 used
the wrong text template. It should have read: this request has been
reviewed by Product Management and is not planned for inclusion
in the current minor release of Red Hat Enterprise Linux.

If you would like this request to be reviewed for the next minor
release, ask your support representative to set the next rhel-x.y
flag to "?" or raise an exception.
Comment 5 RHEL Product and Program Management 2008-12-09 16:16:47 EST
Development Management has reviewed and declined this request.  You may appeal
this decision by reopening this request.

Note You need to log in before you can comment on or make changes to this bug.