Bug 560688

Summary: System locks up and must be booted
Product: Red Hat Enterprise Linux 5 Reporter: Albert Flügel <albert.fluegel>
Component: kernelAssignee: Steve Dickson <steved>
Status: CLOSED DUPLICATE QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: medium Docs Contact:
Priority: low    
Version: 5.4CC: a.drevet, dev_nll, lwoodman
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-10-21 18:55:00 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Albert Flügel 2010-02-01 16:03:33 UTC 
Description of problem:
about once a week the system locks up and the messages below are scrolling over the console repeatedly every few seconds. It happens on two machines, who are
mainly samba servers with > 400 clients each and > 300 NFS (auto-) mounts each

Version-Release number of selected component (if applicable):
kernel 2.6.18-128.1.14.el5

How reproducible:
Configure as samba server for many clients with homes over NFS (not sure, if this is really necessary but other bug reports with similar logs might have to do with the filesystems and probably locking, too). Samba setup is not complicated, mainly [homes] is used, no printers.

Steps to Reproduce:
1. run many clients / user "drives" on this machin
2. wait 2 weeks or so
  
Actual results:
system locks up and must be reset (sysrq - break to reset is not working either)

Expected results:
system continues to run normally and serve SMB requests

Additional info:
The logs appearing every few seconds on the console:
BUG: soft lockup - CPU#1 stuck for 10s! [kswapd0:520]


CPU 1:


Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd


Pid: 520, comm: kswapd0 Tainted: G      2.6.18-164.6.1.el5 #1


RIP: 0010:[<ffffffff80064bcc>]  [<ffffffff80064bcc>] .text.lock.spinlock+0x2/0x30


RSP: 0018:ffff81081f8fbd38  EFLAGS: 00000282


RAX: ffff81081f8fbd50 RBX: 0000000000000000 RCX: 00000000003939f2


RDX: 0000000000000000 RSI: 00000000000000d0 RDI: ffffffff88424f50


RBP: 0000000000000000 R08: 00000000000c9ebc R09: 00000000007e3359


R10: 000000000000005e R11: 0000000000000002 R12: 0000000000000000


R13: 0000000100000002 R14: 0000000000000000 R15: 0000000000000000


FS:  00002b4c2108fdd0(0000) GS:ffff81010e7993c0(0000) knlGS:00000000f7e0aac0


CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b


CR2: 000000000efeb138 CR3: 0000000000201000 CR4: 00000000000006a0





Call Trace:


 [<ffffffff883eb33a>] :nfs:nfs_access_cache_shrinker+0x2d/0x1da


 [<ffffffff8003f33a>] shrink_slab+0x60/0x153


 [<ffffffff80057da5>] kswapd+0x343/0x46c


 [<ffffffff8009fc08>] autoremove_wake_function+0x0/0x2e


 [<ffffffff80057a62>] kswapd+0x0/0x46c


 [<ffffffff8009f9f0>] keventd_create_kthread+0x0/0xc4


 [<ffffffff8003297c>] kthread+0xfe/0x132


 [<ffffffff8005dfb1>] child_rip+0xa/0x11


 [<ffffffff8009f9f0>] keventd_create_kthread+0x0/0xc4


 [<ffffffff8003287e>] kthread+0x0/0x132


 [<ffffffff8005dfa7>] child_rip+0x0/0x11





BUG: soft lockup - CPU#3 stuck for 10s! [smbd:16110]


CPU 3:


Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd


Pid: 16110, comm: smbd Tainted: G      2.6.18-164.6.1.el5 #1


RIP: 0010:[<ffffffff80064bcc>]  [<ffffffff80064bcc>] .text.lock.spinlock+0x2/0x30


RSP: 0018:ffff8100657379b0  EFLAGS: 00000282


RAX: ffff8100657379c8 RBX: 0000000000000000 RCX: 00000000007446cf


RDX: 0000000000000000 RSI: 00000000000200d2 RDI: ffffffff88424f50


RBP: 0000000000001680 R08: ffff810000033600 R09: ffff81019dbe6770


R10: 000000000000005e R11: ffffffff883edf6d R12: 0000000000000246


R13: ffff8104144061c0 R14: 0000000000000000 R15: ffff81041e5aaa40


FS:  00002b342d854570(0000) GS:ffff81042e1b76c0(0000) knlGS:00000000f7e0aac0


CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b


CR2: 00002b19416622f1 CR3: 000000005186f000 CR4: 00000000000006a0





Call Trace:


 [<ffffffff883eb33a>] :nfs:nfs_access_cache_shrinker+0x2d/0x1da


 [<ffffffff8003f33a>] shrink_slab+0x60/0x153


 [<ffffffff800ca905>] try_to_free_pages+0x1da/0x2d7


 [<ffffffff800cb003>] zone_statistics+0x3e/0x6d


 [<ffffffff8000f40d>] __alloc_pages+0x1cb/0x2ce


 [<ffffffff883f6927>] :nfs:nfs_update_request+0x188/0x271


 [<ffffffff800c4b08>] grab_cache_page_write_begin+0x4a/0x89


 [<ffffffff883ecf81>] :nfs:nfs_write_begin+0x41/0xf8


 [<ffffffff8000fcc1>] generic_file_buffered_write+0x14b/0x675


 [<ffffffff80030e7f>] release_sock+0x13/0xaa


 [<ffffffff80016513>] __generic_file_aio_write_nolock+0x369/0x3b6


 [<ffffffff80045c75>] do_sock_read+0xcf/0x110


 [<ffffffff8002156f>] generic_file_aio_write+0x65/0xc1


 [<ffffffff883ed54f>] :nfs:nfs_file_write+0xab/0x124


 [<ffffffff80018123>] do_sync_write+0xc7/0x104


 [<ffffffff8002cafe>] mntput_no_expire+0x19/0x89


 [<ffffffff8009fc08>] autoremove_wake_function+0x0/0x2e


 [<ffffffff80062486>] __sched_text_start+0xf6/0xbd6


 [<ffffffff8001691b>] vfs_write+0xce/0x174


 [<ffffffff80043e0e>] sys_pwrite64+0x50/0x70


 [<ffffffff8005d229>] tracesys+0x71/0xe0


 [<ffffffff8005d28d>] tracesys+0xd5/0xe0





BUG: soft lockup - CPU#0 stuck for 10s! [smbd:24399]


CPU 0:


Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd


Pid: 24399, comm: smbd Tainted: G      2.6.18-164.6.1.el5 #1


RIP: 0010:[<ffffffff80064bcf>]  [<ffffffff80064bcf>] .text.lock.spinlock+0x5/0x30


RSP: 0018:ffff810399f67d20  EFLAGS: 00000282


RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000


RDX: ffff8107dbf1d670 RSI: ffff8107dbf1d670 RDI: ffffffff88424f50


RBP: 000000000e9b1527 R08: ffff81010e776c00 R09: ffff81010e777000


R10: ffff81042e1f8038 R11: 000000d000000000 R12: ffff810600000000


R13: 00000000000fb000 R14: 00000000000f923d R15: 0000000000004937


FS:  00002b342d854570(0000) GS:ffffffff803c1000(0000) knlGS:00000000f7e0aac0


CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b


CR2: 0000000040480fe0 CR3: 0000000240ba7000 CR4: 00000000000006a0





Call Trace:


 [<ffffffff883eaf8e>] :nfs:nfs_access_add_cache+0x13d/0x16d


 [<ffffffff883eb289>] :nfs:nfs_permission+0x147/0x1cb


 [<ffffffff8000ea30>] link_path_walk+0xa6/0xb2


 [<ffffffff8000d902>] permission+0x81/0xc8


 [<ffffffff80012469>] may_open+0x65/0x22f


 [<ffffffff8001afed>] open_namei+0x2c4/0x6d5


 [<ffffffff800e8cd5>] __posix_lock_file_conf+0x396/0x3e0


 [<ffffffff80027308>] do_filp_open+0x1c/0x38


 [<ffffffff80019cdb>] do_sys_open+0x44/0xbe


 [<ffffffff8005d28d>] tracesys+0xd5/0xe0





BUG: soft lockup - CPU#1 stuck for 10s! [kswapd0:520]


CPU 1:


Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd


Pid: 520, comm: kswapd0 Tainted: G      2.6.18-164.6.1.el5 #1


RIP: 0010:[<ffffffff80064bca>]  [<ffffffff80064bca>] .text.lock.spinlock+0x0/0x30


RSP: 0018:ffff81081f8fbd38  EFLAGS: 00000282


RAX: ffff81081f8fbd50 RBX: 0000000000000000 RCX: 00000000003939f2


RDX: 0000000000000000 RSI: 00000000000000d0 RDI: ffffffff88424f50


RBP: 0000000000000000 R08: 00000000000c9ebc R09: 00000000007e3359


R10: 000000000000005e R11: 0000000000000002 R12: 0000000000000000


R13: 0000000100000002 R14: 0000000000000000 R15: 0000000000000000


FS:  00002b4c2108fdd0(0000) GS:ffff81010e7993c0(0000) knlGS:00000000f7e0aac0


CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b


CR2: 000000000efeb138 CR3: 0000000000201000 CR4: 00000000000006a0





Call Trace:


 [<ffffffff883eb33a>] :nfs:nfs_access_cache_shrinker+0x2d/0x1da


 [<ffffffff8003f33a>] shrink_slab+0x60/0x153


 [<ffffffff80057da5>] kswapd+0x343/0x46c


 [<ffffffff8009fc08>] autoremove_wake_function+0x0/0x2e


 [<ffffffff80057a62>] kswapd+0x0/0x46c


 [<ffffffff8009f9f0>] keventd_create_kthread+0x0/0xc4


 [<ffffffff8003297c>] kthread+0xfe/0x132


 [<ffffffff8005dfb1>] child_rip+0xa/0x11


 [<ffffffff8009f9f0>] keventd_create_kthread+0x0/0xc4


 [<ffffffff8003287e>] kthread+0x0/0x132


 [<ffffffff8005dfa7>] child_rip+0x0/0x11





BUG: soft lockup - CPU#3 stuck for 10s! [smbd:16110]


CPU 3:


Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd


Pid: 16110, comm: smbd Tainted: G      2.6.18-164.6.1.el5 #1


RIP: 0010:[<ffffffff80064bca>]  [<ffffffff80064bca>] .text.lock.spinlock+0x0/0x30


RSP: 0018:ffff8100657379b0  EFLAGS: 00000282


RAX: ffff8100657379c8 RBX: 0000000000000000 RCX: 00000000007446cf


RDX: 0000000000000000 RSI: 00000000000200d2 RDI: ffffffff88424f50


RBP: 0000000000001680 R08: ffff810000033600 R09: ffff81019dbe6770


R10: 000000000000005e R11: ffffffff883edf6d R12: 0000000000000246


R13: ffff8104144061c0 R14: 0000000000000000 R15: ffff81041e5aaa40


FS:  00002b342d854570(0000) GS:ffff81042e1b76c0(0000) knlGS:00000000f7e0aac0


CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b


CR2: 00002b19416622f1 CR3: 000000005186f000 CR4: 00000000000006a0





Call Trace:


 [<ffffffff883eb33a>] :nfs:nfs_access_cache_shrinker+0x2d/0x1da


 [<ffffffff8003f33a>] shrink_slab+0x60/0x153


 [<ffffffff800ca905>] try_to_free_pages+0x1da/0x2d7


 [<ffffffff800cb003>] zone_statistics+0x3e/0x6d


 [<ffffffff8000f40d>] __alloc_pages+0x1cb/0x2ce


 [<ffffffff883f6927>] :nfs:nfs_update_request+0x188/0x271


 [<ffffffff800c4b08>] grab_cache_page_write_begin+0x4a/0x89


 [<ffffffff883ecf81>] :nfs:nfs_write_begin+0x41/0xf8


 [<ffffffff8000fcc1>] generic_file_buffered_write+0x14b/0x675


 [<ffffffff80030e7f>] release_sock+0x13/0xaa


 [<ffffffff80016513>] __generic_file_aio_write_nolock+0x369/0x3b6


 [<ffffffff80045c75>] do_sock_read+0xcf/0x110


 [<ffffffff8002156f>] generic_file_aio_write+0x65/0xc1


 [<ffffffff883ed54f>] :nfs:nfs_file_write+0xab/0x124


 [<ffffffff80018123>] do_sync_write+0xc7/0x104


 [<ffffffff8002cafe>] mntput_no_expire+0x19/0x89


 [<ffffffff8009fc08>] autoremove_wake_function+0x0/0x2e


 [<ffffffff80062486>] __sched_text_start+0xf6/0xbd6


 [<ffffffff8001691b>] vfs_write+0xce/0x174


 [<ffffffff80043e0e>] sys_pwrite64+0x50/0x70


 [<ffffffff8005d229>] tracesys+0x71/0xe0


 [<ffffffff8005d28d>] tracesys+0xd5/0xe0





BUG: soft lockup - CPU#0 stuck for 10s! [smbd:24399]


CPU 0:


Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd


Pid: 24399, comm: smbd Tainted: G      2.6.18-164.6.1.el5 #1


RIP: 0010:[<ffffffff80064bcc>]  [<ffffffff80064bcc>] .text.lock.spinlock+0x2/0x30


RSP: 0018:ffff810399f67d20  EFLAGS: 00000282


RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000


RDX: ffff8107dbf1d670 RSI: ffff8107dbf1d670 RDI: ffffffff88424f50


RBP: 000000000e9b1527 R08: ffff81010e776c00 R09: ffff81010e777000


R10: ffff81042e1f8038 R11: 000000d000000000 R12: ffff810600000000


R13: 00000000000fb000 R14: 00000000000f923d R15: 0000000000004937


FS:  00002b342d854570(0000) GS:ffffffff803c1000(0000) knlGS:00000000f7e0aac0


CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b


CR2: 0000000040480fe0 CR3: 0000000240ba7000 CR4: 00000000000006a0





Call Trace:


 [<ffffffff883eaf8e>] :nfs:nfs_access_add_cache+0x13d/0x16d


 [<ffffffff883eb289>] :nfs:nfs_permission+0x147/0x1cb


 [<ffffffff8000ea30>] link_path_walk+0xa6/0xb2


 [<ffffffff8000d902>] permission+0x81/0xc8


 [<ffffffff80012469>] may_open+0x65/0x22f


 [<ffffffff8001afed>] open_namei+0x2c4/0x6d5


 [<ffffffff800e8cd5>] __posix_lock_file_conf+0x396/0x3e0


 [<ffffffff80027308>] do_filp_open+0x1c/0x38


 [<ffffffff80019cdb>] do_sys_open+0x44/0xbe


 [<ffffffff8005d28d>] tracesys+0xd5/0xe0





BUG: soft lockup - CPU#1 stuck for 10s! [kswapd0:520]


CPU 1:


Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd


Pid: 520, comm: kswapd0 Tainted: G      2.6.18-164.6.1.el5 #1


RIP: 0010:[<ffffffff80064bcc>]  [<ffffffff80064bcc>] .text.lock.spinlock+0x2/0x30


RSP: 0018:ffff81081f8fbd38  EFLAGS: 00000282


RAX: ffff81081f8fbd50 RBX: 0000000000000000 RCX: 00000000003939f2


RDX: 0000000000000000 RSI: 00000000000000d0 RDI: ffffffff88424f50


RBP: 0000000000000000 R08: 00000000000c9ebc R09: 00000000007e3359


R10: 000000000000005e R11: 0000000000000002 R12: 0000000000000000


R13: 0000000100000002 R14: 0000000000000000 R15: 0000000000000000


FS:  00002b4c2108fdd0(0000) GS:ffff81010e7993c0(0000) knlGS:00000000f7e0aac0


CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b


CR2: 000000000efeb138 CR3: 0000000000201000 CR4: 00000000000006a0





Call Trace:


 [<ffffffff883eb33a>] :nfs:nfs_access_cache_shrinker+0x2d/0x1da


 [<ffffffff8003f33a>] shrink_slab+0x60/0x153


 [<ffffffff80057da5>] kswapd+0x343/0x46c


 [<ffffffff8009fc08>] autoremove_wake_function+0x0/0x2e


 [<ffffffff80057a62>] kswapd+0x0/0x46c


 [<ffffffff8009f9f0>] keventd_create_kthread+0x0/0xc4


 [<ffffffff8003297c>] kthread+0xfe/0x132


 [<ffffffff8005dfb1>] child_rip+0xa/0x11


 [<ffffffff8009f9f0>] keventd_create_kthread+0x0/0xc4


 [<ffffffff8003287e>] kthread+0x0/0x132


 [<ffffffff8005dfa7>] child_rip+0x0/0x11





BUG: soft lockup - CPU#3 stuck for 10s! [smbd:16110]


CPU 3:


Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd


Pid: 16110, comm: smbd Tainted: G      2.6.18-164.6.1.el5 #1


RIP: 0010:[<ffffffff80064bcc>]  [<ffffffff80064bcc>] .text.lock.spinlock+0x2/0x30


RSP: 0018:ffff8100657379b0  EFLAGS: 00000282


RAX: ffff8100657379c8 RBX: 0000000000000000 RCX: 00000000007446cf


RDX: 0000000000000000 RSI: 00000000000200d2 RDI: ffffffff88424f50


RBP: 0000000000001680 R08: ffff810000033600 R09: ffff81019dbe6770


R10: 000000000000005e R11: ffffffff883edf6d R12: 0000000000000246


R13: ffff8104144061c0 R14: 0000000000000000 R15: ffff81041e5aaa40


FS:  00002b342d854570(0000) GS:ffff81042e1b76c0(0000) knlGS:00000000f7e0aac0


CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b


CR2: 00002b19416622f1 CR3: 000000005186f000 CR4: 00000000000006a0





Call Trace:


 [<ffffffff883eb33a>] :nfs:nfs_access_cache_shrinker+0x2d/0x1da


 [<ffffffff8003f33a>] shrink_slab+0x60/0x153


 [<ffffffff800ca905>] try_to_free_pages+0x1da/0x2d7


 [<ffffffff800cb003>] zone_statistics+0x3e/0x6d


 [<ffffffff8000f40d>] __alloc_pages+0x1cb/0x2ce


 [<ffffffff883f6927>] :nfs:nfs_update_request+0x188/0x271


 [<ffffffff800c4b08>] grab_cache_page_write_begin+0x4a/0x89


 [<ffffffff883ecf81>] :nfs:nfs_write_begin+0x41/0xf8


 [<ffffffff8000fcc1>] generic_file_buffered_write+0x14b/0x675


 [<ffffffff80030e7f>] release_sock+0x13/0xaa


 [<ffffffff80016513>] __generic_file_aio_write_nolock+0x369/0x3b6


 [<ffffffff80045c75>] do_sock_read+0xcf/0x110


 [<ffffffff8002156f>] generic_file_aio_write+0x65/0xc1


 [<ffffffff883ed54f>] :nfs:nfs_file_write+0xab/0x124


 [<ffffffff80018123>] do_sync_write+0xc7/0x104


 [<ffffffff8002cafe>] mntput_no_expire+0x19/0x89


 [<ffffffff8009fc08>] autoremove_wake_function+0x0/0x2e


 [<ffffffff80062486>] __sched_text_start+0xf6/0xbd6


 [<ffffffff8001691b>] vfs_write+0xce/0x174


 [<ffffffff80043e0e>] sys_pwrite64+0x50/0x70


 [<ffffffff8005d229>] tracesys+0x71/0xe0


 [<ffffffff8005d28d>] tracesys+0xd5/0xe0





BUG: soft lockup - CPU#0 stuck for 10s! [smbd:24399]


CPU 0:


Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd


Pid: 24399, comm: smbd Tainted: G      2.6.18-164.6.1.el5 #1


RIP: 0010:[<ffffffff80064bcc>]  [<ffffffff80064bcc>] .text.lock.spinlock+0x2/0x30


RSP: 0018:ffff810399f67d20  EFLAGS: 00000282


RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000


RDX: ffff8107dbf1d670 RSI: ffff8107dbf1d670 RDI: ffffffff88424f50


RBP: 000000000e9b1527 R08: ffff81010e776c00 R09: ffff81010e777000


R10: ffff81042e1f8038 R11: 000000d000000000 R12: ffff810600000000


R13: 00000000000fb000 R14: 00000000000f923d R15: 0000000000004937


FS:  00002b342d854570(0000) GS:ffffffff803c1000(0000) knlGS:00000000f7e0aac0


CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b


CR2: 0000000040480fe0 CR3: 0000000240ba7000 CR4: 00000000000006a0





Call Trace:


 [<ffffffff883eaf8e>] :nfs:nfs_access_add_cache+0x13d/0x16d


 [<ffffffff883eb289>] :nfs:nfs_permission+0x147/0x1cb


 [<ffffffff8000ea30>] link_path_walk+0xa6/0xb2


 [<ffffffff8000d902>] permission+0x81/0xc8


 [<ffffffff80012469>] may_open+0x65/0x22f


 [<ffffffff8001afed>] open_namei+0x2c4/0x6d5


 [<ffffffff800e8cd5>] __posix_lock_file_conf+0x396/0x3e0


 [<ffffffff80027308>] do_filp_open+0x1c/0x38


 [<ffffffff80019cdb>] do_sys_open+0x44/0xbe


 [<ffffffff8005d28d>] tracesys+0xd5/0xe0





BUG: soft lockup - CPU#1 stuck for 10s! [kswapd0:520]


CPU 1:


Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd


Pid: 520, comm: kswapd0 Tainted: G      2.6.18-164.6.1.el5 #1


RIP: 0010:[<ffffffff80064bcc>]  [<ffffffff80064bcc>] .text.lock.spinlock+0x2/0x30


RSP: 0018:ffff81081f8fbd38  EFLAGS: 00000282


RAX: ffff81081f8fbd50 RBX: 0000000000000000 RCX: 00000000003939f2


RDX: 0000000000000000 RSI: 00000000000000d0 RDI: ffffffff88424f50


RBP: 0000000000000000 R08: 00000000000c9ebc R09: 00000000007e3359


R10: 000000000000005e R11: 0000000000000002 R12: 0000000000000000


R13: 0000000100000002 R14: 0000000000000000 R15: 0000000000000000


FS:  00002b4c2108fdd0(0000) GS:ffff81010e7993c0(0000) knlGS:00000000f7e0aac0


CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b


CR2: 000000000efeb138 CR3: 0000000000201000 CR4: 00000000000006a0





Call Trace:


 [<ffffffff883eb33a>] :nfs:nfs_access_cache_shrinker+0x2d/0x1da


 [<ffffffff8003f33a>] shrink_slab+0x60/0x153


 [<ffffffff80057da5>] kswapd+0x343/0x46c


 [<ffffffff8009fc08>] autoremove_wake_function+0x0/0x2e


 [<ffffffff80057a62>] kswapd+0x0/0x46c


 [<ffffffff8009f9f0>] keventd_create_kthread+0x0/0xc4


 [<ffffffff8003297c>] kthread+0xfe/0x132


 [<ffffffff8005dfb1>] child_rip+0xa/0x11


 [<ffffffff8009f9f0>] keventd_create_kthread+0x0/0xc4


 [<ffffffff8003287e>] kthread+0x0/0x132


 [<ffffffff8005dfa7>] child_rip+0x0/0x11





BUG: soft lockup - CPU#3 stuck for 10s! [smbd:16110]


CPU 3:


Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd


Pid: 16110, comm: smbd Tainted: G      2.6.18-164.6.1.el5 #1


RIP: 0010:[<ffffffff80064bca>]  [<ffffffff80064bca>] .text.lock.spinlock+0x0/0x30


RSP: 0018:ffff8100657379b0  EFLAGS: 00000282


RAX: ffff8100657379c8 RBX: 0000000000000000 RCX: 00000000007446cf


RDX: 0000000000000000 RSI: 00000000000200d2 RDI: ffffffff88424f50


RBP: 0000000000001680 R08: ffff810000033600 R09: ffff81019dbe6770


R10: 000000000000005e R11: ffffffff883edf6d R12: 0000000000000246


R13: ffff8104144061c0 R14: 0000000000000000 R15: ffff81041e5aaa40


FS:  00002b342d854570(0000) GS:ffff81042e1b76c0(0000) knlGS:00000000f7e0aac0


CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b


CR2: 00002b19416622f1 CR3: 000000005186f000 CR4: 00000000000006a0





Call Trace:


 [<ffffffff883eb33a>] :nfs:nfs_access_cache_shrinker+0x2d/0x1da


 [<ffffffff8003f33a>] shrink_slab+0x60/0x153


 [<ffffffff800ca905>] try_to_free_pages+0x1da/0x2d7


 [<ffffffff800cb003>] zone_statistics+0x3e/0x6d


 [<ffffffff8000f40d>] __alloc_pages+0x1cb/0x2ce


 [<ffffffff883f6927>] :nfs:nfs_update_request+0x188/0x271


 [<ffffffff800c4b08>] grab_cache_page_write_begin+0x4a/0x89


 [<ffffffff883ecf81>] :nfs:nfs_write_begin+0x41/0xf8


 [<ffffffff8000fcc1>] generic_file_buffered_write+0x14b/0x675


 [<ffffffff80030e7f>] release_sock+0x13/0xaa


 [<ffffffff80016513>] __generic_file_aio_write_nolock+0x369/0x3b6


 [<ffffffff80045c75>] do_sock_read+0xcf/0x110


 [<ffffffff8002156f>] generic_file_aio_write+0x65/0xc1


 [<ffffffff883ed54f>] :nfs:nfs_file_write+0xab/0x124


 [<ffffffff80018123>] do_sync_write+0xc7/0x104


 [<ffffffff8002cafe>] mntput_no_expire+0x19/0x89


 [<ffffffff8009fc08>] autoremove_wake_function+0x0/0x2e


 [<ffffffff80062486>] __sched_text_start+0xf6/0xbd6


 [<ffffffff8001691b>] vfs_write+0xce/0x174


 [<ffffffff80043e0e>] sys_pwrite64+0x50/0x70


 [<ffffffff8005d229>] tracesys+0x71/0xe0


 [<ffffffff8005d28d>] tracesys+0xd5/0xe0





BUG: soft lockup - CPU#0 stuck for 10s! [smbd:24399]


CPU 0:


Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd


Pid: 24399, comm: smbd Tainted: G      2.6.18-164.6.1.el5 #1


RIP: 0010:[<ffffffff80064bca>]  [<ffffffff80064bca>] .text.lock.spinlock+0x0/0x30


RSP: 0018:ffff810399f67d20  EFLAGS: 00000282


RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000


RDX: ffff8107dbf1d670 RSI: ffff8107dbf1d670 RDI: ffffffff88424f50


RBP: 000000000e9b1527 R08: ffff81010e776c00 R09: ffff81010e777000


R10: ffff81042e1f8038 R11: 000000d000000000 R12: ffff810600000000


R13: 00000000000fb000 R14: 00000000000f923d R15: 0000000000004937


FS:  00002b342d854570(0000) GS:ffffffff803c1000(0000) knlGS:00000000f7e0aac0


CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b


CR2: 0000000040480fe0 CR3: 0000000240ba7000 CR4: 00000000000006a0





Call Trace:


 [<ffffffff883eaf8e>] :nfs:nfs_access_add_cache+0x13d/0x16d


 [<ffffffff883eb289>] :nfs:nfs_permission+0x147/0x1cb


 [<ffffffff8000ea30>] link_path_walk+0xa6/0xb2


 [<ffffffff8000d902>] permission+0x81/0xc8


 [<ffffffff80012469>] may_open+0x65/0x22f


 [<ffffffff8001afed>] open_namei+0x2c4/0x6d5


 [<ffffffff800e8cd5>] __posix_lock_file_conf+0x396/0x3e0


 [<ffffffff80027308>] do_filp_open+0x1c/0x38


 [<ffffffff80019cdb>] do_sys_open+0x44/0xbe


 [<ffffffff8005d28d>] tracesys+0xd5/0xe0





BUG: soft lockup - CPU#1 stuck for 10s! [kswapd0:520]


CPU 1:


Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd


Pid: 520, comm: kswapd0 Tainted: G      2.6.18-164.6.1.el5 #1


RIP: 0010:[<ffffffff80064bcc>]  [<ffffffff80064bcc>] .text.lock.spinlock+0x2/0x30


RSP: 0018:ffff81081f8fbd38  EFLAGS: 00000282


RAX: ffff81081f8fbd50 RBX: 0000000000000000 RCX: 00000000003939f2


RDX: 0000000000000000 RSI: 00000000000000d0 RDI: ffffffff88424f50


RBP: 0000000000000000 R08: 00000000000c9ebc R09: 00000000007e3359


R10: 000000000000005e R11: 0000000000000002 R12: 0000000000000000


R13: 0000000100000002 R14: 0000000000000000 R15: 0000000000000000


FS:  00002b4c2108fdd0(0000) GS:ffff81010e7993c0(0000) knlGS:00000000f7e0aac0


CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b


CR2: 000000000efeb138 CR3: 0000000000201000 CR4: 00000000000006a0





Call Trace:


 [<ffffffff883eb33a>] :nfs:nfs_access_cache_shrinker+0x2d/0x1da


 [<ffffffff8003f33a>] shrink_slab+0x60/0x153


 [<ffffffff80057da5>] kswapd+0x343/0x46c


 [<ffffffff8009fc08>] autoremove_wake_function+0x0/0x2e


 [<ffffffff80057a62>] kswapd+0x0/0x46c


 [<ffffffff8009f9f0>] keventd_create_kthread+0x0/0xc4


 [<ffffffff8003297c>] kthread+0xfe/0x132


 [<ffffffff8005dfb1>] child_rip+0xa/0x11


 [<ffffffff8009f9f0>] keventd_create_kthread+0x0/0xc4


 [<ffffffff8003287e>] kthread+0x0/0x132


 [<ffffffff8005dfa7>] child_rip+0x0/0x11





BUG: soft lockup - CPU#3 stuck for 10s! [smbd:16110]


CPU 3:


Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd


Pid: 16110, comm: smbd Tainted: G      2.6.18-164.6.1.el5 #1


RIP: 0010:[<ffffffff80064bcf>]  [<ffffffff80064bcf>] .text.lock.spinlock+0x5/0x30


RSP: 0018:ffff8100657379b0  EFLAGS: 00000282


RAX: ffff8100657379c8 RBX: 0000000000000000 RCX: 00000000007446cf


RDX: 0000000000000000 RSI: 00000000000200d2 RDI: ffffffff88424f50


RBP: 0000000000001680 R08: ffff810000033600 R09: ffff81019dbe6770


R10: 000000000000005e R11: ffffffff883edf6d R12: 0000000000000246


R13: ffff8104144061c0 R14: 0000000000000000 R15: ffff81041e5aaa40


FS:  00002b342d854570(0000) GS:ffff81042e1b76c0(0000) knlGS:00000000f7e0aac0


CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b


CR2: 00002b19416622f1 CR3: 000000005186f000 CR4: 00000000000006a0





Call Trace:


 [<ffffffff883eb33a>] :nfs:nfs_access_cache_shrinker+0x2d/0x1da


 [<ffffffff8003f33a>] shrink_slab+0x60/0x153


 [<ffffffff800ca905>] try_to_free_pages+0x1da/0x2d7


 [<ffffffff800cb003>] zone_statistics+0x3e/0x6d


 [<ffffffff8000f40d>] __alloc_pages+0x1cb/0x2ce


 [<ffffffff883f6927>] :nfs:nfs_update_request+0x188/0x271


 [<ffffffff800c4b08>] grab_cache_page_write_begin+0x4a/0x89


 [<ffffffff883ecf81>] :nfs:nfs_write_begin+0x41/0xf8


 [<ffffffff8000fcc1>] generic_file_buffered_write+0x14b/0x675


 [<ffffffff80030e7f>] release_sock+0x13/0xaa


 [<ffffffff80016513>] __generic_file_aio_write_nolock+0x369/0x3b6


 [<ffffffff80045c75>] do_sock_read+0xcf/0x110


 [<ffffffff8002156f>] generic_file_aio_write+0x65/0xc1


 [<ffffffff883ed54f>] :nfs:nfs_file_write+0xab/0x124


 [<ffffffff80018123>] do_sync_write+0xc7/0x104


 [<ffffffff8002cafe>] mntput_no_expire+0x19/0x89


 [<ffffffff8009fc08>] autoremove_wake_function+0x0/0x2e


 [<ffffffff80062486>] __sched_text_start+0xf6/0xbd6


 [<ffffffff8001691b>] vfs_write+0xce/0x174


 [<ffffffff80043e0e>] sys_pwrite64+0x50/0x70


 [<ffffffff8005d229>] tracesys+0x71/0xe0


 [<ffffffff8005d28d>] tracesys+0xd5/0xe0





...
Comment 1 Larry Woodman 2010-02-19 21:21:52 UTC 
Problem appears to be in the nfs_access_cache_shrinker(), stuck on the 
spin_lock(&nfs_access_lru_lock).

Larry
Comment 2 Rob Moser 2010-02-24 17:27:52 UTC 
It seems like this could be related to https://bugzilla.redhat.com/show_bug.cgi?id=525898 - similar symptoms, at least.  I too am having something like this happen.  Essentially, we got kswapd0 eating all of the system CPU and locking up the system while doing a large amount of reads from NFS.  I posted additional details on the thread for that other bug, but I won't cross-post them all here in case you decide they're not the same after all.
Comment 3 Steve Dickson 2010-10-21 18:55:00 UTC 

*** This bug has been marked as a duplicate of bug 585935 ***