Description of problem: about once a week the system locks up and the messages below are scrolling over the console repeatedly every few seconds. It happens on two machines, who are mainly samba servers with > 400 clients each and > 300 NFS (auto-) mounts each Version-Release number of selected component (if applicable): kernel 2.6.18-128.1.14.el5 How reproducible: Configure as samba server for many clients with homes over NFS (not sure, if this is really necessary but other bug reports with similar logs might have to do with the filesystems and probably locking, too). Samba setup is not complicated, mainly [homes] is used, no printers. Steps to Reproduce: 1. run many clients / user "drives" on this machin 2. wait 2 weeks or so Actual results: system locks up and must be reset (sysrq - break to reset is not working either) Expected results: system continues to run normally and serve SMB requests Additional info: The logs appearing every few seconds on the console: BUG: soft lockup - CPU#1 stuck for 10s! [kswapd0:520] CPU 1: Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Pid: 520, comm: kswapd0 Tainted: G 2.6.18-164.6.1.el5 #1 RIP: 0010:[<ffffffff80064bcc>] [<ffffffff80064bcc>] .text.lock.spinlock+0x2/0x30 RSP: 0018:ffff81081f8fbd38 EFLAGS: 00000282 RAX: ffff81081f8fbd50 RBX: 0000000000000000 RCX: 00000000003939f2 RDX: 0000000000000000 RSI: 00000000000000d0 RDI: ffffffff88424f50 RBP: 0000000000000000 R08: 00000000000c9ebc R09: 00000000007e3359 R10: 000000000000005e R11: 0000000000000002 R12: 0000000000000000 R13: 0000000100000002 R14: 0000000000000000 R15: 0000000000000000 FS: 00002b4c2108fdd0(0000) GS:ffff81010e7993c0(0000) knlGS:00000000f7e0aac0 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 000000000efeb138 CR3: 0000000000201000 CR4: 00000000000006a0 Call Trace: [<ffffffff883eb33a>] :nfs:nfs_access_cache_shrinker+0x2d/0x1da [<ffffffff8003f33a>] shrink_slab+0x60/0x153 [<ffffffff80057da5>] kswapd+0x343/0x46c [<ffffffff8009fc08>] autoremove_wake_function+0x0/0x2e [<ffffffff80057a62>] kswapd+0x0/0x46c [<ffffffff8009f9f0>] keventd_create_kthread+0x0/0xc4 [<ffffffff8003297c>] kthread+0xfe/0x132 [<ffffffff8005dfb1>] child_rip+0xa/0x11 [<ffffffff8009f9f0>] keventd_create_kthread+0x0/0xc4 [<ffffffff8003287e>] kthread+0x0/0x132 [<ffffffff8005dfa7>] child_rip+0x0/0x11 BUG: soft lockup - CPU#3 stuck for 10s! [smbd:16110] CPU 3: Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Pid: 16110, comm: smbd Tainted: G 2.6.18-164.6.1.el5 #1 RIP: 0010:[<ffffffff80064bcc>] [<ffffffff80064bcc>] .text.lock.spinlock+0x2/0x30 RSP: 0018:ffff8100657379b0 EFLAGS: 00000282 RAX: ffff8100657379c8 RBX: 0000000000000000 RCX: 00000000007446cf RDX: 0000000000000000 RSI: 00000000000200d2 RDI: ffffffff88424f50 RBP: 0000000000001680 R08: ffff810000033600 R09: ffff81019dbe6770 R10: 000000000000005e R11: ffffffff883edf6d R12: 0000000000000246 R13: ffff8104144061c0 R14: 0000000000000000 R15: ffff81041e5aaa40 FS: 00002b342d854570(0000) GS:ffff81042e1b76c0(0000) knlGS:00000000f7e0aac0 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00002b19416622f1 CR3: 000000005186f000 CR4: 00000000000006a0 Call Trace: [<ffffffff883eb33a>] :nfs:nfs_access_cache_shrinker+0x2d/0x1da [<ffffffff8003f33a>] shrink_slab+0x60/0x153 [<ffffffff800ca905>] try_to_free_pages+0x1da/0x2d7 [<ffffffff800cb003>] zone_statistics+0x3e/0x6d [<ffffffff8000f40d>] __alloc_pages+0x1cb/0x2ce [<ffffffff883f6927>] :nfs:nfs_update_request+0x188/0x271 [<ffffffff800c4b08>] grab_cache_page_write_begin+0x4a/0x89 [<ffffffff883ecf81>] :nfs:nfs_write_begin+0x41/0xf8 [<ffffffff8000fcc1>] generic_file_buffered_write+0x14b/0x675 [<ffffffff80030e7f>] release_sock+0x13/0xaa [<ffffffff80016513>] __generic_file_aio_write_nolock+0x369/0x3b6 [<ffffffff80045c75>] do_sock_read+0xcf/0x110 [<ffffffff8002156f>] generic_file_aio_write+0x65/0xc1 [<ffffffff883ed54f>] :nfs:nfs_file_write+0xab/0x124 [<ffffffff80018123>] do_sync_write+0xc7/0x104 [<ffffffff8002cafe>] mntput_no_expire+0x19/0x89 [<ffffffff8009fc08>] autoremove_wake_function+0x0/0x2e [<ffffffff80062486>] __sched_text_start+0xf6/0xbd6 [<ffffffff8001691b>] vfs_write+0xce/0x174 [<ffffffff80043e0e>] sys_pwrite64+0x50/0x70 [<ffffffff8005d229>] tracesys+0x71/0xe0 [<ffffffff8005d28d>] tracesys+0xd5/0xe0 BUG: soft lockup - CPU#0 stuck for 10s! [smbd:24399] CPU 0: Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Pid: 24399, comm: smbd Tainted: G 2.6.18-164.6.1.el5 #1 RIP: 0010:[<ffffffff80064bcf>] [<ffffffff80064bcf>] .text.lock.spinlock+0x5/0x30 RSP: 0018:ffff810399f67d20 EFLAGS: 00000282 RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 RDX: ffff8107dbf1d670 RSI: ffff8107dbf1d670 RDI: ffffffff88424f50 RBP: 000000000e9b1527 R08: ffff81010e776c00 R09: ffff81010e777000 R10: ffff81042e1f8038 R11: 000000d000000000 R12: ffff810600000000 R13: 00000000000fb000 R14: 00000000000f923d R15: 0000000000004937 FS: 00002b342d854570(0000) GS:ffffffff803c1000(0000) knlGS:00000000f7e0aac0 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000040480fe0 CR3: 0000000240ba7000 CR4: 00000000000006a0 Call Trace: [<ffffffff883eaf8e>] :nfs:nfs_access_add_cache+0x13d/0x16d [<ffffffff883eb289>] :nfs:nfs_permission+0x147/0x1cb [<ffffffff8000ea30>] link_path_walk+0xa6/0xb2 [<ffffffff8000d902>] permission+0x81/0xc8 [<ffffffff80012469>] may_open+0x65/0x22f [<ffffffff8001afed>] open_namei+0x2c4/0x6d5 [<ffffffff800e8cd5>] __posix_lock_file_conf+0x396/0x3e0 [<ffffffff80027308>] do_filp_open+0x1c/0x38 [<ffffffff80019cdb>] do_sys_open+0x44/0xbe [<ffffffff8005d28d>] tracesys+0xd5/0xe0 BUG: soft lockup - CPU#1 stuck for 10s! [kswapd0:520] CPU 1: Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Pid: 520, comm: kswapd0 Tainted: G 2.6.18-164.6.1.el5 #1 RIP: 0010:[<ffffffff80064bca>] [<ffffffff80064bca>] .text.lock.spinlock+0x0/0x30 RSP: 0018:ffff81081f8fbd38 EFLAGS: 00000282 RAX: ffff81081f8fbd50 RBX: 0000000000000000 RCX: 00000000003939f2 RDX: 0000000000000000 RSI: 00000000000000d0 RDI: ffffffff88424f50 RBP: 0000000000000000 R08: 00000000000c9ebc R09: 00000000007e3359 R10: 000000000000005e R11: 0000000000000002 R12: 0000000000000000 R13: 0000000100000002 R14: 0000000000000000 R15: 0000000000000000 FS: 00002b4c2108fdd0(0000) GS:ffff81010e7993c0(0000) knlGS:00000000f7e0aac0 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 000000000efeb138 CR3: 0000000000201000 CR4: 00000000000006a0 Call Trace: [<ffffffff883eb33a>] :nfs:nfs_access_cache_shrinker+0x2d/0x1da [<ffffffff8003f33a>] shrink_slab+0x60/0x153 [<ffffffff80057da5>] kswapd+0x343/0x46c [<ffffffff8009fc08>] autoremove_wake_function+0x0/0x2e [<ffffffff80057a62>] kswapd+0x0/0x46c [<ffffffff8009f9f0>] keventd_create_kthread+0x0/0xc4 [<ffffffff8003297c>] kthread+0xfe/0x132 [<ffffffff8005dfb1>] child_rip+0xa/0x11 [<ffffffff8009f9f0>] keventd_create_kthread+0x0/0xc4 [<ffffffff8003287e>] kthread+0x0/0x132 [<ffffffff8005dfa7>] child_rip+0x0/0x11 BUG: soft lockup - CPU#3 stuck for 10s! [smbd:16110] CPU 3: Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Pid: 16110, comm: smbd Tainted: G 2.6.18-164.6.1.el5 #1 RIP: 0010:[<ffffffff80064bca>] [<ffffffff80064bca>] .text.lock.spinlock+0x0/0x30 RSP: 0018:ffff8100657379b0 EFLAGS: 00000282 RAX: ffff8100657379c8 RBX: 0000000000000000 RCX: 00000000007446cf RDX: 0000000000000000 RSI: 00000000000200d2 RDI: ffffffff88424f50 RBP: 0000000000001680 R08: ffff810000033600 R09: ffff81019dbe6770 R10: 000000000000005e R11: ffffffff883edf6d R12: 0000000000000246 R13: ffff8104144061c0 R14: 0000000000000000 R15: ffff81041e5aaa40 FS: 00002b342d854570(0000) GS:ffff81042e1b76c0(0000) knlGS:00000000f7e0aac0 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00002b19416622f1 CR3: 000000005186f000 CR4: 00000000000006a0 Call Trace: [<ffffffff883eb33a>] :nfs:nfs_access_cache_shrinker+0x2d/0x1da [<ffffffff8003f33a>] shrink_slab+0x60/0x153 [<ffffffff800ca905>] try_to_free_pages+0x1da/0x2d7 [<ffffffff800cb003>] zone_statistics+0x3e/0x6d [<ffffffff8000f40d>] __alloc_pages+0x1cb/0x2ce [<ffffffff883f6927>] :nfs:nfs_update_request+0x188/0x271 [<ffffffff800c4b08>] grab_cache_page_write_begin+0x4a/0x89 [<ffffffff883ecf81>] :nfs:nfs_write_begin+0x41/0xf8 [<ffffffff8000fcc1>] generic_file_buffered_write+0x14b/0x675 [<ffffffff80030e7f>] release_sock+0x13/0xaa [<ffffffff80016513>] __generic_file_aio_write_nolock+0x369/0x3b6 [<ffffffff80045c75>] do_sock_read+0xcf/0x110 [<ffffffff8002156f>] generic_file_aio_write+0x65/0xc1 [<ffffffff883ed54f>] :nfs:nfs_file_write+0xab/0x124 [<ffffffff80018123>] do_sync_write+0xc7/0x104 [<ffffffff8002cafe>] mntput_no_expire+0x19/0x89 [<ffffffff8009fc08>] autoremove_wake_function+0x0/0x2e [<ffffffff80062486>] __sched_text_start+0xf6/0xbd6 [<ffffffff8001691b>] vfs_write+0xce/0x174 [<ffffffff80043e0e>] sys_pwrite64+0x50/0x70 [<ffffffff8005d229>] tracesys+0x71/0xe0 [<ffffffff8005d28d>] tracesys+0xd5/0xe0 BUG: soft lockup - CPU#0 stuck for 10s! [smbd:24399] CPU 0: Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Pid: 24399, comm: smbd Tainted: G 2.6.18-164.6.1.el5 #1 RIP: 0010:[<ffffffff80064bcc>] [<ffffffff80064bcc>] .text.lock.spinlock+0x2/0x30 RSP: 0018:ffff810399f67d20 EFLAGS: 00000282 RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 RDX: ffff8107dbf1d670 RSI: ffff8107dbf1d670 RDI: ffffffff88424f50 RBP: 000000000e9b1527 R08: ffff81010e776c00 R09: ffff81010e777000 R10: ffff81042e1f8038 R11: 000000d000000000 R12: ffff810600000000 R13: 00000000000fb000 R14: 00000000000f923d R15: 0000000000004937 FS: 00002b342d854570(0000) GS:ffffffff803c1000(0000) knlGS:00000000f7e0aac0 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000040480fe0 CR3: 0000000240ba7000 CR4: 00000000000006a0 Call Trace: [<ffffffff883eaf8e>] :nfs:nfs_access_add_cache+0x13d/0x16d [<ffffffff883eb289>] :nfs:nfs_permission+0x147/0x1cb [<ffffffff8000ea30>] link_path_walk+0xa6/0xb2 [<ffffffff8000d902>] permission+0x81/0xc8 [<ffffffff80012469>] may_open+0x65/0x22f [<ffffffff8001afed>] open_namei+0x2c4/0x6d5 [<ffffffff800e8cd5>] __posix_lock_file_conf+0x396/0x3e0 [<ffffffff80027308>] do_filp_open+0x1c/0x38 [<ffffffff80019cdb>] do_sys_open+0x44/0xbe [<ffffffff8005d28d>] tracesys+0xd5/0xe0 BUG: soft lockup - CPU#1 stuck for 10s! [kswapd0:520] CPU 1: Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Pid: 520, comm: kswapd0 Tainted: G 2.6.18-164.6.1.el5 #1 RIP: 0010:[<ffffffff80064bcc>] [<ffffffff80064bcc>] .text.lock.spinlock+0x2/0x30 RSP: 0018:ffff81081f8fbd38 EFLAGS: 00000282 RAX: ffff81081f8fbd50 RBX: 0000000000000000 RCX: 00000000003939f2 RDX: 0000000000000000 RSI: 00000000000000d0 RDI: ffffffff88424f50 RBP: 0000000000000000 R08: 00000000000c9ebc R09: 00000000007e3359 R10: 000000000000005e R11: 0000000000000002 R12: 0000000000000000 R13: 0000000100000002 R14: 0000000000000000 R15: 0000000000000000 FS: 00002b4c2108fdd0(0000) GS:ffff81010e7993c0(0000) knlGS:00000000f7e0aac0 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 000000000efeb138 CR3: 0000000000201000 CR4: 00000000000006a0 Call Trace: [<ffffffff883eb33a>] :nfs:nfs_access_cache_shrinker+0x2d/0x1da [<ffffffff8003f33a>] shrink_slab+0x60/0x153 [<ffffffff80057da5>] kswapd+0x343/0x46c [<ffffffff8009fc08>] autoremove_wake_function+0x0/0x2e [<ffffffff80057a62>] kswapd+0x0/0x46c [<ffffffff8009f9f0>] keventd_create_kthread+0x0/0xc4 [<ffffffff8003297c>] kthread+0xfe/0x132 [<ffffffff8005dfb1>] child_rip+0xa/0x11 [<ffffffff8009f9f0>] keventd_create_kthread+0x0/0xc4 [<ffffffff8003287e>] kthread+0x0/0x132 [<ffffffff8005dfa7>] child_rip+0x0/0x11 BUG: soft lockup - CPU#3 stuck for 10s! [smbd:16110] CPU 3: Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Pid: 16110, comm: smbd Tainted: G 2.6.18-164.6.1.el5 #1 RIP: 0010:[<ffffffff80064bcc>] [<ffffffff80064bcc>] .text.lock.spinlock+0x2/0x30 RSP: 0018:ffff8100657379b0 EFLAGS: 00000282 RAX: ffff8100657379c8 RBX: 0000000000000000 RCX: 00000000007446cf RDX: 0000000000000000 RSI: 00000000000200d2 RDI: ffffffff88424f50 RBP: 0000000000001680 R08: ffff810000033600 R09: ffff81019dbe6770 R10: 000000000000005e R11: ffffffff883edf6d R12: 0000000000000246 R13: ffff8104144061c0 R14: 0000000000000000 R15: ffff81041e5aaa40 FS: 00002b342d854570(0000) GS:ffff81042e1b76c0(0000) knlGS:00000000f7e0aac0 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00002b19416622f1 CR3: 000000005186f000 CR4: 00000000000006a0 Call Trace: [<ffffffff883eb33a>] :nfs:nfs_access_cache_shrinker+0x2d/0x1da [<ffffffff8003f33a>] shrink_slab+0x60/0x153 [<ffffffff800ca905>] try_to_free_pages+0x1da/0x2d7 [<ffffffff800cb003>] zone_statistics+0x3e/0x6d [<ffffffff8000f40d>] __alloc_pages+0x1cb/0x2ce [<ffffffff883f6927>] :nfs:nfs_update_request+0x188/0x271 [<ffffffff800c4b08>] grab_cache_page_write_begin+0x4a/0x89 [<ffffffff883ecf81>] :nfs:nfs_write_begin+0x41/0xf8 [<ffffffff8000fcc1>] generic_file_buffered_write+0x14b/0x675 [<ffffffff80030e7f>] release_sock+0x13/0xaa [<ffffffff80016513>] __generic_file_aio_write_nolock+0x369/0x3b6 [<ffffffff80045c75>] do_sock_read+0xcf/0x110 [<ffffffff8002156f>] generic_file_aio_write+0x65/0xc1 [<ffffffff883ed54f>] :nfs:nfs_file_write+0xab/0x124 [<ffffffff80018123>] do_sync_write+0xc7/0x104 [<ffffffff8002cafe>] mntput_no_expire+0x19/0x89 [<ffffffff8009fc08>] autoremove_wake_function+0x0/0x2e [<ffffffff80062486>] __sched_text_start+0xf6/0xbd6 [<ffffffff8001691b>] vfs_write+0xce/0x174 [<ffffffff80043e0e>] sys_pwrite64+0x50/0x70 [<ffffffff8005d229>] tracesys+0x71/0xe0 [<ffffffff8005d28d>] tracesys+0xd5/0xe0 BUG: soft lockup - CPU#0 stuck for 10s! [smbd:24399] CPU 0: Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Pid: 24399, comm: smbd Tainted: G 2.6.18-164.6.1.el5 #1 RIP: 0010:[<ffffffff80064bcc>] [<ffffffff80064bcc>] .text.lock.spinlock+0x2/0x30 RSP: 0018:ffff810399f67d20 EFLAGS: 00000282 RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 RDX: ffff8107dbf1d670 RSI: ffff8107dbf1d670 RDI: ffffffff88424f50 RBP: 000000000e9b1527 R08: ffff81010e776c00 R09: ffff81010e777000 R10: ffff81042e1f8038 R11: 000000d000000000 R12: ffff810600000000 R13: 00000000000fb000 R14: 00000000000f923d R15: 0000000000004937 FS: 00002b342d854570(0000) GS:ffffffff803c1000(0000) knlGS:00000000f7e0aac0 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000040480fe0 CR3: 0000000240ba7000 CR4: 00000000000006a0 Call Trace: [<ffffffff883eaf8e>] :nfs:nfs_access_add_cache+0x13d/0x16d [<ffffffff883eb289>] :nfs:nfs_permission+0x147/0x1cb [<ffffffff8000ea30>] link_path_walk+0xa6/0xb2 [<ffffffff8000d902>] permission+0x81/0xc8 [<ffffffff80012469>] may_open+0x65/0x22f [<ffffffff8001afed>] open_namei+0x2c4/0x6d5 [<ffffffff800e8cd5>] __posix_lock_file_conf+0x396/0x3e0 [<ffffffff80027308>] do_filp_open+0x1c/0x38 [<ffffffff80019cdb>] do_sys_open+0x44/0xbe [<ffffffff8005d28d>] tracesys+0xd5/0xe0 BUG: soft lockup - CPU#1 stuck for 10s! [kswapd0:520] CPU 1: Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Pid: 520, comm: kswapd0 Tainted: G 2.6.18-164.6.1.el5 #1 RIP: 0010:[<ffffffff80064bcc>] [<ffffffff80064bcc>] .text.lock.spinlock+0x2/0x30 RSP: 0018:ffff81081f8fbd38 EFLAGS: 00000282 RAX: ffff81081f8fbd50 RBX: 0000000000000000 RCX: 00000000003939f2 RDX: 0000000000000000 RSI: 00000000000000d0 RDI: ffffffff88424f50 RBP: 0000000000000000 R08: 00000000000c9ebc R09: 00000000007e3359 R10: 000000000000005e R11: 0000000000000002 R12: 0000000000000000 R13: 0000000100000002 R14: 0000000000000000 R15: 0000000000000000 FS: 00002b4c2108fdd0(0000) GS:ffff81010e7993c0(0000) knlGS:00000000f7e0aac0 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 000000000efeb138 CR3: 0000000000201000 CR4: 00000000000006a0 Call Trace: [<ffffffff883eb33a>] :nfs:nfs_access_cache_shrinker+0x2d/0x1da [<ffffffff8003f33a>] shrink_slab+0x60/0x153 [<ffffffff80057da5>] kswapd+0x343/0x46c [<ffffffff8009fc08>] autoremove_wake_function+0x0/0x2e [<ffffffff80057a62>] kswapd+0x0/0x46c [<ffffffff8009f9f0>] keventd_create_kthread+0x0/0xc4 [<ffffffff8003297c>] kthread+0xfe/0x132 [<ffffffff8005dfb1>] child_rip+0xa/0x11 [<ffffffff8009f9f0>] keventd_create_kthread+0x0/0xc4 [<ffffffff8003287e>] kthread+0x0/0x132 [<ffffffff8005dfa7>] child_rip+0x0/0x11 BUG: soft lockup - CPU#3 stuck for 10s! [smbd:16110] CPU 3: Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Pid: 16110, comm: smbd Tainted: G 2.6.18-164.6.1.el5 #1 RIP: 0010:[<ffffffff80064bca>] [<ffffffff80064bca>] .text.lock.spinlock+0x0/0x30 RSP: 0018:ffff8100657379b0 EFLAGS: 00000282 RAX: ffff8100657379c8 RBX: 0000000000000000 RCX: 00000000007446cf RDX: 0000000000000000 RSI: 00000000000200d2 RDI: ffffffff88424f50 RBP: 0000000000001680 R08: ffff810000033600 R09: ffff81019dbe6770 R10: 000000000000005e R11: ffffffff883edf6d R12: 0000000000000246 R13: ffff8104144061c0 R14: 0000000000000000 R15: ffff81041e5aaa40 FS: 00002b342d854570(0000) GS:ffff81042e1b76c0(0000) knlGS:00000000f7e0aac0 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00002b19416622f1 CR3: 000000005186f000 CR4: 00000000000006a0 Call Trace: [<ffffffff883eb33a>] :nfs:nfs_access_cache_shrinker+0x2d/0x1da [<ffffffff8003f33a>] shrink_slab+0x60/0x153 [<ffffffff800ca905>] try_to_free_pages+0x1da/0x2d7 [<ffffffff800cb003>] zone_statistics+0x3e/0x6d [<ffffffff8000f40d>] __alloc_pages+0x1cb/0x2ce [<ffffffff883f6927>] :nfs:nfs_update_request+0x188/0x271 [<ffffffff800c4b08>] grab_cache_page_write_begin+0x4a/0x89 [<ffffffff883ecf81>] :nfs:nfs_write_begin+0x41/0xf8 [<ffffffff8000fcc1>] generic_file_buffered_write+0x14b/0x675 [<ffffffff80030e7f>] release_sock+0x13/0xaa [<ffffffff80016513>] __generic_file_aio_write_nolock+0x369/0x3b6 [<ffffffff80045c75>] do_sock_read+0xcf/0x110 [<ffffffff8002156f>] generic_file_aio_write+0x65/0xc1 [<ffffffff883ed54f>] :nfs:nfs_file_write+0xab/0x124 [<ffffffff80018123>] do_sync_write+0xc7/0x104 [<ffffffff8002cafe>] mntput_no_expire+0x19/0x89 [<ffffffff8009fc08>] autoremove_wake_function+0x0/0x2e [<ffffffff80062486>] __sched_text_start+0xf6/0xbd6 [<ffffffff8001691b>] vfs_write+0xce/0x174 [<ffffffff80043e0e>] sys_pwrite64+0x50/0x70 [<ffffffff8005d229>] tracesys+0x71/0xe0 [<ffffffff8005d28d>] tracesys+0xd5/0xe0 BUG: soft lockup - CPU#0 stuck for 10s! [smbd:24399] CPU 0: Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Pid: 24399, comm: smbd Tainted: G 2.6.18-164.6.1.el5 #1 RIP: 0010:[<ffffffff80064bca>] [<ffffffff80064bca>] .text.lock.spinlock+0x0/0x30 RSP: 0018:ffff810399f67d20 EFLAGS: 00000282 RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 RDX: ffff8107dbf1d670 RSI: ffff8107dbf1d670 RDI: ffffffff88424f50 RBP: 000000000e9b1527 R08: ffff81010e776c00 R09: ffff81010e777000 R10: ffff81042e1f8038 R11: 000000d000000000 R12: ffff810600000000 R13: 00000000000fb000 R14: 00000000000f923d R15: 0000000000004937 FS: 00002b342d854570(0000) GS:ffffffff803c1000(0000) knlGS:00000000f7e0aac0 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000040480fe0 CR3: 0000000240ba7000 CR4: 00000000000006a0 Call Trace: [<ffffffff883eaf8e>] :nfs:nfs_access_add_cache+0x13d/0x16d [<ffffffff883eb289>] :nfs:nfs_permission+0x147/0x1cb [<ffffffff8000ea30>] link_path_walk+0xa6/0xb2 [<ffffffff8000d902>] permission+0x81/0xc8 [<ffffffff80012469>] may_open+0x65/0x22f [<ffffffff8001afed>] open_namei+0x2c4/0x6d5 [<ffffffff800e8cd5>] __posix_lock_file_conf+0x396/0x3e0 [<ffffffff80027308>] do_filp_open+0x1c/0x38 [<ffffffff80019cdb>] do_sys_open+0x44/0xbe [<ffffffff8005d28d>] tracesys+0xd5/0xe0 BUG: soft lockup - CPU#1 stuck for 10s! [kswapd0:520] CPU 1: Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Pid: 520, comm: kswapd0 Tainted: G 2.6.18-164.6.1.el5 #1 RIP: 0010:[<ffffffff80064bcc>] [<ffffffff80064bcc>] .text.lock.spinlock+0x2/0x30 RSP: 0018:ffff81081f8fbd38 EFLAGS: 00000282 RAX: ffff81081f8fbd50 RBX: 0000000000000000 RCX: 00000000003939f2 RDX: 0000000000000000 RSI: 00000000000000d0 RDI: ffffffff88424f50 RBP: 0000000000000000 R08: 00000000000c9ebc R09: 00000000007e3359 R10: 000000000000005e R11: 0000000000000002 R12: 0000000000000000 R13: 0000000100000002 R14: 0000000000000000 R15: 0000000000000000 FS: 00002b4c2108fdd0(0000) GS:ffff81010e7993c0(0000) knlGS:00000000f7e0aac0 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 000000000efeb138 CR3: 0000000000201000 CR4: 00000000000006a0 Call Trace: [<ffffffff883eb33a>] :nfs:nfs_access_cache_shrinker+0x2d/0x1da [<ffffffff8003f33a>] shrink_slab+0x60/0x153 [<ffffffff80057da5>] kswapd+0x343/0x46c [<ffffffff8009fc08>] autoremove_wake_function+0x0/0x2e [<ffffffff80057a62>] kswapd+0x0/0x46c [<ffffffff8009f9f0>] keventd_create_kthread+0x0/0xc4 [<ffffffff8003297c>] kthread+0xfe/0x132 [<ffffffff8005dfb1>] child_rip+0xa/0x11 [<ffffffff8009f9f0>] keventd_create_kthread+0x0/0xc4 [<ffffffff8003287e>] kthread+0x0/0x132 [<ffffffff8005dfa7>] child_rip+0x0/0x11 BUG: soft lockup - CPU#3 stuck for 10s! [smbd:16110] CPU 3: Modules linked in: nfsd exportfs auth_rpcgss nfs fscache nfs_acl autofs4 lockd sunrpc ipv6 xfrm_nalgo crypto_api dm_mirror dm_multipath scsi_dh video backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_nforce2 forcedeth k8_edac shpchp i2c_core serio_raw edac_mc k8temp tg3 hwmon pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache sata_nv libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Pid: 16110, comm: smbd Tainted: G 2.6.18-164.6.1.el5 #1 RIP: 0010:[<ffffffff80064bcf>] [<ffffffff80064bcf>] .text.lock.spinlock+0x5/0x30 RSP: 0018:ffff8100657379b0 EFLAGS: 00000282 RAX: ffff8100657379c8 RBX: 0000000000000000 RCX: 00000000007446cf RDX: 0000000000000000 RSI: 00000000000200d2 RDI: ffffffff88424f50 RBP: 0000000000001680 R08: ffff810000033600 R09: ffff81019dbe6770 R10: 000000000000005e R11: ffffffff883edf6d R12: 0000000000000246 R13: ffff8104144061c0 R14: 0000000000000000 R15: ffff81041e5aaa40 FS: 00002b342d854570(0000) GS:ffff81042e1b76c0(0000) knlGS:00000000f7e0aac0 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00002b19416622f1 CR3: 000000005186f000 CR4: 00000000000006a0 Call Trace: [<ffffffff883eb33a>] :nfs:nfs_access_cache_shrinker+0x2d/0x1da [<ffffffff8003f33a>] shrink_slab+0x60/0x153 [<ffffffff800ca905>] try_to_free_pages+0x1da/0x2d7 [<ffffffff800cb003>] zone_statistics+0x3e/0x6d [<ffffffff8000f40d>] __alloc_pages+0x1cb/0x2ce [<ffffffff883f6927>] :nfs:nfs_update_request+0x188/0x271 [<ffffffff800c4b08>] grab_cache_page_write_begin+0x4a/0x89 [<ffffffff883ecf81>] :nfs:nfs_write_begin+0x41/0xf8 [<ffffffff8000fcc1>] generic_file_buffered_write+0x14b/0x675 [<ffffffff80030e7f>] release_sock+0x13/0xaa [<ffffffff80016513>] __generic_file_aio_write_nolock+0x369/0x3b6 [<ffffffff80045c75>] do_sock_read+0xcf/0x110 [<ffffffff8002156f>] generic_file_aio_write+0x65/0xc1 [<ffffffff883ed54f>] :nfs:nfs_file_write+0xab/0x124 [<ffffffff80018123>] do_sync_write+0xc7/0x104 [<ffffffff8002cafe>] mntput_no_expire+0x19/0x89 [<ffffffff8009fc08>] autoremove_wake_function+0x0/0x2e [<ffffffff80062486>] __sched_text_start+0xf6/0xbd6 [<ffffffff8001691b>] vfs_write+0xce/0x174 [<ffffffff80043e0e>] sys_pwrite64+0x50/0x70 [<ffffffff8005d229>] tracesys+0x71/0xe0 [<ffffffff8005d28d>] tracesys+0xd5/0xe0 ...
Problem appears to be in the nfs_access_cache_shrinker(), stuck on the spin_lock(&nfs_access_lru_lock). Larry
It seems like this could be related to https://bugzilla.redhat.com/show_bug.cgi?id=525898 - similar symptoms, at least. I too am having something like this happen. Essentially, we got kswapd0 eating all of the system CPU and locking up the system while doing a large amount of reads from NFS. I posted additional details on the thread for that other bug, but I won't cross-post them all here in case you decide they're not the same after all.
*** This bug has been marked as a duplicate of bug 585935 ***