Bug 469496

Summary: BUG: soft lockup - CPU stuck
Product: [Fedora] Fedora Reporter: Albert Strasheim <fullung>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED WONTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: medium    
Version: 9CC: fullung, jaswinder, kernel-maint, quintela, ravikumar.d25
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-07-14 14:24:12 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Albert Strasheim 2008-11-01 15:37:39 UTC 
Description of problem:

BUG: soft lockup - CPU#2 stuck for 182s! [someprog:8489]
Modules linked in: coretemp hwmon tun ipt_MASQUERADE iptable_nat nf_nat ipt_REJECT nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6 loop dm_multipath sr_mod cdrom bnx2 ata_piix dcdbas ata_generic pata_acpi libata ses serio_raw enclosure iTCO_wdt pcspkr iTCO_vendor_support i5000_edac edac_core sg dm_snapshot dm_zero dm_mirror dm_log dm_mod shpchp megaraid_sas sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table]
CPU 2:
Modules linked in: coretemp hwmon tun ipt_MASQUERADE iptable_nat nf_nat ipt_REJECT nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6 loop dm_multipath sr_mod cdrom bnx2 ata_piix dcdbas ata_generic pata_acpi libata ses serio_raw enclosure iTCO_wdt pcspkr iTCO_vendor_support i5000_edac edac_core sg dm_snapshot dm_zero dm_mirror dm_log dm_mod shpchp megaraid_sas sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table]
Pid: 8489, comm: test_unizar_lib Not tainted 2.6.26.6-79.fc9.x86_64 #1
RIP: 0010:[<ffffffff81086544>]  [<ffffffff81086544>] __pagevec_lru_add_active+0xd6/0xfa
RSP: 0000:ffff81042d9d9ab8  EFLAGS: 00000202
RAX: 000000000000000e RBX: ffff81042d9d9ae8 RCX: ffff81000001b000
RDX: 0000000000000015 RSI: 0000000000000002 RDI: ffff81000001b000
RBP: 0000000000000000 R08: ffff81043fc04a80 R09: 0000000000000000
R10: 0000000000009ace R11: 0000000000000002 R12: 0000008000000000
R13: 0000000000000000 R14: ffffffff81083dde R15: ffff81042d9d9aa8
FS:  000000007ffd8000(0063) GS:ffff81043fc04c00(006b) knlGS:00000000602f1a00
CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
CR2: 0000000000af8000 CR3: 000000024213a000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

Call Trace:
 [<ffffffff8108651f>] ? __pagevec_lru_add_active+0xb1/0xfa
 [<ffffffff810865ae>] ? lru_cache_add_active+0x46/0x48
 [<ffffffff8108e01f>] ? handle_mm_fault+0x3a4/0x933
 [<ffffffff8129fdb3>] ? do_page_fault+0x676/0xabf
 [<ffffffff81083281>] ? __pagevec_free+0x29/0x3d
 [<ffffffff810862bf>] ? release_pages+0x1be/0x1d0
 [<ffffffff81092431>] ? vma_adjust+0x1f6/0x442
 [<ffffffff81092cbf>] ? vma_merge+0x1ce/0x307
 [<ffffffff8101a99b>] ? flush_tlb_mm+0x49/0x78
 [<ffffffff8109430e>] ? mprotect_fixup+0x5aa/0x6f2
 [<ffffffff81072afe>] ? audit_syscall_exit+0x331/0x353
 [<ffffffff8129d9f9>] ? error_exit+0x0/0x60

BUG: soft lockup - CPU#3 stuck for 182s! [someprog:8490]
Modules linked in: coretemp hwmon tun ipt_MASQUERADE iptable_nat nf_nat ipt_REJECT nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6 loop dm_multipath sr_mod cdrom bnx2 ata_piix dcdbas ata_generic pata_acpi libata ses serio_raw enclosure iTCO_wdt pcspkr iTCO_vendor_support i5000_edac edac_core sg dm_snapshot dm_zero dm_mirror dm_log dm_mod shpchp megaraid_sas sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table]
CPU 3:
Modules linked in: coretemp hwmon tun ipt_MASQUERADE iptable_nat nf_nat ipt_REJECT nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6 loop dm_multipath sr_mod cdrom bnx2 ata_piix dcdbas ata_generic pata_acpi libata ses serio_raw enclosure iTCO_wdt pcspkr iTCO_vendor_support i5000_edac edac_core sg dm_snapshot dm_zero dm_mirror dm_log dm_mod shpchp megaraid_sas sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table]
Pid: 8490, comm: test_unizar_lib Not tainted 2.6.26.6-79.fc9.x86_64 #1
RIP: 0010:[<ffffffff810837e0>]  [<ffffffff810837e0>] get_page_from_freelist+0x3f5/0x673
RSP: 0000:ffff81042e1f9918  EFLAGS: 00000246
RAX: 0000000000000040 RBX: ffff81042e1f9a08 RCX: 0000000000000000
RDX: 00000000000000f7 RSI: 0000000000000013 RDI: 0000000000000246
RBP: ffffe20006dfd720 R08: ffff81043fc04e00 R09: 0000000000000000
R10: 0000000000009ace R11: 0000000000000002 R12: 000000000000001e
R13: 00000000000000fe R14: 0000000000000000 R15: 0000000000000040
FS:  000000007ffd8000(0063) GS:ffff81043fc04f80(006b) knlGS:00000000602f1a00
CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
CR2: 00000000013ad000 CR3: 00000002c8458000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

Call Trace:
 [<ffffffff81083dde>] ? __alloc_pages_internal+0xfe/0x44b
 [<ffffffff8109f759>] ? alloc_pages_current+0xb9/0xc2
 [<ffffffff81084134>] ? __alloc_pages_nodemask+0x9/0xb
 [<ffffffff8109f8d7>] ? alloc_page_vma+0x175/0x17a
 [<ffffffff8108df18>] ? handle_mm_fault+0x29d/0x933
 [<ffffffff810862bf>] ? release_pages+0x1be/0x1d0
 [<ffffffff81141d19>] ? list_add+0xc/0xf
 [<ffffffff81081ed8>] ? free_pages_bulk+0x1d4/0x246
 [<ffffffff8129fdb3>] ? do_page_fault+0x676/0xabf
 [<ffffffff81092431>] ? vma_adjust+0x1f6/0x442
 [<ffffffff81092cbf>] ? vma_merge+0x1ce/0x307
 [<ffffffff8101a99b>] ? flush_tlb_mm+0x49/0x78
 [<ffffffff8109430e>] ? mprotect_fixup+0x5aa/0x6f2
 [<ffffffff81072afe>] ? audit_syscall_exit+0x331/0x353
 [<ffffffff8129d9f9>] ? error_exit+0x0/0x60

BUG: soft lockup - CPU#5 stuck for 183s! [rsyslogd:2538]
Modules linked in: coretemp hwmon tun ipt_MASQUERADE iptable_nat nf_nat ipt_REJECT nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6 loop dm_multipath sr_mod cdrom bnx2 ata_piix dcdbas ata_generic pata_acpi libata ses serio_raw enclosure iTCO_wdt pcspkr iTCO_vendor_support i5000_edac edac_core sg dm_snapshot dm_zero dm_mirror dm_log dm_mod shpchp megaraid_sas sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table]
CPU 5:
Modules linked in: coretemp hwmon tun ipt_MASQUERADE iptable_nat nf_nat ipt_REJECT nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6 loop dm_multipath sr_mod cdrom bnx2 ata_piix dcdbas ata_generic pata_acpi libata ses serio_raw enclosure iTCO_wdt pcspkr iTCO_vendor_support i5000_edac edac_core sg dm_snapshot dm_zero dm_mirror dm_log dm_mod shpchp megaraid_sas sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table]
Pid: 2538, comm: rsyslogd Not tainted 2.6.26.6-79.fc9.x86_64 #1
RIP: 0010:[<ffffffff81037ebf>]  [<ffffffff81037ebf>] do_syslog+0x168/0x37a
RSP: 0000:ffff81043ccfde38  EFLAGS: 00000206
RAX: 000000000000bee4 RBX: ffff81043ccfdea8 RCX: 000000000000bee3
RDX: ffffffff815bbe20 RSI: 000000000000026e RDI: ffffffff815bb8e4
RBP: 000000050000000c R08: ffffffff813d3dc8 R09: 0000000000000000
R10: 0005b43d3a6888f4 R11: ffff810434952d88 R12: ffff810434953090
R13: ffff81043e56da80 R14: ffff810434952d40 R15: ffffffff81585700
FS:  0000000042030950(0063) GS:ffff81043e45d680(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000007d07e25c CR3: 000000043cd84000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

Call Trace:
 [<ffffffff81037ed9>] ? do_syslog+0x182/0x37a
 [<ffffffff8104a388>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff810f46fc>] ? kmsg_read+0x3b/0x47
 [<ffffffff810ec4c5>] ? proc_reg_read+0x7b/0x95
 [<ffffffff810ad915>] ? vfs_read+0xab/0x154
 [<ffffffff810ada82>] ? sys_read+0x47/0x70
 [<ffffffff8100c2c1>] ? tracesys+0xd0/0xd5

Version-Release number of selected component (if applicable):

kernel-2.6.26.6-79.fc9.x86_64

Additional info:

Hardware is a Dell PowerEdge 2950
Comment 1 Albert Strasheim 2008-11-03 16:16:18 UTC 
I had another lockup today with a new log message:

Clocksource tsc unstable (delta = 3004558313002 ns)

Here's the rest of the dump:

BUG: soft lockup - CPU#6 stuck for 15125s! [sge_shepherd:26981]
Modules linked in: coretemp hwmon tun ipt_MASQUERADE iptable_nat nf_nat ipt_REJECT nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6 loop dm_multipath sr_mod cdrom bnx2 ata_piix dcdbas ata_generic pata_acpi libata ses serio_raw enclosure iTCO_wdt pcspkr iTCO_vendor_support i5000_edac edac_core sg dm_snapshot dm_zero dm_mirror dm_log dm_mod shpchp megaraid_sas sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table]
CPU 6:
Modules linked in: coretemp hwmon tun ipt_MASQUERADE iptable_nat nf_nat ipt_REJECT nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6 loop dm_multipath sr_mod cdrom bnx2 ata_piix dcdbas ata_generic pata_acpi libata ses serio_raw enclosure iTCO_wdt pcspkr iTCO_vendor_support i5000_edac edac_core sg dm_snapshot dm_zero dm_mirror dm_log dm_mod shpchp megaraid_sas sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table]
Pid: 26981, comm: sge_shepherd Not tainted 2.6.26.6-79.fc9.x86_64 #1
RIP: 0010:[<ffffffff81191f2b>]  [<ffffffff81191f2b>] add_timer_randomness+0x9f/0x10c
RSP: 0018:ffff81043e5ffd10  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff81043e5ffd50 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000286
RBP: ffff81043e5ffc90 R08: 000000000000007f R09: 00000000355baa09
R10: 0000000000000286 R11: 0000000000000001 R12: ffffffff8100cf42
R13: ffff81043e5ffc90 R14: 0000000000800100 R15: ffff81043e5a2fa0
FS:  00007f61c56016f0(0000) GS:ffff81043e45da00(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000071fe44 CR3: 000000008b2ae000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

Call Trace:
 <IRQ>  [<ffffffff81191fc3>] ? add_disk_randomness+0x2b/0x2d
 [<ffffffff81129cf9>] ? blk_end_io+0x6a/0xa8
 [<ffffffff81129d63>] ? blk_end_request+0xe/0x10
 [<ffffffffa0058114>] ? :scsi_mod:scsi_end_request+0x30/0x90
 [<ffffffffa0058721>] ? :scsi_mod:scsi_io_completion+0x1b3/0x3b0
 [<ffffffffa005263b>] ? :scsi_mod:scsi_finish_command+0xce/0xd7
 [<ffffffffa0058b8f>] ? :scsi_mod:scsi_softirq_done+0xe4/0xed
 [<ffffffff81128c39>] ? blk_done_softirq+0x77/0x87
 [<ffffffff8103c21a>] ? __do_softirq+0x6d/0xe1
 [<ffffffffa008339d>] ? :megaraid_sas:megasas_isr+0x47/0x52
 [<ffffffff8100d51c>] ? call_softirq+0x1c/0x28
 [<ffffffff8100e7b4>] ? do_softirq+0x44/0x8b
 [<ffffffff8103bfeb>] ? irq_exit+0x3f/0x80
 [<ffffffff8100ea9c>] ? do_IRQ+0x147/0x169
 [<ffffffff8100c6bd>] ? ret_from_intr+0x0/0x19
 <EOI>  [<ffffffffa003ab13>] ? :ext3:ext3_create+0x7e/0x111
 [<ffffffffa003aaeb>] ? :ext3:ext3_create+0x56/0x111
 [<ffffffff810b523c>] ? vfs_create+0xde/0x153
 [<ffffffff810b82a6>] ? do_filp_open+0x216/0x8ba
 [<ffffffff8113ed95>] ? __strncpy_from_user+0x2c/0x53
 [<ffffffff810ab4a4>] ? get_unused_fd_flags+0x8b/0x11f
 [<ffffffff810ab589>] ? do_sys_open+0x51/0xd1
 [<ffffffff810ab632>] ? sys_open+0x1b/0x1d
 [<ffffffff8100c2c1>] ? tracesys+0xd0/0xd5
Comment 2 ravikumar.d 2009-04-17 06:23:34 UTC 
Apr 15 09:04:42 loginserver10.oscardomain kernel: BUG: soft lockup - CPU#2 stuck for 10s! [hald-addon-stor:11473] 
Apr 15 09:04:42 loginserver10.oscardomain kernel: CPU 2: 
Apr 15 09:04:42 loginserver10.oscardomain kernel: Modules linked in: obdfilter(U) fsfilt_ldiskfs(U) ost(U) mgc(U) ldiskfs(U) crc16(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) libcfs(U) deflate(U) zlib_deflate(U) af_key(U) autofs4(U) hidp(U) rfcomm(U) l2cap(U) bluetooth(U) sunrpc(U) xt_tcpudp(U) ipt_tos(U) ipt_owner(U) xt_multiport(U) xt_mark(U) xt_mac(U) xt_limit(U) ipt_TOS(U) ipt_REJECT(U) xt_MARK(U) ipt_LOG(U) x_tables(U) ip_queue(U) ib_iser(U) iscsi_tcp(U) libiscsi(U) scsi_transport_iscsi(U) rdma_ucm(U) qlgc_vnic(U) ib_sdp(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ipoib_helper(U) ib_cm(U) ib_sa(U) ipv6(U) xfrm_nalgo(U) crypto_api(U) ib_uverbs(U) ib_umad(U) iw_cxgb3(U) cxgb3(U) ib_ipath(U) mlx4_ib(U) mlx4_core(U) dm_mirror(U) dm_multipath(U) dm_mod(U) video(U) sbs(U) backlight(U) i2c_ec(U) button(U) battery(U) asus_acpi(U) acpi_memhotplug(U) ac(U) parport_pc(U) lp(U) parport(U) joydev(U) ib_mthca(U) i2c_i801(U) ib_mad(U) sg(U) pata_acpi(U) ib_core(U) shpchp(U) i5000_edac(U)  
Apr 15 09:04:42 loginserver10.oscardomain kernel: nx2(U) edac_mc(U) pcspkr(U) ide_cd(U) i2c_core(U) cdrom(U) serio_raw(U) ata_piix(U) libata(U) aacraid(U) sd_mod(U) scsi_mod(U) ext3(U) jbd(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U) 
Apr 15 09:04:42 loginserver10.oscardomain kernel: Pid: 11473, comm: hald-addon-stor Tainted: G      2.6.18-92.1.10.el5_lustre.1.6.6smp #1 
Apr 15 09:04:42 loginserver10.oscardomain kernel: RIP: 0010:[<ffffffff80064ad0>]  [<ffffffff80064ad0>] _spin_unlock_irqrestore+0x8/0x9 
Apr 15 09:04:42 loginserver10.oscardomain kernel: RSP: 0018:ffff81042672f8b0  EFLAGS: 00000246 
Apr 15 09:04:42 loginserver10.oscardomain kernel: RAX: 0000000000000001 RBX: ffffffff80509f18 RCX: 0000000000000001 
Apr 15 09:04:42 loginserver10.oscardomain kernel: RDX: 0000000000000001 RSI: 0000000000000246 RDI: ffffffff80412600 
Apr 15 09:04:42 loginserver10.oscardomain kernel: RBP: ffff8103dc9618e8 R08: 000000000d7ffae0 R09: ffff81043ea930d0 
Apr 15 09:04:42 loginserver10.oscardomain kernel: R10: ffff81010d7ffae0 R11: 000000000000007b R12: ffff81043eaaf2c0 
Apr 15 09:04:42 loginserver10.oscardomain kernel: R13: ffff8103dc641a40 R14: ffff8103dc641a40 R15: 0000000100000000 
Apr 15 09:04:42 loginserver10.oscardomain kernel: FS:  00002b515074d2b0(0000) GS:ffff81010ef1acc0(0000) knlGS:0000000000000000 
Apr 15 09:04:42 loginserver10.oscardomain kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b 
Apr 15 09:04:42 loginserver10.oscardomain kernel: CR2: 00002aaaac010000 CR3: 0000000428932000 CR4: 00000000000006e0 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  
Apr 15 09:04:42 loginserver10.oscardomain kernel: Call Trace: 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8000af01>] ide_end_request+0xf0/0xfc 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8000ef72>] ide_do_request+0x711/0x790 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8000af01>] ide_end_request+0xf0/0xfc 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8003d50e>] lock_timer_base+0x1b/0x3c 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff80031f1c>] del_timer+0x4e/0x57 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff800c6029>] zone_statistics+0x3e/0x6d 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff80138184>] elv_insert+0xd6/0x1f7 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff80139a74>] blk_execute_rq_nowait+0x7e/0x9a 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff80139b2e>] blk_execute_rq+0x9e/0xce 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8013ceac>] sg_io+0x235/0x333 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8013d431>] scsi_cmd_ioctl+0x1d2/0x3b5 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8811dd98>] :cdrom:cdrom_ioctl+0x31/0xc18 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8008939c>] __wake_up_common+0x3e/0x68 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8813cd4e>] :ide_cd:idecd_ioctl+0x13f/0x159 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff80064991>] __down+0x99/0xd8 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8008ad8c>] default_wake_function+0x0/0xe 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8013b43f>] blkdev_driver_ioctl+0x5d/0x72 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8013ba93>] blkdev_ioctl+0x63f/0x69a 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff80064c4f>] .text.lock.kernel_lock+0x5/0x1c 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff800dce86>] do_open+0x250/0x30f 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff800dc3df>] bd_claim+0x18/0x73 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff800dd199>] blkdev_open+0x0/0x4f 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff800dd1d3>] blkdev_open+0x3a/0x4f 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8001e2db>] __dentry_open+0x101/0x1dc 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff800dc610>] block_ioctl+0x1b/0x1f 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff80041919>] do_ioctl+0x21/0x6b 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8002fea7>] vfs_ioctl+0x248/0x261 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8004bd45>] sys_ioctl+0x59/0x78 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0 
Apr 15 09:04:42 loginserver10.oscardomain kernel:
Comment 3 ravikumar.d 2009-04-17 06:32:27 UTC 
Hello all iam using IBM System X3650 server with lustre 1.6.6 patched kernel of of RHEL5.2, and when i am running any benchmarks on this system i am getting this error log, 

Apr 15 09:04:42 loginserver10.oscardomain kernel: BUG: soft lockup - CPU#2 stuck for 10s! [hald-addon-stor:11473] 
Apr 15 09:04:42 loginserver10.oscardomain kernel: CPU 2: 
Apr 15 09:04:42 loginserver10.oscardomain kernel: Modules linked in: obdfilter(U) fsfilt_ldiskfs(U) ost(U) mgc(U) ldiskfs(U) crc16(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) libcfs(U) deflate(U) zlib_deflate(U) af_key(U) autofs4(U) hidp(U) rfcomm(U) l2cap(U) bluetooth(U) sunrpc(U) xt_tcpudp(U) ipt_tos(U) ipt_owner(U) xt_multiport(U) xt_mark(U) xt_mac(U) xt_limit(U) ipt_TOS(U) ipt_REJECT(U) xt_MARK(U) ipt_LOG(U) x_tables(U) ip_queue(U) ib_iser(U) iscsi_tcp(U) libiscsi(U) scsi_transport_iscsi(U) rdma_ucm(U) qlgc_vnic(U) ib_sdp(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ipoib_helper(U) ib_cm(U) ib_sa(U) ipv6(U) xfrm_nalgo(U) crypto_api(U) ib_uverbs(U) ib_umad(U) iw_cxgb3(U) cxgb3(U) ib_ipath(U) mlx4_ib(U) mlx4_core(U) dm_mirror(U) dm_multipath(U) dm_mod(U) video(U) sbs(U) backlight(U) i2c_ec(U) button(U) battery(U) asus_acpi(U) acpi_memhotplug(U) ac(U) parport_pc(U) lp(U) parport(U) joydev(U) ib_mthca(U) i2c_i801(U) ib_mad(U) sg(U) pata_acpi(U) ib_core(U) shpchp(U) i5000_edac(U)  
Apr 15 09:04:42 loginserver10.oscardomain kernel: nx2(U) edac_mc(U) pcspkr(U) ide_cd(U) i2c_core(U) cdrom(U) serio_raw(U) ata_piix(U) libata(U) aacraid(U) sd_mod(U) scsi_mod(U) ext3(U) jbd(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U) 
Apr 15 09:04:42 loginserver10.oscardomain kernel: Pid: 11473, comm: hald-addon-stor Tainted: G      2.6.18-92.1.10.el5_lustre.1.6.6smp #1 
Apr 15 09:04:42 loginserver10.oscardomain kernel: RIP: 0010:[<ffffffff80064ad0>]  [<ffffffff80064ad0>] _spin_unlock_irqrestore+0x8/0x9 
Apr 15 09:04:42 loginserver10.oscardomain kernel: RSP: 0018:ffff81042672f8b0  EFLAGS: 00000246 
Apr 15 09:04:42 loginserver10.oscardomain kernel: RAX: 0000000000000001 RBX: ffffffff80509f18 RCX: 0000000000000001 
Apr 15 09:04:42 loginserver10.oscardomain kernel: RDX: 0000000000000001 RSI: 0000000000000246 RDI: ffffffff80412600 
Apr 15 09:04:42 loginserver10.oscardomain kernel: RBP: ffff8103dc9618e8 R08: 000000000d7ffae0 R09: ffff81043ea930d0 
Apr 15 09:04:42 loginserver10.oscardomain kernel: R10: ffff81010d7ffae0 R11: 000000000000007b R12: ffff81043eaaf2c0 
Apr 15 09:04:42 loginserver10.oscardomain kernel: R13: ffff8103dc641a40 R14: ffff8103dc641a40 R15: 0000000100000000 
Apr 15 09:04:42 loginserver10.oscardomain kernel: FS:  00002b515074d2b0(0000) GS:ffff81010ef1acc0(0000) knlGS:0000000000000000 
Apr 15 09:04:42 loginserver10.oscardomain kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b 
Apr 15 09:04:42 loginserver10.oscardomain kernel: CR2: 00002aaaac010000 CR3: 0000000428932000 CR4: 00000000000006e0 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  
Apr 15 09:04:42 loginserver10.oscardomain kernel: Call Trace: 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8000af01>] ide_end_request+0xf0/0xfc 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8000ef72>] ide_do_request+0x711/0x790 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8000af01>] ide_end_request+0xf0/0xfc 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8003d50e>] lock_timer_base+0x1b/0x3c 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff80031f1c>] del_timer+0x4e/0x57 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff800c6029>] zone_statistics+0x3e/0x6d 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff80138184>] elv_insert+0xd6/0x1f7 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff80139a74>] blk_execute_rq_nowait+0x7e/0x9a 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff80139b2e>] blk_execute_rq+0x9e/0xce 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8013ceac>] sg_io+0x235/0x333 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8013d431>] scsi_cmd_ioctl+0x1d2/0x3b5 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8811dd98>] :cdrom:cdrom_ioctl+0x31/0xc18 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8008939c>] __wake_up_common+0x3e/0x68 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8813cd4e>] :ide_cd:idecd_ioctl+0x13f/0x159 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff80064991>] __down+0x99/0xd8 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8008ad8c>] default_wake_function+0x0/0xe 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8013b43f>] blkdev_driver_ioctl+0x5d/0x72 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8013ba93>] blkdev_ioctl+0x63f/0x69a 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff80064c4f>] .text.lock.kernel_lock+0x5/0x1c 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff800dce86>] do_open+0x250/0x30f 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff800dc3df>] bd_claim+0x18/0x73 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff800dd199>] blkdev_open+0x0/0x4f 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff800dd1d3>] blkdev_open+0x3a/0x4f 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8001e2db>] __dentry_open+0x101/0x1dc 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff800dc610>] block_ioctl+0x1b/0x1f 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff80041919>] do_ioctl+0x21/0x6b 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8002fea7>] vfs_ioctl+0x248/0x261 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8004bd45>] sys_ioctl+0x59/0x78 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0 
Apr 15 09:04:42 loginserver10.oscardomain kernel:  


During this state i am unable to do any of the ssh to this particular system.
Please any on help me to solve this problem
Comment 4 Bug Zapper 2009-06-10 03:08:05 UTC 
This message is a reminder that Fedora 9 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 9.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '9'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 9's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 9 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 5 Bug Zapper 2009-07-14 14:24:12 UTC 
Fedora 9 changed to end-of-life (EOL) status on 2009-07-10. Fedora 9 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.