Description of problem: BUG: soft lockup - CPU#2 stuck for 182s! [someprog:8489] Modules linked in: coretemp hwmon tun ipt_MASQUERADE iptable_nat nf_nat ipt_REJECT nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6 loop dm_multipath sr_mod cdrom bnx2 ata_piix dcdbas ata_generic pata_acpi libata ses serio_raw enclosure iTCO_wdt pcspkr iTCO_vendor_support i5000_edac edac_core sg dm_snapshot dm_zero dm_mirror dm_log dm_mod shpchp megaraid_sas sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table] CPU 2: Modules linked in: coretemp hwmon tun ipt_MASQUERADE iptable_nat nf_nat ipt_REJECT nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6 loop dm_multipath sr_mod cdrom bnx2 ata_piix dcdbas ata_generic pata_acpi libata ses serio_raw enclosure iTCO_wdt pcspkr iTCO_vendor_support i5000_edac edac_core sg dm_snapshot dm_zero dm_mirror dm_log dm_mod shpchp megaraid_sas sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table] Pid: 8489, comm: test_unizar_lib Not tainted 2.6.26.6-79.fc9.x86_64 #1 RIP: 0010:[<ffffffff81086544>] [<ffffffff81086544>] __pagevec_lru_add_active+0xd6/0xfa RSP: 0000:ffff81042d9d9ab8 EFLAGS: 00000202 RAX: 000000000000000e RBX: ffff81042d9d9ae8 RCX: ffff81000001b000 RDX: 0000000000000015 RSI: 0000000000000002 RDI: ffff81000001b000 RBP: 0000000000000000 R08: ffff81043fc04a80 R09: 0000000000000000 R10: 0000000000009ace R11: 0000000000000002 R12: 0000008000000000 R13: 0000000000000000 R14: ffffffff81083dde R15: ffff81042d9d9aa8 FS: 000000007ffd8000(0063) GS:ffff81043fc04c00(006b) knlGS:00000000602f1a00 CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033 CR2: 0000000000af8000 CR3: 000000024213a000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [<ffffffff8108651f>] ? __pagevec_lru_add_active+0xb1/0xfa [<ffffffff810865ae>] ? lru_cache_add_active+0x46/0x48 [<ffffffff8108e01f>] ? handle_mm_fault+0x3a4/0x933 [<ffffffff8129fdb3>] ? do_page_fault+0x676/0xabf [<ffffffff81083281>] ? __pagevec_free+0x29/0x3d [<ffffffff810862bf>] ? release_pages+0x1be/0x1d0 [<ffffffff81092431>] ? vma_adjust+0x1f6/0x442 [<ffffffff81092cbf>] ? vma_merge+0x1ce/0x307 [<ffffffff8101a99b>] ? flush_tlb_mm+0x49/0x78 [<ffffffff8109430e>] ? mprotect_fixup+0x5aa/0x6f2 [<ffffffff81072afe>] ? audit_syscall_exit+0x331/0x353 [<ffffffff8129d9f9>] ? error_exit+0x0/0x60 BUG: soft lockup - CPU#3 stuck for 182s! [someprog:8490] Modules linked in: coretemp hwmon tun ipt_MASQUERADE iptable_nat nf_nat ipt_REJECT nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6 loop dm_multipath sr_mod cdrom bnx2 ata_piix dcdbas ata_generic pata_acpi libata ses serio_raw enclosure iTCO_wdt pcspkr iTCO_vendor_support i5000_edac edac_core sg dm_snapshot dm_zero dm_mirror dm_log dm_mod shpchp megaraid_sas sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table] CPU 3: Modules linked in: coretemp hwmon tun ipt_MASQUERADE iptable_nat nf_nat ipt_REJECT nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6 loop dm_multipath sr_mod cdrom bnx2 ata_piix dcdbas ata_generic pata_acpi libata ses serio_raw enclosure iTCO_wdt pcspkr iTCO_vendor_support i5000_edac edac_core sg dm_snapshot dm_zero dm_mirror dm_log dm_mod shpchp megaraid_sas sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table] Pid: 8490, comm: test_unizar_lib Not tainted 2.6.26.6-79.fc9.x86_64 #1 RIP: 0010:[<ffffffff810837e0>] [<ffffffff810837e0>] get_page_from_freelist+0x3f5/0x673 RSP: 0000:ffff81042e1f9918 EFLAGS: 00000246 RAX: 0000000000000040 RBX: ffff81042e1f9a08 RCX: 0000000000000000 RDX: 00000000000000f7 RSI: 0000000000000013 RDI: 0000000000000246 RBP: ffffe20006dfd720 R08: ffff81043fc04e00 R09: 0000000000000000 R10: 0000000000009ace R11: 0000000000000002 R12: 000000000000001e R13: 00000000000000fe R14: 0000000000000000 R15: 0000000000000040 FS: 000000007ffd8000(0063) GS:ffff81043fc04f80(006b) knlGS:00000000602f1a00 CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033 CR2: 00000000013ad000 CR3: 00000002c8458000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [<ffffffff81083dde>] ? __alloc_pages_internal+0xfe/0x44b [<ffffffff8109f759>] ? alloc_pages_current+0xb9/0xc2 [<ffffffff81084134>] ? __alloc_pages_nodemask+0x9/0xb [<ffffffff8109f8d7>] ? alloc_page_vma+0x175/0x17a [<ffffffff8108df18>] ? handle_mm_fault+0x29d/0x933 [<ffffffff810862bf>] ? release_pages+0x1be/0x1d0 [<ffffffff81141d19>] ? list_add+0xc/0xf [<ffffffff81081ed8>] ? free_pages_bulk+0x1d4/0x246 [<ffffffff8129fdb3>] ? do_page_fault+0x676/0xabf [<ffffffff81092431>] ? vma_adjust+0x1f6/0x442 [<ffffffff81092cbf>] ? vma_merge+0x1ce/0x307 [<ffffffff8101a99b>] ? flush_tlb_mm+0x49/0x78 [<ffffffff8109430e>] ? mprotect_fixup+0x5aa/0x6f2 [<ffffffff81072afe>] ? audit_syscall_exit+0x331/0x353 [<ffffffff8129d9f9>] ? error_exit+0x0/0x60 BUG: soft lockup - CPU#5 stuck for 183s! [rsyslogd:2538] Modules linked in: coretemp hwmon tun ipt_MASQUERADE iptable_nat nf_nat ipt_REJECT nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6 loop dm_multipath sr_mod cdrom bnx2 ata_piix dcdbas ata_generic pata_acpi libata ses serio_raw enclosure iTCO_wdt pcspkr iTCO_vendor_support i5000_edac edac_core sg dm_snapshot dm_zero dm_mirror dm_log dm_mod shpchp megaraid_sas sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table] CPU 5: Modules linked in: coretemp hwmon tun ipt_MASQUERADE iptable_nat nf_nat ipt_REJECT nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6 loop dm_multipath sr_mod cdrom bnx2 ata_piix dcdbas ata_generic pata_acpi libata ses serio_raw enclosure iTCO_wdt pcspkr iTCO_vendor_support i5000_edac edac_core sg dm_snapshot dm_zero dm_mirror dm_log dm_mod shpchp megaraid_sas sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table] Pid: 2538, comm: rsyslogd Not tainted 2.6.26.6-79.fc9.x86_64 #1 RIP: 0010:[<ffffffff81037ebf>] [<ffffffff81037ebf>] do_syslog+0x168/0x37a RSP: 0000:ffff81043ccfde38 EFLAGS: 00000206 RAX: 000000000000bee4 RBX: ffff81043ccfdea8 RCX: 000000000000bee3 RDX: ffffffff815bbe20 RSI: 000000000000026e RDI: ffffffff815bb8e4 RBP: 000000050000000c R08: ffffffff813d3dc8 R09: 0000000000000000 R10: 0005b43d3a6888f4 R11: ffff810434952d88 R12: ffff810434953090 R13: ffff81043e56da80 R14: ffff810434952d40 R15: ffffffff81585700 FS: 0000000042030950(0063) GS:ffff81043e45d680(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000007d07e25c CR3: 000000043cd84000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [<ffffffff81037ed9>] ? do_syslog+0x182/0x37a [<ffffffff8104a388>] ? autoremove_wake_function+0x0/0x38 [<ffffffff810f46fc>] ? kmsg_read+0x3b/0x47 [<ffffffff810ec4c5>] ? proc_reg_read+0x7b/0x95 [<ffffffff810ad915>] ? vfs_read+0xab/0x154 [<ffffffff810ada82>] ? sys_read+0x47/0x70 [<ffffffff8100c2c1>] ? tracesys+0xd0/0xd5 Version-Release number of selected component (if applicable): kernel-2.6.26.6-79.fc9.x86_64 Additional info: Hardware is a Dell PowerEdge 2950
I had another lockup today with a new log message: Clocksource tsc unstable (delta = 3004558313002 ns) Here's the rest of the dump: BUG: soft lockup - CPU#6 stuck for 15125s! [sge_shepherd:26981] Modules linked in: coretemp hwmon tun ipt_MASQUERADE iptable_nat nf_nat ipt_REJECT nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6 loop dm_multipath sr_mod cdrom bnx2 ata_piix dcdbas ata_generic pata_acpi libata ses serio_raw enclosure iTCO_wdt pcspkr iTCO_vendor_support i5000_edac edac_core sg dm_snapshot dm_zero dm_mirror dm_log dm_mod shpchp megaraid_sas sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table] CPU 6: Modules linked in: coretemp hwmon tun ipt_MASQUERADE iptable_nat nf_nat ipt_REJECT nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6 loop dm_multipath sr_mod cdrom bnx2 ata_piix dcdbas ata_generic pata_acpi libata ses serio_raw enclosure iTCO_wdt pcspkr iTCO_vendor_support i5000_edac edac_core sg dm_snapshot dm_zero dm_mirror dm_log dm_mod shpchp megaraid_sas sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table] Pid: 26981, comm: sge_shepherd Not tainted 2.6.26.6-79.fc9.x86_64 #1 RIP: 0010:[<ffffffff81191f2b>] [<ffffffff81191f2b>] add_timer_randomness+0x9f/0x10c RSP: 0018:ffff81043e5ffd10 EFLAGS: 00000246 RAX: 0000000000000000 RBX: ffff81043e5ffd50 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000286 RBP: ffff81043e5ffc90 R08: 000000000000007f R09: 00000000355baa09 R10: 0000000000000286 R11: 0000000000000001 R12: ffffffff8100cf42 R13: ffff81043e5ffc90 R14: 0000000000800100 R15: ffff81043e5a2fa0 FS: 00007f61c56016f0(0000) GS:ffff81043e45da00(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000000071fe44 CR3: 000000008b2ae000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: <IRQ> [<ffffffff81191fc3>] ? add_disk_randomness+0x2b/0x2d [<ffffffff81129cf9>] ? blk_end_io+0x6a/0xa8 [<ffffffff81129d63>] ? blk_end_request+0xe/0x10 [<ffffffffa0058114>] ? :scsi_mod:scsi_end_request+0x30/0x90 [<ffffffffa0058721>] ? :scsi_mod:scsi_io_completion+0x1b3/0x3b0 [<ffffffffa005263b>] ? :scsi_mod:scsi_finish_command+0xce/0xd7 [<ffffffffa0058b8f>] ? :scsi_mod:scsi_softirq_done+0xe4/0xed [<ffffffff81128c39>] ? blk_done_softirq+0x77/0x87 [<ffffffff8103c21a>] ? __do_softirq+0x6d/0xe1 [<ffffffffa008339d>] ? :megaraid_sas:megasas_isr+0x47/0x52 [<ffffffff8100d51c>] ? call_softirq+0x1c/0x28 [<ffffffff8100e7b4>] ? do_softirq+0x44/0x8b [<ffffffff8103bfeb>] ? irq_exit+0x3f/0x80 [<ffffffff8100ea9c>] ? do_IRQ+0x147/0x169 [<ffffffff8100c6bd>] ? ret_from_intr+0x0/0x19 <EOI> [<ffffffffa003ab13>] ? :ext3:ext3_create+0x7e/0x111 [<ffffffffa003aaeb>] ? :ext3:ext3_create+0x56/0x111 [<ffffffff810b523c>] ? vfs_create+0xde/0x153 [<ffffffff810b82a6>] ? do_filp_open+0x216/0x8ba [<ffffffff8113ed95>] ? __strncpy_from_user+0x2c/0x53 [<ffffffff810ab4a4>] ? get_unused_fd_flags+0x8b/0x11f [<ffffffff810ab589>] ? do_sys_open+0x51/0xd1 [<ffffffff810ab632>] ? sys_open+0x1b/0x1d [<ffffffff8100c2c1>] ? tracesys+0xd0/0xd5
Apr 15 09:04:42 loginserver10.oscardomain kernel: BUG: soft lockup - CPU#2 stuck for 10s! [hald-addon-stor:11473] Apr 15 09:04:42 loginserver10.oscardomain kernel: CPU 2: Apr 15 09:04:42 loginserver10.oscardomain kernel: Modules linked in: obdfilter(U) fsfilt_ldiskfs(U) ost(U) mgc(U) ldiskfs(U) crc16(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) libcfs(U) deflate(U) zlib_deflate(U) af_key(U) autofs4(U) hidp(U) rfcomm(U) l2cap(U) bluetooth(U) sunrpc(U) xt_tcpudp(U) ipt_tos(U) ipt_owner(U) xt_multiport(U) xt_mark(U) xt_mac(U) xt_limit(U) ipt_TOS(U) ipt_REJECT(U) xt_MARK(U) ipt_LOG(U) x_tables(U) ip_queue(U) ib_iser(U) iscsi_tcp(U) libiscsi(U) scsi_transport_iscsi(U) rdma_ucm(U) qlgc_vnic(U) ib_sdp(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ipoib_helper(U) ib_cm(U) ib_sa(U) ipv6(U) xfrm_nalgo(U) crypto_api(U) ib_uverbs(U) ib_umad(U) iw_cxgb3(U) cxgb3(U) ib_ipath(U) mlx4_ib(U) mlx4_core(U) dm_mirror(U) dm_multipath(U) dm_mod(U) video(U) sbs(U) backlight(U) i2c_ec(U) button(U) battery(U) asus_acpi(U) acpi_memhotplug(U) ac(U) parport_pc(U) lp(U) parport(U) joydev(U) ib_mthca(U) i2c_i801(U) ib_mad(U) sg(U) pata_acpi(U) ib_core(U) shpchp(U) i5000_edac(U) Apr 15 09:04:42 loginserver10.oscardomain kernel: nx2(U) edac_mc(U) pcspkr(U) ide_cd(U) i2c_core(U) cdrom(U) serio_raw(U) ata_piix(U) libata(U) aacraid(U) sd_mod(U) scsi_mod(U) ext3(U) jbd(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U) Apr 15 09:04:42 loginserver10.oscardomain kernel: Pid: 11473, comm: hald-addon-stor Tainted: G 2.6.18-92.1.10.el5_lustre.1.6.6smp #1 Apr 15 09:04:42 loginserver10.oscardomain kernel: RIP: 0010:[<ffffffff80064ad0>] [<ffffffff80064ad0>] _spin_unlock_irqrestore+0x8/0x9 Apr 15 09:04:42 loginserver10.oscardomain kernel: RSP: 0018:ffff81042672f8b0 EFLAGS: 00000246 Apr 15 09:04:42 loginserver10.oscardomain kernel: RAX: 0000000000000001 RBX: ffffffff80509f18 RCX: 0000000000000001 Apr 15 09:04:42 loginserver10.oscardomain kernel: RDX: 0000000000000001 RSI: 0000000000000246 RDI: ffffffff80412600 Apr 15 09:04:42 loginserver10.oscardomain kernel: RBP: ffff8103dc9618e8 R08: 000000000d7ffae0 R09: ffff81043ea930d0 Apr 15 09:04:42 loginserver10.oscardomain kernel: R10: ffff81010d7ffae0 R11: 000000000000007b R12: ffff81043eaaf2c0 Apr 15 09:04:42 loginserver10.oscardomain kernel: R13: ffff8103dc641a40 R14: ffff8103dc641a40 R15: 0000000100000000 Apr 15 09:04:42 loginserver10.oscardomain kernel: FS: 00002b515074d2b0(0000) GS:ffff81010ef1acc0(0000) knlGS:0000000000000000 Apr 15 09:04:42 loginserver10.oscardomain kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Apr 15 09:04:42 loginserver10.oscardomain kernel: CR2: 00002aaaac010000 CR3: 0000000428932000 CR4: 00000000000006e0 Apr 15 09:04:42 loginserver10.oscardomain kernel: Apr 15 09:04:42 loginserver10.oscardomain kernel: Call Trace: Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8000af01>] ide_end_request+0xf0/0xfc Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8000ef72>] ide_do_request+0x711/0x790 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8000af01>] ide_end_request+0xf0/0xfc Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8003d50e>] lock_timer_base+0x1b/0x3c Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff80031f1c>] del_timer+0x4e/0x57 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff800c6029>] zone_statistics+0x3e/0x6d Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff80138184>] elv_insert+0xd6/0x1f7 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff80139a74>] blk_execute_rq_nowait+0x7e/0x9a Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff80139b2e>] blk_execute_rq+0x9e/0xce Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8013ceac>] sg_io+0x235/0x333 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8013d431>] scsi_cmd_ioctl+0x1d2/0x3b5 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8811dd98>] :cdrom:cdrom_ioctl+0x31/0xc18 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8008939c>] __wake_up_common+0x3e/0x68 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8813cd4e>] :ide_cd:idecd_ioctl+0x13f/0x159 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff80064991>] __down+0x99/0xd8 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8008ad8c>] default_wake_function+0x0/0xe Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8013b43f>] blkdev_driver_ioctl+0x5d/0x72 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8013ba93>] blkdev_ioctl+0x63f/0x69a Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff80064c4f>] .text.lock.kernel_lock+0x5/0x1c Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff800dce86>] do_open+0x250/0x30f Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff800dc3df>] bd_claim+0x18/0x73 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff800dd199>] blkdev_open+0x0/0x4f Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff800dd1d3>] blkdev_open+0x3a/0x4f Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8001e2db>] __dentry_open+0x101/0x1dc Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff800dc610>] block_ioctl+0x1b/0x1f Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff80041919>] do_ioctl+0x21/0x6b Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8002fea7>] vfs_ioctl+0x248/0x261 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8004bd45>] sys_ioctl+0x59/0x78 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Apr 15 09:04:42 loginserver10.oscardomain kernel:
Hello all iam using IBM System X3650 server with lustre 1.6.6 patched kernel of of RHEL5.2, and when i am running any benchmarks on this system i am getting this error log, Apr 15 09:04:42 loginserver10.oscardomain kernel: BUG: soft lockup - CPU#2 stuck for 10s! [hald-addon-stor:11473] Apr 15 09:04:42 loginserver10.oscardomain kernel: CPU 2: Apr 15 09:04:42 loginserver10.oscardomain kernel: Modules linked in: obdfilter(U) fsfilt_ldiskfs(U) ost(U) mgc(U) ldiskfs(U) crc16(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) libcfs(U) deflate(U) zlib_deflate(U) af_key(U) autofs4(U) hidp(U) rfcomm(U) l2cap(U) bluetooth(U) sunrpc(U) xt_tcpudp(U) ipt_tos(U) ipt_owner(U) xt_multiport(U) xt_mark(U) xt_mac(U) xt_limit(U) ipt_TOS(U) ipt_REJECT(U) xt_MARK(U) ipt_LOG(U) x_tables(U) ip_queue(U) ib_iser(U) iscsi_tcp(U) libiscsi(U) scsi_transport_iscsi(U) rdma_ucm(U) qlgc_vnic(U) ib_sdp(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ipoib_helper(U) ib_cm(U) ib_sa(U) ipv6(U) xfrm_nalgo(U) crypto_api(U) ib_uverbs(U) ib_umad(U) iw_cxgb3(U) cxgb3(U) ib_ipath(U) mlx4_ib(U) mlx4_core(U) dm_mirror(U) dm_multipath(U) dm_mod(U) video(U) sbs(U) backlight(U) i2c_ec(U) button(U) battery(U) asus_acpi(U) acpi_memhotplug(U) ac(U) parport_pc(U) lp(U) parport(U) joydev(U) ib_mthca(U) i2c_i801(U) ib_mad(U) sg(U) pata_acpi(U) ib_core(U) shpchp(U) i5000_edac(U) Apr 15 09:04:42 loginserver10.oscardomain kernel: nx2(U) edac_mc(U) pcspkr(U) ide_cd(U) i2c_core(U) cdrom(U) serio_raw(U) ata_piix(U) libata(U) aacraid(U) sd_mod(U) scsi_mod(U) ext3(U) jbd(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U) Apr 15 09:04:42 loginserver10.oscardomain kernel: Pid: 11473, comm: hald-addon-stor Tainted: G 2.6.18-92.1.10.el5_lustre.1.6.6smp #1 Apr 15 09:04:42 loginserver10.oscardomain kernel: RIP: 0010:[<ffffffff80064ad0>] [<ffffffff80064ad0>] _spin_unlock_irqrestore+0x8/0x9 Apr 15 09:04:42 loginserver10.oscardomain kernel: RSP: 0018:ffff81042672f8b0 EFLAGS: 00000246 Apr 15 09:04:42 loginserver10.oscardomain kernel: RAX: 0000000000000001 RBX: ffffffff80509f18 RCX: 0000000000000001 Apr 15 09:04:42 loginserver10.oscardomain kernel: RDX: 0000000000000001 RSI: 0000000000000246 RDI: ffffffff80412600 Apr 15 09:04:42 loginserver10.oscardomain kernel: RBP: ffff8103dc9618e8 R08: 000000000d7ffae0 R09: ffff81043ea930d0 Apr 15 09:04:42 loginserver10.oscardomain kernel: R10: ffff81010d7ffae0 R11: 000000000000007b R12: ffff81043eaaf2c0 Apr 15 09:04:42 loginserver10.oscardomain kernel: R13: ffff8103dc641a40 R14: ffff8103dc641a40 R15: 0000000100000000 Apr 15 09:04:42 loginserver10.oscardomain kernel: FS: 00002b515074d2b0(0000) GS:ffff81010ef1acc0(0000) knlGS:0000000000000000 Apr 15 09:04:42 loginserver10.oscardomain kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Apr 15 09:04:42 loginserver10.oscardomain kernel: CR2: 00002aaaac010000 CR3: 0000000428932000 CR4: 00000000000006e0 Apr 15 09:04:42 loginserver10.oscardomain kernel: Apr 15 09:04:42 loginserver10.oscardomain kernel: Call Trace: Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8000af01>] ide_end_request+0xf0/0xfc Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8000ef72>] ide_do_request+0x711/0x790 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8000af01>] ide_end_request+0xf0/0xfc Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8003d50e>] lock_timer_base+0x1b/0x3c Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff80031f1c>] del_timer+0x4e/0x57 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff800c6029>] zone_statistics+0x3e/0x6d Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff80138184>] elv_insert+0xd6/0x1f7 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff80139a74>] blk_execute_rq_nowait+0x7e/0x9a Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff80139b2e>] blk_execute_rq+0x9e/0xce Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8013ceac>] sg_io+0x235/0x333 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8013d431>] scsi_cmd_ioctl+0x1d2/0x3b5 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8811dd98>] :cdrom:cdrom_ioctl+0x31/0xc18 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8008939c>] __wake_up_common+0x3e/0x68 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8813cd4e>] :ide_cd:idecd_ioctl+0x13f/0x159 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff80064991>] __down+0x99/0xd8 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8008ad8c>] default_wake_function+0x0/0xe Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8013b43f>] blkdev_driver_ioctl+0x5d/0x72 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8013ba93>] blkdev_ioctl+0x63f/0x69a Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff80064c4f>] .text.lock.kernel_lock+0x5/0x1c Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff800dce86>] do_open+0x250/0x30f Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff800dc3df>] bd_claim+0x18/0x73 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff800dd199>] blkdev_open+0x0/0x4f Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff800dd1d3>] blkdev_open+0x3a/0x4f Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8001e2db>] __dentry_open+0x101/0x1dc Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff800dc610>] block_ioctl+0x1b/0x1f Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff80041919>] do_ioctl+0x21/0x6b Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8002fea7>] vfs_ioctl+0x248/0x261 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8004bd45>] sys_ioctl+0x59/0x78 Apr 15 09:04:42 loginserver10.oscardomain kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Apr 15 09:04:42 loginserver10.oscardomain kernel: During this state i am unable to do any of the ssh to this particular system. Please any on help me to solve this problem
This message is a reminder that Fedora 9 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 9. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '9'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 9's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 9 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Fedora 9 changed to end-of-life (EOL) status on 2009-07-10. Fedora 9 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. Thank you for reporting this bug and we are sorry it could not be fixed.