Bug 595630
Summary: | GFS2 - Kernel Panics while file copying | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 5 | Reporter: | veliogluh | ||||||
Component: | kernel | Assignee: | Steve Whitehouse <swhiteho> | ||||||
Status: | CLOSED DUPLICATE | QA Contact: | Cluster QE <mspqa-list> | ||||||
Severity: | high | Docs Contact: | |||||||
Priority: | low | ||||||||
Version: | 5.5 | CC: | adas, bmarzins, rpeterso, swhiteho | ||||||
Target Milestone: | --- | ||||||||
Target Release: | --- | ||||||||
Hardware: | x86_64 | ||||||||
OS: | Linux | ||||||||
Whiteboard: | |||||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||||
Doc Text: | Story Points: | --- | |||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2010-05-26 08:29:21 UTC | Type: | --- | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Attachments: |
|
Created attachment 416321 [details]
SUN ILO Screen While Kernel Panics Second Screen
I copy a small protion of my production data to another node with NFS. Then I only mount all 13 GFS2 disks to new node which is using iSCSI SAN. When I start file copy same kernel panic occured. This hardware is an old HP DL 580 G2 machine. I think this problem may not be an hardware related problem. No GFS2 functions appear in the call trace, so I have a hard time believing GFS2 is at fault here. I recommend you temporarily change your /etc/sysctl.conf file to include the following options: kernel.sysrq = 1 kernel.printk = 7 kernel.panic_on_oops = 0 and then reboot, and recreate the scenario that fails. Then see if you can get all of the kernel messages with the dmesg command. If that doesn't work and the kernel still panics, you should redirect kernel messages to a serial port and attach a system to monitor that port, to capture the complete console output, then attach that output to the bugzilla record. Thanks. Looks like it might be either dm or the underlying device driver to me. I set your kernel settings and this time I can get kernel messges from /var/log/messages. When I checked all my kernel panic messages I realized that all of them starts with same message list_add corruption. next->prev should be ffff810211426ed0, but was 0000000000000000 What does this means? Here are the syslog messages when the problem starts: May 25 18:01:13 xentest kernel: list_add corruption. next->prev should be ffff810211426ed0, but was 0000000000000000 May 25 18:01:13 xentest kernel: ----------- [cut here ] --------- [please bite here ] --------- May 25 18:01:18 xentest kernel: Kernel BUG at lib/list_debug.c:26 May 25 18:01:18 xentest kernel: invalid opcode: 0000 [1] SMP May 25 18:01:18 xentest kernel: last sysfs file: /devices/pci0000:00/0000:00:02.0/0000:01:00.0/0000:02:00.0/0000:03:00.0/host0/target0:0:1/0:0:1:12/timeout May 25 18:01:18 xentest kernel: CPU 4 May 25 18:01:18 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U) May 25 18:01:18 xentest kernel: Pid: 14910, comm: cp Tainted: G 2.6.18-194.3.1.el5 #1 May 25 18:01:18 xentest kernel: RIP: 0010:[<ffffffff80154004>] [<ffffffff80154004>] __list_add+0x24/0x68 May 25 18:01:18 xentest kernel: RSP: 0018:ffff8101afe11b58 EFLAGS: 00010082 May 25 18:01:18 xentest kernel: RAX: 0000000000000058 RBX: ffff81008469b000 RCX: ffffffff80311da8 May 25 18:01:18 xentest kernel: RDX: ffffffff80311da8 RSI: 0000000000000000 RDI: ffffffff80311da0 May 25 18:01:18 xentest kernel: RBP: ffff810211426ed0 R08: ffffffff80311da8 R09: 0000000000000039 May 25 18:01:18 xentest kernel: R10: ffff8101afe117f8 R11: 0000000000000180 R12: ffff810072f5d000 May 25 18:01:18 xentest kernel: R13: ffff810211426ec0 R14: 0000000000000014 R15: ffff8101ad2701c0 May 25 18:01:18 xentest kernel: FS: 00002ae4d4b77f80(0000) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000 May 25 18:01:18 xentest kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b May 25 18:01:18 xentest kernel: CR2: 0000003572292215 CR3: 000000020daee000 CR4: 00000000000006e0 May 25 18:01:18 xentest kernel: Process cp (pid: 14910, threadinfo ffff8101afe10000, task ffff81021e483100) May 25 18:01:18 xentest kernel: Stack: 0000000000000028 ffff810072f5d000 ffff81021d833c00 ffffffff8005c382 May 25 18:01:18 xentest kernel: 0000085000000000 0000000000000246 0000000000000850 ffff8101ad2701c0 May 25 18:01:18 xentest kernel: 0000000000000001 ffff81010a52b000 00000000000022d8 ffffffff800db8fa May 25 18:01:18 xentest kernel: Call Trace: May 25 18:01:18 xentest kernel: [<ffffffff8005c382>] cache_alloc_refill+0x106/0x186 May 25 18:01:18 xentest kernel: [<ffffffff800db8fa>] kmem_cache_zalloc+0x6f/0x94 May 25 18:01:18 xentest kernel: [<ffffffff8871171f>] :gfs2:gfs2_trans_add_bh+0x84/0x104 May 25 18:01:18 xentest kernel: [<ffffffff886fdba4>] :gfs2:gfs2_createi+0x6ea/0xd28 May 25 18:01:18 xentest kernel: [<ffffffff886f4be6>] :gfs2:gfs2_dirent_find+0x0/0x4e May 25 18:01:18 xentest kernel: [<ffffffff886f4be6>] :gfs2:gfs2_dirent_find+0x0/0x4e May 25 18:01:18 xentest kernel: [<ffffffff886f498b>] :gfs2:gfs2_dirent_search+0x147/0x16e May 25 18:01:18 xentest kernel: [<ffffffff886f9e7a>] :gfs2:gfs2_glock_schedule_for_reclaim+0x5d/0x9a May 25 18:01:18 xentest kernel: [<ffffffff8002d035>] wake_up_bit+0x11/0x22 May 25 18:01:18 xentest kernel: [<ffffffff88708d59>] :gfs2:gfs2_create+0x65/0x143 May 25 18:01:18 xentest kernel: [<ffffffff886fd51d>] :gfs2:gfs2_createi+0x63/0xd28 May 25 18:01:18 xentest kernel: [<ffffffff886fb6aa>] :gfs2:gfs2_glock_nq_num+0x3b/0x68 May 25 18:01:18 xentest kernel: [<ffffffff8003a55a>] vfs_create+0xe6/0x158 May 25 18:01:18 xentest kernel: [<ffffffff8001b04d>] open_namei+0x19d/0x6d5 May 25 18:01:18 xentest kernel: [<ffffffff800274a4>] do_filp_open+0x1c/0x38 May 25 18:01:18 xentest kernel: [<ffffffff80019dd1>] do_sys_open+0x44/0xbe May 25 18:01:18 xentest kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0 May 25 18:01:18 xentest kernel: May 25 18:01:18 xentest kernel: May 25 18:01:18 xentest kernel: Code: 0f 0b 68 06 bb 2b 80 c2 1a 00 48 8b 55 00 48 39 da 74 1b 48 May 25 18:01:18 xentest kernel: RIP [<ffffffff80154004>] __list_add+0x24/0x68 May 25 18:01:18 xentest kernel: RSP <ffff8101afe11b58> May 25 18:01:28 xentest kernel: <3>BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151] May 25 18:01:28 xentest kernel: CPU 4: May 25 18:01:28 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U) May 25 18:01:28 xentest kernel: Pid: 4151, comm: multipathd Tainted: G 2.6.18-194.3.1.el5 #1 May 25 18:01:28 xentest kernel: RIP: 0010:[<ffffffff8007665f>] [<ffffffff8007665f>] __smp_call_function_many+0x9c/0xbc May 25 18:01:28 xentest kernel: RSP: 0018:ffff810218c8fbf8 EFLAGS: 00000293 May 25 18:01:28 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000 May 25 18:01:28 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0 May 25 18:01:28 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038 May 25 18:01:28 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000 May 25 18:01:28 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd May 25 18:01:28 xentest kernel: FS: 0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000 May 25 18:01:28 xentest kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 25 18:01:28 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0 May 25 18:01:28 xentest kernel: May 25 18:01:28 xentest kernel: Call Trace: May 25 18:01:28 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:01:28 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:01:28 xentest kernel: [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c May 25 18:01:28 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:01:28 xentest kernel: [<ffffffff8007684c>] smp_call_function+0x4e/0x5e May 25 18:01:28 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:01:28 xentest kernel: [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38 May 25 18:01:28 xentest kernel: [<ffffffff800947db>] on_each_cpu+0x10/0x22 May 25 18:01:28 xentest kernel: [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42 May 25 18:01:28 xentest kernel: [<ffffffff800d0f60>] remove_vm_area+0x18/0x25 May 25 18:01:28 xentest kernel: [<ffffffff800d0fb4>] __vunmap+0x47/0xed May 25 18:01:28 xentest kernel: [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b May 25 18:01:28 xentest kernel: [<ffffffff80042175>] do_ioctl+0x55/0x6b May 25 18:01:28 xentest kernel: [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9 May 25 18:01:28 xentest kernel: [<ffffffff8000d3ba>] dput+0x2c/0x114 May 25 18:01:28 xentest kernel: [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22 May 25 18:01:28 xentest kernel: [<ffffffff8004c870>] sys_ioctl+0x59/0x78 May 25 18:01:28 xentest kernel: [<ffffffff8005d116>] system_call+0x7e/0x83 May 25 18:01:28 xentest kernel: May 25 18:01:38 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151] May 25 18:01:38 xentest kernel: CPU 4: May 25 18:01:38 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U) May 25 18:01:38 xentest kernel: Pid: 4151, comm: multipathd Tainted: G 2.6.18-194.3.1.el5 #1 May 25 18:01:39 xentest kernel: RIP: 0010:[<ffffffff80076659>] [<ffffffff80076659>] __smp_call_function_many+0x96/0xbc May 25 18:01:39 xentest kernel: RSP: 0018:ffff810218c8fbf8 EFLAGS: 00000293 May 25 18:01:39 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000 May 25 18:01:39 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0 May 25 18:01:39 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038 May 25 18:01:39 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000 May 25 18:01:39 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd May 25 18:01:39 xentest kernel: FS: 0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000 May 25 18:01:39 xentest kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 25 18:01:39 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0 May 25 18:01:39 xentest kernel: May 25 18:01:39 xentest kernel: Call Trace: May 25 18:01:39 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:01:39 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:01:39 xentest kernel: [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c May 25 18:01:39 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:01:39 xentest kernel: [<ffffffff8007684c>] smp_call_function+0x4e/0x5e May 25 18:01:39 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:01:39 xentest kernel: [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38 May 25 18:01:39 xentest kernel: [<ffffffff800947db>] on_each_cpu+0x10/0x22 May 25 18:01:39 xentest kernel: [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42 May 25 18:01:39 xentest kernel: [<ffffffff800d0f60>] remove_vm_area+0x18/0x25 May 25 18:01:39 xentest kernel: [<ffffffff800d0fb4>] __vunmap+0x47/0xed May 25 18:01:39 xentest kernel: [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b May 25 18:01:39 xentest kernel: [<ffffffff80042175>] do_ioctl+0x55/0x6b May 25 18:01:39 xentest kernel: [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9 May 25 18:01:39 xentest kernel: [<ffffffff8000d3ba>] dput+0x2c/0x114 May 25 18:01:39 xentest kernel: [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22 May 25 18:01:39 xentest kernel: [<ffffffff8004c870>] sys_ioctl+0x59/0x78 May 25 18:01:39 xentest kernel: [<ffffffff8005d116>] system_call+0x7e/0x83 May 25 18:01:39 xentest kernel: May 25 18:01:49 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151] May 25 18:01:49 xentest kernel: CPU 4: May 25 18:01:49 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U) May 25 18:01:49 xentest kernel: Pid: 4151, comm: multipathd Tainted: G 2.6.18-194.3.1.el5 #1 May 25 18:01:49 xentest kernel: RIP: 0010:[<ffffffff8007665d>] [<ffffffff8007665d>] __smp_call_function_many+0x9a/0xbc May 25 18:01:49 xentest kernel: RSP: 0018:ffff810218c8fbf8 EFLAGS: 00000293 May 25 18:01:49 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000 May 25 18:01:49 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0 May 25 18:01:49 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038 May 25 18:01:49 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000 May 25 18:01:49 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd May 25 18:01:49 xentest kernel: FS: 0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000 May 25 18:01:49 xentest kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 25 18:01:50 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0 May 25 18:01:50 xentest kernel: May 25 18:01:50 xentest kernel: Call Trace: May 25 18:01:50 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:01:50 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:01:50 xentest kernel: [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c May 25 18:01:50 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:01:50 xentest kernel: [<ffffffff8007684c>] smp_call_function+0x4e/0x5e May 25 18:01:50 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:01:50 xentest kernel: [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38 May 25 18:01:50 xentest kernel: [<ffffffff800947db>] on_each_cpu+0x10/0x22 May 25 18:01:50 xentest kernel: [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42 May 25 18:01:50 xentest kernel: [<ffffffff800d0f60>] remove_vm_area+0x18/0x25 May 25 18:01:50 xentest kernel: [<ffffffff800d0fb4>] __vunmap+0x47/0xed May 25 18:01:50 xentest kernel: [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b May 25 18:01:50 xentest kernel: [<ffffffff80042175>] do_ioctl+0x55/0x6b May 25 18:01:50 xentest kernel: [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9 May 25 18:01:50 xentest kernel: [<ffffffff8000d3ba>] dput+0x2c/0x114 May 25 18:01:50 xentest kernel: [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22 May 25 18:01:50 xentest kernel: [<ffffffff8004c870>] sys_ioctl+0x59/0x78 May 25 18:01:50 xentest kernel: [<ffffffff8005d116>] system_call+0x7e/0x83 May 25 18:01:50 xentest kernel: May 25 18:02:00 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151] May 25 18:02:00 xentest kernel: CPU 4: May 25 18:02:00 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U) May 25 18:02:00 xentest kernel: Pid: 4151, comm: multipathd Tainted: G 2.6.18-194.3.1.el5 #1 May 25 18:02:00 xentest kernel: RIP: 0010:[<ffffffff80076659>] [<ffffffff80076659>] __smp_call_function_many+0x96/0xbc May 25 18:02:00 xentest kernel: RSP: 0018:ffff810218c8fbf8 EFLAGS: 00000293 May 25 18:02:00 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000 May 25 18:02:00 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0 May 25 18:02:00 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038 May 25 18:02:00 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000 May 25 18:02:00 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd May 25 18:02:00 xentest kernel: FS: 0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000 May 25 18:02:00 xentest kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 25 18:02:01 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0 May 25 18:02:01 xentest kernel: May 25 18:02:01 xentest kernel: Call Trace: May 25 18:02:01 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:01 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:01 xentest kernel: [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c May 25 18:02:01 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:01 xentest kernel: [<ffffffff8007684c>] smp_call_function+0x4e/0x5e May 25 18:02:01 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:01 xentest kernel: [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38 May 25 18:02:01 xentest kernel: [<ffffffff800947db>] on_each_cpu+0x10/0x22 May 25 18:02:01 xentest kernel: [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42 May 25 18:02:01 xentest kernel: [<ffffffff800d0f60>] remove_vm_area+0x18/0x25 May 25 18:02:01 xentest kernel: [<ffffffff800d0fb4>] __vunmap+0x47/0xed May 25 18:02:01 xentest kernel: [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b May 25 18:02:01 xentest kernel: [<ffffffff80042175>] do_ioctl+0x55/0x6b May 25 18:02:01 xentest kernel: [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9 May 25 18:02:01 xentest kernel: [<ffffffff8000d3ba>] dput+0x2c/0x114 May 25 18:02:01 xentest kernel: [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22 May 25 18:02:01 xentest kernel: [<ffffffff8004c870>] sys_ioctl+0x59/0x78 May 25 18:02:01 xentest kernel: [<ffffffff8005d116>] system_call+0x7e/0x83 May 25 18:02:01 xentest kernel: May 25 18:02:11 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151] May 25 18:02:11 xentest kernel: CPU 4: May 25 18:02:11 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U) May 25 18:02:11 xentest kernel: Pid: 4151, comm: multipathd Tainted: G 2.6.18-194.3.1.el5 #1 May 25 18:02:11 xentest kernel: RIP: 0010:[<ffffffff80076659>] [<ffffffff80076659>] __smp_call_function_many+0x96/0xbc May 25 18:02:11 xentest kernel: RSP: 0018:ffff810218c8fbf8 EFLAGS: 00000293 May 25 18:02:11 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000 May 25 18:02:11 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0 May 25 18:02:11 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038 May 25 18:02:11 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000 May 25 18:02:11 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd May 25 18:02:11 xentest kernel: FS: 0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000 May 25 18:02:11 xentest kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 25 18:02:12 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0 May 25 18:02:12 xentest kernel: May 25 18:02:12 xentest kernel: Call Trace: May 25 18:02:12 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:12 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:12 xentest kernel: [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c May 25 18:02:12 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:12 xentest kernel: [<ffffffff8007684c>] smp_call_function+0x4e/0x5e May 25 18:02:12 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:12 xentest kernel: [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38 May 25 18:02:12 xentest kernel: [<ffffffff800947db>] on_each_cpu+0x10/0x22 May 25 18:02:12 xentest kernel: [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42 May 25 18:02:12 xentest kernel: [<ffffffff800d0f60>] remove_vm_area+0x18/0x25 May 25 18:02:12 xentest kernel: [<ffffffff800d0fb4>] __vunmap+0x47/0xed May 25 18:02:12 xentest kernel: [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b May 25 18:02:12 xentest kernel: [<ffffffff80042175>] do_ioctl+0x55/0x6b May 25 18:02:12 xentest kernel: [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9 May 25 18:02:12 xentest kernel: [<ffffffff8000d3ba>] dput+0x2c/0x114 May 25 18:02:12 xentest kernel: [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22 May 25 18:02:12 xentest kernel: [<ffffffff8004c870>] sys_ioctl+0x59/0x78 May 25 18:02:12 xentest kernel: [<ffffffff8005d116>] system_call+0x7e/0x83 May 25 18:02:12 xentest kernel: May 25 18:02:22 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151] May 25 18:02:22 xentest kernel: CPU 4: May 25 18:02:22 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U) May 25 18:02:22 xentest kernel: Pid: 4151, comm: multipathd Tainted: G 2.6.18-194.3.1.el5 #1 May 25 18:02:22 xentest kernel: RIP: 0010:[<ffffffff80076659>] [<ffffffff80076659>] __smp_call_function_many+0x96/0xbc May 25 18:02:22 xentest kernel: RSP: 0018:ffff810218c8fbf8 EFLAGS: 00000293 May 25 18:02:22 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000 May 25 18:02:22 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0 May 25 18:02:22 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038 May 25 18:02:22 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000 May 25 18:02:22 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd May 25 18:02:22 xentest kernel: FS: 0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000 May 25 18:02:23 xentest kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 25 18:02:23 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0 May 25 18:02:23 xentest kernel: May 25 18:02:23 xentest kernel: Call Trace: May 25 18:02:23 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:23 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:23 xentest kernel: [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c May 25 18:02:23 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:23 xentest kernel: [<ffffffff8007684c>] smp_call_function+0x4e/0x5e May 25 18:02:23 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:23 xentest kernel: [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38 May 25 18:02:23 xentest kernel: [<ffffffff800947db>] on_each_cpu+0x10/0x22 May 25 18:02:23 xentest kernel: [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42 May 25 18:02:23 xentest kernel: [<ffffffff800d0f60>] remove_vm_area+0x18/0x25 May 25 18:02:23 xentest kernel: [<ffffffff800d0fb4>] __vunmap+0x47/0xed May 25 18:02:23 xentest kernel: [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b May 25 18:02:23 xentest kernel: [<ffffffff80042175>] do_ioctl+0x55/0x6b May 25 18:02:23 xentest kernel: [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9 May 25 18:02:23 xentest kernel: [<ffffffff8000d3ba>] dput+0x2c/0x114 May 25 18:02:23 xentest kernel: [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22 May 25 18:02:23 xentest kernel: [<ffffffff8004c870>] sys_ioctl+0x59/0x78 May 25 18:02:23 xentest kernel: [<ffffffff8005d116>] system_call+0x7e/0x83 May 25 18:02:23 xentest kernel: May 25 18:02:33 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151] May 25 18:02:33 xentest kernel: CPU 4: May 25 18:02:33 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U) May 25 18:02:33 xentest kernel: Pid: 4151, comm: multipathd Tainted: G 2.6.18-194.3.1.el5 #1 May 25 18:02:33 xentest kernel: RIP: 0010:[<ffffffff8007665d>] [<ffffffff8007665d>] __smp_call_function_many+0x9a/0xbc May 25 18:02:33 xentest kernel: RSP: 0018:ffff810218c8fbf8 EFLAGS: 00000293 May 25 18:02:33 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000 May 25 18:02:33 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0 May 25 18:02:33 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038 May 25 18:02:33 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000 May 25 18:02:33 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd May 25 18:02:33 xentest kernel: FS: 0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000 May 25 18:02:34 xentest kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 25 18:02:34 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0 May 25 18:02:34 xentest kernel: May 25 18:02:34 xentest kernel: Call Trace: May 25 18:02:34 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:34 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:34 xentest kernel: [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c May 25 18:02:34 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:34 xentest kernel: [<ffffffff8007684c>] smp_call_function+0x4e/0x5e May 25 18:02:34 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:34 xentest kernel: [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38 May 25 18:02:34 xentest kernel: [<ffffffff800947db>] on_each_cpu+0x10/0x22 May 25 18:02:34 xentest kernel: [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42 May 25 18:02:34 xentest kernel: [<ffffffff800d0f60>] remove_vm_area+0x18/0x25 May 25 18:02:34 xentest kernel: [<ffffffff800d0fb4>] __vunmap+0x47/0xed May 25 18:02:34 xentest kernel: [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b May 25 18:02:34 xentest kernel: [<ffffffff80042175>] do_ioctl+0x55/0x6b May 25 18:02:34 xentest kernel: [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9 May 25 18:02:34 xentest kernel: [<ffffffff8000d3ba>] dput+0x2c/0x114 May 25 18:02:34 xentest kernel: [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22 May 25 18:02:34 xentest kernel: [<ffffffff8004c870>] sys_ioctl+0x59/0x78 May 25 18:02:34 xentest kernel: [<ffffffff8005d116>] system_call+0x7e/0x83 May 25 18:02:34 xentest kernel: May 25 18:02:46 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151] May 25 18:02:46 xentest kernel: CPU 4: May 25 18:02:47 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U) May 25 18:02:47 xentest kernel: Pid: 4151, comm: multipathd Tainted: G 2.6.18-194.3.1.el5 #1 May 25 18:02:47 xentest kernel: RIP: 0010:[<ffffffff8007665f>] [<ffffffff8007665f>] __smp_call_function_many+0x9c/0xbc May 25 18:02:47 xentest kernel: RSP: 0018:ffff810218c8fbf8 EFLAGS: 00000293 May 25 18:02:47 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000 May 25 18:02:47 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0 May 25 18:02:47 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038 May 25 18:02:47 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000 May 25 18:02:47 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd May 25 18:02:47 xentest kernel: FS: 0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000 May 25 18:02:47 xentest kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 25 18:02:47 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0 May 25 18:02:47 xentest kernel: May 25 18:02:47 xentest kernel: Call Trace: May 25 18:02:47 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:47 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:47 xentest kernel: [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c May 25 18:02:47 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:47 xentest kernel: [<ffffffff8007684c>] smp_call_function+0x4e/0x5e May 25 18:02:47 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:47 xentest kernel: [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38 May 25 18:02:47 xentest kernel: [<ffffffff800947db>] on_each_cpu+0x10/0x22 May 25 18:02:47 xentest kernel: [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42 May 25 18:02:47 xentest kernel: [<ffffffff800d0f60>] remove_vm_area+0x18/0x25 May 25 18:02:47 xentest kernel: [<ffffffff800d0fb4>] __vunmap+0x47/0xed May 25 18:02:47 xentest kernel: [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b May 25 18:02:47 xentest kernel: [<ffffffff80042175>] do_ioctl+0x55/0x6b May 25 18:02:47 xentest kernel: [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9 May 25 18:02:47 xentest kernel: [<ffffffff8000d3ba>] dput+0x2c/0x114 May 25 18:02:47 xentest kernel: [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22 May 25 18:02:47 xentest kernel: [<ffffffff8004c870>] sys_ioctl+0x59/0x78 May 25 18:02:47 xentest kernel: [<ffffffff8005d116>] system_call+0x7e/0x83 May 25 18:02:47 xentest kernel: May 25 18:02:57 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151] May 25 18:02:57 xentest kernel: CPU 4: May 25 18:02:58 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U) May 25 18:02:58 xentest kernel: Pid: 4151, comm: multipathd Tainted: G 2.6.18-194.3.1.el5 #1 May 25 18:02:58 xentest kernel: RIP: 0010:[<ffffffff8007665f>] [<ffffffff8007665f>] __smp_call_function_many+0x9c/0xbc May 25 18:02:58 xentest kernel: RSP: 0018:ffff810218c8fbf8 EFLAGS: 00000293 May 25 18:02:58 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000 May 25 18:02:58 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0 May 25 18:02:58 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038 May 25 18:02:58 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000 May 25 18:02:58 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd May 25 18:02:58 xentest kernel: FS: 0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000 May 25 18:02:58 xentest kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 25 18:02:58 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0 May 25 18:02:58 xentest kernel: May 25 18:02:58 xentest kernel: Call Trace: May 25 18:02:58 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:58 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:58 xentest kernel: [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c May 25 18:02:58 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:58 xentest kernel: [<ffffffff8007684c>] smp_call_function+0x4e/0x5e May 25 18:02:58 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:02:58 xentest kernel: [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38 May 25 18:02:58 xentest kernel: [<ffffffff800947db>] on_each_cpu+0x10/0x22 May 25 18:02:58 xentest kernel: [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42 May 25 18:02:58 xentest kernel: [<ffffffff800d0f60>] remove_vm_area+0x18/0x25 May 25 18:02:58 xentest kernel: [<ffffffff800d0fb4>] __vunmap+0x47/0xed May 25 18:02:58 xentest kernel: [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b May 25 18:02:58 xentest kernel: [<ffffffff80042175>] do_ioctl+0x55/0x6b May 25 18:02:58 xentest kernel: [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9 May 25 18:02:58 xentest kernel: [<ffffffff8000d3ba>] dput+0x2c/0x114 May 25 18:02:58 xentest kernel: [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22 May 25 18:02:58 xentest kernel: [<ffffffff8004c870>] sys_ioctl+0x59/0x78 May 25 18:02:58 xentest kernel: [<ffffffff8005d116>] system_call+0x7e/0x83 May 25 18:02:58 xentest kernel: May 25 18:03:09 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151] May 25 18:03:09 xentest kernel: CPU 4: May 25 18:03:09 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U) May 25 18:03:09 xentest kernel: Pid: 4151, comm: multipathd Tainted: G 2.6.18-194.3.1.el5 #1 May 25 18:03:09 xentest kernel: RIP: 0010:[<ffffffff80076659>] [<ffffffff80076659>] __smp_call_function_many+0x96/0xbc May 25 18:03:09 xentest kernel: RSP: 0018:ffff810218c8fbf8 EFLAGS: 00000293 May 25 18:03:09 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000 May 25 18:03:09 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0 May 25 18:03:09 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038 May 25 18:03:09 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000 May 25 18:03:09 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd May 25 18:03:09 xentest kernel: FS: 0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000 May 25 18:03:09 xentest kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 25 18:03:10 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0 May 25 18:03:10 xentest kernel: May 25 18:03:10 xentest kernel: Call Trace: May 25 18:03:10 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:03:10 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:03:10 xentest kernel: [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c May 25 18:03:10 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:03:10 xentest kernel: [<ffffffff8007684c>] smp_call_function+0x4e/0x5e May 25 18:03:10 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:03:10 xentest kernel: [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38 May 25 18:03:10 xentest kernel: [<ffffffff800947db>] on_each_cpu+0x10/0x22 May 25 18:03:10 xentest kernel: [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42 May 25 18:03:10 xentest kernel: [<ffffffff800d0f60>] remove_vm_area+0x18/0x25 May 25 18:03:10 xentest kernel: [<ffffffff800d0fb4>] __vunmap+0x47/0xed May 25 18:03:10 xentest kernel: [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b May 25 18:03:10 xentest kernel: [<ffffffff80042175>] do_ioctl+0x55/0x6b May 25 18:03:10 xentest kernel: [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9 May 25 18:03:10 xentest kernel: [<ffffffff8000d3ba>] dput+0x2c/0x114 May 25 18:03:10 xentest kernel: [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22 May 25 18:03:10 xentest kernel: [<ffffffff8004c870>] sys_ioctl+0x59/0x78 May 25 18:03:10 xentest kernel: [<ffffffff8005d116>] system_call+0x7e/0x83 May 25 18:03:10 xentest kernel: May 25 18:03:24 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151] May 25 18:03:24 xentest kernel: CPU 4: May 25 18:03:24 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U) May 25 18:03:24 xentest kernel: Pid: 4151, comm: multipathd Tainted: G 2.6.18-194.3.1.el5 #1 May 25 18:03:24 xentest kernel: RIP: 0010:[<ffffffff80076659>] [<ffffffff80076659>] __smp_call_function_many+0x96/0xbc May 25 18:03:24 xentest kernel: RSP: 0018:ffff810218c8fbf8 EFLAGS: 00000293 May 25 18:03:25 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000 May 25 18:03:25 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0 May 25 18:03:25 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038 May 25 18:03:25 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000 May 25 18:03:25 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd May 25 18:03:25 xentest kernel: FS: 0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000 May 25 18:03:25 xentest kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 25 18:03:25 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0 May 25 18:03:25 xentest kernel: May 25 18:03:25 xentest kernel: Call Trace: May 25 18:03:25 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:03:25 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:03:25 xentest kernel: [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c May 25 18:03:25 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:03:25 xentest kernel: [<ffffffff8007684c>] smp_call_function+0x4e/0x5e May 25 18:03:25 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:03:25 xentest kernel: [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38 May 25 18:03:25 xentest kernel: [<ffffffff800947db>] on_each_cpu+0x10/0x22 May 25 18:03:25 xentest kernel: [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42 May 25 18:03:25 xentest kernel: [<ffffffff800d0f60>] remove_vm_area+0x18/0x25 May 25 18:03:25 xentest kernel: [<ffffffff800d0fb4>] __vunmap+0x47/0xed May 25 18:03:25 xentest kernel: [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b May 25 18:03:25 xentest kernel: [<ffffffff80042175>] do_ioctl+0x55/0x6b May 25 18:03:25 xentest kernel: [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9 May 25 18:03:25 xentest kernel: [<ffffffff8000d3ba>] dput+0x2c/0x114 May 25 18:03:25 xentest kernel: [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22 May 25 18:03:25 xentest kernel: [<ffffffff8004c870>] sys_ioctl+0x59/0x78 May 25 18:03:25 xentest kernel: [<ffffffff8005d116>] system_call+0x7e/0x83 May 25 18:03:25 xentest kernel: May 25 18:03:35 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151] May 25 18:03:35 xentest kernel: CPU 4: May 25 18:03:35 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U) May 25 18:03:35 xentest kernel: Pid: 4151, comm: multipathd Tainted: G 2.6.18-194.3.1.el5 #1 May 25 18:03:35 xentest kernel: RIP: 0010:[<ffffffff80076659>] [<ffffffff80076659>] __smp_call_function_many+0x96/0xbc May 25 18:03:35 xentest kernel: RSP: 0018:ffff810218c8fbf8 EFLAGS: 00000293 May 25 18:03:36 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000 May 25 18:03:36 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0 May 25 18:03:36 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038 May 25 18:03:36 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000 May 25 18:03:36 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd May 25 18:03:36 xentest kernel: FS: 0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000 May 25 18:03:36 xentest kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 25 18:03:36 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0 May 25 18:03:36 xentest kernel: May 25 18:03:36 xentest kernel: Call Trace: May 25 18:03:36 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:03:36 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:03:36 xentest kernel: [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c May 25 18:03:36 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:03:36 xentest kernel: [<ffffffff8007684c>] smp_call_function+0x4e/0x5e May 25 18:03:36 xentest kernel: [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a May 25 18:03:36 xentest kernel: [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38 May 25 18:03:36 xentest kernel: [<ffffffff800947db>] on_each_cpu+0x10/0x22 May 25 18:03:36 xentest kernel: [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42 May 25 18:03:36 xentest kernel: [<ffffffff800d0f60>] remove_vm_area+0x18/0x25 May 25 18:03:36 xentest kernel: [<ffffffff800d0fb4>] __vunmap+0x47/0xed May 25 18:03:36 xentest kernel: [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b May 25 18:03:36 xentest kernel: [<ffffffff80042175>] do_ioctl+0x55/0x6b May 25 18:03:36 xentest kernel: [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9 May 25 18:03:36 xentest kernel: [<ffffffff8000d3ba>] dput+0x2c/0x114 May 25 18:03:36 xentest kernel: [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22 May 25 18:03:36 xentest kernel: [<ffffffff8004c870>] sys_ioctl+0x59/0x78 May 25 18:03:36 xentest kernel: [<ffffffff8005d116>] system_call+0x7e/0x83 May 25 18:03:36 xentest kernel: May 25 18:07:25 xentest syslogd 1.4.1: restart. This looks nothing like the last report, and much more like a gfs2 issue. Are you using quotas? Can you paste your /proc/mounts here? Yes I am mounting my GFS2 disks with quota option enabled. Here is my fstab and mounts. [root@xentest ~]# cat /proc/mounts rootfs / rootfs rw 0 0 /dev/root / ext3 rw,data=ordered 0 0 /dev /dev tmpfs rw 0 0 /proc /proc proc rw 0 0 /sys /sys sysfs rw 0 0 /proc/bus/usb /proc/bus/usb usbfs rw 0 0 devpts /dev/pts devpts rw 0 0 tmpfs /dev/shm tmpfs rw 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0 none /sys/kernel/config configfs rw 0 0 /dev/mapper/akademik1 /users/akademik1 gfs2 rw,hostdata=jid=0:id=1114116:first=1,quota=on 0 0 /dev/mapper/akademik2 /users/akademik2 gfs2 rw,hostdata=jid=0:id=1245188:first=1,quota=on 0 0 /dev/mapper/akademik3 /users/akademik3 gfs2 rw,hostdata=jid=0:id=1376260:first=1,quota=on 0 0 /dev/mapper/akademik4 /users/akademik4 gfs2 rw,hostdata=jid=0:id=1507332:first=1,quota=on 0 0 /dev/mapper/idari1 /users/idari1 gfs2 rw,hostdata=jid=0:id=1638404:first=1,quota=on 0 0 /dev/mapper/ogrenci1 /users/ogrenci1 gfs2 rw,hostdata=jid=0:id=1769476:first=1,quota=on 0 0 /dev/mapper/ogrenci2 /users/ogrenci2 gfs2 rw,hostdata=jid=0:id=1900548:first=1,quota=on 0 0 /dev/mapper/ogrenci3 /users/ogrenci3 gfs2 rw,hostdata=jid=0:id=2031620:first=1,quota=on 0 0 /dev/mapper/ogrenci4 /users/ogrenci4 gfs2 rw,hostdata=jid=0:id=2162692:first=1,quota=on 0 0 /dev/mapper/ogrenci5 /users/ogrenci5 gfs2 rw,hostdata=jid=0:id=2293764:first=1,quota=on 0 0 /dev/mapper/ogrenci6 /users/ogrenci6 gfs2 rw,hostdata=jid=0:id=2424836:first=1,quota=on 0 0 /dev/mapper/sk1 /users/sk1 gfs2 rw,hostdata=jid=0:id=2555908:first=1,quota=on 0 0 /dev/mapper/diger1 /users/diger1 gfs2 rw,hostdata=jid=0:id=2686980:first=1,quota=on 0 0 [root@xentest ~]# By the way after every crash I check my GFS2 disks with fsck.gfs2 and recover problems. If you turn quota off does it work correctly? If so that is almost certainly a known issue for which we have a fix. The problems that you see reported by fsck are unlikely to be serious and are most likely just a consequence of an unclean shutdown. If you are worried post the results here and we can take a look at them. You are right Steve it worked perfect without quota enabled. My 5 day fight finally end :) Where could I find the fix. It is in kernel 2.6.18-194.4.1.el5. I'll close this as a dup of the 5.5.z bug so that you can see the full details. *** This bug has been marked as a duplicate of bug 586008 *** |
Created attachment 416320 [details] SUN ILO Screen While Kernel Panics Description of problem: I have a SUN X4150 server which has two Qlogic QLE2460 HBA card. My server is connected to EMC CX4-480 Storage over Fiber Channel. I got 13 Thin Luns each size is 5TB. My cluster has 5 node but only one is using GFS2 disks while I am testing. This is my test environment with GFS2 and I want to copy my production data to 13 GFS2 disks. After I start copying my data (most of is small files <5K), kernel gives a panic message and crashes. I tried copying to only one disk, using iSCSI instead of FC, changed Qlogic multipath to dm-mpio but all had the same crash. Version-Release number of selected component (if applicable): I am using Red Hat 5.5 and all of my components (including the server ILO firmware and HBA firmwares) are latest version. How reproducible: I cope with this problem for 5 days. However I got only one FC connected machine so I couldn't try this on another machine. By the way I had used this machine for storage testing and it worked perfect. Steps to Reproduce: 1. Start cman service and connect to cluster. 2. Mount 13 GFS2 disks. 3. Start cp process in background to copy production data to GFS2 disks. Actual results: Kernel panics Expected results: Copying files Additional info: This machine is my old XEN test machine but I am using regular kernel instead of xen kernel in this test. I can give any data if required.