Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 595630 - GFS2 - Kernel Panics while file copying
GFS2 - Kernel Panics while file copying
Status: CLOSED DUPLICATE of bug 586008
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel (Show other bugs)
5.5
x86_64 Linux
low Severity high
: ---
: ---
Assigned To: Steve Whitehouse
Cluster QE
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-05-25 04:47 EDT by veliogluh
Modified: 2010-05-26 04:29 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-05-26 04:29:21 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
SUN ILO Screen While Kernel Panics (195.16 KB, image/jpeg)
2010-05-25 04:47 EDT, veliogluh
no flags Details
SUN ILO Screen While Kernel Panics Second Screen (198.31 KB, image/jpeg)
2010-05-25 04:48 EDT, veliogluh
no flags Details

  None (edit)
Description veliogluh 2010-05-25 04:47:17 EDT
Created attachment 416320 [details]
SUN ILO Screen While Kernel Panics

Description of problem:
I have a SUN X4150 server which has two Qlogic QLE2460 HBA card. My server is connected to EMC CX4-480 Storage over Fiber Channel. I got 13 Thin Luns each size is 5TB. My cluster has 5 node but only one is using GFS2 disks while I am testing.

This is my test environment with GFS2 and I want to copy my production data to 13 GFS2 disks. After I start copying my data (most of is small files <5K), kernel gives a panic message and crashes.

I tried copying to only one disk, using iSCSI instead of FC, changed Qlogic multipath to dm-mpio but all had the same crash.


Version-Release number of selected component (if applicable):
I am using Red Hat 5.5 and all of my components (including the server ILO firmware and HBA firmwares) are latest version.

How reproducible:
I cope with this problem for 5 days.
However I got only one FC connected machine so I couldn't try this on another machine. By the way I had used this machine for storage testing and it worked perfect.

Steps to Reproduce:
1. Start cman service and connect to cluster.
2. Mount 13 GFS2 disks.
3. Start cp process in background to copy  production data to GFS2 disks.
  
Actual results:
Kernel panics

Expected results:
Copying files

Additional info:
This machine is my old XEN test machine but I am using regular kernel instead of xen kernel in this test.

I can give any data if required.
Comment 1 veliogluh 2010-05-25 04:48:20 EDT
Created attachment 416321 [details]
SUN ILO Screen While Kernel Panics Second Screen
Comment 2 veliogluh 2010-05-25 06:36:33 EDT
I copy a small protion of my production data to another node with NFS. Then I only mount all 13 GFS2 disks to new node which is using iSCSI SAN. When I start file copy same kernel panic occured. This hardware is an old HP DL 580 G2 machine. I think this problem may not be an hardware related problem.
Comment 3 Robert Peterson 2010-05-25 09:20:27 EDT
No GFS2 functions appear in the call trace, so I have a hard
time believing GFS2 is at fault here.  I recommend you temporarily
change your /etc/sysctl.conf file to include the following options:

kernel.sysrq = 1
kernel.printk = 7
kernel.panic_on_oops = 0

and then reboot, and recreate the scenario that fails.
Then see if you can get all of the kernel messages with the dmesg
command.  If that doesn't work and the kernel still panics, you
should redirect kernel messages to a serial port and attach a
system to monitor that port, to capture the complete console output,
then attach that output to the bugzilla record.  Thanks.
Comment 4 Steve Whitehouse 2010-05-25 09:34:13 EDT
Looks like it might be either dm or the underlying device driver to me.
Comment 5 veliogluh 2010-05-25 11:22:11 EDT
I set your kernel settings and this time I can get kernel messges from /var/log/messages. When I checked all my kernel panic messages I realized that all of them starts with same message
list_add corruption. next->prev should be ffff810211426ed0, but was 0000000000000000
What does this means?

Here are the syslog messages when the problem starts:

May 25 18:01:13 xentest kernel: list_add corruption. next->prev should be ffff810211426ed0, but was 0000000000000000
May 25 18:01:13 xentest kernel: ----------- [cut here ] --------- [please bite here ] ---------
May 25 18:01:18 xentest kernel: Kernel BUG at lib/list_debug.c:26
May 25 18:01:18 xentest kernel: invalid opcode: 0000 [1] SMP
May 25 18:01:18 xentest kernel: last sysfs file: /devices/pci0000:00/0000:00:02.0/0000:01:00.0/0000:02:00.0/0000:03:00.0/host0/target0:0:1/0:0:1:12/timeout
May 25 18:01:18 xentest kernel: CPU 4
May 25 18:01:18 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U)
May 25 18:01:18 xentest kernel: Pid: 14910, comm: cp Tainted: G      2.6.18-194.3.1.el5 #1
May 25 18:01:18 xentest kernel: RIP: 0010:[<ffffffff80154004>]  [<ffffffff80154004>] __list_add+0x24/0x68
May 25 18:01:18 xentest kernel: RSP: 0018:ffff8101afe11b58  EFLAGS: 00010082
May 25 18:01:18 xentest kernel: RAX: 0000000000000058 RBX: ffff81008469b000 RCX: ffffffff80311da8
May 25 18:01:18 xentest kernel: RDX: ffffffff80311da8 RSI: 0000000000000000 RDI: ffffffff80311da0
May 25 18:01:18 xentest kernel: RBP: ffff810211426ed0 R08: ffffffff80311da8 R09: 0000000000000039
May 25 18:01:18 xentest kernel: R10: ffff8101afe117f8 R11: 0000000000000180 R12: ffff810072f5d000
May 25 18:01:18 xentest kernel: R13: ffff810211426ec0 R14: 0000000000000014 R15: ffff8101ad2701c0
May 25 18:01:18 xentest kernel: FS:  00002ae4d4b77f80(0000) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000
May 25 18:01:18 xentest kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
May 25 18:01:18 xentest kernel: CR2: 0000003572292215 CR3: 000000020daee000 CR4: 00000000000006e0
May 25 18:01:18 xentest kernel: Process cp (pid: 14910, threadinfo ffff8101afe10000, task ffff81021e483100)
May 25 18:01:18 xentest kernel: Stack:  0000000000000028 ffff810072f5d000 ffff81021d833c00 ffffffff8005c382
May 25 18:01:18 xentest kernel:  0000085000000000 0000000000000246 0000000000000850 ffff8101ad2701c0
May 25 18:01:18 xentest kernel:  0000000000000001 ffff81010a52b000 00000000000022d8 ffffffff800db8fa
May 25 18:01:18 xentest kernel: Call Trace:
May 25 18:01:18 xentest kernel:  [<ffffffff8005c382>] cache_alloc_refill+0x106/0x186
May 25 18:01:18 xentest kernel:  [<ffffffff800db8fa>] kmem_cache_zalloc+0x6f/0x94
May 25 18:01:18 xentest kernel:  [<ffffffff8871171f>] :gfs2:gfs2_trans_add_bh+0x84/0x104
May 25 18:01:18 xentest kernel:  [<ffffffff886fdba4>] :gfs2:gfs2_createi+0x6ea/0xd28
May 25 18:01:18 xentest kernel:  [<ffffffff886f4be6>] :gfs2:gfs2_dirent_find+0x0/0x4e
May 25 18:01:18 xentest kernel:  [<ffffffff886f4be6>] :gfs2:gfs2_dirent_find+0x0/0x4e
May 25 18:01:18 xentest kernel:  [<ffffffff886f498b>] :gfs2:gfs2_dirent_search+0x147/0x16e
May 25 18:01:18 xentest kernel:  [<ffffffff886f9e7a>] :gfs2:gfs2_glock_schedule_for_reclaim+0x5d/0x9a
May 25 18:01:18 xentest kernel:  [<ffffffff8002d035>] wake_up_bit+0x11/0x22
May 25 18:01:18 xentest kernel:  [<ffffffff88708d59>] :gfs2:gfs2_create+0x65/0x143
May 25 18:01:18 xentest kernel:  [<ffffffff886fd51d>] :gfs2:gfs2_createi+0x63/0xd28
May 25 18:01:18 xentest kernel:  [<ffffffff886fb6aa>] :gfs2:gfs2_glock_nq_num+0x3b/0x68
May 25 18:01:18 xentest kernel:  [<ffffffff8003a55a>] vfs_create+0xe6/0x158
May 25 18:01:18 xentest kernel:  [<ffffffff8001b04d>] open_namei+0x19d/0x6d5
May 25 18:01:18 xentest kernel:  [<ffffffff800274a4>] do_filp_open+0x1c/0x38
May 25 18:01:18 xentest kernel:  [<ffffffff80019dd1>] do_sys_open+0x44/0xbe
May 25 18:01:18 xentest kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
May 25 18:01:18 xentest kernel:
May 25 18:01:18 xentest kernel:
May 25 18:01:18 xentest kernel: Code: 0f 0b 68 06 bb 2b 80 c2 1a 00 48 8b 55 00 48 39 da 74 1b 48
May 25 18:01:18 xentest kernel: RIP  [<ffffffff80154004>] __list_add+0x24/0x68
May 25 18:01:18 xentest kernel:  RSP <ffff8101afe11b58>
May 25 18:01:28 xentest kernel:  <3>BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151]
May 25 18:01:28 xentest kernel: CPU 4:
May 25 18:01:28 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U)
May 25 18:01:28 xentest kernel: Pid: 4151, comm: multipathd Tainted: G      2.6.18-194.3.1.el5 #1
May 25 18:01:28 xentest kernel: RIP: 0010:[<ffffffff8007665f>]  [<ffffffff8007665f>] __smp_call_function_many+0x9c/0xbc
May 25 18:01:28 xentest kernel: RSP: 0018:ffff810218c8fbf8  EFLAGS: 00000293
May 25 18:01:28 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000
May 25 18:01:28 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0
May 25 18:01:28 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038
May 25 18:01:28 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000
May 25 18:01:28 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd
May 25 18:01:28 xentest kernel: FS:  0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000
May 25 18:01:28 xentest kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 25 18:01:28 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0
May 25 18:01:28 xentest kernel:
May 25 18:01:28 xentest kernel: Call Trace:
May 25 18:01:28 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:01:28 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:01:28 xentest kernel:  [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c
May 25 18:01:28 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:01:28 xentest kernel:  [<ffffffff8007684c>] smp_call_function+0x4e/0x5e
May 25 18:01:28 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:01:28 xentest kernel:  [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38
May 25 18:01:28 xentest kernel:  [<ffffffff800947db>] on_each_cpu+0x10/0x22
May 25 18:01:28 xentest kernel:  [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42
May 25 18:01:28 xentest kernel:  [<ffffffff800d0f60>] remove_vm_area+0x18/0x25
May 25 18:01:28 xentest kernel:  [<ffffffff800d0fb4>] __vunmap+0x47/0xed
May 25 18:01:28 xentest kernel:  [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b
May 25 18:01:28 xentest kernel:  [<ffffffff80042175>] do_ioctl+0x55/0x6b
May 25 18:01:28 xentest kernel:  [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9
May 25 18:01:28 xentest kernel:  [<ffffffff8000d3ba>] dput+0x2c/0x114
May 25 18:01:28 xentest kernel:  [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22
May 25 18:01:28 xentest kernel:  [<ffffffff8004c870>] sys_ioctl+0x59/0x78
May 25 18:01:28 xentest kernel:  [<ffffffff8005d116>] system_call+0x7e/0x83
May 25 18:01:28 xentest kernel:
May 25 18:01:38 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151]
May 25 18:01:38 xentest kernel: CPU 4:
May 25 18:01:38 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U)
May 25 18:01:38 xentest kernel: Pid: 4151, comm: multipathd Tainted: G      2.6.18-194.3.1.el5 #1
May 25 18:01:39 xentest kernel: RIP: 0010:[<ffffffff80076659>]  [<ffffffff80076659>] __smp_call_function_many+0x96/0xbc
May 25 18:01:39 xentest kernel: RSP: 0018:ffff810218c8fbf8  EFLAGS: 00000293
May 25 18:01:39 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000
May 25 18:01:39 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0
May 25 18:01:39 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038
May 25 18:01:39 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000
May 25 18:01:39 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd
May 25 18:01:39 xentest kernel: FS:  0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000
May 25 18:01:39 xentest kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 25 18:01:39 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0
May 25 18:01:39 xentest kernel:
May 25 18:01:39 xentest kernel: Call Trace:
May 25 18:01:39 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:01:39 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:01:39 xentest kernel:  [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c
May 25 18:01:39 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:01:39 xentest kernel:  [<ffffffff8007684c>] smp_call_function+0x4e/0x5e
May 25 18:01:39 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:01:39 xentest kernel:  [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38
May 25 18:01:39 xentest kernel:  [<ffffffff800947db>] on_each_cpu+0x10/0x22
May 25 18:01:39 xentest kernel:  [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42
May 25 18:01:39 xentest kernel:  [<ffffffff800d0f60>] remove_vm_area+0x18/0x25
May 25 18:01:39 xentest kernel:  [<ffffffff800d0fb4>] __vunmap+0x47/0xed
May 25 18:01:39 xentest kernel:  [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b
May 25 18:01:39 xentest kernel:  [<ffffffff80042175>] do_ioctl+0x55/0x6b
May 25 18:01:39 xentest kernel:  [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9
May 25 18:01:39 xentest kernel:  [<ffffffff8000d3ba>] dput+0x2c/0x114
May 25 18:01:39 xentest kernel:  [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22
May 25 18:01:39 xentest kernel:  [<ffffffff8004c870>] sys_ioctl+0x59/0x78
May 25 18:01:39 xentest kernel:  [<ffffffff8005d116>] system_call+0x7e/0x83
May 25 18:01:39 xentest kernel:
May 25 18:01:49 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151]
May 25 18:01:49 xentest kernel: CPU 4:
May 25 18:01:49 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U)
May 25 18:01:49 xentest kernel: Pid: 4151, comm: multipathd Tainted: G      2.6.18-194.3.1.el5 #1
May 25 18:01:49 xentest kernel: RIP: 0010:[<ffffffff8007665d>]  [<ffffffff8007665d>] __smp_call_function_many+0x9a/0xbc
May 25 18:01:49 xentest kernel: RSP: 0018:ffff810218c8fbf8  EFLAGS: 00000293
May 25 18:01:49 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000
May 25 18:01:49 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0
May 25 18:01:49 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038
May 25 18:01:49 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000
May 25 18:01:49 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd
May 25 18:01:49 xentest kernel: FS:  0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000
May 25 18:01:49 xentest kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 25 18:01:50 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0
May 25 18:01:50 xentest kernel:
May 25 18:01:50 xentest kernel: Call Trace:
May 25 18:01:50 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:01:50 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:01:50 xentest kernel:  [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c
May 25 18:01:50 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:01:50 xentest kernel:  [<ffffffff8007684c>] smp_call_function+0x4e/0x5e
May 25 18:01:50 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:01:50 xentest kernel:  [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38
May 25 18:01:50 xentest kernel:  [<ffffffff800947db>] on_each_cpu+0x10/0x22
May 25 18:01:50 xentest kernel:  [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42
May 25 18:01:50 xentest kernel:  [<ffffffff800d0f60>] remove_vm_area+0x18/0x25
May 25 18:01:50 xentest kernel:  [<ffffffff800d0fb4>] __vunmap+0x47/0xed
May 25 18:01:50 xentest kernel:  [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b
May 25 18:01:50 xentest kernel:  [<ffffffff80042175>] do_ioctl+0x55/0x6b
May 25 18:01:50 xentest kernel:  [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9
May 25 18:01:50 xentest kernel:  [<ffffffff8000d3ba>] dput+0x2c/0x114
May 25 18:01:50 xentest kernel:  [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22
May 25 18:01:50 xentest kernel:  [<ffffffff8004c870>] sys_ioctl+0x59/0x78
May 25 18:01:50 xentest kernel:  [<ffffffff8005d116>] system_call+0x7e/0x83
May 25 18:01:50 xentest kernel:
May 25 18:02:00 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151]
May 25 18:02:00 xentest kernel: CPU 4:
May 25 18:02:00 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U)
May 25 18:02:00 xentest kernel: Pid: 4151, comm: multipathd Tainted: G      2.6.18-194.3.1.el5 #1
May 25 18:02:00 xentest kernel: RIP: 0010:[<ffffffff80076659>]  [<ffffffff80076659>] __smp_call_function_many+0x96/0xbc
May 25 18:02:00 xentest kernel: RSP: 0018:ffff810218c8fbf8  EFLAGS: 00000293
May 25 18:02:00 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000
May 25 18:02:00 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0
May 25 18:02:00 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038
May 25 18:02:00 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000
May 25 18:02:00 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd
May 25 18:02:00 xentest kernel: FS:  0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000
May 25 18:02:00 xentest kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 25 18:02:01 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0
May 25 18:02:01 xentest kernel:
May 25 18:02:01 xentest kernel: Call Trace:
May 25 18:02:01 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:01 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:01 xentest kernel:  [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c
May 25 18:02:01 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:01 xentest kernel:  [<ffffffff8007684c>] smp_call_function+0x4e/0x5e
May 25 18:02:01 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:01 xentest kernel:  [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38
May 25 18:02:01 xentest kernel:  [<ffffffff800947db>] on_each_cpu+0x10/0x22
May 25 18:02:01 xentest kernel:  [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42
May 25 18:02:01 xentest kernel:  [<ffffffff800d0f60>] remove_vm_area+0x18/0x25
May 25 18:02:01 xentest kernel:  [<ffffffff800d0fb4>] __vunmap+0x47/0xed
May 25 18:02:01 xentest kernel:  [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b
May 25 18:02:01 xentest kernel:  [<ffffffff80042175>] do_ioctl+0x55/0x6b
May 25 18:02:01 xentest kernel:  [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9
May 25 18:02:01 xentest kernel:  [<ffffffff8000d3ba>] dput+0x2c/0x114
May 25 18:02:01 xentest kernel:  [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22
May 25 18:02:01 xentest kernel:  [<ffffffff8004c870>] sys_ioctl+0x59/0x78
May 25 18:02:01 xentest kernel:  [<ffffffff8005d116>] system_call+0x7e/0x83
May 25 18:02:01 xentest kernel:
May 25 18:02:11 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151]
May 25 18:02:11 xentest kernel: CPU 4:
May 25 18:02:11 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U)
May 25 18:02:11 xentest kernel: Pid: 4151, comm: multipathd Tainted: G      2.6.18-194.3.1.el5 #1
May 25 18:02:11 xentest kernel: RIP: 0010:[<ffffffff80076659>]  [<ffffffff80076659>] __smp_call_function_many+0x96/0xbc
May 25 18:02:11 xentest kernel: RSP: 0018:ffff810218c8fbf8  EFLAGS: 00000293
May 25 18:02:11 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000
May 25 18:02:11 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0
May 25 18:02:11 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038
May 25 18:02:11 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000
May 25 18:02:11 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd
May 25 18:02:11 xentest kernel: FS:  0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000
May 25 18:02:11 xentest kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 25 18:02:12 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0
May 25 18:02:12 xentest kernel:
May 25 18:02:12 xentest kernel: Call Trace:
May 25 18:02:12 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:12 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:12 xentest kernel:  [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c
May 25 18:02:12 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:12 xentest kernel:  [<ffffffff8007684c>] smp_call_function+0x4e/0x5e
May 25 18:02:12 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:12 xentest kernel:  [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38
May 25 18:02:12 xentest kernel:  [<ffffffff800947db>] on_each_cpu+0x10/0x22
May 25 18:02:12 xentest kernel:  [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42
May 25 18:02:12 xentest kernel:  [<ffffffff800d0f60>] remove_vm_area+0x18/0x25
May 25 18:02:12 xentest kernel:  [<ffffffff800d0fb4>] __vunmap+0x47/0xed
May 25 18:02:12 xentest kernel:  [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b
May 25 18:02:12 xentest kernel:  [<ffffffff80042175>] do_ioctl+0x55/0x6b
May 25 18:02:12 xentest kernel:  [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9
May 25 18:02:12 xentest kernel:  [<ffffffff8000d3ba>] dput+0x2c/0x114
May 25 18:02:12 xentest kernel:  [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22
May 25 18:02:12 xentest kernel:  [<ffffffff8004c870>] sys_ioctl+0x59/0x78
May 25 18:02:12 xentest kernel:  [<ffffffff8005d116>] system_call+0x7e/0x83
May 25 18:02:12 xentest kernel:
May 25 18:02:22 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151]
May 25 18:02:22 xentest kernel: CPU 4:
May 25 18:02:22 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U)
May 25 18:02:22 xentest kernel: Pid: 4151, comm: multipathd Tainted: G      2.6.18-194.3.1.el5 #1
May 25 18:02:22 xentest kernel: RIP: 0010:[<ffffffff80076659>]  [<ffffffff80076659>] __smp_call_function_many+0x96/0xbc
May 25 18:02:22 xentest kernel: RSP: 0018:ffff810218c8fbf8  EFLAGS: 00000293
May 25 18:02:22 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000
May 25 18:02:22 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0
May 25 18:02:22 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038
May 25 18:02:22 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000
May 25 18:02:22 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd
May 25 18:02:22 xentest kernel: FS:  0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000
May 25 18:02:23 xentest kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 25 18:02:23 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0
May 25 18:02:23 xentest kernel:
May 25 18:02:23 xentest kernel: Call Trace:
May 25 18:02:23 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:23 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:23 xentest kernel:  [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c
May 25 18:02:23 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:23 xentest kernel:  [<ffffffff8007684c>] smp_call_function+0x4e/0x5e
May 25 18:02:23 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:23 xentest kernel:  [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38
May 25 18:02:23 xentest kernel:  [<ffffffff800947db>] on_each_cpu+0x10/0x22
May 25 18:02:23 xentest kernel:  [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42
May 25 18:02:23 xentest kernel:  [<ffffffff800d0f60>] remove_vm_area+0x18/0x25
May 25 18:02:23 xentest kernel:  [<ffffffff800d0fb4>] __vunmap+0x47/0xed
May 25 18:02:23 xentest kernel:  [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b
May 25 18:02:23 xentest kernel:  [<ffffffff80042175>] do_ioctl+0x55/0x6b
May 25 18:02:23 xentest kernel:  [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9
May 25 18:02:23 xentest kernel:  [<ffffffff8000d3ba>] dput+0x2c/0x114
May 25 18:02:23 xentest kernel:  [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22
May 25 18:02:23 xentest kernel:  [<ffffffff8004c870>] sys_ioctl+0x59/0x78
May 25 18:02:23 xentest kernel:  [<ffffffff8005d116>] system_call+0x7e/0x83
May 25 18:02:23 xentest kernel:
May 25 18:02:33 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151]
May 25 18:02:33 xentest kernel: CPU 4:
May 25 18:02:33 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U)
May 25 18:02:33 xentest kernel: Pid: 4151, comm: multipathd Tainted: G      2.6.18-194.3.1.el5 #1
May 25 18:02:33 xentest kernel: RIP: 0010:[<ffffffff8007665d>]  [<ffffffff8007665d>] __smp_call_function_many+0x9a/0xbc
May 25 18:02:33 xentest kernel: RSP: 0018:ffff810218c8fbf8  EFLAGS: 00000293
May 25 18:02:33 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000
May 25 18:02:33 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0
May 25 18:02:33 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038
May 25 18:02:33 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000
May 25 18:02:33 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd
May 25 18:02:33 xentest kernel: FS:  0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000
May 25 18:02:34 xentest kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 25 18:02:34 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0
May 25 18:02:34 xentest kernel:
May 25 18:02:34 xentest kernel: Call Trace:
May 25 18:02:34 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:34 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:34 xentest kernel:  [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c
May 25 18:02:34 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:34 xentest kernel:  [<ffffffff8007684c>] smp_call_function+0x4e/0x5e
May 25 18:02:34 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:34 xentest kernel:  [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38
May 25 18:02:34 xentest kernel:  [<ffffffff800947db>] on_each_cpu+0x10/0x22
May 25 18:02:34 xentest kernel:  [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42
May 25 18:02:34 xentest kernel:  [<ffffffff800d0f60>] remove_vm_area+0x18/0x25
May 25 18:02:34 xentest kernel:  [<ffffffff800d0fb4>] __vunmap+0x47/0xed
May 25 18:02:34 xentest kernel:  [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b
May 25 18:02:34 xentest kernel:  [<ffffffff80042175>] do_ioctl+0x55/0x6b
May 25 18:02:34 xentest kernel:  [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9
May 25 18:02:34 xentest kernel:  [<ffffffff8000d3ba>] dput+0x2c/0x114
May 25 18:02:34 xentest kernel:  [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22
May 25 18:02:34 xentest kernel:  [<ffffffff8004c870>] sys_ioctl+0x59/0x78
May 25 18:02:34 xentest kernel:  [<ffffffff8005d116>] system_call+0x7e/0x83
May 25 18:02:34 xentest kernel:
May 25 18:02:46 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151]
May 25 18:02:46 xentest kernel: CPU 4:
May 25 18:02:47 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U)
May 25 18:02:47 xentest kernel: Pid: 4151, comm: multipathd Tainted: G      2.6.18-194.3.1.el5 #1
May 25 18:02:47 xentest kernel: RIP: 0010:[<ffffffff8007665f>]  [<ffffffff8007665f>] __smp_call_function_many+0x9c/0xbc
May 25 18:02:47 xentest kernel: RSP: 0018:ffff810218c8fbf8  EFLAGS: 00000293
May 25 18:02:47 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000
May 25 18:02:47 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0
May 25 18:02:47 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038
May 25 18:02:47 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000
May 25 18:02:47 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd
May 25 18:02:47 xentest kernel: FS:  0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000
May 25 18:02:47 xentest kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 25 18:02:47 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0
May 25 18:02:47 xentest kernel:
May 25 18:02:47 xentest kernel: Call Trace:
May 25 18:02:47 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:47 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:47 xentest kernel:  [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c
May 25 18:02:47 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:47 xentest kernel:  [<ffffffff8007684c>] smp_call_function+0x4e/0x5e
May 25 18:02:47 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:47 xentest kernel:  [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38
May 25 18:02:47 xentest kernel:  [<ffffffff800947db>] on_each_cpu+0x10/0x22
May 25 18:02:47 xentest kernel:  [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42
May 25 18:02:47 xentest kernel:  [<ffffffff800d0f60>] remove_vm_area+0x18/0x25
May 25 18:02:47 xentest kernel:  [<ffffffff800d0fb4>] __vunmap+0x47/0xed
May 25 18:02:47 xentest kernel:  [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b
May 25 18:02:47 xentest kernel:  [<ffffffff80042175>] do_ioctl+0x55/0x6b
May 25 18:02:47 xentest kernel:  [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9
May 25 18:02:47 xentest kernel:  [<ffffffff8000d3ba>] dput+0x2c/0x114
May 25 18:02:47 xentest kernel:  [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22
May 25 18:02:47 xentest kernel:  [<ffffffff8004c870>] sys_ioctl+0x59/0x78
May 25 18:02:47 xentest kernel:  [<ffffffff8005d116>] system_call+0x7e/0x83
May 25 18:02:47 xentest kernel:
May 25 18:02:57 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151]
May 25 18:02:57 xentest kernel: CPU 4:
May 25 18:02:58 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U)
May 25 18:02:58 xentest kernel: Pid: 4151, comm: multipathd Tainted: G      2.6.18-194.3.1.el5 #1
May 25 18:02:58 xentest kernel: RIP: 0010:[<ffffffff8007665f>]  [<ffffffff8007665f>] __smp_call_function_many+0x9c/0xbc
May 25 18:02:58 xentest kernel: RSP: 0018:ffff810218c8fbf8  EFLAGS: 00000293
May 25 18:02:58 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000
May 25 18:02:58 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0
May 25 18:02:58 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038
May 25 18:02:58 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000
May 25 18:02:58 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd
May 25 18:02:58 xentest kernel: FS:  0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000
May 25 18:02:58 xentest kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 25 18:02:58 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0
May 25 18:02:58 xentest kernel:
May 25 18:02:58 xentest kernel: Call Trace:
May 25 18:02:58 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:58 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:58 xentest kernel:  [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c
May 25 18:02:58 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:58 xentest kernel:  [<ffffffff8007684c>] smp_call_function+0x4e/0x5e
May 25 18:02:58 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:02:58 xentest kernel:  [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38
May 25 18:02:58 xentest kernel:  [<ffffffff800947db>] on_each_cpu+0x10/0x22
May 25 18:02:58 xentest kernel:  [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42
May 25 18:02:58 xentest kernel:  [<ffffffff800d0f60>] remove_vm_area+0x18/0x25
May 25 18:02:58 xentest kernel:  [<ffffffff800d0fb4>] __vunmap+0x47/0xed
May 25 18:02:58 xentest kernel:  [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b
May 25 18:02:58 xentest kernel:  [<ffffffff80042175>] do_ioctl+0x55/0x6b
May 25 18:02:58 xentest kernel:  [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9
May 25 18:02:58 xentest kernel:  [<ffffffff8000d3ba>] dput+0x2c/0x114
May 25 18:02:58 xentest kernel:  [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22
May 25 18:02:58 xentest kernel:  [<ffffffff8004c870>] sys_ioctl+0x59/0x78
May 25 18:02:58 xentest kernel:  [<ffffffff8005d116>] system_call+0x7e/0x83
May 25 18:02:58 xentest kernel:
May 25 18:03:09 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151]
May 25 18:03:09 xentest kernel: CPU 4:
May 25 18:03:09 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U)
May 25 18:03:09 xentest kernel: Pid: 4151, comm: multipathd Tainted: G      2.6.18-194.3.1.el5 #1
May 25 18:03:09 xentest kernel: RIP: 0010:[<ffffffff80076659>]  [<ffffffff80076659>] __smp_call_function_many+0x96/0xbc
May 25 18:03:09 xentest kernel: RSP: 0018:ffff810218c8fbf8  EFLAGS: 00000293
May 25 18:03:09 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000
May 25 18:03:09 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0
May 25 18:03:09 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038
May 25 18:03:09 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000
May 25 18:03:09 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd
May 25 18:03:09 xentest kernel: FS:  0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000
May 25 18:03:09 xentest kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 25 18:03:10 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0
May 25 18:03:10 xentest kernel:
May 25 18:03:10 xentest kernel: Call Trace:
May 25 18:03:10 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:03:10 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:03:10 xentest kernel:  [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c
May 25 18:03:10 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:03:10 xentest kernel:  [<ffffffff8007684c>] smp_call_function+0x4e/0x5e
May 25 18:03:10 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:03:10 xentest kernel:  [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38
May 25 18:03:10 xentest kernel:  [<ffffffff800947db>] on_each_cpu+0x10/0x22
May 25 18:03:10 xentest kernel:  [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42
May 25 18:03:10 xentest kernel:  [<ffffffff800d0f60>] remove_vm_area+0x18/0x25
May 25 18:03:10 xentest kernel:  [<ffffffff800d0fb4>] __vunmap+0x47/0xed
May 25 18:03:10 xentest kernel:  [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b
May 25 18:03:10 xentest kernel:  [<ffffffff80042175>] do_ioctl+0x55/0x6b
May 25 18:03:10 xentest kernel:  [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9
May 25 18:03:10 xentest kernel:  [<ffffffff8000d3ba>] dput+0x2c/0x114
May 25 18:03:10 xentest kernel:  [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22
May 25 18:03:10 xentest kernel:  [<ffffffff8004c870>] sys_ioctl+0x59/0x78
May 25 18:03:10 xentest kernel:  [<ffffffff8005d116>] system_call+0x7e/0x83
May 25 18:03:10 xentest kernel:
May 25 18:03:24 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151]
May 25 18:03:24 xentest kernel: CPU 4:
May 25 18:03:24 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U)
May 25 18:03:24 xentest kernel: Pid: 4151, comm: multipathd Tainted: G      2.6.18-194.3.1.el5 #1
May 25 18:03:24 xentest kernel: RIP: 0010:[<ffffffff80076659>]  [<ffffffff80076659>] __smp_call_function_many+0x96/0xbc
May 25 18:03:24 xentest kernel: RSP: 0018:ffff810218c8fbf8  EFLAGS: 00000293
May 25 18:03:25 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000
May 25 18:03:25 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0
May 25 18:03:25 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038
May 25 18:03:25 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000
May 25 18:03:25 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd
May 25 18:03:25 xentest kernel: FS:  0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000
May 25 18:03:25 xentest kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 25 18:03:25 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0
May 25 18:03:25 xentest kernel:
May 25 18:03:25 xentest kernel: Call Trace:
May 25 18:03:25 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:03:25 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:03:25 xentest kernel:  [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c
May 25 18:03:25 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:03:25 xentest kernel:  [<ffffffff8007684c>] smp_call_function+0x4e/0x5e
May 25 18:03:25 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:03:25 xentest kernel:  [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38
May 25 18:03:25 xentest kernel:  [<ffffffff800947db>] on_each_cpu+0x10/0x22
May 25 18:03:25 xentest kernel:  [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42
May 25 18:03:25 xentest kernel:  [<ffffffff800d0f60>] remove_vm_area+0x18/0x25
May 25 18:03:25 xentest kernel:  [<ffffffff800d0fb4>] __vunmap+0x47/0xed
May 25 18:03:25 xentest kernel:  [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b
May 25 18:03:25 xentest kernel:  [<ffffffff80042175>] do_ioctl+0x55/0x6b
May 25 18:03:25 xentest kernel:  [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9
May 25 18:03:25 xentest kernel:  [<ffffffff8000d3ba>] dput+0x2c/0x114
May 25 18:03:25 xentest kernel:  [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22
May 25 18:03:25 xentest kernel:  [<ffffffff8004c870>] sys_ioctl+0x59/0x78
May 25 18:03:25 xentest kernel:  [<ffffffff8005d116>] system_call+0x7e/0x83
May 25 18:03:25 xentest kernel:
May 25 18:03:35 xentest kernel: BUG: soft lockup - CPU#4 stuck for 10s! [multipathd:4151]
May 25 18:03:35 xentest kernel: CPU 4:
May 25 18:03:35 xentest kernel: Modules linked in: lock_dlm gfs2 dlm configfs lockd sunrpc bridge 8021q bonding ipv6 xfrm_nalgo crypto_api ip_conntrack_netbios_ns ipt_REJECT xt_state xt_tcpudp xt_multiport iptable_filter ipt_MASQUERADE iptable_nat ip_nat ip_conntrack nfnetlink ip_tables x_tables dm_mirror dm_round_robin scsi_dh_alua dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom tpm_infineon tpm i2c_i801 tpm_bios i5000_edac e1000e i2c_core serio_raw pcspkr sg edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ahci ata_piix libata shpchp aacraid ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(U) sd_mod scsi_mod qla2xxx_conf(U) intermodule(U)
May 25 18:03:35 xentest kernel: Pid: 4151, comm: multipathd Tainted: G      2.6.18-194.3.1.el5 #1
May 25 18:03:35 xentest kernel: RIP: 0010:[<ffffffff80076659>]  [<ffffffff80076659>] __smp_call_function_many+0x96/0xbc
May 25 18:03:35 xentest kernel: RSP: 0018:ffff810218c8fbf8  EFLAGS: 00000293
May 25 18:03:36 xentest kernel: RAX: 0000000000000004 RBX: 0000000000000007 RCX: 0000000000000000
May 25 18:03:36 xentest kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0
May 25 18:03:36 xentest kernel: RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000038
May 25 18:03:36 xentest kernel: R10: ffff810218c8fb98 R11: 0000000000000000 R12: 0000000000000000
May 25 18:03:36 xentest kernel: R13: 0000000000000000 R14: ffff81021e3c7400 R15: ffffffff800767bd
May 25 18:03:36 xentest kernel: FS:  0000000041a73940(0063) GS:ffff81021fc7dd40(0000) knlGS:0000000000000000
May 25 18:03:36 xentest kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 25 18:03:36 xentest kernel: CR2: 0000003572292215 CR3: 00000002155f1000 CR4: 00000000000006e0
May 25 18:03:36 xentest kernel:
May 25 18:03:36 xentest kernel: Call Trace:
May 25 18:03:36 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:03:36 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:03:36 xentest kernel:  [<ffffffff8007675b>] smp_call_function_many+0x38/0x4c
May 25 18:03:36 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:03:36 xentest kernel:  [<ffffffff8007684c>] smp_call_function+0x4e/0x5e
May 25 18:03:36 xentest kernel:  [<ffffffff80076530>] do_flush_tlb_all+0x0/0x6a
May 25 18:03:36 xentest kernel:  [<ffffffff882c1b28>] :dm_mod:dev_status+0x0/0x38
May 25 18:03:36 xentest kernel:  [<ffffffff800947db>] on_each_cpu+0x10/0x22
May 25 18:03:36 xentest kernel:  [<ffffffff800d0f31>] __remove_vm_area+0x2b/0x42
May 25 18:03:36 xentest kernel:  [<ffffffff800d0f60>] remove_vm_area+0x18/0x25
May 25 18:03:36 xentest kernel:  [<ffffffff800d0fb4>] __vunmap+0x47/0xed
May 25 18:03:36 xentest kernel:  [<ffffffff882c2eff>] :dm_mod:ctl_ioctl+0x237/0x25b
May 25 18:03:36 xentest kernel:  [<ffffffff80042175>] do_ioctl+0x55/0x6b
May 25 18:03:36 xentest kernel:  [<ffffffff8003018e>] vfs_ioctl+0x457/0x4b9
May 25 18:03:36 xentest kernel:  [<ffffffff8000d3ba>] dput+0x2c/0x114
May 25 18:03:36 xentest kernel:  [<ffffffff800a2a47>] hrtimer_wakeup+0x0/0x22
May 25 18:03:36 xentest kernel:  [<ffffffff8004c870>] sys_ioctl+0x59/0x78
May 25 18:03:36 xentest kernel:  [<ffffffff8005d116>] system_call+0x7e/0x83
May 25 18:03:36 xentest kernel:
May 25 18:07:25 xentest syslogd 1.4.1: restart.
Comment 6 Steve Whitehouse 2010-05-25 11:37:51 EDT
This looks nothing like the last report, and much more like a gfs2 issue. Are you using quotas? Can you paste your /proc/mounts here?
Comment 7 veliogluh 2010-05-25 12:21:42 EDT
Yes I am mounting my GFS2 disks with quota option enabled.
Here is my fstab and mounts.
[root@xentest ~]# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 rw,data=ordered 0 0
/dev /dev tmpfs rw 0 0
/proc /proc proc rw 0 0
/sys /sys sysfs rw 0 0
/proc/bus/usb /proc/bus/usb usbfs rw 0 0
devpts /dev/pts devpts rw 0 0
tmpfs /dev/shm tmpfs rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
none /sys/kernel/config configfs rw 0 0
/dev/mapper/akademik1 /users/akademik1 gfs2 rw,hostdata=jid=0:id=1114116:first=1,quota=on 0 0
/dev/mapper/akademik2 /users/akademik2 gfs2 rw,hostdata=jid=0:id=1245188:first=1,quota=on 0 0
/dev/mapper/akademik3 /users/akademik3 gfs2 rw,hostdata=jid=0:id=1376260:first=1,quota=on 0 0
/dev/mapper/akademik4 /users/akademik4 gfs2 rw,hostdata=jid=0:id=1507332:first=1,quota=on 0 0
/dev/mapper/idari1 /users/idari1 gfs2 rw,hostdata=jid=0:id=1638404:first=1,quota=on 0 0
/dev/mapper/ogrenci1 /users/ogrenci1 gfs2 rw,hostdata=jid=0:id=1769476:first=1,quota=on 0 0
/dev/mapper/ogrenci2 /users/ogrenci2 gfs2 rw,hostdata=jid=0:id=1900548:first=1,quota=on 0 0
/dev/mapper/ogrenci3 /users/ogrenci3 gfs2 rw,hostdata=jid=0:id=2031620:first=1,quota=on 0 0
/dev/mapper/ogrenci4 /users/ogrenci4 gfs2 rw,hostdata=jid=0:id=2162692:first=1,quota=on 0 0
/dev/mapper/ogrenci5 /users/ogrenci5 gfs2 rw,hostdata=jid=0:id=2293764:first=1,quota=on 0 0
/dev/mapper/ogrenci6 /users/ogrenci6 gfs2 rw,hostdata=jid=0:id=2424836:first=1,quota=on 0 0
/dev/mapper/sk1 /users/sk1 gfs2 rw,hostdata=jid=0:id=2555908:first=1,quota=on 0 0
/dev/mapper/diger1 /users/diger1 gfs2 rw,hostdata=jid=0:id=2686980:first=1,quota=on 0 0
[root@xentest ~]#





By the way after every crash I check my GFS2 disks with fsck.gfs2 and recover problems.
Comment 8 Steve Whitehouse 2010-05-25 12:27:50 EDT
If you turn quota off does it work correctly? If so that is almost certainly a known issue for which we have a fix.

The problems that you see reported by fsck are unlikely to be serious and are most likely just a consequence of an unclean shutdown. If you are worried post the results here and we can take a look at them.
Comment 10 veliogluh 2010-05-25 16:45:17 EDT
You are right Steve it worked perfect without quota enabled. My 5 day fight finally end :)

Where could I find the fix.
Comment 11 Steve Whitehouse 2010-05-26 04:28:43 EDT
It is in kernel 2.6.18-194.4.1.el5. I'll close this as a dup of the 5.5.z bug so that you can see the full details.
Comment 12 Steve Whitehouse 2010-05-26 04:29:21 EDT

*** This bug has been marked as a duplicate of bug 586008 ***

Note You need to log in before you can comment on or make changes to this bug.