Attachment 915160 Details for Bug 643165 – Comment

Comment

comment_3215285.txt (text/plain), 65.50 KB, created by Adam Drew on 2010-10-14 20:12:11 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Adam Drew

Created: 2010-10-14 20:12:11 UTC

Size: 65.50 KB

patch

obsolete

>Description of problem:
>Running 2.6.18-226 for testing purposes related to BZ595397. With 2.6.18-226 we're hitting kernel panics in gfs2_glock_hold.
>
>crash data:
>crash> bt
>PID: 30177  TASK: ffff810798ef1820  CPU: 2   COMMAND: "dcs_output_mgr"
> #0 [ffff810798f0f7a0] crash_kexec at ffffffff800af83a
> #1 [ffff810798f0f860] __die at ffffffff80065117
> #2 [ffff810798f0f8a0] die at ffffffff8006c73a
> #3 [ffff810798f0f8d0] do_invalid_op at ffffffff8006ccfa
> #4 [ffff810798f0f990] error_exit at ffffffff8005dde9
>    [exception RIP: gfs2_glock_hold+21]
>    RIP: ffffffff88844351  RSP: ffff810798f0fa48  RFLAGS: 00010292
>    RAX: 0000000000000000  RBX: ffff81074fab7d98  RCX: ffffffff80318f28
>    RDX: ffffffff80318f28  RSI: 0000000000000096  RDI: ffffffff80318f20
>    RBP: 000000000007cd19   R8: ffffffff80318f28   R9: 0000000000000001
>    R10: 0000000000000000  R11: 0000000000000052  R12: ffff810744dc38f0
>    R13: ffff81081dc79000  R14: ffff81078b23c9a8  R15: 0000000000000000
>    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0000
> #5 [ffff810798f0fa50] gfs2_process_unlinked_inode at ffffffff8884837e
> #6 [ffff810798f0fad0] gfs2_inplace_reserve_i at ffffffff88858ffa
> #7 [ffff810798f0fbb0] gfs2_write_begin at ffffffff8884d5fc
> #8 [ffff810798f0fc20] gfs2_file_buffered_write at ffffffff8884edc6
> #9 [ffff810798f0fcf0] __gfs2_file_aio_write_nolock at ffffffff8884f1ec
>#10 [ffff810798f0fdb0] gfs2_file_write_nolock at ffffffff8884f38e
>#11 [ffff810798f0fec0] gfs2_file_write at ffffffff8884f4de
>#12 [ffff810798f0ff10] vfs_write at ffffffff80016af0
>#13 [ffff810798f0ff40] sys_write at ffffffff800173a8
>#14 [ffff810798f0ff80] tracesys at ffffffff8005d28d (via system_call)
>    RIP: 00000039462c60ab  RSP: 00007fff2a7aac70  RFLAGS: 00000202
>    RAX: ffffffffffffffda  RBX: ffffffff8005d28d  RCX: ffffffffffffffff
>    RDX: 0000000000001000  RSI: 00002aaaab967000  RDI: 0000000000000019
>    RBP: 0000000000001000   R8: 2020202020202020   R9: 38314b5254202020
>    R10: 2020203634373239  R11: 0000000000000202  R12: 000000000f048bd0
>    R13: 00002aaaab967000  R14: 0000000000001000  R15: 0000000100000000
>    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b
>
>From dmesg:
>----------- [cut here ] --------- [please bite here ] ---------
>Kernel BUG at fs/gfs2/glock.c:209
>invalid opcode: 0000 [1] SMP 
>last sysfs file: /devices/pci0000:00/0000:00:00.0/irq
>CPU 2 
>Modules linked in: ipt_MASQUERADE iptable_nat ip_nat xt_state ip_conntrack nfnetlink ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge nfs fscache nfsd exportfs nfs_acl auth_rpcgss autofs4 hidp l2cap bluetooth dm_log_clustered(U) lock_dlm gfs2 dlm configfs lockd sunrpc be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_addr iscsi_tcp bnx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i iw_cxgb3 ib_core cxgb3 8021q libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport ksm(U) kvm_intel(U) kvm(U) i5000_edac tpm_tis edac_mc pcspkr tpm bnx2 tpm_bios dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod mppVhba(U) usb_storage qla2xxx scsi_transport_fc ata_piix libata shpchp mptsas mptscsih mptbase scsi_transport_sas mppUpper(U) sg sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
>Pid: 30177, comm: dcs_output_mgr Tainted: G      2.6.18-226.el5 #1
>RIP: 0010:[<ffffffff88844351>]  [<ffffffff88844351>] :gfs2:gfs2_glock_hold+0x15/0x25
>RSP: 0000:ffff810798f0fa48  EFLAGS: 00010292
>RAX: 0000000000000000 RBX: ffff81074fab7d98 RCX: ffffffff80318f28
>RDX: ffffffff80318f28 RSI: 0000000000000096 RDI: ffffffff80318f20
>RBP: 000000000007cd19 R08: ffffffff80318f28 R09: 0000000000000001
>R10: 0000000000000000 R11: 0000000000000052 R12: ffff810744dc38f0
>R13: ffff81081dc79000 R14: ffff81078b23c9a8 R15: 0000000000000000
>FS:  00002ac607435b00(0000) GS:ffff81011cb9ce40(0000) knlGS:0000000000000000
>CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>CR2: 000000001920e588 CR3: 0000000798951000 CR4: 00000000000026e0
>Process dcs_output_mgr (pid: 30177, threadinfo ffff810798f0e000, task ffff810798ef1820)
>Stack:  ffff810744dc3688 ffffffff8884837e ffff810798f0fa58 ffff810798f0fa58
> ffff81074fab7d98 00000001000075e1 0000000000000001 0000000000000000
> ffffffff8884837e ffffffff88843e8a 000000000007cd19 ffff810700000000
>Call Trace:
> [<ffffffff8884837e>] :gfs2:gfs2_process_unlinked_inode+0x140/0x1b4
> [<ffffffff8884837e>] :gfs2:gfs2_process_unlinked_inode+0x140/0x1b4
> [<ffffffff88843e8a>] :gfs2:gfs2_glock_schedule_for_reclaim+0x5d/0x9a
> [<ffffffff88858ffa>] :gfs2:gfs2_inplace_reserve_i+0x663/0x6f4
> [<ffffffff88843deb>] :gfs2:do_promote+0xf5/0x137
> [<ffffffff8884d5fc>] :gfs2:gfs2_write_begin+0x16c/0x33e
> [<ffffffff8884edc6>] :gfs2:gfs2_file_buffered_write+0xf3/0x26c
> [<ffffffff8884f1ec>] :gfs2:__gfs2_file_aio_write_nolock+0x2ad/0x2e4
> [<ffffffff8884f38e>] :gfs2:gfs2_file_write_nolock+0xaa/0x10f
> [<ffffffff800c8bc7>] generic_file_read+0xac/0xc5
> [<ffffffff800a2896>] autoremove_wake_function+0x0/0x2e
> [<ffffffff800a2896>] autoremove_wake_function+0x0/0x2e
> [<ffffffff8884f4de>] :gfs2:gfs2_file_write+0x49/0xa7
> [<ffffffff80016af0>] vfs_write+0xce/0x174
> [<ffffffff800173a8>] sys_write+0x45/0x6e
> [<ffffffff8005d28d>] tracesys+0xd5/0xe0
>
>
>Code: 0f 0b 68 78 df 85 88 c2 d1 00 f0 ff 43 28 5b c3 53 8b 47 78 
>RIP  [<ffffffff88844351>] :gfs2:gfs2_glock_hold+0x15/0x25
> RSP <ffff810798f0fa48>
>
>What's strange is we show that we're calling gfs2_glock_hold from within gfs2_process_unlinked_inode but looking through the code we don't directly call gfs2_glock_hold from within gfs2_process_unlinked_inode.
>
>The panic itself looks to be a handled exception in gfs2_glock_hold because we started to dump a glock and there's a GLOCK_BUG_ON macro call in gfs2_glock_hold. But how did we end up in gfs2_glock_hold from gfs2_process_unlinked_inode in the first place?
>
>Version-Release number of selected component (if applicable):
>2.6.18-226
>
>How reproducible:
>Within an hour of use
>
>Steps to Reproduce:
>1. Run on the 226 kernel
>2. Start application running against GFS
>3. Catch panic
>  
>Actual results:
>Panic
>
>Expected results:
>No Panic
>
>Additional info:
>Linux version 2.6.18-226.el5 (mockbuild@ls20-bc2-13.build.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Thu Oct 7 20:55:39 EDT 2010
>Command line: ro root=/dev/sda1 rhgb quiet crashkernel=128M@16M
>BIOS-provided physical RAM map:
> BIOS-e820: 0000000000010000 - 000000000009d000 (usable)
> BIOS-e820: 000000000009d000 - 00000000000a0000 (reserved)
> BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
> BIOS-e820: 0000000000100000 - 00000000cffbcdc0 (usable)
> BIOS-e820: 00000000cffbcdc0 - 00000000cffd0000 (ACPI data)
> BIOS-e820: 00000000cffd0000 - 00000000d0000000 (reserved)
> BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
> BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved)
> BIOS-e820: 0000000100000000 - 0000000830000000 (usable)
>DMI 2.4 present.
>ACPI: RSDP (v002 IBM                                   ) @ 0x00000000000fdfd0
>ACPI: XSDT (v001 IBM    SERBLADE 0x00001001 IBM  0x45444f43) @ 0x00000000cffcff00
>ACPI: FADT (v002 IBM    SERBLADE 0x00001001 IBM  0x45444f43) @ 0x00000000cffcfe40
>ACPI: MADT (v001 IBM    SERBLADE 0x00001001 IBM  0x45444f43) @ 0x00000000cffcfd80
>ACPI: MCFG (v001 IBM    SERBLADE 0x00001001 IBM  0x45444f43) @ 0x00000000cffcfd40
>ACPI: ERST (v001 IBM    SERBLADE 0x00001001 IBM  0x45444f43) @ 0x00000000cffcbf00
>ACPI: DSDT (v002 IBM    SERBLADE 0x00001000 INTL 0x20060707) @ 0x0000000000000000
>No NUMA configuration found
>Faking a node at 0000000000000000-0000000830000000
>Bootmem setup node 0 0000000000000000-0000000830000000
>On node 0 totalpages: 8269531
>  DMA zone: 2407 pages, LIFO batch:0
>  DMA32 zone: 833524 pages, LIFO batch:31
>  Normal zone: 7433600 pages, LIFO batch:31
>ACPI: PM-Timer IO Port: 0x588
>ACPI: Local APIC address 0xfee00000
>ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
>Processor #0 6:15 APIC version 20
>ACPI: LAPIC (acpi_id[0x01] lapic_id[0x06] enabled)
>Processor #6 6:15 APIC version 20
>ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
>Processor #1 6:15 APIC version 20
>ACPI: LAPIC (acpi_id[0x03] lapic_id[0x07] enabled)
>Processor #7 6:15 APIC version 20
>ACPI: LAPIC_NMI (acpi_id[0x00] dfl dfl lint[0x1])
>ACPI: LAPIC_NMI (acpi_id[0x01] dfl dfl lint[0x1])
>ACPI: LAPIC_NMI (acpi_id[0x02] dfl dfl lint[0x1])
>ACPI: LAPIC_NMI (acpi_id[0x03] dfl dfl lint[0x1])
>ACPI: IOAPIC (id[0x0e] address[0xfec00000] gsi_base[0])
>IOAPIC[0]: apic_id 14, version 32, address 0xfec00000, GSI 0-23
>ACPI: IOAPIC (id[0x0d] address[0xfec80000] gsi_base[24])
>IOAPIC[1]: apic_id 13, version 32, address 0xfec80000, GSI 24-47
>ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
>ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
>ACPI: IRQ0 used by override.
>ACPI: IRQ2 used by override.
>ACPI: IRQ9 used by override.
>Setting APIC routing to physical flat
>Using ACPI (MADT) for SMP configuration information
>Nosave address range: 000000000009d000 - 00000000000a0000
>Nosave address range: 00000000000a0000 - 00000000000e0000
>Nosave address range: 00000000000e0000 - 0000000000100000
>Nosave address range: 00000000cffbc000 - 00000000cffbd000
>Nosave address range: 00000000cffbd000 - 00000000cffd0000
>Nosave address range: 00000000cffd0000 - 00000000d0000000
>Nosave address range: 00000000d0000000 - 00000000e0000000
>Nosave address range: 00000000e0000000 - 00000000f0000000
>Nosave address range: 00000000f0000000 - 00000000fec00000
>Nosave address range: 00000000fec00000 - 0000000100000000
>Allocating PCI resources starting at d1000000 (gap: d0000000:10000000)
>SMP: Allowing 4 CPUs, 0 hotplug CPUs
>Built 1 zonelists.  Total pages: 8269531
>Kernel command line: ro root=/dev/sda1 rhgb quiet crashkernel=128M@16M
>Initializing CPU#0
>PID hash table entries: 4096 (order: 12, 32768 bytes)
>Console: colour VGA+ 80x25
>Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes)
>Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes)
>Checking aperture...
>ACPI: DMAR not present
>PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
>Placing software IO TLB between 0xc03c000 - 0x1003c000
>Memory: 32827944k/34340864k available (2591k kernel code, 725756k reserved, 1645k data, 224k init)
>Calibrating delay loop (skipped), value calculated using timer frequency.. 6000.40 BogoMIPS (lpj=3000201)
>Security Framework v1.0.0 initialized
>SELinux:  Initializing.
>SELinux:  Starting in permissive mode
>selinux_register_security:  Registering secondary module capability
>Capability LSM initialized as secondary
>Mount-cache hash table entries: 256
>CPU: L1 I cache: 32K, L1 D cache: 32K
>CPU: L2 cache: 4096K
>using mwait in idle threads.
>CPU: Physical Processor ID: 0
>CPU: Processor Core ID: 0
>CPU0: Thermal monitoring enabled (TM1)
>SMP alternatives: switching to UP code
>ACPI: Core revision 20060707
>Using local APIC timer interrupts.
>Detected 20.834 MHz APIC timer.
>SMP alternatives: switching to SMP code
>Booting processor 1/4 APIC 0x6
>Initializing CPU#1
>Calibrating delay using timer specific routine.. 5999.38 BogoMIPS (lpj=2999693)
>CPU: L1 I cache: 32K, L1 D cache: 32K
>CPU: L2 cache: 4096K
>CPU: Physical Processor ID: 3
>CPU: Processor Core ID: 0
>CPU1: Thermal monitoring enabled (TM1)
>Intel(R) Xeon(R) CPU            5160  @ 3.00GHz stepping 0b
>SMP alternatives: switching to SMP code
>Booting processor 2/4 APIC 0x1
>Initializing CPU#2
>Calibrating delay using timer specific routine.. 5999.37 BogoMIPS (lpj=2999688)
>CPU: L1 I cache: 32K, L1 D cache: 32K
>CPU: L2 cache: 4096K
>CPU: Physical Processor ID: 0
>CPU: Processor Core ID: 1
>CPU2: Thermal monitoring enabled (TM1)
>Intel(R) Xeon(R) CPU            5160  @ 3.00GHz stepping 0b
>SMP alternatives: switching to SMP code
>Booting processor 3/4 APIC 0x7
>Initializing CPU#3
>Calibrating delay using timer specific routine.. 5999.46 BogoMIPS (lpj=2999734)
>CPU: L1 I cache: 32K, L1 D cache: 32K
>CPU: L2 cache: 4096K
>CPU: Physical Processor ID: 3
>CPU: Processor Core ID: 1
>CPU3: Thermal monitoring enabled (TM1)
>Intel(R) Xeon(R) CPU            5160  @ 3.00GHz stepping 0b
>Brought up 4 CPUs
>CPU#0: NMI watchdog performance counter calibration - 259->279
>CPU#1: NMI watchdog performance counter calibration - 60->80
>CPU#2: NMI watchdog performance counter calibration - 60->80
>CPU#3: NMI watchdog performance counter calibration - 60->80
>NMI watchdog testing PASSED.
>time.c: Using 3.579545 MHz WALL PM GTOD PIT/TSC timer.
>time.c: Detected 3000.201 MHz processor.
>sizeof(vma)=176 bytes
>sizeof(page)=56 bytes
>sizeof(inode)=560 bytes
>sizeof(dentry)=216 bytes
>sizeof(ext3inode)=760 bytes
>sizeof(buffer_head)=96 bytes
>sizeof(skbuff)=248 bytes
>migration_cost=14,9255
>checking if image is initramfs... it is
>Freeing initrd memory: 3458k freed
>NET: Registered protocol family 16
>ACPI: bus type pci registered
>Warning: pci_mmcfg_init marking 256MB space uncacheable.
>MCFG table requires 16MB uncacheable only. Try booting with acpi_mcfg_max_pci_bus_num=on
>PCI: Using MMCONFIG at e0000000
>ACPI: Interpreter enabled
>ACPI: Using IOAPIC for interrupt routing
>ACPI: No dock devices found.
>ACPI: PCI Root Bridge [PCI0] (0000:00)
>PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.2
>PCI: Transparent bridge - 0000:00:1e.0
>ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
>ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI7.PCIX._PRT]
>ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI7.PCI9.PCIA._PRT]
>ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI7.PCI9.PCIB._PRT]
>ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI3._PRT]
>ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI4._PRT]
>ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCIE._PRT]
>ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCIF._PRT]
>ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCIS._PRT]
>ACPI: PCI Interrupt Link [LP00] (IRQs *4)
>ACPI: PCI Interrupt Link [LP01] (IRQs *5)
>ACPI: PCI Interrupt Link [LP02] (IRQs *11)
>ACPI: PCI Interrupt Link [LP03] (IRQs *4)
>ACPI: Blank IRQ resource
>ACPI: Resource is not an IRQ entry
>ACPI: PCI Interrupt Link [LP04] (IRQs) *0, disabled.
>ACPI: PCI Interrupt Link [LP05] (IRQs *5)
>ACPI: PCI Interrupt Link [LP06] (IRQs *11)
>ACPI: PCI Interrupt Link [LP07] (IRQs *4)
>Linux Plug and Play Support v0.97 (c) Adam Belay
>pnp: PnP ACPI init
>pnp: PnP ACPI: found 10 devices
>usbcore: registered new driver usbfs
>usbcore: registered new driver hub
>PCI: Using ACPI for IRQ routing
>PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
>NetLabel: Initializing
>NetLabel:  domain hash size = 128
>NetLabel:  protocols = UNLABELED CIPSOv4
>NetLabel:  unlabeled traffic allowed by default
>ACPI: DMAR not present
>PCI-GART: No AMD northbridge found.
>pnp: 00:01: ioport range 0xc00-0xc1f has been reserved
>pnp: 00:01: ioport range 0xc60-0xc67 has been reserved
>pnp: 00:08: ioport range 0xc28-0xc28 has been reserved
>pnp: 00:08: ioport range 0xc2c-0xc2c has been reserved
>pnp: 00:09: iomem range 0xe0000000-0xefffffff could not be reserved
>pnp: 00:09: iomem range 0xfed1c000-0xfed1ffff could not be reserved
>pnp: 00:09: iomem range 0xfe000000-0xfe01ffff has been reserved
>pnp: 00:09: iomem range 0xfe020000-0xfe6fffff has been reserved
>PCI: Bridge: 0000:09:00.0
>  IO window: disabled.
>  MEM window: disabled.
>  PREFETCH window: disabled.
>PCI: Bridge: 0000:09:01.0
>  IO window: disabled.
>  MEM window: disabled.
>  PREFETCH window: disabled.
>PCI: Bridge: 0000:07:00.0
>  IO window: disabled.
>  MEM window: disabled.
>  PREFETCH window: disabled.
>PCI: Bridge: 0000:07:00.3
>  IO window: 5000-5fff
>  MEM window: d5000000-d6ffffff
>  PREFETCH window 0x00000000d1200000-0x00000000d12fffff
>PCI: Bridge: 0000:00:02.0
>  IO window: 5000-5fff
>  MEM window: d5000000-d6ffffff
>  PREFETCH window 0x00000000d1200000-0x00000000d12fffff
>PCI: Bridge: 0000:00:03.0
>  IO window: disabled.
>  MEM window: disabled.
>  PREFETCH window: disabled.
>PCI: Bridge: 0000:00:04.0
>  IO window: disabled.
>  MEM window: disabled.
>  PREFETCH window: disabled.
>PCI: Bridge: 0000:00:05.0
>  IO window: 6000-ffff
>  MEM window: disabled.
>  PREFETCH window: disabled.
>PCI: Bridge: 0000:03:00.0
>  IO window: disabled.
>  MEM window: da000000-dcffffff
>  PREFETCH window: disabled.
>PCI: Bridge: 0000:00:06.0
>  IO window: disabled.
>  MEM window: da000000-dcffffff
>  PREFETCH window: disabled.
>PCI: Bridge: 0000:05:00.0
>  IO window: disabled.
>  MEM window: d7000000-d9ffffff
>  PREFETCH window: disabled.
>PCI: Bridge: 0000:00:07.0
>  IO window: disabled.
>  MEM window: d7000000-d9ffffff
>  PREFETCH window: disabled.
>PCI: Bridge: 0000:00:1c.0
>  IO window: 4000-4fff
>  MEM window: dd000000-deffffff
>  PREFETCH window 0x00000000d1000000-0x00000000d11fffff
>PCI: Bridge: 0000:00:1e.0
>  IO window: 3000-3fff
>  MEM window: f8000000-f8ffffff
>  PREFETCH window 0x00000000f0000000-0x00000000f7ffffff
>PCI: Setting latency timer of device 0000:00:02.0 to 64
>PCI: Setting latency timer of device 0000:07:00.0 to 64
>PCI: Setting latency timer of device 0000:09:00.0 to 64
>PCI: Setting latency timer of device 0000:09:01.0 to 64
>PCI: Setting latency timer of device 0000:07:00.3 to 64
>PCI: Setting latency timer of device 0000:00:03.0 to 64
>PCI: Setting latency timer of device 0000:00:04.0 to 64
>PCI: Setting latency timer of device 0000:00:05.0 to 64
>PCI: Setting latency timer of device 0000:00:06.0 to 64
>PCI: Setting latency timer of device 0000:03:00.0 to 64
>PCI: Setting latency timer of device 0000:00:07.0 to 64
>PCI: Setting latency timer of device 0000:05:00.0 to 64
>GSI 16 sharing vector 0xA9 and IRQ 16
>ACPI: PCI Interrupt 0000:00:1c.0[A] -> GSI 16 (level, low) -> IRQ 169
>PCI: Setting latency timer of device 0000:00:1c.0 to 64
>PCI: Setting latency timer of device 0000:00:1e.0 to 64
>NET: Registered protocol family 2
>IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
>TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
>TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
>TCP: Hash tables configured (established 262144 bind 65536)
>TCP reno registered
>audit: initializing netlink socket (disabled)
>type=2000 audit(1287003683.724:1): initialized
>Total HugeTLB memory allocated, 0
>VFS: Disk quotas dquot_6.5.1
>Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
>SELinux:  Registering netfilter hooks
>Initializing Cryptographic API
>alg: No test for crc32c (crc32c-generic)
>ksign: Installing public key data
>Loading keyring
>- Added public key 71EDA2525418B92B
>- User ID: Red Hat, Inc. (Kernel Module GPG key)
>io scheduler noop registered
>io scheduler anticipatory registered
>io scheduler deadline registered
>io scheduler cfq registered (default)
>Boot video device is 0000:01:01.0
>PCI: Setting latency timer of device 0000:00:02.0 to 64
>PCI: Setting latency timer of device 0000:00:03.0 to 64
>PCI: Setting latency timer of device 0000:00:04.0 to 64
>PCI: Setting latency timer of device 0000:00:05.0 to 64
>PCI: Setting latency timer of device 0000:00:06.0 to 64
>PCI: Setting latency timer of device 0000:00:07.0 to 64
>PCI: Setting latency timer of device 0000:00:1c.0 to 64
>PCI: Setting latency timer of device 0000:07:00.0 to 64
>PCI: Setting latency timer of device 0000:09:00.0 to 64
>PCI: Setting latency timer of device 0000:09:01.0 to 64
>pci_hotplug: PCI Hot Plug PCI Core version: 0.5
>ACPI: Processor [CPU0] (supports 8 throttling states)
>ACPI: Processor [CPU1] (supports 8 throttling states)
>ACPI: Processor [CPU2] (supports 8 throttling states)
>ACPI: Processor [CPU3] (supports 8 throttling states)
>Real Time Clock Driver v1.12ac
>Non-volatile memory driver v1.2
>Linux agpgart interface v0.101 (c) Dave Jones
>Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
>brd: module loaded
>Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
>ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
>ide0: I/O resource 0x1F0-0x1F7 not free.
>ide0: ports already in use, skipping probe
>Probing IDE interface ide1...
>ide-floppy driver 0.99.newide
>usbcore: registered new driver hiddev
>usbcore: registered new driver usbhid
>drivers/usb/input/hid-core.c: v2.6:USB HID core driver
>PNP: No PS/2 controller found. Probing ports directly.
>serio: i8042 KBD port at 0x60,0x64 irq 1
>mice: PS/2 mouse device common for all mice
>md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
>md: bitmap version 4.39
>TCP bic registered
>Initializing IPsec netlink socket
>NET: Registered protocol family 1
>NET: Registered protocol family 17
>ACPI: (supports S0 S4 S5)
>Initalizing network drop monitor service
>Freeing unused kernel memory: 224k freed
>Write protecting the kernel read-only data: 518k
>GSI 17 sharing vector 0x42 and IRQ 17
>ACPI: PCI Interrupt 0000:00:1d.7[D] -> GSI 23 (level, low) -> IRQ 66
>PCI: Setting latency timer of device 0000:00:1d.7 to 64
>ehci_hcd 0000:00:1d.7: EHCI Host Controller
>ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1
>ehci_hcd 0000:00:1d.7: debug port 1
>PCI: cache line size of 32 is not supported by device 0000:00:1d.7
>ehci_hcd 0000:00:1d.7: irq 66, io mem 0xf9000000
>ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
>usb usb1: configuration #1 chosen from 1 choice
>hub 1-0:1.0: USB hub found
>hub 1-0:1.0: 5 ports detected
>ohci_hcd: 2005 April 22 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI)
>USB Universal Host Controller Interface driver v3.0
>GSI 18 sharing vector 0x4A and IRQ 18
>ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 18 (level, low) -> IRQ 74
>PCI: Setting latency timer of device 0000:00:1d.0 to 64
>uhci_hcd 0000:00:1d.0: UHCI Host Controller
>uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
>uhci_hcd 0000:00:1d.0: irq 74, io base 0x00002200
>usb usb2: configuration #1 chosen from 1 choice
>hub 2-0:1.0: USB hub found
>hub 2-0:1.0: 2 ports detected
>GSI 19 sharing vector 0x52 and IRQ 19
>ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 21 (level, low) -> IRQ 82
>PCI: Setting latency timer of device 0000:00:1d.1 to 64
>uhci_hcd 0000:00:1d.1: UHCI Host Controller
>uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3
>uhci_hcd 0000:00:1d.1: irq 82, io base 0x00002600
>usb usb3: configuration #1 chosen from 1 choice
>hub 3-0:1.0: USB hub found
>hub 3-0:1.0: 2 ports detected
>usb 1-3: new high speed USB device using ehci_hcd and address 2
>GSI 20 sharing vector 0x5A and IRQ 20
>ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 19 (level, low) -> IRQ 90
>PCI: Setting latency timer of device 0000:00:1d.2 to 64
>uhci_hcd 0000:00:1d.2: UHCI Host Controller
>uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4
>uhci_hcd 0000:00:1d.2: irq 90, io base 0x00002800
>usb usb4: configuration #1 chosen from 1 choice
>hub 4-0:1.0: USB hub found
>hub 4-0:1.0: 2 ports detected
>usb 1-3: configuration #1 chosen from 1 choice
>hub 1-3:1.0: USB hub found
>hub 1-3:1.0: 2 ports detected
>SCSI subsystem initialized
>Fusion MPT base driver 3.04.15rh
>Copyright (c) 1999-2008 LSI Corporation
>Fusion MPT SAS Host driver 3.04.15rh
>ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 16 (level, low) -> IRQ 169
>mptbase: ioc0: Initiating bringup
>ioc0: LSISAS1064E B1: Capabilities={Initiator}
>PCI: Setting latency timer of device 0000:02:00.0 to 64
>usb 3-2: new full speed USB device using uhci_hcd and address 2
>usb 3-2: configuration #1 chosen from 1 choice
>input: IBM IBM MM2 as /class/input/input0
>input: USB HID v1.10 Keyboard [IBM IBM MM2] on usb-0000:00:1d.1-2
>input: IBM IBM MM2 as /class/input/input1
>input: USB HID v1.10 Mouse [IBM IBM MM2] on usb-0000:00:1d.1-2
>scsi0 : ioc0: LSISAS1064E B1, FwRev=011b5600h, Ports=1, MaxQ=277, IRQ=169
>mptsas: ioc0: attaching ssp device: fw_channel 0, fw_id 1, phy 0, sas_addr 0x5000c5000638dde9
>  Vendor: IBM-ESXS  Model: ST9146802SS       Rev: B522
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>62 [RAIDarray.mpp]Device 0x2f9d5000 is not an array
>SCSI device sda: 286748000 512-byte hdwr sectors (146815 MB)
>sda: Write Protect is off
>sda: Mode Sense: bb 00 10 08
>SCSI device sda: drive cache: write through w/ FUA
>SCSI device sda: 286748000 512-byte hdwr sectors (146815 MB)
>sda: Write Protect is off
>sda: Mode Sense: bb 00 10 08
>SCSI device sda: drive cache: write through w/ FUA
> sda: sda1 sda2
>sd 0:0:0:0: Attached scsi disk sda
>62 [RAIDarray.mpp]Device 0x2f9d5000 is not an array
>sd 0:0:0:0: Attached scsi generic sg0 type 0
>shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
>libata version 3.00 loaded.
>ata_piix 0000:00:1f.2: version 2.12
>ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 23 (level, low) -> IRQ 66
>ata_piix 0000:00:1f.2: MAP [ P0 P2 IDE IDE ]
>PCI: Setting latency timer of device 0000:00:1f.2 to 64
>scsi1 : ata_piix
>scsi2 : ata_piix
>ata1: SATA max UDMA/133 bmdma 0x2a00 irq 14
>ata2: PATA max UDMA/100 bmdma 0x2a08 irq 15
>ata2: port disabled. ignoring.
>QLogic Fibre Channel HBA Driver
>GSI 21 sharing vector 0x62 and IRQ 21
>ACPI: PCI Interrupt 0000:08:01.0[A] -> GSI 25 (level, low) -> IRQ 98
>qla2xxx 0000:08:01.0: Found an ISP2422, irq 98, iobase 0xffffc2000001c000
>qla2xxx 0000:08:01.0: Configuring PCI space...
>qla2xxx 0000:08:01.0: Configure NVRAM parameters...
>qla2xxx 0000:08:01.0: Verifying loaded RISC code...
>qla2xxx 0000:08:01.0: Allocated (64 KB) for EFT...
>qla2xxx 0000:08:01.0: Allocated (1413 KB) for firmware dump...
>qla2xxx 0000:08:01.0: Unable to read FCP priority data.
>scsi3 : qla2xxx
>qla2xxx 0000:08:01.0: 
> QLogic Fibre Channel HBA Driver: 8.03.01.05.05.06-k
>  QLogic QMC2462S - PCI-X 2.0 Dual Channel 4Gb Fibre Channel SFF Mezzanine HBA
>  ISP2422: PCI-X Mode 1 (133 MHz) @ 0000:08:01.0 hdma+, host#=3, fw=5.03.02 (496)
>GSI 22 sharing vector 0x6A and IRQ 22
>ACPI: PCI Interrupt 0000:08:01.1[B] -> GSI 26 (level, low) -> IRQ 106
>qla2xxx 0000:08:01.1: Found an ISP2422, irq 106, iobase 0xffffc2000001e000
>qla2xxx 0000:08:01.1: Configuring PCI space...
>qla2xxx 0000:08:01.1: Configure NVRAM parameters...
>qla2xxx 0000:08:01.1: Verifying loaded RISC code...
>qla2xxx 0000:08:01.1: Allocated (64 KB) for EFT...
>qla2xxx 0000:08:01.1: Allocated (1413 KB) for firmware dump...
>qla2xxx 0000:08:01.1: Unable to read FCP priority data.
>scsi4 : qla2xxx
>qla2xxx 0000:08:01.1: 
> QLogic Fibre Channel HBA Driver: 8.03.01.05.05.06-k
>  QLogic QMC2462S - PCI-X 2.0 Dual Channel 4Gb Fibre Channel SFF Mezzanine HBA
>  ISP2422: PCI-X Mode 1 (133 MHz) @ 0000:08:01.1 hdma+, host#=4, fw=5.03.02 (496)
>Initializing USB Mass Storage driver...
>usbcore: registered new driver usb-storage
>USB Mass Storage support registered.
>qla2xxx 0000:08:01.0: LOOP UP detected (4 Gbps).
>qla2xxx 0000:08:01.1: LOOP UP detected (4 Gbps).
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: Universal Xport   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>scsi 3:0:0:31: Attached scsi generic sg1 type 0
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: 1814      FAStT   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>  Vendor: IBM       Model: Universal Xport   Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>scsi 4:0:0:31: Attached scsi generic sg2 type 0
>scsi5 : mpp virtual bus adapter :version:09.03.0C05.0233,timestamp:Fri Aug 21 11:36:33 CDT 2009
>  Vendor: IBM       Model: VirtualDisk       Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>scsi(5:0:0:0): Enabled tagged queuing, queue depth 30.
>SCSI device sdb: 20971520 512-byte hdwr sectors (10737 MB)
>sdb: Write Protect is off
>sdb: Mode Sense: 77 00 10 08
>SCSI device sdb: drive cache: write back w/ FUA
>SCSI device sdb: 20971520 512-byte hdwr sectors (10737 MB)
>sdb: Write Protect is off
>sdb: Mode Sense: 77 00 10 08
>SCSI device sdb: drive cache: write back w/ FUA
> sdb: unknown partition table
>sd 5:0:0:0: Attached scsi disk sdb
>sd 5:0:0:0: Attached scsi generic sg3 type 0
>  Vendor: IBM       Model: VirtualDisk       Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>scsi(5:0:0:1): Enabled tagged queuing, queue depth 30.
>SCSI device sdc: 20971520 512-byte hdwr sectors (10737 MB)
>sdc: Write Protect is off
>sdc: Mode Sense: 77 00 10 08
>SCSI device sdc: drive cache: write back w/ FUA
>SCSI device sdc: 20971520 512-byte hdwr sectors (10737 MB)
>sdc: Write Protect is off
>sdc: Mode Sense: 77 00 10 08
>SCSI device sdc: drive cache: write back w/ FUA
> sdc: unknown partition table
>sd 5:0:0:1: Attached scsi disk sdc
>sd 5:0:0:1: Attached scsi generic sg4 type 0
>  Vendor: IBM       Model: VirtualDisk       Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>scsi(5:0:0:2): Enabled tagged queuing, queue depth 30.
>SCSI device sdd: 20971520 512-byte hdwr sectors (10737 MB)
>sdd: Write Protect is off
>sdd: Mode Sense: 77 00 10 08
>SCSI device sdd: drive cache: write back w/ FUA
>SCSI device sdd: 20971520 512-byte hdwr sectors (10737 MB)
>sdd: Write Protect is off
>sdd: Mode Sense: 77 00 10 08
>SCSI device sdd: drive cache: write back w/ FUA
> sdd: unknown partition table
>sd 5:0:0:2: Attached scsi disk sdd
>sd 5:0:0:2: Attached scsi generic sg5 type 0
>  Vendor: IBM       Model: VirtualDisk       Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>scsi(5:0:0:3): Enabled tagged queuing, queue depth 30.
>SCSI device sde: 20971520 512-byte hdwr sectors (10737 MB)
>sde: Write Protect is off
>sde: Mode Sense: 77 00 10 08
>SCSI device sde: drive cache: write back w/ FUA
>SCSI device sde: 20971520 512-byte hdwr sectors (10737 MB)
>sde: Write Protect is off
>sde: Mode Sense: 77 00 10 08
>SCSI device sde: drive cache: write back w/ FUA
> sde: unknown partition table
>sd 5:0:0:3: Attached scsi disk sde
>sd 5:0:0:3: Attached scsi generic sg6 type 0
>  Vendor: IBM       Model: VirtualDisk       Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>scsi(5:0:0:4): Enabled tagged queuing, queue depth 30.
>SCSI device sdf: 20971520 512-byte hdwr sectors (10737 MB)
>sdf: Write Protect is off
>sdf: Mode Sense: 77 00 10 08
>SCSI device sdf: drive cache: write back w/ FUA
>SCSI device sdf: 20971520 512-byte hdwr sectors (10737 MB)
>sdf: Write Protect is off
>sdf: Mode Sense: 77 00 10 08
>SCSI device sdf: drive cache: write back w/ FUA
> sdf: unknown partition table
>sd 5:0:0:4: Attached scsi disk sdf
>sd 5:0:0:4: Attached scsi generic sg7 type 0
>  Vendor: IBM       Model: VirtualDisk       Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>scsi(5:0:0:5): Enabled tagged queuing, queue depth 30.
>SCSI device sdg: 20971520 512-byte hdwr sectors (10737 MB)
>sdg: Write Protect is off
>sdg: Mode Sense: 77 00 10 08
>SCSI device sdg: drive cache: write back w/ FUA
>SCSI device sdg: 20971520 512-byte hdwr sectors (10737 MB)
>sdg: Write Protect is off
>sdg: Mode Sense: 77 00 10 08
>SCSI device sdg: drive cache: write back w/ FUA
> sdg: unknown partition table
>sd 5:0:0:5: Attached scsi disk sdg
>sd 5:0:0:5: Attached scsi generic sg8 type 0
>  Vendor: IBM       Model: VirtualDisk       Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>scsi(5:0:0:6): Enabled tagged queuing, queue depth 30.
>SCSI device sdh: 20971520 512-byte hdwr sectors (10737 MB)
>sdh: Write Protect is off
>sdh: Mode Sense: 77 00 10 08
>SCSI device sdh: drive cache: write back w/ FUA
>SCSI device sdh: 20971520 512-byte hdwr sectors (10737 MB)
>sdh: Write Protect is off
>sdh: Mode Sense: 77 00 10 08
>SCSI device sdh: drive cache: write back w/ FUA
> sdh: unknown partition table
>sd 5:0:0:6: Attached scsi disk sdh
>sd 5:0:0:6: Attached scsi generic sg9 type 0
>  Vendor: IBM       Model: VirtualDisk       Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>scsi(5:0:0:7): Enabled tagged queuing, queue depth 30.
>SCSI device sdi: 20971520 512-byte hdwr sectors (10737 MB)
>sdi: Write Protect is off
>sdi: Mode Sense: 77 00 10 08
>SCSI device sdi: drive cache: write back w/ FUA
>SCSI device sdi: 20971520 512-byte hdwr sectors (10737 MB)
>sdi: Write Protect is off
>sdi: Mode Sense: 77 00 10 08
>SCSI device sdi: drive cache: write back w/ FUA
> sdi: unknown partition table
>sd 5:0:0:7: Attached scsi disk sdi
>sd 5:0:0:7: Attached scsi generic sg10 type 0
>  Vendor: IBM       Model: VirtualDisk       Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>scsi(5:0:0:8): Enabled tagged queuing, queue depth 30.
>SCSI device sdj: 20971520 512-byte hdwr sectors (10737 MB)
>sdj: Write Protect is off
>sdj: Mode Sense: 77 00 10 08
>SCSI device sdj: drive cache: write back w/ FUA
>SCSI device sdj: 20971520 512-byte hdwr sectors (10737 MB)
>sdj: Write Protect is off
>sdj: Mode Sense: 77 00 10 08
>SCSI device sdj: drive cache: write back w/ FUA
> sdj: unknown partition table
>sd 5:0:0:8: Attached scsi disk sdj
>sd 5:0:0:8: Attached scsi generic sg11 type 0
>  Vendor: IBM       Model: VirtualDisk       Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>scsi(5:0:0:9): Enabled tagged queuing, queue depth 30.
>SCSI device sdk: 20971520 512-byte hdwr sectors (10737 MB)
>sdk: Write Protect is off
>sdk: Mode Sense: 77 00 10 08
>SCSI device sdk: drive cache: write back w/ FUA
>SCSI device sdk: 20971520 512-byte hdwr sectors (10737 MB)
>sdk: Write Protect is off
>sdk: Mode Sense: 77 00 10 08
>SCSI device sdk: drive cache: write back w/ FUA
> sdk: unknown partition table
>sd 5:0:0:9: Attached scsi disk sdk
>sd 5:0:0:9: Attached scsi generic sg12 type 0
>  Vendor: IBM       Model: VirtualDisk       Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>scsi(5:0:0:10): Enabled tagged queuing, queue depth 30.
>SCSI device sdl: 20971520 512-byte hdwr sectors (10737 MB)
>sdl: Write Protect is off
>sdl: Mode Sense: 77 00 10 08
>SCSI device sdl: drive cache: write back w/ FUA
>SCSI device sdl: 20971520 512-byte hdwr sectors (10737 MB)
>sdl: Write Protect is off
>sdl: Mode Sense: 77 00 10 08
>SCSI device sdl: drive cache: write back w/ FUA
> sdl: sdl1
>sd 5:0:0:10: Attached scsi disk sdl
>sd 5:0:0:10: Attached scsi generic sg13 type 0
>  Vendor: IBM       Model: VirtualDisk       Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>scsi(5:0:0:11): Enabled tagged queuing, queue depth 30.
>SCSI device sdm: 20971520 512-byte hdwr sectors (10737 MB)
>sdm: Write Protect is off
>sdm: Mode Sense: 77 00 10 08
>SCSI device sdm: drive cache: write back w/ FUA
>SCSI device sdm: 20971520 512-byte hdwr sectors (10737 MB)
>sdm: Write Protect is off
>sdm: Mode Sense: 77 00 10 08
>SCSI device sdm: drive cache: write back w/ FUA
> sdm: sdm1
>sd 5:0:0:11: Attached scsi disk sdm
>sd 5:0:0:11: Attached scsi generic sg14 type 0
>  Vendor: IBM       Model: VirtualDisk       Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>scsi(5:0:0:12): Enabled tagged queuing, queue depth 30.
>SCSI device sdn: 20971520 512-byte hdwr sectors (10737 MB)
>sdn: Write Protect is off
>sdn: Mode Sense: 77 00 10 08
>SCSI device sdn: drive cache: write back w/ FUA
>SCSI device sdn: 20971520 512-byte hdwr sectors (10737 MB)
>sdn: Write Protect is off
>sdn: Mode Sense: 77 00 10 08
>SCSI device sdn: drive cache: write back w/ FUA
> sdn: sdn1
>sd 5:0:0:12: Attached scsi disk sdn
>sd 5:0:0:12: Attached scsi generic sg15 type 0
>  Vendor: IBM       Model: VirtualDisk       Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>scsi(5:0:0:13): Enabled tagged queuing, queue depth 30.
>SCSI device sdo: 20971520 512-byte hdwr sectors (10737 MB)
>sdo: Write Protect is off
>sdo: Mode Sense: 77 00 10 08
>SCSI device sdo: drive cache: write back w/ FUA
>SCSI device sdo: 20971520 512-byte hdwr sectors (10737 MB)
>sdo: Write Protect is off
>sdo: Mode Sense: 77 00 10 08
>SCSI device sdo: drive cache: write back w/ FUA
> sdo: sdo1
>sd 5:0:0:13: Attached scsi disk sdo
>sd 5:0:0:13: Attached scsi generic sg16 type 0
>  Vendor: IBM       Model: VirtualDisk       Rev: 0916
>  Type:   Direct-Access                      ANSI SCSI revision: 05
>scsi(5:0:0:14): Enabled tagged queuing, queue depth 30.
>SCSI device sdp: 20971520 512-byte hdwr sectors (10737 MB)
>sdp: Write Protect is off
>sdp: Mode Sense: 77 00 10 08
>SCSI device sdp: drive cache: write back w/ FUA
>SCSI device sdp: 20971520 512-byte hdwr sectors (10737 MB)
>sdp: Write Protect is off
>sdp: Mode Sense: 77 00 10 08
>SCSI device sdp: drive cache: write back w/ FUA
> sdp: sdp1
>sd 5:0:0:14: Attached scsi disk sdp
>sd 5:0:0:14: Attached scsi generic sg17 type 0
>device-mapper: uevent: version 1.0.3
>device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com
>device-mapper: dm-raid45: initialized v0.2594l
>usb 3-2: USB disconnect, address 2
>EXT3-fs: INFO: recovery required on readonly filesystem.
>EXT3-fs: write access will be enabled during recovery.
>kjournald starting.  Commit interval 5 seconds
>EXT3-fs: recovery complete.
>EXT3-fs: mounted filesystem with ordered data mode.
>SELinux:  Disabled at runtime.
>SELinux:  Unregistering netfilter hooks
>type=1404 audit(1287003732.071:2): selinux=0 auid=4294967295 ses=4294967295
>Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.0.8 (Feb 15, 2010)
>GSI 23 sharing vector 0x72 and IRQ 23
>ACPI: PCI Interrupt 0000:04:00.0[A] -> GSI 17 (level, low) -> IRQ 114
>eth0: Broadcom NetXtreme II BCM5708 1000Base-SX (B2) PCI-X 64-bit 133MHz found at mem da000000, IRQ 114, node addr 001a64598fe6
>ACPI: PCI Interrupt 0000:06:00.0[A] -> GSI 19 (level, low) -> IRQ 90
>eth1: Broadcom NetXtreme II BCM5708 1000Base-SX (B2) PCI-X 64-bit 133MHz found at mem d8000000, IRQ 90, node addr 001a64598fe8
>intel_rng: FWH not detected
>input: PC Speaker as /class/input/input2
>EDAC MC: Ver: 2.0.1 Oct  7 2010
>EDAC MC0: Giving out device to i5000_edac.c I5000: DEV 0000:00:10.0
>Floppy drive(s): fd0 is 1.44M
>floppy0: no floppy controllers found
>work still pending
>kvm: virtualization flags detected on this hardware: vmx tpr_shadow vnmi flexpriority
>loaded kvm module (kvm-83-164.el5)
>ksm loaded
>Floppy drive(s): fd0 is 1.44M
>floppy0: no floppy controllers found
>work still pending
>lp: driver loaded but no devices found
>ACPI: Power Button (FF) [PWRF]
>ACPI: Mapper loaded
>dell-wmi: No known WMI GUID found
>md: Autodetecting RAID arrays.
>md: autorun ...
>md: ... autorun DONE.
>device-mapper: multipath: version 1.0.5 loaded
>EXT3 FS on sda1, internal journal
>Adding 2096472k swap on /dev/sda2.  Priority:-1 extents:1 across:2096472k
>IA-32 Microcode Update Driver: v1.14a <tigran@veritas.com>
>microcode: CPU2 updated from revision 0xb7 to 0xb9, date = 05112009 
>microcode: CPU3 updated from revision 0xb7 to 0xb9, date = 05112009 
>microcode: CPU1 updated from revision 0xb7 to 0xb9, date = 05112009 
>microcode: CPU0 updated from revision 0xb7 to 0xb9, date = 05112009 
>Loading iSCSI transport class v2.0-871.
>802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>
>All bugs added by David S. Miller <davem@redhat.com>
>cxgb3i: tag itt 0x1fff, 13 bits, age 0xf, 4 bits.
>iscsi: registered transport (cxgb3i)
>NET: Registered protocol family 10
>lo: Disabled Privacy Extensions
>IPv6 over IPv4 tunneling driver
>Broadcom NetXtreme II CNIC Driver cnic v2.1.2 (May 26, 2010)
>cnic: Added CNIC device: eth0
>cnic: Added CNIC device: eth1
>Broadcom NetXtreme II iSCSI Driver bnx2i v2.1.3 (Aug 10, 2010)
>iscsi: registered transport (bnx2i)
>scsi6 : Broadcom Offload iSCSI Initiator
>scsi7 : Broadcom Offload iSCSI Initiator
>iscsi: registered transport (tcp)
>iscsi: registered transport (iser)
>iscsi: registered transport (be2iscsi)
>bnx2: eth0: using MSI
>ADDRCONF(NETDEV_UP): eth0: link is not ready
>bnx2i: iSCSI not supported, dev=eth0
>bnx2: eth0 NIC SerDes Link is Up, 1000 Mbps full duplex
>ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>bnx2i: iSCSI not supported, dev=eth0
>bnx2: eth1: using MSI
>ADDRCONF(NETDEV_UP): eth1: link is not ready
>bnx2i: iSCSI not supported, dev=eth1
>bnx2: eth1 NIC SerDes Link is Up, 1000 Mbps full duplex
>ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
>bnx2i: iSCSI not supported, dev=eth1
>DLM (built Oct  7 2010 20:56:18) installed
>GFS2 (built Oct  7 2010 20:56:59) installed
>Lock_DLM (built Oct  7 2010 20:57:06) installed
>eth0: no IPv6 routers present
>eth1: no IPv6 routers present
>device-mapper: dm-log-clustered: (built Dec 22 2009 14:39:43) installed
>GFS2: fsid=: Trying to join cluster "lock_dlm", "rhel55:appl01"
>dlm: Using TCP for communications
>dlm: connecting to 4
>dlm: connecting to 3
>dlm: connecting to 2
>dlm: got connection from 2
>dlm: got connection from 4
>dlm: got connection from 3
>GFS2: fsid=rhel55:appl01.3: Joined cluster. Now mounting FS...
>GFS2: fsid=rhel55:appl01.3: jid=3, already locked for use
>GFS2: fsid=rhel55:appl01.3: jid=3: Looking at journal...
>GFS2: fsid=rhel55:appl01.3: jid=3: Done
>GFS2: fsid=: Trying to join cluster "lock_dlm", "rhel55:data01"
>GFS2: fsid=rhel55:data01.1: Joined cluster. Now mounting FS...
>GFS2: fsid=rhel55:data01.1: jid=1, already locked for use
>GFS2: fsid=rhel55:data01.1: jid=1: Looking at journal...
>GFS2: fsid=rhel55:data01.1: jid=1: Done
>GFS2: fsid=: Trying to join cluster "lock_dlm", "rhel55:data02"
>GFS2: fsid=rhel55:data02.1: Joined cluster. Now mounting FS...
>GFS2: fsid=rhel55:data02.1: jid=1, already locked for use
>GFS2: fsid=rhel55:data02.1: jid=1: Looking at journal...
>GFS2: fsid=rhel55:data02.1: jid=1: Done
>GFS2: fsid=: Trying to join cluster "lock_dlm", "rhel55:data03"
>GFS2: fsid=rhel55:data03.1: Joined cluster. Now mounting FS...
>GFS2: fsid=rhel55:data03.1: jid=1, already locked for use
>GFS2: fsid=rhel55:data03.1: jid=1: Looking at journal...
>GFS2: fsid=rhel55:data03.1: jid=1: Done
>GFS2: fsid=: Trying to join cluster "lock_dlm", "rhel55:data04"
>GFS2: fsid=rhel55:data04.1: Joined cluster. Now mounting FS...
>GFS2: fsid=rhel55:data04.1: jid=1, already locked for use
>GFS2: fsid=rhel55:data04.1: jid=1: Looking at journal...
>GFS2: fsid=rhel55:data04.1: jid=1: Done
>Bluetooth: Core ver 2.10
>NET: Registered protocol family 31
>Bluetooth: HCI device and connection manager initialized
>Bluetooth: HCI socket layer initialized
>Bluetooth: L2CAP ver 2.8
>Bluetooth: L2CAP socket layer initialized
>Bluetooth: HIDP (Human Interface Emulation) ver 1.1
>Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
>NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
>NFSD: starting 90-second grace period
>FS-Cache: Loaded
>Bridge firewalling registered
>ip_tables: (C) 2000-2006 Netfilter Core Team
>Netfilter messages via NETLINK v0.30.
>ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack
>99 [RAIDarray.mpp]RHEL_DS4700:1:0:0 mpp status 7
>7 [RAIDarray.mpp]RHEL_DS4700:1:0 Path Failed
>495 [RAIDarray.mpp]RHEL_DS4700:1:0:0 Cmnd failed-retry on a new path. vcmnd SN 2430 pdev H3:C0:T0:L0 0x00/0x00/0x00 0x00070000 mpp_status:7
>99 [RAIDarray.mpp]RHEL_DS4700:1:1:0 mpp status 7
>492 [RAIDarray.mpp]RHEL_DS4700:1:1:0 IO FAILURE. vcmnd SN 2430 pdev H4:C0:T0:L0 0x00/0x00/0x00 0x00070000 mpp_status:7
>99 [RAIDarray.mpp]RHEL_DS4700:1:1:1 mpp status 7
>492 [RAIDarray.mpp]RHEL_DS4700:1:1:1 IO FAILURE. vcmnd SN 2436 pdev H4:C0:T0:L1 0x00/0x00/0x00 0x00070000 mpp_status:7
>99 [RAIDarray.mpp]RHEL_DS4700:1:1:2 mpp status 7
>492 [RAIDarray.mpp]RHEL_DS4700:1:1:2 IO FAILURE. vcmnd SN 2442 pdev H4:C0:T0:L2 0x00/0x00/0x00 0x00070000 mpp_status:7
>99 [RAIDarray.mpp]RHEL_DS4700:1:1:3 mpp status 7
>492 [RAIDarray.mpp]RHEL_DS4700:1:1:3 IO FAILURE. vcmnd SN 2448 pdev H4:C0:T0:L3 0x00/0x00/0x00 0x00070000 mpp_status:7
>99 [RAIDarray.mpp]RHEL_DS4700:1:1:4 mpp status 7
>492 [RAIDarray.mpp]RHEL_DS4700:1:1:4 IO FAILURE. vcmnd SN 2454 pdev H4:C0:T0:L4 0x00/0x00/0x00 0x00070000 mpp_status:7
>99 [RAIDarray.mpp]RHEL_DS4700:1:1:5 mpp status 7
>492 [RAIDarray.mpp]RHEL_DS4700:1:1:5 IO FAILURE. vcmnd SN 2460 pdev H4:C0:T0:L5 0x00/0x00/0x00 0x00070000 mpp_status:7
>99 [RAIDarray.mpp]RHEL_DS4700:1:1:6 mpp status 7
>492 [RAIDarray.mpp]RHEL_DS4700:1:1:6 IO FAILURE. vcmnd SN 2466 pdev H4:C0:T0:L6 0x00/0x00/0x00 0x00070000 mpp_status:7
>99 [RAIDarray.mpp]RHEL_DS4700:1:1:7 mpp status 7
>492 [RAIDarray.mpp]RHEL_DS4700:1:1:7 IO FAILURE. vcmnd SN 2472 pdev H4:C0:T0:L7 0x00/0x00/0x00 0x00070000 mpp_status:7
>99 [RAIDarray.mpp]RHEL_DS4700:1:1:8 mpp status 7
>492 [RAIDarray.mpp]RHEL_DS4700:1:1:8 IO FAILURE. vcmnd SN 2478 pdev H4:C0:T0:L8 0x00/0x00/0x00 0x00070000 mpp_status:7
>99 [RAIDarray.mpp]RHEL_DS4700:1:1:9 mpp status 7
>492 [RAIDarray.mpp]RHEL_DS4700:1:1:9 IO FAILURE. vcmnd SN 2484 pdev H4:C0:T0:L9 0x00/0x00/0x00 0x00070000 mpp_status:7
>99 [RAIDarray.mpp]RHEL_DS4700:1:1:10 mpp status 7
>492 [RAIDarray.mpp]RHEL_DS4700:1:1:10 IO FAILURE. vcmnd SN 2490 pdev H4:C0:T0:L10 0x00/0x00/0x00 0x00070000 mpp_status:7
>99 [RAIDarray.mpp]RHEL_DS4700:1:1:11 mpp status 7
>492 [RAIDarray.mpp]RHEL_DS4700:1:1:11 IO FAILURE. vcmnd SN 2496 pdev H4:C0:T0:L11 0x00/0x00/0x00 0x00070000 mpp_status:7
>99 [RAIDarray.mpp]RHEL_DS4700:1:1:12 mpp status 7
>492 [RAIDarray.mpp]RHEL_DS4700:1:1:12 IO FAILURE. vcmnd SN 2502 pdev H4:C0:T0:L12 0x00/0x00/0x00 0x00070000 mpp_status:7
>99 [RAIDarray.mpp]RHEL_DS4700:1:1:13 mpp status 7
>492 [RAIDarray.mpp]RHEL_DS4700:1:1:13 IO FAILURE. vcmnd SN 2508 pdev H4:C0:T0:L13 0x00/0x00/0x00 0x00070000 mpp_status:7
>99 [RAIDarray.mpp]RHEL_DS4700:1:1:14 mpp status 7
>492 [RAIDarray.mpp]RHEL_DS4700:1:1:14 IO FAILURE. vcmnd SN 2514 pdev H4:C0:T0:L14 0x00/0x00/0x00 0x00070000 mpp_status:7
>usb 3-2: new full speed USB device using uhci_hcd and address 3
>usb 3-2: configuration #1 chosen from 1 choice
>input: IBM IBM MM2 as /class/input/input3
>input: USB HID v1.10 Keyboard [IBM IBM MM2] on usb-0000:00:1d.1-2
>input: IBM IBM MM2 as /class/input/input4
>input: USB HID v1.10 Mouse [IBM IBM MM2] on usb-0000:00:1d.1-2
>virbr0: no IPv6 routers present
>usb 3-2: USB disconnect, address 3
>INFO: task dcs_operation_m:25109 blocked for more than 120 seconds.
>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>dcs_operation D ffff8107d2fd5420     0 25109  17854         25118 25085 (NOTLB)
> ffff8107ce993c78 0000000000000086 0000000000000002 ffffffff88843ef7
> ffff8107ce993c28 0000000000000009 ffff8107ce8ec100 ffff8107f5b917e0
> 0000010ee490f84b 000000000000159d ffff8107ce8ec2e8 000000020000000a
>Call Trace:
> [<ffffffff88843ef7>] :gfs2:just_schedule+0x0/0xe
> [<ffffffff88843ef7>] :gfs2:just_schedule+0x0/0xe
> [<ffffffff80063a90>] out_of_line_wait_on_bit+0x6c/0x78
> [<ffffffff80063c4f>] __mutex_lock_slowpath+0x60/0x9b
> [<ffffffff888539fe>] :gfs2:gfs2_permission+0xaf/0xd5
> [<ffffffff80063c99>] .text.lock.mutex+0xf/0x14
> [<ffffffff8000cfb3>] do_lookup+0x90/0x1e6
> [<ffffffff8000a2c5>] __link_path_walk+0xa2a/0xfb9
> [<ffffffff8000ea74>] link_path_walk+0x42/0xb2
> [<ffffffff8000cda3>] do_path_lookup+0x275/0x2f1
> [<ffffffff8001288f>] getname+0x15b/0x1c2
> [<ffffffff8002399e>] __user_walk_fd+0x37/0x4c
> [<ffffffff800288a2>] vfs_stat_fd+0x1b/0x4a
> [<ffffffff800236d0>] sys_newstat+0x19/0x31
> [<ffffffff8005d229>] tracesys+0x71/0xe0
> [<ffffffff8005d28d>] tracesys+0xd5/0xe0
>
>INFO: task dcs_output_mgr:30119 blocked for more than 120 seconds.
>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>dcs_output_mg D ffff81011cac29a8     0 30119  17854         30167 30089 (NOTLB)
> ffff81079a873ba8 0000000000000082 0000000000000018 ffffffff888184f8
> 0000000000000292 0000000000000009 ffff81079af53080 ffff81079ba490c0
> 0000025ab603485e 0000000000028c79 ffff81079af53268 0000000188819e5f
>Call Trace:
> [<ffffffff888184f8>] :dlm:request_lock+0x93/0xa0
> [<ffffffff888bf556>] :lock_dlm:gdlm_ast+0x0/0x311
> [<ffffffff888bf2c1>] :lock_dlm:gdlm_bast+0x0/0x8d
> [<ffffffff88843ef7>] :gfs2:just_schedule+0x0/0xe
> [<ffffffff88843f00>] :gfs2:just_schedule+0x9/0xe
> [<ffffffff800639f6>] __wait_on_bit+0x40/0x6e
> [<ffffffff88843ef7>] :gfs2:just_schedule+0x0/0xe
> [<ffffffff80063a90>] out_of_line_wait_on_bit+0x6c/0x78
> [<ffffffff800a28c4>] wake_bit_function+0x0/0x23
> [<ffffffff88843ef2>] :gfs2:gfs2_glock_wait+0x2b/0x30
> [<ffffffff888539d2>] :gfs2:gfs2_permission+0x83/0xd5
> [<ffffffff888539ca>] :gfs2:gfs2_permission+0x7b/0xd5
> [<ffffffff8000cf88>] do_lookup+0x65/0x1e6
> [<ffffffff8000da09>] permission+0x81/0xc8
> [<ffffffff80009a0e>] __link_path_walk+0x173/0xfb9
> [<ffffffff8000ea74>] link_path_walk+0x42/0xb2
> [<ffffffff8000cda3>] do_path_lookup+0x275/0x2f1
> [<ffffffff800237b2>] __path_lookup_intent_open+0x56/0x97
> [<ffffffff8001b0bf>] open_namei+0x73/0x712
> [<ffffffff800274e7>] do_filp_open+0x1c/0x38
> [<ffffffff80019f75>] do_sys_open+0x44/0xbe
> [<ffffffff8005d28d>] tracesys+0xd5/0xe0
>
>INFO: task dcs_output_mgr:30167 blocked for more than 120 seconds.
>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>dcs_output_mg D ffff81011cac29a8     0 30167  17854         30177 30119 (NOTLB)
> ffff810799145ba8 0000000000000086 0000000000000018 ffffffff888184f8
> 0000000000000292 0000000000000009 ffff81079a89f7e0 ffff8107a521d7a0
> 00000259c430267d 0000000000000451 ffff81079a89f9c8 0000000188819e5f
>Call Trace:
> [<ffffffff888184f8>] :dlm:request_lock+0x93/0xa0
> [<ffffffff88843ef7>] :gfs2:just_schedule+0x0/0xe
> [<ffffffff88843f00>] :gfs2:just_schedule+0x9/0xe
> [<ffffffff800639f6>] __wait_on_bit+0x40/0x6e
> [<ffffffff88843ef7>] :gfs2:just_schedule+0x0/0xe
> [<ffffffff80063a90>] out_of_line_wait_on_bit+0x6c/0x78
> [<ffffffff800a28c4>] wake_bit_function+0x0/0x23
> [<ffffffff88843ef2>] :gfs2:gfs2_glock_wait+0x2b/0x30
> [<ffffffff888539d2>] :gfs2:gfs2_permission+0x83/0xd5
> [<ffffffff888539ca>] :gfs2:gfs2_permission+0x7b/0xd5
> [<ffffffff8000cf88>] do_lookup+0x65/0x1e6
> [<ffffffff8000da09>] permission+0x81/0xc8
> [<ffffffff80009a0e>] __link_path_walk+0x173/0xfb9
> [<ffffffff8000ea74>] link_path_walk+0x42/0xb2
> [<ffffffff8000cda3>] do_path_lookup+0x275/0x2f1
> [<ffffffff800237b2>] __path_lookup_intent_open+0x56/0x97
> [<ffffffff8001b0bf>] open_namei+0x73/0x712
> [<ffffffff800274e7>] do_filp_open+0x1c/0x38
> [<ffffffff80019f75>] do_sys_open+0x44/0xbe
> [<ffffffff8005d28d>] tracesys+0xd5/0xe0
>
>INFO: task dcs_operation_m:432 blocked for more than 120 seconds.
>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>dcs_operation D ffff81000901d7a0     0   432  17854           517 19773 (NOTLB)
> ffff8107a1e13bc8 0000000000000086 0000000000000018 ffffffff888184f8
> 0000000000000296 0000000000000009 ffff8107b5929080 ffff81011cba9100
> 000002519776d0a1 0000000000013a72 ffff8107b5929268 0000000388819e5f
>Call Trace:
> [<ffffffff888184f8>] :dlm:request_lock+0x93/0xa0
> [<ffffffff88843ef7>] :gfs2:just_schedule+0x0/0xe
> [<ffffffff88843f00>] :gfs2:just_schedule+0x9/0xe
> [<ffffffff800639f6>] __wait_on_bit+0x40/0x6e
> [<ffffffff88843ef7>] :gfs2:just_schedule+0x0/0xe
> [<ffffffff80063a90>] out_of_line_wait_on_bit+0x6c/0x78
> [<ffffffff800a28c4>] wake_bit_function+0x0/0x23
> [<ffffffff88843ef2>] :gfs2:gfs2_glock_wait+0x2b/0x30
> [<ffffffff888539d2>] :gfs2:gfs2_permission+0x83/0xd5
> [<ffffffff888539ca>] :gfs2:gfs2_permission+0x7b/0xd5
> [<ffffffff8000cf88>] do_lookup+0x65/0x1e6
> [<ffffffff8000da09>] permission+0x81/0xc8
> [<ffffffff80009a0e>] __link_path_walk+0x173/0xfb9
> [<ffffffff8000ea74>] link_path_walk+0x42/0xb2
> [<ffffffff8000cda3>] do_path_lookup+0x275/0x2f1
> [<ffffffff8001288f>] getname+0x15b/0x1c2
> [<ffffffff8002399e>] __user_walk_fd+0x37/0x4c
> [<ffffffff800288a2>] vfs_stat_fd+0x1b/0x4a
> [<ffffffff8000e50c>] free_pages_and_swap_cache+0x67/0x7e
> [<ffffffff800236d0>] sys_newstat+0x19/0x31
> [<ffffffff8005d229>] tracesys+0x71/0xe0
> [<ffffffff8005d28d>] tracesys+0xd5/0xe0
>
>INFO: task dcs_operation_m:3979 blocked for more than 120 seconds.
>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>dcs_operation D ffff81011cb6c0c0     0  3979  17854          4080  1635 (NOTLB)
> ffff8107b926fbc8 0000000000000082 0000000000000018 ffffffff888184f8
> 0000000000000296 0000000000000009 ffff81079f3bc040 ffff81011cb6c0c0
> 0000025fe51d35f4 0000000000004bd7 ffff81079f3bc228 0000000188819e5f
>Call Trace:
> [<ffffffff888184f8>] :dlm:request_lock+0x93/0xa0
> [<ffffffff888bf556>] :lock_dlm:gdlm_ast+0x0/0x311
> [<ffffffff888bf2c1>] :lock_dlm:gdlm_bast+0x0/0x8d
> [<ffffffff88843ef7>] :gfs2:just_schedule+0x0/0xe
> [<ffffffff88843f00>] :gfs2:just_schedule+0x9/0xe
> [<ffffffff800639f6>] __wait_on_bit+0x40/0x6e
> [<ffffffff88843ef7>] :gfs2:just_schedule+0x0/0xe
> [<ffffffff80063a90>] out_of_line_wait_on_bit+0x6c/0x78
> [<ffffffff800a28c4>] wake_bit_function+0x0/0x23
> [<ffffffff88843ef2>] :gfs2:gfs2_glock_wait+0x2b/0x30
> [<ffffffff888539d2>] :gfs2:gfs2_permission+0x83/0xd5
> [<ffffffff888539ca>] :gfs2:gfs2_permission+0x7b/0xd5
> [<ffffffff8000cf88>] do_lookup+0x65/0x1e6
> [<ffffffff8000da09>] permission+0x81/0xc8
> [<ffffffff80009a0e>] __link_path_walk+0x173/0xfb9
> [<ffffffff8000ea74>] link_path_walk+0x42/0xb2
> [<ffffffff8000cda3>] do_path_lookup+0x275/0x2f1
> [<ffffffff8001288f>] getname+0x15b/0x1c2
> [<ffffffff8002399e>] __user_walk_fd+0x37/0x4c
> [<ffffffff800288a2>] vfs_stat_fd+0x1b/0x4a
> [<ffffffff80062ff0>] thread_return+0x62/0xfe
> [<ffffffff800236d0>] sys_newstat+0x19/0x31
> [<ffffffff8005d229>] tracesys+0x71/0xe0
> [<ffffffff8005d28d>] tracesys+0xd5/0xe0
>
>INFO: task dcs_operation_m:9031 blocked for more than 120 seconds.
>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>dcs_operation D ffff81011cabc9a8     0  9031  17854          9034  9025 (NOTLB)
> ffff81079eba7bc8 0000000000000086 ffff81079eba7ba8 0000000000060f83
> ffff81011b7d0600 0000000000000009 ffff8107a094b820 ffff81011cb6c0c0
> 0000025522117055 0000000000013d2f ffff8107a094ba08 0000000100000010
>Call Trace:
> [<ffffffff8883ec2c>] :gfs2:gfs2_dirent_find+0x32/0x4e
> [<ffffffff8883ebfa>] :gfs2:gfs2_dirent_find+0x0/0x4e
> [<ffffffff8883e453>] :gfs2:gfs2_dirent_scan+0xb1/0x175
> [<ffffffff8883ebfa>] :gfs2:gfs2_dirent_find+0x0/0x4e
> [<ffffffff88843ef7>] :gfs2:just_schedule+0x0/0xe
> [<ffffffff88843f00>] :gfs2:just_schedule+0x9/0xe
> [<ffffffff800639f6>] __wait_on_bit+0x40/0x6e
> [<ffffffff88843ef7>] :gfs2:just_schedule+0x0/0xe
> [<ffffffff80063a90>] out_of_line_wait_on_bit+0x6c/0x78
> [<ffffffff800a28c4>] wake_bit_function+0x0/0x23
> [<ffffffff88843ef2>] :gfs2:gfs2_glock_wait+0x2b/0x30
> [<ffffffff888539d2>] :gfs2:gfs2_permission+0x83/0xd5
> [<ffffffff888539ca>] :gfs2:gfs2_permission+0x7b/0xd5
> [<ffffffff8000cf88>] do_lookup+0x65/0x1e6
> [<ffffffff8000da09>] permission+0x81/0xc8
> [<ffffffff80009a0e>] __link_path_walk+0x173/0xfb9
> [<ffffffff8000ea74>] link_path_walk+0x42/0xb2
> [<ffffffff8000cda3>] do_path_lookup+0x275/0x2f1
> [<ffffffff8001288f>] getname+0x15b/0x1c2
> [<ffffffff8002399e>] __user_walk_fd+0x37/0x4c
> [<ffffffff800288a2>] vfs_stat_fd+0x1b/0x4a
> [<ffffffff8006723e>] do_page_fault+0x4fe/0x874
> [<ffffffff80062ff0>] thread_return+0x62/0xfe
> [<ffffffff800236d0>] sys_newstat+0x19/0x31
> [<ffffffff8005d229>] tracesys+0x71/0xe0
> [<ffffffff8005d28d>] tracesys+0xd5/0xe0
>
>INFO: task dcs_operation_m:9034 blocked for more than 120 seconds.
>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>dcs_operation D ffff81011cabc990     0  9034  17854          9041  9031 (NOTLB)
> ffff8107ac9ffbc8 0000000000000082 ffff8107ac9ffba8 0000000000060f83
> ffff81011b7d0600 0000000000000009 ffff8107bb00e040 ffff8107cc5830c0
> 00000254a5401478 0000000000000661 ffff8107bb00e228 0000000100000010
>Call Trace:
> [<ffffffff88843ef7>] :gfs2:just_schedule+0x0/0xe
> [<ffffffff88843f00>] :gfs2:just_schedule+0x9/0xe
> [<ffffffff800639f6>] __wait_on_bit+0x40/0x6e
> [<ffffffff88843ef7>] :gfs2:just_schedule+0x0/0xe
> [<ffffffff80063a90>] out_of_line_wait_on_bit+0x6c/0x78
> [<ffffffff800a28c4>] wake_bit_function+0x0/0x23
> [<ffffffff88843ef2>] :gfs2:gfs2_glock_wait+0x2b/0x30
> [<ffffffff888539d2>] :gfs2:gfs2_permission+0x83/0xd5
> [<ffffffff888539ca>] :gfs2:gfs2_permission+0x7b/0xd5
> [<ffffffff8000cf88>] do_lookup+0x65/0x1e6
> [<ffffffff8000da09>] permission+0x81/0xc8
> [<ffffffff80009a0e>] __link_path_walk+0x173/0xfb9
> [<ffffffff8000ea74>] link_path_walk+0x42/0xb2
> [<ffffffff8000cda3>] do_path_lookup+0x275/0x2f1
> [<ffffffff8001288f>] getname+0x15b/0x1c2
> [<ffffffff8002399e>] __user_walk_fd+0x37/0x4c
> [<ffffffff800288a2>] vfs_stat_fd+0x1b/0x4a
> [<ffffffff8006723e>] do_page_fault+0x4fe/0x874
> [<ffffffff800236d0>] sys_newstat+0x19/0x31
> [<ffffffff8005d229>] tracesys+0x71/0xe0
> [<ffffffff8005d28d>] tracesys+0xd5/0xe0
>
>INFO: task dcs_operation_m:9041 blocked for more than 120 seconds.
>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>dcs_operation D ffff810009015120     0  9041  17854          9049  9034 (NOTLB)
> ffff8107d1dafbc8 0000000000000086 ffff8107d1dafba8 0000000000060f83
> ffff81011b7d0600 0000000000000009 ffff810790d57820 ffff81011cb9d080
> 00000254ab315bd1 0000000000023304 ffff810790d57a08 0000000200000010
>Call Trace:
> [<ffffffff8883ec2c>] :gfs2:gfs2_dirent_find+0x32/0x4e
> [<ffffffff88843ef7>] :gfs2:just_schedule+0x0/0xe
> [<ffffffff88843f00>] :gfs2:just_schedule+0x9/0xe
> [<ffffffff800639f6>] __wait_on_bit+0x40/0x6e
> [<ffffffff88843ef7>] :gfs2:just_schedule+0x0/0xe
> [<ffffffff80063a90>] out_of_line_wait_on_bit+0x6c/0x78
> [<ffffffff800a28c4>] wake_bit_function+0x0/0x23
> [<ffffffff88843ef2>] :gfs2:gfs2_glock_wait+0x2b/0x30
> [<ffffffff888539d2>] :gfs2:gfs2_permission+0x83/0xd5
> [<ffffffff888539ca>] :gfs2:gfs2_permission+0x7b/0xd5
> [<ffffffff8000cf88>] do_lookup+0x65/0x1e6
> [<ffffffff8000da09>] permission+0x81/0xc8
> [<ffffffff80009a0e>] __link_path_walk+0x173/0xfb9
> [<ffffffff8000ea74>] link_path_walk+0x42/0xb2
> [<ffffffff8000cda3>] do_path_lookup+0x275/0x2f1
> [<ffffffff8001288f>] getname+0x15b/0x1c2
> [<ffffffff8002399e>] __user_walk_fd+0x37/0x4c
> [<ffffffff800288a2>] vfs_stat_fd+0x1b/0x4a
> [<ffffffff8006723e>] do_page_fault+0x4fe/0x874
> [<ffffffff800236d0>] sys_newstat+0x19/0x31
> [<ffffffff8005d229>] tracesys+0x71/0xe0
> [<ffffffff8005d28d>] tracesys+0xd5/0xe0
>
>INFO: task dcs_operation_m:9943 blocked for more than 120 seconds.
>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>dcs_operation D ffff810009015120     0  9943  17854          9948  9105 (NOTLB)
> ffff810788b17b28 0000000000000082 00000000fffffff5 ffff81082dc9e800
> 0000000000000296 0000000000000009 ffff81078c6bc860 ffff81011cb9d080
> 0000024a164a548f 0000000000000f29 ffff81078c6bca48 0000000288819e5f
>Call Trace:
> [<ffffffff88843ef7>] :gfs2:just_schedule+0x0/0xe
> [<ffffffff88843f00>] :gfs2:just_schedule+0x9/0xe
> [<ffffffff800639f6>] __wait_on_bit+0x40/0x6e
> [<ffffffff88843ef7>] :gfs2:just_schedule+0x0/0xe
> [<ffffffff80063a90>] out_of_line_wait_on_bit+0x6c/0x78
> [<ffffffff800a28c4>] wake_bit_function+0x0/0x23
> [<ffffffff88843ef2>] :gfs2:gfs2_glock_wait+0x2b/0x30
> [<ffffffff88847589>] :gfs2:gfs2_createi+0x6d/0xd22
> [<ffffffff8883ebfa>] :gfs2:gfs2_dirent_find+0x0/0x4e
> [<ffffffff8884bc61>] :gfs2:gfs2_meta_read+0x17/0x65
> [<ffffffff8883e453>] :gfs2:gfs2_dirent_scan+0xb1/0x175
> [<ffffffff8883ebfa>] :gfs2:gfs2_dirent_find+0x0/0x4e
> [<ffffffff8002d0ac>] wake_up_bit+0x11/0x22
> [<ffffffff88843deb>] :gfs2:do_promote+0xf5/0x137
> [<ffffffff88852f8d>] :gfs2:gfs2_create+0x65/0x143
> [<ffffffff8884757f>] :gfs2:gfs2_createi+0x63/0xd22
> [<ffffffff8000984c>] __d_lookup+0xb0/0xff
> [<ffffffff8003a670>] vfs_create+0xe6/0x158
> [<ffffffff8001b1e9>] open_namei+0x19d/0x712
> [<ffffffff800274e7>] do_filp_open+0x1c/0x38
> [<ffffffff80019f75>] do_sys_open+0x44/0xbe
> [<ffffffff8005d28d>] tracesys+0xd5/0xe0
>
>INFO: task dcs_output_mgr:30119 blocked for more than 120 seconds.
>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>dcs_output_mg D ffff81011cac29a8     0 30119  17854         30167 30089 (NOTLB)
> ffff81079a873ba8 0000000000000082 0000000000000018 ffffffff888184f8
> 0000000000000292 0000000000000009 ffff81079af53080 ffff81079ba490c0
> 0000025ab603485e 0000000000028c79 ffff81079af53268 0000000188819e5f
>Call Trace:
> [<ffffffff888184f8>] :dlm:request_lock+0x93/0xa0
> [<ffffffff888bf556>] :lock_dlm:gdlm_ast+0x0/0x311
> [<ffffffff888bf2c1>] :lock_dlm:gdlm_bast+0x0/0x8d
> [<ffffffff88843ef7>] :gfs2:just_schedule+0x0/0xe
> [<ffffffff88843f00>] :gfs2:just_schedule+0x9/0xe
> [<ffffffff800639f6>] __wait_on_bit+0x40/0x6e
> [<ffffffff88843ef7>] :gfs2:just_schedule+0x0/0xe
> [<ffffffff80063a90>] out_of_line_wait_on_bit+0x6c/0x78
> [<ffffffff800a28c4>] wake_bit_function+0x0/0x23
> [<ffffffff88843ef2>] :gfs2:gfs2_glock_wait+0x2b/0x30
> [<ffffffff888539d2>] :gfs2:gfs2_permission+0x83/0xd5
> [<ffffffff888539ca>] :gfs2:gfs2_permission+0x7b/0xd5
> [<ffffffff8000cf88>] do_lookup+0x65/0x1e6
> [<ffffffff8000da09>] permission+0x81/0xc8
> [<ffffffff80009a0e>] __link_path_walk+0x173/0xfb9
> [<ffffffff8000ea74>] link_path_walk+0x42/0xb2
> [<ffffffff8000cda3>] do_path_lookup+0x275/0x2f1
> [<ffffffff800237b2>] __path_lookup_intent_open+0x56/0x97
> [<ffffffff8001b0bf>] open_namei+0x73/0x712
> [<ffffffff800274e7>] do_filp_open+0x1c/0x38
> [<ffffffff80019f75>] do_sys_open+0x44/0xbe
> [<ffffffff8005d28d>] tracesys+0xd5/0xe0
>
> G:  s:UN n:2/7cd19 f: t:UN d:EX/0 l:0 a:0 r:0
>----------- [cut here ] --------- [please bite here ] ---------
>Kernel BUG at fs/gfs2/glock.c:209
>invalid opcode: 0000 [1] SMP 
>last sysfs file: /devices/pci0000:00/0000:00:00.0/irq
>CPU 2 
>Modules linked in: ipt_MASQUERADE iptable_nat ip_nat xt_state ip_conntrack nfnetlink ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge nfs fscache nfsd exportfs nfs_acl auth_rpcgss autofs4 hidp l2cap bluetooth dm_log_clustered(U) lock_dlm gfs2 dlm configfs lockd sunrpc be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_addr iscsi_tcp bnx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i iw_cxgb3 ib_core cxgb3 8021q libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport ksm(U) kvm_intel(U) kvm(U) i5000_edac tpm_tis edac_mc pcspkr tpm bnx2 tpm_bios dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod mppVhba(U) usb_storage qla2xxx scsi_transport_fc ata_piix libata shpchp mptsas mptscsih mptbase scsi_transport_sas mppUpper(U) sg sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
>Pid: 30177, comm: dcs_output_mgr Tainted: G      2.6.18-226.el5 #1
>RIP: 0010:[<ffffffff88844351>]  [<ffffffff88844351>] :gfs2:gfs2_glock_hold+0x15/0x25
>RSP: 0000:ffff810798f0fa48  EFLAGS: 00010292
>RAX: 0000000000000000 RBX: ffff81074fab7d98 RCX: ffffffff80318f28
>RDX: ffffffff80318f28 RSI: 0000000000000096 RDI: ffffffff80318f20
>RBP: 000000000007cd19 R08: ffffffff80318f28 R09: 0000000000000001
>R10: 0000000000000000 R11: 0000000000000052 R12: ffff810744dc38f0
>R13: ffff81081dc79000 R14: ffff81078b23c9a8 R15: 0000000000000000
>FS:  00002ac607435b00(0000) GS:ffff81011cb9ce40(0000) knlGS:0000000000000000
>CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>CR2: 000000001920e588 CR3: 0000000798951000 CR4: 00000000000026e0
>Process dcs_output_mgr (pid: 30177, threadinfo ffff810798f0e000, task ffff810798ef1820)
>Stack:  ffff810744dc3688 ffffffff8884837e ffff810798f0fa58 ffff810798f0fa58
> ffff81074fab7d98 00000001000075e1 0000000000000001 0000000000000000
> ffffffff8884837e ffffffff88843e8a 000000000007cd19 ffff810700000000
>Call Trace:
> [<ffffffff8884837e>] :gfs2:gfs2_process_unlinked_inode+0x140/0x1b4
> [<ffffffff8884837e>] :gfs2:gfs2_process_unlinked_inode+0x140/0x1b4
> [<ffffffff88843e8a>] :gfs2:gfs2_glock_schedule_for_reclaim+0x5d/0x9a
> [<ffffffff88858ffa>] :gfs2:gfs2_inplace_reserve_i+0x663/0x6f4
> [<ffffffff88843deb>] :gfs2:do_promote+0xf5/0x137
> [<ffffffff8884d5fc>] :gfs2:gfs2_write_begin+0x16c/0x33e
> [<ffffffff8884edc6>] :gfs2:gfs2_file_buffered_write+0xf3/0x26c
> [<ffffffff8884f1ec>] :gfs2:__gfs2_file_aio_write_nolock+0x2ad/0x2e4
> [<ffffffff8884f38e>] :gfs2:gfs2_file_write_nolock+0xaa/0x10f
> [<ffffffff800c8bc7>] generic_file_read+0xac/0xc5
> [<ffffffff800a2896>] autoremove_wake_function+0x0/0x2e
> [<ffffffff800a2896>] autoremove_wake_function+0x0/0x2e
> [<ffffffff8884f4de>] :gfs2:gfs2_file_write+0x49/0xa7
> [<ffffffff80016af0>] vfs_write+0xce/0x174
> [<ffffffff800173a8>] sys_write+0x45/0x6e
> [<ffffffff8005d28d>] tracesys+0xd5/0xe0
>
>
>Code: 0f 0b 68 78 df 85 88 c2 d1 00 f0 ff 43 28 5b c3 53 8b 47 78 
>RIP  [<ffffffff88844351>] :gfs2:gfs2_glock_hold+0x15/0x25
> RSP <ffff810798f0fa48>

Actions: View

Attachments on bug 643165: 455771 | 456362 | 456544 | 459442 | 915160