Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

For bugs related to Red Hat Enterprise Linux 5 product line. The current stable release is 5.10. For Red Hat Enterprise Linux 6 and above, please visit Red Hat JIRA https://issues.redhat.com/secure/CreateIssue!default.jspa?pid=12332745 to report new issues.

Bug 460789

Summary:

Filesystem EXT3 Error

Product:

Red Hat Enterprise Linux 5

Reporter:

Nick Cheng <nickchengster>

Component:

kernel

Assignee:

Tomas Henzl <thenzl>

Status:

CLOSED DUPLICATE

QA Contact:

Martin Jenner <mjenner>

Severity:

high

Docs Contact:

Priority:

medium

Version:

5.2

CC:

andriusb, coughlan, esandeen, ferry.tap, kevin34, thenzl

Target Milestone:

Keywords:

Regression

Target Release:

---

Hardware:

i386

OS:

Linux

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2008-09-24 09:31:14 UTC

Type:

---

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
removed patches	none

Description Nick Cheng 2008-09-01 04:19:20 UTC

Description of problem:
Dump error messages while the IO operation by Areca RAID HBA and 3W-ware RAID card.

Version-Release number of selected component (if applicable):
RHEL5.2

How reproducible:


Steps to Reproduce:
1.Create EXT3 filesystem on the Areca or 3W-ware RAID volumes 
2.Mount the specific volumes
3.Execute any IO operation
  
Actual results:
As the attachment

Expected results:


Additional info:
It happens also on the nirmal HDs not only on RAID volumes.
Does any bugs on RHEL5.2?

Comment 1 Tomas Henzl 2008-09-01 10:31:17 UTC

Nick,
please attach here the appropriate error messages.
Do you see the messages also when you are not using the raid controller ?

Comment 2 Nick Cheng 2008-09-01 11:42:45 UTC

Tomas,
Because the system got hang, therefore can not dump the message.
Are there any customers reporting this kind of questions?

Comment 3 Tomas Henzl 2008-09-01 13:03:54 UTC

Nick,
I've just created new partition using Areca controller and RHEL5.2, then created ext3 filesystem there and did some i/o without problems. Besides that it successfully booted into it (I'm using the default setup so ext3 is the default filesystem).

Is your problem reproducible ? - Does it happen also on another computer ?

Do you see the messages also when you are not using the raid controller ?

Comment 4 Nick Cheng 2008-09-02 01:48:14 UTC

Tomas,
I will introduce our FAE, Kevin, to answer your questions.
Kevin,
Would you please answer Tomas's questions?
Thank you,

Comment 5 Kevin 2008-09-02 04:27:26 UTC

Tomas, 

I had tested with our controller and 3Ware 9500S, same motherboard and same hard drive set. system installed in a separate SATA disk, and do multiple copy/compare tasks in array.

with Areca 1220, the kernel will dump errors as Nick attached file.
with 3Ware 9500S, the kernel will panic.


I didn't verify without raid card, because i create a raid5 array with 7 drives.

We found this problem because some customers have same behavior in their machine. and the problem disappeared if they use RHEL5.1

Comment 6 Tomas Henzl 2008-09-02 11:43:25 UTC

(In reply to comment #5)
Nick, Kevin,

please remove the latest areca patch which was used in RHEL5.2, rebuild the kernel and test it so we know if the problem is in the drivers code or elsewhere.

> with Areca 1220, the kernel will dump errors as Nick attached file.
I do not see an attachment here.

Comment 7 Nick Cheng 2008-09-02 12:16:10 UTC

Tomas,
I have attached the log two times.
I paste it on this page directly.
Please check it.
BTW, this situation happens on Areca and 3ware RAID HBA.
Have you ever tried 3ware's RAID card?
Thanks,



Linux version 2.6.18-92.el5 (mockbuild.org) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)) #1 SMP Tue Jun 10 18:51:06 EDT 2008
Command line: ro root=/dev/VolGroup00/LogVol00 rhgb quiet
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000002fbf4000 (usable)
 BIOS-e820: 000000002fbf4000 - 000000002fc5f000 (reserved)
 BIOS-e820: 000000002fc5f000 - 000000002fcfd000 (usable)
 BIOS-e820: 000000002fcfd000 - 000000002fedf000 (ACPI NVS)
 BIOS-e820: 000000002fedf000 - 000000002feed000 (usable)
 BIOS-e820: 000000002feed000 - 000000002feff000 (ACPI data)
 BIOS-e820: 000000002feff000 - 000000002ff00000 (usable)
 BIOS-e820: 00000000fffc0000 - 00000000fffd0000 (reserved)
DMI 2.4 present.
ACPI: RSDP (v002 INTEL                                 ) @ 0x00000000000f0350
ACPI: XSDT (v001 INTEL  S3000AHX 0x00000000 INTL 0x01000013) @ 0x000000002fefe120
ACPI: FADT (v003 INTEL  S3000AHX 0x00000000 MSFT 0x01000013) @ 0x000000002fefb000
ACPI: MADT (v001 INTEL  S3000AHX 0x00000000 MSFT 0x01000013) @ 0x000000002fef5000
ACPI: WDDT (v001 INTEL  S3000AHX 0x00000000 MSFT 0x01000013) @ 0x000000002fef4000
ACPI: MCFG (v001 INTEL  S3000AHX 0x00000000 MSFT 0x01000013) @ 0x000000002fef3000
ACPI: ASF! (v032 INTEL  S3000AHX 0x00000001 MSFT 0x01000013) @ 0x000000002fef2000
ACPI: SSDT (v001 INTEL   Cpu0Ist 0x00000010 MSFT 0x01000013) @ 0x000000002fef1000
ACPI: SSDT (v001 INTEL   Cpu1Ist 0x00000010 MSFT 0x01000013) @ 0x000000002fef0000
ACPI: SSDT (v001 INTEL   Cpu2Ist 0x00000010 MSFT 0x01000013) @ 0x000000002feef000
ACPI: SSDT (v001 INTEL   Cpu3Ist 0x00000010 MSFT 0x01000013) @ 0x000000002feee000
ACPI: SSDT (v001 INTEL     CpuPm 0x00000010 MSFT 0x01000013) @ 0x000000002feed000
ACPI: DSDT (v001 INTEL  S3000AHX 0x00000000 MSFT 0x01000013) @ 0x0000000000000000
No NUMA configuration found
Faking a node at 0000000000000000-000000002ff00000
Bootmem setup node 0 0000000000000000-000000002ff00000
Memory for crash kernel (0x0 to 0x0) notwithin permissible range
disabling kdump
On node 0 totalpages: 191717
  DMA zone: 2696 pages, LIFO batch:0
  DMA32 zone: 189021 pages, LIFO batch:31
ACPI: PM-Timer IO Port: 0x408
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Processor #0 6:15 APIC version 20
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
Processor #1 6:15 APIC version 20
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x82] disabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x83] disabled)
ACPI: LAPIC_NMI (acpi_id[0x01] dfl dfl lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x02] dfl dfl lint[0x1])
ACPI: IOAPIC (id[0x05] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 5, version 32, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x06] address[0xfec10000] gsi_base[30])
IOAPIC[1]: apic_id 6, version 32, address 0xfec10000, GSI 30-53
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Setting APIC routing to physical flat
Using ACPI (MADT) for SMP configuration information
Nosave address range: 000000000009f000 - 00000000000a0000
Nosave address range: 00000000000a0000 - 0000000000100000
Nosave address range: 000000002fbf4000 - 000000002fc5f000
Nosave address range: 000000002fcfd000 - 000000002fedf000
Nosave address range: 000000002feed000 - 000000002feff000
Allocating PCI resources starting at 30000000 (gap: 2ff00000:d00c0000)
SMP: Allowing 4 CPUs, 2 hotplug CPUs
Built 1 zonelists.  Total pages: 191717
Kernel command line: ro root=/dev/VolGroup00/LogVol00 rhgb quiet
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 32768 bytes)
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
Checking aperture...
Memory: 761708k/785408k available (2457k kernel code, 20884k reserved, 1246k data, 196k init)
Calibrating delay using timer specific routine.. 3735.16 BogoMIPS (lpj=1867581)
Security Framework v1.0.0 initialized
SELinux:  Initializing.
SELinux:  Starting in permissive mode
selinux_register_security:  Registering secondary module capability
Capability LSM initialized as secondary
Mount-cache hash table entries: 256
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 2048K
using mwait in idle threads.
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
CPU0: Thermal monitoring enabled (TM2)
SMP alternatives: switching to UP code
ACPI: Core revision 20060707
Using local APIC timer interrupts.
result 16667707
Detected 16.667 MHz APIC timer.
SMP alternatives: switching to SMP code
Booting processor 1/2 APIC 0x1
Initializing CPU#1
Calibrating delay using timer specific routine.. 3732.94 BogoMIPS (lpj=1866471)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 2048K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 1
CPU1: Thermal monitoring enabled (TM2)
Intel(R) Core(TM)2 CPU          6300  @ 1.86GHz stepping 06
Brought up 2 CPUs
testing NMI watchdog ... OK.
time.c: Using 3.579545 MHz WALL PM GTOD PIT/TSC timer.
time.c: Detected 1866.784 MHz processor.
sizeof(vma)=176 bytes
sizeof(page)=56 bytes
sizeof(inode)=560 bytes
sizeof(dentry)=216 bytes
sizeof(ext3inode)=760 bytes
sizeof(buffer_head)=96 bytes
sizeof(skbuff)=240 bytes
migration_cost=148
checking if image is initramfs... it is
Freeing initrd memory: 3136k freed
NET: Registered protocol family 16
No dock devices found.
ACPI: bus type pci registered
PCI: BIOS Bug: MCFG area at f0000000 is not E820-reserved
PCI: Not using MMCONFIG.
PCI: Using configuration type 1
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI: Probing PCI hardware (bus 00)
PCI quirk: region 0400-047f claimed by ICH6 ACPI/GPIO/TCO
PCI quirk: region 0500-053f claimed by ICH6 GPIO
PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.1
PCI: PXH quirk detected, disabling MSI for SHPC device
Boot video device is 0000:08:04.0
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P32_._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 7 9 10 *11 12)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 7 *9 10 11 12)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 7 9 10 *11 12)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 7 9 *10 11 12)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 7 9 10 11 12) *0, disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 7 9 10 11 12) *0, disabled.
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 7 9 10 11 12) *0, disabled.
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 7 9 10 *11 12)
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEX0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEX0.PXS1._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEX4._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEX5._PRT]
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
pnp: PnP ACPI: found 12 devices
usbcore: registered new driver usbfs
usbcore: registered new driver hub
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
PCI-GART: No AMD northbridge found.
pnp: 00:01: ioport range 0x779-0x780 has been reserved
pnp: 00:06: ioport range 0x500-0x53f has been reserved
pnp: 00:06: ioport range 0x400-0x47f could not be reserved
pnp: 00:06: ioport range 0x680-0x6ff has been reserved
PCI: Bridge: 0000:01:00.0
  IO window: disabled.
  MEM window: 38600000-386fffff
  PREFETCH window: 38000000-383fffff
PCI: Bridge: 0000:01:00.2
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:01.0
  IO window: disabled.
  MEM window: 38600000-386fffff
  PREFETCH window: 38000000-383fffff
PCI: Bridge: 0000:04:00.0
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:1c.0
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:1c.4
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:1c.5
  IO window: 2000-2fff
  MEM window: 38500000-385fffff
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:1e.0
  IO window: 1000-1fff
  MEM window: 38400000-384fffff
  PREFETCH window: 30000000-37ffffff
GSI 16 sharing vector 0xA9 and IRQ 16
ACPI: PCI Interrupt 0000:00:01.0[A] -> GSI 16 (level, low) -> IRQ 169
PCI: Setting latency timer of device 0000:00:01.0 to 64
PCI: Setting latency timer of device 0000:01:00.0 to 64
PCI: Setting latency timer of device 0000:01:00.2 to 64
GSI 17 sharing vector 0xB1 and IRQ 17
ACPI: PCI Interrupt 0000:00:1c.0[A] -> GSI 17 (level, low) -> IRQ 177
PCI: Setting latency timer of device 0000:00:1c.0 to 64
PCI: Setting latency timer of device 0000:04:00.0 to 64
ACPI: PCI Interrupt 0000:00:1c.4[A] -> GSI 17 (level, low) -> IRQ 177
PCI: Setting latency timer of device 0000:00:1c.4 to 64
ACPI: PCI Interrupt 0000:00:1c.5[B] -> GSI 16 (level, low) -> IRQ 169
PCI: Setting latency timer of device 0000:00:1c.5 to 64
PCI: Setting latency timer of device 0000:00:1e.0 to 64
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 6, 262144 bytes)
TCP established hash table entries: 131072 (order: 9, 2097152 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
audit: initializing netlink socket (disabled)
audit(1219200395.851:1): initialized
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
SELinux:  Registering netfilter hooks
Initializing Cryptographic API
ksign: Installing public key data
Loading keyring
- Added public key 52394A11B61A91
- User ID: CentOS (Kernel Module GPG key)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
0000:00:1d.7 EHCI: BIOS handoff failed (BIOS bug ?) 01010001
PCI: Setting latency timer of device 0000:00:01.0 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:01.0:pcie00]
PCI: Setting latency timer of device 0000:00:1c.0 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:1c.0:pcie00]
Allocate Port Service[0000:00:1c.0:pcie02]
PCI: Setting latency timer of device 0000:00:1c.4 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:1c.4:pcie00]
Allocate Port Service[0000:00:1c.4:pcie02]
PCI: Setting latency timer of device 0000:00:1c.5 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:1c.5:pcie00]
Allocate Port Service[0000:00:1c.5:pcie02]
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707]
ACPI: Getting cpuindex for acpiid 0x3
ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707]
ACPI: Getting cpuindex for acpiid 0x4
Real Time Clock Driver v1.12ac
Non-volatile memory driver v1.2
Linux agpgart interface v0.101 (c) Dave Jones
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:0a: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
GSI 18 sharing vector 0xE1 and IRQ 18
ACPI: PCI Interrupt 0000:07:00.3[C] -> GSI 19 (level, low) -> IRQ 225
0000:07:00.3: ttyS1 at I/O 0x2020 (irq = 225) is a 16550A
RAMDISK driver initialized: 16 RAM disks of 16384K size 4096 blocksize
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH7: IDE controller at PCI slot 0000:00:1f.1
GSI 19 sharing vector 0xE9 and IRQ 19
ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 233
ICH7: chipset revision 1
ICH7: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0x30b0-0x30b7, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0x30b8-0x30bf, BIOS settings: hdc:pio, hdd:pio
Probing IDE interface ide0...
hda: HL-DT-STDVD-ROM GDR8164B, ATAPI CD/DVD-ROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
Probing IDE interface ide1...
ide-floppy driver 0.99.newide
usbcore: registered new driver hiddev
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.6:USB HID core driver
PNP: PS/2 Controller [PNP0303:PS2K,PNP0f03:PS2M] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
TCP bic registered
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
ACPI: (supports S0 S1 S4 S5)
Freeing unused kernel memory: 196k freed
Write protecting the kernel read-only data: 475k
input: AT Translated Set 2 keyboard as /class/input/input0
GSI 20 sharing vector 0x32 and IRQ 20
ACPI: PCI Interrupt 0000:00:1d.7[A] -> GSI 23 (level, low) -> IRQ 50
PCI: Setting latency timer of device 0000:00:1d.7 to 64
ehci_hcd 0000:00:1d.7: EHCI Host Controller
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1
ehci_hcd 0000:00:1d.7: debug port 1
PCI: cache line size of 32 is not supported by device 0000:00:1d.7
ehci_hcd 0000:00:1d.7: irq 50, io mem 0x38700400
ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 8 ports detected
logips2pp: Detected unknown logitech mouse model 1
ohci_hcd: 2005 April 22 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI)
USB Universal Host Controller Interface driver v3.0
ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 23 (level, low) -> IRQ 50
PCI: Setting latency timer of device 0000:00:1d.0 to 64
uhci_hcd 0000:00:1d.0: UHCI Host Controller
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
uhci_hcd 0000:00:1d.0: irq 50, io base 0x00003080
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
input: ImExPS/2 Logitech Explorer Mouse as /class/input/input1
ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 225
PCI: Setting latency timer of device 0000:00:1d.1 to 64
uhci_hcd 0000:00:1d.1: UHCI Host Controller
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:1d.1: irq 225, io base 0x00003060
usb usb3: configuration #1 chosen from 1 choice
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 233
PCI: Setting latency timer of device 0000:00:1d.2 to 64
uhci_hcd 0000:00:1d.2: UHCI Host Controller
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4
uhci_hcd 0000:00:1d.2: irq 233, io base 0x00003040
usb usb4: configuration #1 chosen from 1 choice
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
ACPI: PCI Interrupt 0000:00:1d.3[D] -> GSI 16 (level, low) -> IRQ 169
PCI: Setting latency timer of device 0000:00:1d.3 to 64
uhci_hcd 0000:00:1d.3: UHCI Host Controller
uhci_hcd 0000:00:1d.3: new USB bus registered, assigned bus number 5
uhci_hcd 0000:00:1d.3: irq 169, io base 0x00003020
usb usb5: configuration #1 chosen from 1 choice
hub 5-0:1.0: USB hub found
hub 5-0:1.0: 2 ports detected
SCSI subsystem initialized
libata version 3.00 loaded.
ata_piix 0000:00:1f.2: version 2.12
ata_piix 0000:00:1f.2: MAP [ P0 P2 P1 P3 ]
ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 225
PCI: Setting latency timer of device 0000:00:1f.2 to 64
scsi0 : ata_piix
scsi1 : ata_piix
ata1: SATA max UDMA/133 cmd 0x30c8 ctl 0x30e4 bmdma 0x30a0 irq 225
ata2: SATA max UDMA/133 cmd 0x30c0 ctl 0x30e0 bmdma 0x30a8 irq 225
ata1.00: ATA-7: WDC WD4000KD-00NAB0, 01.06A01, max UDMA/133
ata1.00: 781422768 sectors, multi 16: LBA48 NCQ (depth 0/32)
ata1.00: configured for UDMA/133
  Vendor: ATA       Model: WDC WD4000KD-00N  Rev: 01.0
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sda: 781422768 512-byte hdwr sectors (400088 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: drive cache: write back
SCSI device sda: 781422768 512-byte hdwr sectors (400088 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: drive cache: write back
 sda: sda1 sda2
sd 0:0:0:0: Attached scsi disk sda
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
audit(1219200407.839:2): enforcing=1 old_enforcing=0 auid=4294967295 ses=4294967295
security:  3 users, 6 roles, 1676 types, 213 bools, 1 sens, 1024 cats
security:  61 classes, 60092 rules
SELinux:  Completing initialization.
SELinux:  Setting up existing superblocks.
SELinux: initialized (dev dm-0, type ext3), uses xattr
SELinux: initialized (dev usbfs, type usbfs), uses genfs_contexts
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev debugfs, type debugfs), uses genfs_contexts
SELinux: initialized (dev selinuxfs, type selinuxfs), uses genfs_contexts
SELinux: initialized (dev mqueue, type mqueue), uses transition SIDs
SELinux: initialized (dev hugetlbfs, type hugetlbfs), uses genfs_contexts
SELinux: initialized (dev devpts, type devpts), uses transition SIDs
SELinux: initialized (dev eventpollfs, type eventpollfs), uses task SIDs
SELinux: initialized (dev inotifyfs, type inotifyfs), uses genfs_contexts
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev futexfs, type futexfs), uses genfs_contexts
SELinux: initialized (dev pipefs, type pipefs), uses task SIDs
SELinux: initialized (dev sockfs, type sockfs), uses task SIDs
SELinux: initialized (dev cpuset, type cpuset), uses genfs_contexts
SELinux: initialized (dev proc, type proc), uses genfs_contexts
SELinux: initialized (dev bdev, type bdev), uses genfs_contexts
SELinux: initialized (dev rootfs, type rootfs), uses genfs_contexts
SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts
audit(1219200408.125:3): policy loaded auid=4294967295 ses=4294967295
input: PC Speaker as /class/input/input2
shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
e1000e: Intel(R) PRO/1000 Network Driver - 0.2.0
e1000e: Copyright (c) 1999-2007 Intel Corporation.
ACPI: PCI Interrupt 0000:07:00.0[A] -> GSI 17 (level, low) -> IRQ 177
PCI: Setting latency timer of device 0000:07:00.0 to 64
Intel(R) PRO/1000 Network Driver - version 7.3.20-k2-NAPI
Copyright (c) 1999-2006 Intel Corporation.
intel_rng: Firmware space is locked read-only. <4>intel_rng: If you can't or
 don't want to <4>intel_rng: disable this in firmware setup, and <4>intel_rng: if
 you are certain that your <4>intel_rng: system has a functional
 RNG, try<4>intel_rng: using the 'no_fwh_detect' option.
EDAC MC: Ver: 2.0.1 Jun 10 2008
0000:00:1c.5: eth0: (PCI Express:2.5GB/s:Width x1) 00:15:17:35:65:45
0000:00:1c.5: eth0: Intel(R) PRO/1000 Network Connection
0000:00:1c.5: eth0: MAC: 2, PHY: 2, PBA No: ffffff-0ff
ACPI: PCI Interrupt 0000:08:05.0[A] -> GSI 17 (level, low) -> IRQ 177
Floppy drive(s): fd0 is 1.44M
e1000: 0000:08:05.0: e1000_probe: (PCI:33MHz:32-bit) 00:15:17:35:65:46
FDC 0 is a post-1991 82077
hda: ATAPI 52X DVD-ROM drive, 256kB Cache, UDMA(33)
Uniform CD-ROM driver Revision: 3.20
e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
ACPI: PCI Interrupt 0000:02:0e.0[A] -> GSI 18 (level, low) -> IRQ 233
ACPI: PCI Interrupt 0000:00:1f.3[B] -> GSI 19 (level, low) -> IRQ 225
EDAC MC0: Giving out device to i3000_edac i3000: DEV 0000:00:00.0
ARECA RAID ADAPTER2: FIRMWARE VERSION V1.45 2008-3-14  
scsi2 : Areca SATA Host Adapter RAID Controller( RAID6 capable)
 Driver Version 1.20.00.15.RH 2007/12/24
  Vendor: Areca     Model: ARC-1220-VOL#00   Rev: R001
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sdb: 4294920192 512-byte hdwr sectors (2198999 MB)
sdb: Write Protect is off
sdb: Mode Sense: cb 00 00 08
SCSI device sdb: drive cache: write back
SCSI device sdb: 4294920192 512-byte hdwr sectors (2198999 MB)
sdb: Write Protect is off
sdb: Mode Sense: cb 00 00 08
SCSI device sdb: drive cache: write back
 sdb: unknown partition table
sd 2:0:0:0: Attached scsi disk sdb
  Vendor: Areca     Model: RAID controller   Rev: R001
  Type:   Processor                          ANSI SCSI revision: 00
sd 0:0:0:0: Attached scsi generic sg0 type 0
sd 2:0:0:0: Attached scsi generic sg1 type 0
scsi 2:0:16:0: Attached scsi generic sg2 type 3
lp: driver loaded but no devices found
SELinux: initialized (dev ramfs, type ramfs), uses genfs_contexts
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
IPv6 over IPv4 tunneling driver
ACPI: Power Button (FF) [PWRF]
ACPI: Sleep Button (CM) [SLPB]
ibm_acpi: ec object not found
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
device-mapper: multipath: version 1.0.5 loaded
EXT3 FS on dm-0, internal journal
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
SELinux: initialized (dev sda1, type ext3), uses xattr
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Adding 1540088k swap on /dev/VolGroup00/LogVol01.  Priority:-1 extents:1 across:1540088k
SELinux: initialized (dev binfmt_misc, type binfmt_misc), uses genfs_contexts
IA-32 Microcode Update Driver: v1.14a <tigran>
microcode: CPU1 updated from revision 0xc6 to 0xcb, date = 09162007 
microcode: CPU0 updated from revision 0xc6 to 0xcb, date = 09162007 
ip6_tables: (C) 2000-2006 Netfilter Core Team
ip_tables: (C) 2000-2006 Netfilter Core Team
Netfilter messages via NETLINK v0.30.
ip_conntrack version 2.4 (3068 buckets, 24544 max) - 304 bytes per conntrack
ADDRCONF(NETDEV_UP): eth0: link is not ready
e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Half Duplex, Flow Control: None
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
SELinux: initialized (dev rpc_pipefs, type rpc_pipefs), uses genfs_contexts
Bluetooth: Core ver 2.10
NET: Registered protocol family 31
Bluetooth: HCI device and connection manager initialized
Bluetooth: HCI socket layer initialized
Bluetooth: L2CAP ver 2.8
Bluetooth: L2CAP socket layer initialized
Bluetooth: RFCOMM socket layer initialized
Bluetooth: RFCOMM TTY layer initialized
Bluetooth: RFCOMM ver 1.8
Bluetooth: HIDP (Human Interface Emulation) ver 1.1
SELinux: initialized (dev autofs, type autofs), uses genfs_contexts
SELinux: initialized (dev autofs, type autofs), uses genfs_contexts
SELinux: initialized (dev autofs, type autofs), uses genfs_contexts
eth0: no IPv6 routers present
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sdb, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
SELinux: initialized (dev sdb, type ext3), uses xattr
attempt to access beyond end of device
sdb: rw=0, want=19473909208, limit=4294920192
attempt to access beyond end of device
sdb: rw=0, want=19473909208, limit=4294920192
EXT3-fs error (device sdb): ext3_free_blocks: Freeing blocks not in datazone - block = 2434238650, count = 1
Aborting journal on device sdb.
EXT3-fs error (device sdb) in ext3_free_blocks_sb: Journal has aborted
ext3_abort called.
EXT3-fs error (device sdb) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device sdb): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
EXT3-fs error (device sdb) in ext3_truncate: Journal has aborted
EXT3-fs error (device sdb) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device sdb) in ext3_orphan_del: Journal has aborted
EXT3-fs error (device sdb) in ext3_reserve_inode_write: Journal has aborted
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data

Comment 9 Tomas Henzl 2008-09-02 12:38:32 UTC

(In reply to comment #7)
> BTW, this situation happens on Areca and 3ware RAID HBA.
> Have you ever tried 3ware's RAID card?
No I haven't. The 3ware driver has also been changed and it could happen, even if unlikely that there are two different bugs. 
It would help a little if you could test the RHEL5.2 kernel without the areca patch. If you need some assistance with that I'll help you.
Thanks.

Comment 10 Tomas Henzl 2008-09-02 12:40:32 UTC

*** Bug 459247 has been marked as a duplicate of this bug. ***

Comment 11 Tomas Henzl 2008-09-02 12:42:14 UTC

Comment from closed bug 459247

Server with RHEL 5.2 (kernel-2.6.18-92.1.6.el5, x86_64) and an Areca 1120 RAID
controller hangs after heavy read/write to the RAID set. The problem persists
with kernel-2.6.18-92.1.10.el5. 

It works fine with the older kernel (kernel-2.6.18-53.1.21.el5). 

I am member of the CentOS Community. See the following post:
http://www.centos.org/modules/newbb/viewtopic.php?topic_id=15640&forum=39&post_id=54941#forumpost54941

Comment 15 Nick Cheng 2008-09-03 10:34:49 UTC

Hi Tomas,
Do I still need to try? Or you have figured it out?

Comment 16 Tomas Henzl 2008-09-03 12:31:52 UTC

(In reply to comment #15)
> Hi Tomas,
> Do I still need to try? Or you have figured it out?

Yes please, try it out, we have no other reported problems with other drivers/filesystem, only this one plus the bz459247 which is also areca driver.

Comment 17 Tom Coughlan 2008-09-03 22:25:03 UTC

(Eric, I am adding you as an FYI. It is too early to conclude this is related to ext3, but just in case.)

(In reply to comment #7)

> SCSI device sdb: 4294920192 512-byte hdwr sectors (2198999 MB)

That says it is a 2 TiB disk. This is possible with an underlying HW RAID device. Is this the correct size?

>  sdb: unknown partition table

Is this correct, you did not put a partition table on sdb? 

This may be fine. I am just checking to see if there is trouble reading the disk. 

> EXT3 FS on sdb, internal journal
> EXT3-fs: mounted filesystem with ordered data mode.
> SELinux: initialized (dev sdb, type ext3), uses xattr
> attempt to access beyond end of device
> sdb: rw=0, want=19473909208, limit=4294920192

The "limit" matches what we saw earlier, so that is good. The "want" sector looks ridiculous. I believe this means the filesystem metadata has become corrupt. 

Please describe the exact scenario that lead to this set of messages. For example, you booted the system on sda, the systme configured sdb okay, then you put a filesystem on sdb (did you create a partition, how?), then you mounted sdb and did I/O, the there was a failure (?), then you rebooted and saw the messages in comment 7? Or what exactly did happen? 

You said that with the Areca you get errors (followed by a hang?). With the 3ware you get a crash. Without any more information we should assume these are different problems. Please provide more information on the 3ware problem. Ideally, a crash dump, or at least a system boot log and the error messages leading up to the crash and the stack trace. 

Simmilarly, without more information, we should not assume that the problem reported in bug 459247 is the same as what is described in comment 7. In 459247 we have a system hang after heavy read/write to the RAID set. Comment 7 just shows failure to mount at boot time. Again, was comment 7 preceeded by a hang? while doing heavy I/O? Can you reproduce that (either comment 7 or BZ 459247) and capture a crash dump, or at least a system boot log and the error messages leading up to the crash and the stack trace?

Tom

Comment 18 Eric Sandeen 2008-09-03 22:56:16 UTC

> The "limit" matches what we saw earlier, so that is good. The "want" sector
> looks ridiculous. I believe this means the filesystem metadata has become
> corrupt. 

Not necessarily; that's about 9T right?  It's possible.  :)

however:

> EXT3-fs error (device sdb): ext3_free_blocks: Freeing blocks not in datazone -
block = 2434238650, count = 1

indicates that the block in question is beyond the superblock's s_blocks_count.  Wish we printed that along with the error message.

Nick, how big is your filesystem, and the block device, *supposed* to be?  dumpe2fs -h /dev/sdb would give us the fs geometry info.

e2fsck -fn /dev/sdb might be interesting too, although if there were IO problems it may have resulted in the corruption, rather than the other way around.

-Eric

Comment 19 Nick Cheng 2008-09-04 01:42:13 UTC

Kevin,
Would you please feedback these questions to Tomas and Eric?
Thank you,

Comment 20 Nick Cheng 2008-09-04 02:44:32 UTC

Hi Tomas and Eric,
Because the bug was verified and found by FAE,Kevin, originally, the detailed info needs to be provided by him.
I have setup the system and try by myself.
First, I remove Areca driver and do I/O on the local SATA HDs.
It looks fine but dumped some messages.
I collect them on my site for your information.
Besides, the outcome for "e2fsck -fn /dev/sda1" is nothing wrong.
I also paste the output "dumpe2fs -h /dev/sda1". Please check it out.
I will try to duplicate the issue on my site.


THE OUTPUT OF /var/log/messages:
Sep  3 18:59:37 localhost syslogd 1.4.1: restart.
Sep  3 18:59:37 localhost kernel: klogd 1.4.1, log source = /proc/kmsg started.
Sep  3 18:59:37 localhost kernel: Linux version 2.6.18-92.el5 (mockbuild.org) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)) #1 SMP Tue Jun 10 18:51:06 EDT 2008
Sep  3 18:59:37 localhost kernel: Command line: ro root=/dev/VolGroup00/LogVol00 rhgb quiet
Sep  3 18:59:37 localhost kernel: BIOS-provided physical RAM map:
Sep  3 18:59:37 localhost kernel:  BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
Sep  3 18:59:37 localhost kernel:  BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
Sep  3 18:59:37 localhost kernel:  BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
Sep  3 18:59:37 localhost kernel:  BIOS-e820: 0000000000100000 - 000000001fef0000 (usable)
Sep  3 18:59:37 localhost kernel:  BIOS-e820: 000000001fef0000 - 000000001fef3000 (ACPI NVS)
Sep  3 18:59:37 localhost kernel:  BIOS-e820: 000000001fef3000 - 000000001ff00000 (ACPI data)
Sep  3 18:59:37 localhost kernel:  BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
Sep  3 18:59:37 localhost kernel:  BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved)
Sep  3 18:59:37 localhost kernel: DMI 2.3 present.
Sep  3 18:59:37 localhost kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Sep  3 18:59:37 localhost kernel: SRAT: Node 0 PXM 0 0-a0000
Sep  3 18:59:37 localhost kernel: SRAT: Node 0 PXM 0 0-20000000
Sep  3 18:59:37 localhost kernel: Bootmem setup node 0 0000000000000000-000000001fef0000
Sep  3 18:59:37 localhost kernel: Memory for crash kernel (0x0 to 0x0) notwithin permissible range
Sep  3 18:59:37 localhost kernel: disabling kdump
Sep  3 18:59:37 localhost kernel: ACPI: PM-Timer IO Port: 0x4008
Sep  3 18:59:37 localhost kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Sep  3 18:59:37 localhost kernel: Processor #0 15:15 APIC version 16
Sep  3 18:59:37 localhost kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] disabled)
Sep  3 18:59:37 localhost kernel: ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
Sep  3 18:59:37 localhost kernel: ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
Sep  3 18:59:37 localhost kernel: ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
Sep  3 18:59:37 localhost kernel: IOAPIC[0]: apic_id 2, version 33, address 0xfec00000, GSI 0-23
Sep  3 18:59:37 localhost kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Sep  3 18:59:37 localhost kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 21 low level)
Sep  3 18:59:37 localhost kernel: Setting APIC routing to physical flat
Sep  3 18:59:37 localhost kernel: Using ACPI (MADT) for SMP configuration information
Sep  3 18:59:37 localhost kernel: Nosave address range: 000000000009f000 - 00000000000a0000
Sep  3 18:59:37 localhost kernel: Nosave address range: 00000000000a0000 - 00000000000f0000
Sep  3 18:59:37 localhost kernel: Nosave address range: 00000000000f0000 - 0000000000100000
Sep  3 18:59:37 localhost kernel: Allocating PCI resources starting at 20000000 (gap: 1ff00000:c0100000)
Sep  3 18:59:37 localhost kernel: SMP: Allowing 2 CPUs, 1 hotplug CPUs
Sep  3 18:59:37 localhost kernel: Built 1 zonelists.  Total pages: 127653
Sep  3 18:59:37 localhost kernel: Kernel command line: ro root=/dev/VolGroup00/LogVol00 rhgb quiet
Sep  3 18:59:37 localhost kernel: Initializing CPU#0
Sep  3 18:59:37 localhost kernel: PID hash table entries: 2048 (order: 11, 16384 bytes)
Sep  3 18:59:37 localhost kernel: Console: colour VGA+ 80x25
Sep  3 18:59:37 localhost kernel: Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
Sep  3 18:59:37 localhost kernel: Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
Sep  3 18:59:37 localhost kernel: Checking aperture...
Sep  3 18:59:37 localhost kernel: CPU 0: aperture @ 4000000 size 32 MB
Sep  3 18:59:37 localhost kernel: Aperture too small (32 MB)
Sep  3 18:59:37 localhost kernel: AGP bridge at 00:00:00
Sep  3 18:59:37 localhost kernel: Aperture from AGP @ fe02f00000000000 size 256 MB (APSIZE f00)
Sep  3 18:59:37 localhost kernel: Aperture beyond 4GB. Ignoring.
Sep  3 18:59:37 localhost kernel: Your BIOS doesn't leave a aperture memory hole
Sep  3 18:59:37 localhost kernel: Please enable the IOMMU option in the BIOS setup
Sep  3 18:59:37 localhost kernel: This costs you 64 MB of RAM
Sep  3 18:59:37 localhost kernel: Mapping aperture over 65536 KB of RAM @ 4000000
Sep  3 18:59:37 localhost kernel: Memory: 440832k/523200k available (2457k kernel code, 81980k reserved, 1246k data, 196k init)
Sep  3 18:59:37 localhost kernel: Calibrating delay using timer specific routine.. 4002.03 BogoMIPS (lpj=2001019)
Sep  3 18:59:37 localhost kernel: Security Framework v1.0.0 initialized
Sep  3 18:59:37 localhost kernel: SELinux:  Initializing.
Sep  3 18:59:37 localhost kernel: selinux_register_security:  Registering secondary module capability
Sep  3 18:59:37 localhost kernel: Capability LSM initialized as secondary
Sep  3 18:59:37 localhost kernel: Mount-cache hash table entries: 256
Sep  3 18:59:37 localhost kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Sep  3 18:59:37 localhost kernel: CPU: L2 Cache: 512K (64 bytes/line)
Sep  3 18:59:37 localhost kernel: CPU 0/0 -> Node 0
Sep  3 18:59:37 localhost kernel: SMP alternatives: switching to UP code
Sep  3 18:59:37 localhost kernel: ACPI: Core revision 20060707
Sep  3 18:59:37 localhost kernel: ..MP-BIOS bug: 8254 timer not connected to IO-APIC
Sep  3 18:59:37 localhost rpc.statd[7899]: Version 1.0.9 Starting
Sep  3 18:59:37 localhost kernel: Using local APIC timer interrupts.
Sep  3 18:59:37 localhost kernel: result 12499088
Sep  3 18:59:37 localhost kernel: Detected 12.499 MHz APIC timer.
Sep  3 18:59:37 localhost kernel: Brought up 1 CPUs
Sep  3 18:59:37 localhost kernel: testing NMI watchdog ... OK.
Sep  3 18:59:37 localhost kernel: time.c: Using 3.579545 MHz WALL PM GTOD PIT/TSC timer.
Sep  3 18:59:37 localhost kernel: time.c: Detected 1999.851 MHz processor.
Sep  3 18:59:37 localhost kernel: checking if image is initramfs... it is
Sep  3 18:59:37 localhost kernel: Freeing initrd memory: 3132k freed
Sep  3 18:59:37 localhost kernel: NET: Registered protocol family 16
Sep  3 18:59:37 localhost rpc.statd[7899]: statd running as root. chown /var/lib/nfs/statd/sm to choose different user 
Sep  3 18:59:37 localhost kernel: No dock devices found.
Sep  3 18:59:37 localhost kernel: ACPI: bus type pci registered
Sep  3 18:59:37 localhost kernel: PCI: Using MMCONFIG at e0000000
Sep  3 18:59:37 localhost kernel: PCI: No mmconfig possible on device 0:18
Sep  3 18:59:37 localhost kernel: PCI: Buses that can't use MMCONFIG will use type 1 PCI conf access.
Sep  3 18:59:37 localhost kernel: ACPI: Interpreter enabled
Sep  3 18:59:37 localhost kernel: ACPI: Using IOAPIC for interrupt routing
Sep  3 18:59:37 localhost kernel: ACPI: PCI Root Bridge [PCI0] (0000:00)
Sep  3 18:59:37 localhost kernel: PCI: Ignoring BAR0-3 of IDE controller 0000:00:14.1
Sep  3 18:59:37 localhost kernel: PCI: Transparent bridge - 0000:00:14.4
Sep  3 18:59:37 localhost kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 10 11) *0, disabled.
Sep  3 18:59:37 localhost kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 10 11) *0, disabled.
Sep  3 18:59:37 localhost kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 10 11) *0, disabled.
Sep  3 18:59:37 localhost kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 10 11) *0, disabled.
Sep  3 18:59:37 localhost kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 *10 11)
Sep  3 18:59:37 localhost kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 *5 6 7 10 11)
Sep  3 18:59:37 localhost kernel: ACPI: PCI Interrupt Link [LNK0] (IRQs 3 4 5 6 7 *10 11)
Sep  3 18:59:37 localhost kernel: ACPI: PCI Interrupt Link [LNK1] (IRQs 3 4 5 6 7 10 *11)
Sep  3 18:59:37 localhost kernel: Linux Plug and Play Support v0.97 (c) Adam Belay
Sep  3 18:59:37 localhost kernel: pnp: PnP ACPI init
Sep  3 18:59:37 localhost kernel: pnp: PnP ACPI: found 14 devices
Sep  3 18:59:37 localhost kernel: usbcore: registered new driver usbfs
Sep  3 18:59:37 localhost kernel: usbcore: registered new driver hub
Sep  3 18:59:37 localhost kernel: PCI: Using ACPI for IRQ routing
Sep  3 18:59:37 localhost kernel: PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
Sep  3 18:59:37 localhost kernel: PCI: Cannot allocate resource region 3 of device 0000:00:00.0
Sep  3 18:59:37 localhost kernel: NetLabel: Initializing
Sep  3 18:59:37 localhost kernel: NetLabel:  domain hash size = 128
Sep  3 18:59:37 localhost kernel: NetLabel:  protocols = UNLABELED CIPSOv4
Sep  3 18:59:37 localhost kernel: NetLabel:  unlabeled traffic allowed by default
Sep  3 18:59:37 localhost kernel: PCI-DMA: Disabling IOMMU.
Sep  3 18:59:37 localhost kernel: pnp: 00:01: ioport range 0x140-0x15f has been reserved
Sep  3 18:59:37 localhost kernel: pnp: 00:01: ioport range 0x228-0x22f has been reserved
Sep  3 18:59:37 localhost kernel: pnp: 00:01: ioport range 0x4d6-0x4d6 has been reserved
Sep  3 18:59:37 localhost kernel: pnp: 00:01: ioport range 0xc00-0xc01 has been reserved
Sep  3 18:59:37 localhost kernel: pnp: 00:01: ioport range 0xc14-0xc14 has been reserved
Sep  3 18:59:37 localhost kernel: pnp: 00:01: ioport range 0xc50-0xc52 has been reserved
Sep  3 18:59:37 localhost kernel: pnp: 00:01: ioport range 0xc6c-0xc6d has been reserved
Sep  3 18:59:37 localhost kernel: pnp: 00:01: ioport range 0xc6f-0xc6f has been reserved
Sep  3 18:59:37 localhost kernel: PCI: Bridge: 0000:00:06.0
Sep  3 18:59:37 localhost kernel:   IO window: e000-efff
Sep  3 18:59:37 localhost kernel:   MEM window: fde00000-fdefffff
Sep  3 18:59:37 localhost kernel:   PREFETCH window: fdd00000-fddfffff
Sep  3 18:59:37 localhost kernel: PCI: Bridge: 0000:00:14.4
Sep  3 18:59:37 localhost kernel:   IO window: d000-dfff
Sep  3 18:59:37 localhost kernel:   MEM window: fb000000-fcffffff
Sep  3 18:59:37 localhost kernel:   PREFETCH window: d8000000-dfffffff
Sep  3 18:59:37 localhost kernel: NET: Registered protocol family 2
Sep  3 18:59:37 localhost kernel: IP route cache hash table entries: 4096 (order: 3, 32768 bytes)
Sep  3 18:59:37 localhost kernel: TCP established hash table entries: 16384 (order: 6, 262144 bytes)
Sep  3 18:59:37 localhost kernel: TCP bind hash table entries: 8192 (order: 5, 131072 bytes)
Sep  3 18:59:37 localhost kernel: TCP: Hash tables configured (established 16384 bind 8192)
Sep  3 18:59:37 localhost kernel: TCP reno registered
Sep  3 18:59:37 localhost kernel: audit: initializing netlink socket (disabled)
Sep  3 18:59:37 localhost kernel: audit(1220439528.039:1): initialized
Sep  3 18:59:37 localhost kernel: Total HugeTLB memory allocated, 0
Sep  3 18:59:37 localhost kernel: VFS: Disk quotas dquot_6.5.1
Sep  3 18:59:37 localhost kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Sep  3 18:59:37 localhost kernel: Initializing Cryptographic API
Sep  3 18:59:37 localhost kernel: ksign: Installing public key data
Sep  3 18:59:37 localhost kernel: Loading keyring
Sep  3 18:59:37 localhost kernel: - Added public key 52394A11B61A91
Sep  3 18:59:37 localhost kernel: - User ID: CentOS (Kernel Module GPG key)
Sep  3 18:59:37 localhost kernel: io scheduler noop registered
Sep  3 18:59:37 localhost kernel: io scheduler anticipatory registered
Sep  3 18:59:37 localhost kernel: io scheduler deadline registered
Sep  3 18:59:37 localhost kernel: io scheduler cfq registered (default)
Sep  3 18:59:37 localhost kernel: PCI: MSI quirk detected. MSI deactivated.
Sep  3 18:59:37 localhost kernel: 0000:00:13.2 EHCI: BIOS handoff failed (BIOS bug ?) 01010001
Sep  3 18:59:37 localhost kernel: assign_interrupt_mode Found MSI capability
Sep  3 18:59:37 localhost kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Sep  3 18:59:37 localhost kernel: ACPI: Fan [FAN] (on)
Sep  3 18:59:37 localhost kernel: ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707]
Sep  3 18:59:37 localhost kernel: ACPI: Getting cpuindex for acpiid 0x1
Sep  3 18:59:37 localhost kernel: ACPI: Thermal Zone [THRM] (-127 C)
Sep  3 18:59:37 localhost kernel: Real Time Clock Driver v1.12ac
Sep  3 18:59:37 localhost kernel: Non-volatile memory driver v1.2
Sep  3 18:59:37 localhost kernel: Linux agpgart interface v0.101 (c) Dave Jones
Sep  3 18:59:37 localhost kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
Sep  3 18:59:37 localhost kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Sep  3 18:59:37 localhost kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Sep  3 18:59:37 localhost kernel: 00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Sep  3 18:59:37 localhost kernel: RAMDISK driver initialized: 16 RAM disks of 16384K size 4096 blocksize
Sep  3 18:59:37 localhost kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
Sep  3 18:59:37 localhost kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
Sep  3 18:59:37 localhost kernel: ATIIXP: IDE controller at PCI slot 0000:00:14.1
Sep  3 18:59:37 localhost kernel: GSI 16 sharing vector 0xB1 and IRQ 16
Sep  3 18:59:37 localhost kernel: ACPI: PCI Interrupt 0000:00:14.1[A] -> GSI 16 (level, low) -> IRQ 177
Sep  3 18:59:37 localhost kernel: ATIIXP: chipset revision 128
Sep  3 18:59:37 localhost kernel: ATIIXP: not 100% native mode: will probe irqs later
Sep  3 18:59:37 localhost kernel:     ide0: BM-DMA at 0xf400-0xf407, BIOS settings: hda:DMA, hdb:pio
Sep  3 18:59:37 localhost kernel:     ide1: BM-DMA at 0xf408-0xf40f, BIOS settings: hdc:pio, hdd:pio
Sep  3 18:59:37 localhost kernel: hda: PIONEER DVD-ROM DVD-123P, ATAPI CD/DVD-ROM drive
Sep  3 18:59:37 localhost kernel: ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Sep  3 18:59:37 localhost kernel: ide-floppy driver 0.99.newide
Sep  3 18:59:37 localhost kernel: usbcore: registered new driver hiddev
Sep  3 18:59:37 localhost kernel: usbcore: registered new driver usbhid
Sep  3 18:59:37 localhost kernel: drivers/usb/input/hid-core.c: v2.6:USB HID core driver
Sep  3 18:59:37 localhost kernel: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
Sep  3 18:59:37 localhost kernel: serio: i8042 KBD port at 0x60,0x64 irq 1
Sep  3 18:59:37 localhost kernel: serio: i8042 AUX port at 0x60,0x64 irq 12
Sep  3 18:59:37 localhost kernel: mice: PS/2 mouse device common for all mice
Sep  3 18:59:37 localhost kernel: md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
Sep  3 18:59:37 localhost kernel: md: bitmap version 4.39
Sep  3 18:59:37 localhost kernel: TCP bic registered
Sep  3 18:59:37 localhost kernel: Initializing IPsec netlink socket
Sep  3 18:59:37 localhost kernel: NET: Registered protocol family 1
Sep  3 18:59:37 localhost kernel: NET: Registered protocol family 17
Sep  3 18:59:37 localhost kernel: powernow-k8: Pre-initialization of ACPI failed
Sep  3 18:59:37 localhost kernel: powernow-k8: Found 1 AMD Athlon(tm) 64 Processor 3200+ processors (1 cpu cores) (version 2.20.00)
Sep  3 18:59:37 localhost kernel: powernow-k8: BIOS error - no PSB or ACPI _PSS objects
Sep  3 18:59:37 localhost kernel: ACPI: (supports S0 S3 S4 S5)
Sep  3 18:59:37 localhost kernel: Freeing unused kernel memory: 196k freed
Sep  3 18:59:37 localhost kernel: Write protecting the kernel read-only data: 475k
Sep  3 18:59:37 localhost kernel: input: AT Translated Set 2 keyboard as /class/input/input0
Sep  3 18:59:37 localhost kernel: GSI 17 sharing vector 0xB9 and IRQ 17
Sep  3 18:59:37 localhost kernel: ACPI: PCI Interrupt 0000:00:13.2[A] -> GSI 19 (level, low) -> IRQ 185
Sep  3 18:59:37 localhost kernel: ehci_hcd 0000:00:13.2: EHCI Host Controller
Sep  3 18:59:37 localhost kernel: ehci_hcd 0000:00:13.2: new USB bus registered, assigned bus number 1
Sep  3 18:59:37 localhost kernel: ehci_hcd 0000:00:13.2: irq 185, io mem 0xfe02a000
Sep  3 18:59:37 localhost kernel: ehci_hcd 0000:00:13.2: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
Sep  3 18:59:37 localhost kernel: usb usb1: configuration #1 chosen from 1 choice
Sep  3 18:59:37 localhost kernel: hub 1-0:1.0: USB hub found
Sep  3 18:59:37 localhost kernel: hub 1-0:1.0: 8 ports detected
Sep  3 18:59:37 localhost hcid[7965]: Bluetooth HCI daemon
Sep  3 18:59:37 localhost sdpd[7969]: Bluetooth SDP daemon 
Sep  3 18:59:37 localhost hcid[7965]: Register path:/org/bluez fallback:1
Sep  3 18:59:37 localhost kernel: ACPI: PCI Interrupt 0000:00:13.0[A] -> GSI 19 (level, low) -> IRQ 185
Sep  3 18:59:37 localhost kernel: ohci_hcd 0000:00:13.0: OHCI Host Controller
Sep  3 18:59:37 localhost kernel: ohci_hcd 0000:00:13.0: new USB bus registered, assigned bus number 2
Sep  3 18:59:37 localhost kernel: ohci_hcd 0000:00:13.0: irq 185, io mem 0xfe02c000
Sep  3 18:59:37 localhost kernel: usb usb2: configuration #1 chosen from 1 choice
Sep  3 18:59:37 localhost kernel: hub 2-0:1.0: USB hub found
Sep  3 18:59:37 localhost kernel: hub 2-0:1.0: 4 ports detected
Sep  3 18:59:37 localhost kernel: ACPI: PCI Interrupt 0000:00:13.1[A] -> GSI 19 (level, low) -> IRQ 185
Sep  3 18:59:37 localhost kernel: ohci_hcd 0000:00:13.1: OHCI Host Controller
Sep  3 18:59:37 localhost kernel: ohci_hcd 0000:00:13.1: new USB bus registered, assigned bus number 3
Sep  3 18:59:37 localhost kernel: ohci_hcd 0000:00:13.1: irq 185, io mem 0xfe02b000
Sep  3 18:59:37 localhost kernel: usb usb3: configuration #1 chosen from 1 choice
Sep  3 18:59:37 localhost kernel: hub 3-0:1.0: USB hub found
Sep  3 18:59:37 localhost kernel: hub 3-0:1.0: 4 ports detected
Sep  3 18:59:37 localhost kernel: USB Universal Host Controller Interface driver v3.0
Sep  3 18:59:37 localhost kernel: SCSI subsystem initialized
Sep  3 18:59:37 localhost kernel: GSI 18 sharing vector 0xC1 and IRQ 18
Sep  3 18:59:38 localhost kernel: ACPI: PCI Interrupt 0000:00:11.0[A] -> GSI 23 (level, low) -> IRQ 193
Sep  3 18:59:38 localhost kernel: scsi0 : sata_sil
Sep  3 18:59:38 localhost kernel: scsi1 : sata_sil
Sep  3 18:59:38 localhost kernel: ata1: SATA max UDMA/100 mmio m512@0xfe02e000 tf 0xfe02e080 irq 193
Sep  3 18:59:38 localhost kernel: ata2: SATA max UDMA/100 mmio m512@0xfe02e000 tf 0xfe02e0c0 irq 193
Sep  3 18:59:38 localhost kernel: input: ImExPS/2 Generic Explorer Mouse as /class/input/input1
Sep  3 18:59:38 localhost kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
Sep  3 18:59:38 localhost kernel: ata1.00: ATA-6: ST380013AS, 3.18, max UDMA/133
Sep  3 18:59:38 localhost kernel: ata1.00: 156301488 sectors, multi 16: LBA48 
Sep  3 18:59:38 localhost kernel: ata1.00: configured for UDMA/100
Sep  3 18:59:38 localhost kernel: ata2: SATA link down (SStatus 0 SControl 310)
Sep  3 18:59:38 localhost kernel:   Vendor: ATA       Model: ST380013AS        Rev: 3.18
Sep  3 18:59:38 localhost kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Sep  3 18:59:38 localhost kernel: SCSI device sda: 156301488 512-byte hdwr sectors (80026 MB)
Sep  3 18:59:38 localhost kernel: sda: Write Protect is off
Sep  3 18:59:38 localhost kernel: SCSI device sda: drive cache: write back
Sep  3 18:59:38 localhost kernel: SCSI device sda: 156301488 512-byte hdwr sectors (80026 MB)
Sep  3 18:59:38 localhost kernel: sda: Write Protect is off
Sep  3 18:59:38 localhost kernel: SCSI device sda: drive cache: write back
Sep  3 18:59:38 localhost kernel:  sda: sda1
Sep  3 18:59:38 localhost kernel: sd 0:0:0:0: Attached scsi disk sda
Sep  3 18:59:38 localhost kernel: GSI 19 sharing vector 0xC9 and IRQ 19
Sep  3 18:59:38 localhost kernel: ACPI: PCI Interrupt 0000:00:12.0[A] -> GSI 22 (level, low) -> IRQ 201
Sep  3 18:59:38 localhost kernel: scsi2 : sata_sil
Sep  3 18:59:38 localhost kernel: scsi3 : sata_sil
Sep  3 18:59:38 localhost kernel: ata3: SATA max UDMA/100 mmio m512@0xfe02d000 tf 0xfe02d080 irq 201
Sep  3 18:59:38 localhost kernel: ata4: SATA max UDMA/100 mmio m512@0xfe02d000 tf 0xfe02d0c0 irq 201
Sep  3 18:59:38 localhost kernel: ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
Sep  3 18:59:38 localhost kernel: ata3.00: ATA-7: WDC WD5000YS-01MPB1, 09.02E09, max UDMA/133
Sep  3 18:59:38 localhost kernel: ata3.00: 976773168 sectors, multi 16: LBA48 NCQ (depth 0/32)
Sep  3 18:59:38 localhost kernel: ata3.00: configured for UDMA/100
Sep  3 18:59:38 localhost kernel: ata4: SATA link down (SStatus 0 SControl 310)
Sep  3 18:59:38 localhost kernel:   Vendor: ATA       Model: WDC WD5000YS-01M  Rev: 09.0
Sep  3 18:59:38 localhost kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Sep  3 18:59:38 localhost kernel: SCSI device sdb: 976773168 512-byte hdwr sectors (500108 MB)
Sep  3 18:59:38 localhost kernel: sdb: Write Protect is off
Sep  3 18:59:38 localhost kernel: SCSI device sdb: drive cache: write back
Sep  3 18:59:38 localhost kernel: SCSI device sdb: 976773168 512-byte hdwr sectors (500108 MB)
Sep  3 18:59:38 localhost kernel: sdb: Write Protect is off
Sep  3 18:59:38 localhost kernel: SCSI device sdb: drive cache: write back
Sep  3 18:59:38 localhost kernel:  sdb: sdb1 sdb2
Sep  3 18:59:38 localhost kernel: sd 2:0:0:0: Attached scsi disk sdb
Sep  3 18:59:38 localhost kernel: device-mapper: uevent: version 1.0.3
Sep  3 18:59:38 localhost kernel: device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel
Sep  3 18:59:38 localhost kernel: EXT3-fs: INFO: recovery required on readonly filesystem.
Sep  3 18:59:38 localhost kernel: EXT3-fs: write access will be enabled during recovery.
Sep  3 18:59:38 localhost kernel: kjournald starting.  Commit interval 5 seconds
Sep  3 18:59:38 localhost kernel: EXT3-fs: recovery complete.
Sep  3 18:59:38 localhost kernel: EXT3-fs: mounted filesystem with ordered data mode.
Sep  3 18:59:38 localhost kernel: SELinux:  Disabled at runtime.
Sep  3 18:59:38 localhost kernel: audit(1220439543.032:2): selinux=0 auid=4294967295 ses=4294967295
Sep  3 18:59:38 localhost kernel: input: PC Speaker as /class/input/input2
Sep  3 18:59:38 localhost kernel: hda: ATAPI 40X DVD-ROM drive, 2048kB Cache, UDMA(33)
Sep  3 18:59:38 localhost kernel: Uniform CD-ROM driver Revision: 3.20
Sep  3 18:59:38 localhost kernel: EDAC MC: Ver: 2.0.1 Jun 10 2008
Sep  3 18:59:38 localhost pcscd: pcscdaemon.c:507:main() pcsc-lite 1.4.4 daemon ready.
Sep  3 18:59:38 localhost kernel: EDAC MC0: Giving out device to k8_edac Athlon64/Opteron: DEV 0000:00:18.2
Sep  3 18:59:38 localhost kernel: ACPI: PCI Interrupt 0000:02:0a.0[A] -> GSI 21 (level, low) -> IRQ 169
Sep  3 18:59:38 localhost kernel: skge 1.6 addr 0xfcff8000 irq 169 chip Yukon-Lite rev 9
Sep  3 18:59:38 localhost kernel: skge eth0: addr 00:01:29:15:eb:c3
Sep  3 18:59:38 localhost kernel: Floppy drive(s): fd0 is 1.44M
Sep  3 18:59:38 localhost kernel: FDC 0 is a post-1991 82077
Sep  3 18:59:38 localhost kernel: piix4_smbus 0000:00:14.0: Found 0000:00:14.0 device
Sep  3 18:59:38 localhost kernel: GSI 20 sharing vector 0xD1 and IRQ 20
Sep  3 18:59:38 localhost kernel: ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 18 (level, low) -> IRQ 209
Sep  3 18:59:38 localhost kernel: sky2 0000:01:00.0: v1.14 addr 0xfdefc000 irq 209 Yukon-EC (0xb6) rev 1
Sep  3 18:59:38 localhost kernel: sky2 eth1: addr 00:01:29:15:8a:1b
Sep  3 18:59:38 localhost kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0
Sep  3 18:59:38 localhost kernel: sd 2:0:0:0: Attached scsi generic sg1 type 0
Sep  3 18:59:38 localhost kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
Sep  3 18:59:38 localhost kernel: lp: driver loaded but no devices found
Sep  3 18:59:38 localhost kernel: NET: Registered protocol family 10
Sep  3 18:59:38 localhost kernel: lo: Disabled Privacy Extensions
Sep  3 18:59:38 localhost pcscd: hotplug_libusb.c:402:HPEstablishUSBNotifications() Driver ifd-egate.bundle does not support IFD_GENERATE_HOTPLUG. Using active polling instead.
Sep  3 18:59:38 localhost kernel: IPv6 over IPv4 tunneling driver
Sep  3 18:59:38 localhost kernel: ACPI: Power Button (FF) [PWRF]
Sep  3 18:59:38 localhost kernel: ACPI: Power Button (CM) [PWRB]
Sep  3 18:59:38 localhost kernel: ibm_acpi: ec object not found
Sep  3 18:59:38 localhost kernel: md: Autodetecting RAID arrays.
Sep  3 18:59:38 localhost kernel: md: autorun ...
Sep  3 18:59:38 localhost kernel: md: ... autorun DONE.
Sep  3 18:59:38 localhost kernel: device-mapper: multipath: version 1.0.5 loaded
Sep  3 18:59:38 localhost kernel: EXT3 FS on dm-0, internal journal
Sep  3 18:59:38 localhost kernel: kjournald starting.  Commit interval 5 seconds
Sep  3 18:59:38 localhost kernel: EXT3 FS on sdb1, internal journal
Sep  3 18:59:38 localhost kernel: EXT3-fs: mounted filesystem with ordered data mode.
Sep  3 18:59:38 localhost kernel: Adding 917496k swap on /dev/VolGroup00/LogVol01.  Priority:-1 extents:1 across:917496k
Sep  3 18:59:38 localhost kernel: NET: Registered protocol family 27
Sep  3 18:59:38 localhost kernel: Loading iSCSI transport class v2.0-724.
Sep  3 18:59:38 localhost kernel: iscsi: registered transport (iser)
Sep  3 18:59:38 localhost kernel: skge eth0: enabling interface
Sep  3 18:59:38 localhost kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Sep  3 18:59:38 localhost kernel: skge eth0: disabling interface
Sep  3 18:59:38 localhost kernel: Bluetooth: Core ver 2.10
Sep  3 18:59:38 localhost kernel: NET: Registered protocol family 31
Sep  3 18:59:38 localhost kernel: Bluetooth: HCI device and connection manager initialized
Sep  3 18:59:38 localhost kernel: Bluetooth: HCI socket layer initialized
Sep  3 18:59:38 localhost kernel: Bluetooth: L2CAP ver 2.8
Sep  3 18:59:38 localhost kernel: Bluetooth: L2CAP socket layer initialized
Sep  3 18:59:38 localhost kernel: Bluetooth: RFCOMM socket layer initialized
Sep  3 18:59:38 localhost kernel: Bluetooth: RFCOMM TTY layer initialized
Sep  3 18:59:38 localhost kernel: Bluetooth: RFCOMM ver 1.8
Sep  3 18:59:38 localhost kernel: Bluetooth: HIDP (Human Interface Emulation) ver 1.1
Sep  3 18:59:38 localhost hidd[8046]: Bluetooth HID daemon
Sep  3 18:59:38 localhost pcscd: hotplug_libusb.c:411:HPEstablishUSBNotifications() Polling forced every 1 second(s)
Sep  3 18:59:38 localhost automount[8061]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Sep  3 18:59:39 localhost gpm[8145]: *** info [startup.c(95)]: 
Sep  3 18:59:39 localhost gpm[8145]: Started gpm successfully. Entered daemon mode.
Sep  3 18:59:40 localhost avahi-daemon[8241]: Found user 'avahi' (UID 70) and group 'avahi' (GID 70).
Sep  3 18:59:40 localhost avahi-daemon[8241]: Successfully dropped root privileges.
Sep  3 18:59:40 localhost avahi-daemon[8241]: avahi-daemon 0.6.16 starting up.
Sep  3 18:59:40 localhost avahi-daemon[8241]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Sep  3 18:59:40 localhost avahi-daemon[8241]: Successfully called chroot().
Sep  3 18:59:40 localhost avahi-daemon[8241]: Successfully dropped remaining capabilities.
Sep  3 18:59:40 localhost avahi-daemon[8241]: Loading service file /services/sftp-ssh.service.
Sep  3 18:59:40 localhost avahi-daemon[8241]: Network interface enumeration completed.
Sep  3 18:59:40 localhost avahi-daemon[8241]: Registering HINFO record with values 'X86_64'/'LINUX'.
Sep  3 18:59:40 localhost avahi-daemon[8241]: Server startup complete. Host name is localhost.local. Local service cookie is 2605129156.
Sep  3 18:59:40 localhost avahi-daemon[8241]: Service "SFTP File Transfer on localhost" (/services/sftp-ssh.service) successfully established.
Sep  3 18:59:46 localhost smartd[8350]: smartd version 5.36 [x86_64-redhat-linux-gnu] Copyright (C) 2002-6 Bruce Allen 
Sep  3 18:59:46 localhost smartd[8350]: Home page is http://smartmontools.sourceforge.net/  
Sep  3 18:59:46 localhost smartd[8350]: Opened configuration file /etc/smartd.conf 
Sep  3 18:59:46 localhost smartd[8350]: Configuration file /etc/smartd.conf parsed. 
Sep  3 18:59:46 localhost smartd[8350]: Device: /dev/sda, opened 
Sep  3 18:59:46 localhost smartd[8350]: Device: /dev/sda, found in smartd database. 
Sep  3 18:59:46 localhost smartd[8350]: Device: /dev/sda, is SMART capable. Adding to "monitor" list. 
Sep  3 18:59:46 localhost smartd[8350]: Device: /dev/sdb, opened 
Sep  3 18:59:46 localhost smartd[8350]: Device: /dev/sdb, not found in smartd database. 
Sep  3 18:59:47 localhost smartd[8350]: Device: /dev/sdb, is SMART capable. Adding to "monitor" list. 
Sep  3 18:59:47 localhost smartd[8350]: Monitoring 2 ATA and 0 SCSI devices 
Sep  3 18:59:47 localhost smartd[8350]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  3 18:59:47 localhost smartd[8350]: Sending warning via mail to root ... 
Sep  3 18:59:48 localhost smartd[8350]: Warning via mail to root: successful 
Sep  3 18:59:48 localhost smartd[8350]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  3 18:59:48 localhost smartd[8350]: Sending warning via mail to root ... 
Sep  3 18:59:49 localhost smartd[8350]: Warning via mail to root: successful 
Sep  3 18:59:49 localhost smartd[8367]: smartd has fork()ed into background mode. New PID=8367. 
Sep  3 18:59:50 localhost init: open(/dev/pts/0): No such file or directory
Sep  3 18:59:50 localhost last message repeated 3 times
Sep  3 18:59:50 localhost pcscd: winscard.c:304:SCardConnect() Reader E-Gate 0 0 Not Found
Sep  3 18:59:50 localhost last message repeated 3 times
Sep  3 19:00:02 localhost gdm[8475]: Couldn't authenticate user
Sep  3 19:00:10 localhost gconfd (root-8564): starting (version 2.14.0), pid 8564 user 'root'
Sep  3 19:00:10 localhost gconfd (root-8564): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0
Sep  3 19:00:10 localhost gconfd (root-8564): Resolved address "xml:readwrite:/root/.gconf" to a writable configuration source at position 1
Sep  3 19:00:10 localhost gconfd (root-8564): Resolved address "xml:readonly:/etc/gconf/gconf.xml.defaults" to a read-only configuration source at position 2
Sep  3 19:00:11 localhost hcid[7965]: Default passkey agent (:1.7, /org/bluez/applet) registered
Sep  3 19:00:11 localhost pcscd: winscard.c:304:SCardConnect() Reader E-Gate 0 0 Not Found
Sep  3 19:00:12 localhost last message repeated 4 times
Sep  3 19:00:12 localhost gconfd (root-8564): Resolved address "xml:readwrite:/root/.gconf" to a writable configuration source at position 0
Sep  3 19:00:16 localhost yum-updatesd-helper: error getting update info: Cannot retrieve repository metadata (repomd.xml) for repository: base. Please verify its path and try again
Sep  3 19:09:40 localhost kernel: JBD: no valid journal superblock found
Sep  3 19:09:40 localhost kernel: EXT3-fs: error loading journal.
Sep  3 19:11:29 localhost kernel: SCSI device sda: 156301488 512-byte hdwr sectors (80026 MB)
Sep  3 19:11:29 localhost kernel: sda: Write Protect is off
Sep  3 19:11:29 localhost kernel: SCSI device sda: drive cache: write back
Sep  3 19:11:29 localhost kernel:  sda: sda1
Sep  3 19:11:31 localhost kernel: SCSI device sda: 156301488 512-byte hdwr sectors (80026 MB)
Sep  3 19:11:31 localhost kernel: sda: Write Protect is off
Sep  3 19:11:31 localhost kernel: SCSI device sda: drive cache: write back
Sep  3 19:11:31 localhost kernel:  sda: sda1
Sep  3 19:26:33 localhost kernel: kjournald starting.  Commit interval 5 seconds
Sep  3 19:26:33 localhost kernel: EXT3 FS on sda1, internal journal
Sep  3 19:26:33 localhost kernel: EXT3-fs: mounted filesystem with ordered data mode.
Sep  3 19:27:45 localhost system-config-network[8843]: mkdir //etc/cipe
Sep  3 19:27:45 localhost system-config-network[8843]: ln //etc/sysconfig/network-scripts//ifcfg-eth0 //etc/sysconfig/networking/devices//ifcfg-eth0
Sep  3 19:27:45 localhost system-config-network[8843]: Linking eth0 to devices and putting it in profile default.
Sep  3 19:27:45 localhost system-config-network[8843]: ln //etc/sysconfig/networking/devices//ifcfg-eth0 //etc/sysconfig/networking/profiles//default/ifcfg-eth0
Sep  3 19:27:45 localhost system-config-network[8843]: ln //etc/hosts //etc/sysconfig/networking/profiles//default/hosts
Sep  3 19:27:45 localhost system-config-network[8843]: ln //etc/resolv.conf //etc/sysconfig/networking/profiles//default/resolv.conf
Sep  3 19:27:45 localhost system-config-network[8843]: ln //etc/sysconfig/network-scripts//ifcfg-eth1 //etc/sysconfig/networking/devices//ifcfg-eth1
Sep  3 19:27:45 localhost system-config-network[8843]: Linking eth1 to devices and putting it in profile default.
Sep  3 19:27:45 localhost system-config-network[8843]: ln //etc/sysconfig/networking/devices//ifcfg-eth1 //etc/sysconfig/networking/profiles//default/ifcfg-eth1
Sep  3 19:27:49 localhost kernel: skge eth0: enabling interface
Sep  3 19:27:49 localhost kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Sep  3 19:27:55 localhost kernel: skge eth0: disabling interface
Sep  3 19:27:58 localhost kernel: sky2 eth1: enabling interface
Sep  3 19:27:58 localhost kernel: sky2 eth1: ram buffer 48K
Sep  3 19:27:58 localhost kernel: ADDRCONF(NETDEV_UP): eth1: link is not ready
Sep  3 19:27:59 localhost kernel: sky2 eth1: Link is up at 100 Mbps, full duplex, flow control both
Sep  3 19:27:59 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Sep  3 19:28:00 localhost dhclient: DHCPDISCOVER on eth1 to 255.255.255.255 port 67 interval 6
Sep  3 19:28:00 localhost dhclient: DHCPOFFER from 192.168.0.1
Sep  3 19:28:00 localhost dhclient: DHCPREQUEST on eth1 to 255.255.255.255 port 67
Sep  3 19:28:00 localhost dhclient: DHCPACK from 192.168.0.1
Sep  3 19:28:00 localhost avahi-daemon[8241]: New relevant interface eth1.IPv4 for mDNS.
Sep  3 19:28:00 localhost avahi-daemon[8241]: Joining mDNS multicast group on interface eth1.IPv4 with address 192.168.0.54.
Sep  3 19:28:00 localhost avahi-daemon[8241]: Registering new address record for 192.168.0.54 on eth1.
Sep  3 19:28:00 localhost NET[9229]: /sbin/dhclient-script : updated /etc/resolv.conf
Sep  3 19:28:00 localhost dhclient: bound to 192.168.0.54 -- renewal in 16200 seconds.
Sep  3 19:28:01 localhost avahi-daemon[8241]: New relevant interface eth1.IPv6 for mDNS.
Sep  3 19:28:01 localhost avahi-daemon[8241]: Joining mDNS multicast group on interface eth1.IPv6 with address fe80::201:29ff:fe15:8a1b.
Sep  3 19:28:01 localhost avahi-daemon[8241]: Registering new address record for fe80::201:29ff:fe15:8a1b on eth1.
Sep  3 19:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  3 19:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  3 19:59:50 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  3 19:59:50 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  3 20:14:02 localhost init: Trying to re-exec init
Sep  3 20:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  3 20:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  3 20:59:50 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  3 20:59:50 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  3 21:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  3 21:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  3 21:59:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  3 21:59:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  3 22:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  3 22:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  3 22:59:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  3 22:59:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  3 23:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  3 23:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  3 23:58:00 localhost dhclient: DHCPREQUEST on eth1 to 192.168.0.1 port 67
Sep  3 23:58:00 localhost dhclient: DHCPACK from 192.168.0.1
Sep  3 23:58:00 localhost dhclient: bound to 192.168.0.54 -- renewal in 16200 seconds.
Sep  3 23:59:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  3 23:59:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  4 00:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  4 00:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  4 00:59:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  4 00:59:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  4 01:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  4 01:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  4 01:59:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  4 01:59:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  4 02:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  4 02:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  4 02:59:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  4 02:59:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  4 03:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  4 03:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  4 03:59:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  4 03:59:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  4 04:28:00 localhost dhclient: DHCPREQUEST on eth1 to 192.168.0.1 port 67
Sep  4 04:28:00 localhost dhclient: DHCPACK from 192.168.0.1
Sep  4 04:28:00 localhost dhclient: bound to 192.168.0.54 -- renewal in 16200 seconds.
Sep  4 04:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  4 04:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  4 04:59:50 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  4 04:59:50 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  4 05:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  4 05:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  4 05:59:50 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  4 05:59:50 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  4 06:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  4 06:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  4 06:59:50 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  4 06:59:50 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  4 07:29:50 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  4 07:29:50 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  4 07:59:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  4 07:59:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  4 08:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  4 08:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  4 08:57:59 localhost dhclient: DHCPREQUEST on eth1 to 192.168.0.1 port 67
Sep  4 08:58:00 localhost dhclient: DHCPACK from 192.168.0.1
Sep  4 08:58:00 localhost dhclient: bound to 192.168.0.54 -- renewal in 16200 seconds.
Sep  4 08:59:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  4 08:59:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  4 09:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  4 09:29:49 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 
Sep  4 09:59:50 localhost smartd[8367]: Device: /dev/sda, 20 Currently unreadable (pending) sectors 
Sep  4 09:59:50 localhost smartd[8367]: Device: /dev/sda, 20 Offline uncorrectable sectors 


THE OUTPUT OF "dumpe2fs -h /dev/sda1":
dumpe2fs 1.39 (29-May-2006)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          bd6f9e90-a253-46f4-9474-b13866c2d86c
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal resize_inode dir_index filetype needs_recovery sparse_super large_file
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              9781248
Block count:              19537040
Reserved block count:     976852
Free blocks:              19184049
Free inodes:              9781237
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1019
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16384
Inode blocks per group:   512
Filesystem created:       Wed Sep  3 19:23:04 2008
Last mount time:          Wed Sep  3 19:26:33 2008
Last write time:          Wed Sep  3 19:26:33 2008
Mount count:              1
Maximum mount count:      38
Last checked:             Wed Sep  3 19:23:04 2008
Check interval:           15552000 (6 months)
Next check after:         Mon Mar  2 19:23:04 2009
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      ff6fe2c8-e972-4a15-b667-cdd443a6fe6e
Journal backup:           inode blocks
Journal size:             128M

Comment 21 Kevin 2008-09-04 03:28:14 UTC

Hi,

there have some questions in Comment #17, i will trying to explain it as more as i can.

(In reply to comment #17)
> (Eric, I am adding you as an FYI. It is too early to conclude this is related
> to ext3, but just in case.)
> 
> (In reply to comment #7)
> 
> > SCSI device sdb: 4294920192 512-byte hdwr sectors (2198999 MB)
> 
> That says it is a 2 TiB disk. This is possible with an underlying HW RAID
> device. Is this the correct size?

Yes, the volume size is 2199GB only, i don't want use over 2TB volume to minimize the possible problem.

> >  sdb: unknown partition table
> 
> Is this correct, you did not put a partition table on sdb? 

Yes, i make a filesystem in sdb directly, without any partition.
the command is quite simple
# mkfs.ext3 /dev/sdb


> This may be fine. I am just checking to see if there is trouble reading the
> disk. 
> 
> > EXT3 FS on sdb, internal journal
> > EXT3-fs: mounted filesystem with ordered data mode.
> > SELinux: initialized (dev sdb, type ext3), uses xattr
> > attempt to access beyond end of device
> > sdb: rw=0, want=19473909208, limit=4294920192
> 
> The "limit" matches what we saw earlier, so that is good. The "want" sector
> looks ridiculous. I believe this means the filesystem metadata has become
> corrupt. 
> 
> Please describe the exact scenario that lead to this set of messages. For
> example, you booted the system on sda, the systme configured sdb okay, then you
> put a filesystem on sdb (did you create a partition, how?), then you mounted
> sdb and did I/O, the there was a failure (?), then you rebooted and saw the
> messages in comment 7? Or what exactly did happen? 

1. the system install on sda which attached on motherboard onboard data port.

2. controller have 7 drives configure with a under 2TB volume.(sdb)

3. format sdb with ext3 filesystem ( mkfs.ext3 /dev/sdb )

4. mount sdb and do multiple copy/compare tasks on sdb

5. copy/compare error occured, dump the error message and system log to a text file.

and these error messages below appears after problem happen, not while start up.
attempt to access beyond end of device
sdb: rw=0, want=19473909208, limit=4294920192
attempt to access beyond end of device
sdb: rw=0, want=19473909208, limit=4294920192
EXT3-fs error (device sdb): ext3_free_blocks: Freeing blocks not in datazone -
block = 2434238650, count = 1
Aborting journal on device sdb.
EXT3-fs error (device sdb) in ext3_free_blocks_sb: Journal has aborted
ext3_abort called.
EXT3-fs error (device sdb) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device sdb): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
EXT3-fs error (device sdb) in ext3_truncate: Journal has aborted
EXT3-fs error (device sdb) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device sdb) in ext3_orphan_del: Journal has aborted
EXT3-fs error (device sdb) in ext3_reserve_inode_write: Journal has aborted
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data

when i verify with 3ware controller, same test procedure, but the kernel will panic, so i will not able to dump the error messages.

i will trying to find the message log instead dmesg for easier understanding.


Kevin

> 
> You said that with the Areca you get errors (followed by a hang?). With the
> 3ware you get a crash. Without any more information we should assume these are
> different problems. Please provide more information on the 3ware problem.
> Ideally, a crash dump, or at least a system boot log and the error messages
> leading up to the crash and the stack trace. 
> 
> Simmilarly, without more information, we should not assume that the problem
> reported in bug 459247 is the same as what is described in comment 7. In 459247
> we have a system hang after heavy read/write to the RAID set. Comment 7 just
> shows failure to mount at boot time. Again, was comment 7 preceeded by a hang?
> while doing heavy I/O? Can you reproduce that (either comment 7 or BZ 459247)
> and capture a crash dump, or at least a system boot log and the error messages
> leading up to the crash and the stack trace?
> 
> Tom

Comment 22 Kevin 2008-09-04 03:34:41 UTC

Dear All,

these message below is came from the system message log, these messages happen after heavy loading, not while boot up or format device.


Aug 20 15:17:12 localhost dhclient: DHCPREQUEST on eth0 to 192.168.0.1 port 67
Aug 20 15:17:12 localhost dhclient: DHCPACK from 192.168.0.1
Aug 20 15:17:12 localhost dhclient: bound to 192.168.0.92 -- renewal in 16200 seconds.
Aug 20 15:28:22 localhost kernel: attempt to access beyond end of device
Aug 20 15:28:22 localhost kernel: sdb: rw=0, want=19473909208, limit=4294920192
Aug 20 15:28:22 localhost kernel: attempt to access beyond end of device
Aug 20 15:28:22 localhost kernel: sdb: rw=0, want=19473909208, limit=4294920192
Aug 20 15:28:27 localhost kernel: EXT3-fs error (device sdb): ext3_free_blocks: Freeing blocks not in datazone - block = 2434238650, count = 1
Aug 20 15:28:27 localhost kernel: Aborting journal on device sdb.
Aug 20 15:28:27 localhost kernel: EXT3-fs error (device sdb) in ext3_free_blocks_sb: Journal has aborted
Aug 20 15:28:27 localhost kernel: ext3_abort called.
Aug 20 15:28:27 localhost kernel: EXT3-fs error (device sdb) in ext3_reserve_inode_write: Journal has aborted
Aug 20 15:28:27 localhost kernel: EXT3-fs error (device sdb): ext3_journal_start_sb: Detected aborted journal
Aug 20 15:28:27 localhost kernel: Remounting filesystem read-only
Aug 20 15:28:27 localhost kernel: EXT3-fs error (device sdb) in ext3_truncate: Journal has aborted
Aug 20 15:28:27 localhost kernel: EXT3-fs error (device sdb) in ext3_reserve_inode_write: Journal has aborted
Aug 20 15:28:27 localhost kernel: EXT3-fs error (device sdb) in ext3_orphan_del: Journal has aborted
Aug 20 15:28:27 localhost kernel: EXT3-fs error (device sdb) in ext3_reserve_inode_write: Journal has aborted
Aug 20 15:28:27 localhost kernel: __journal_remove_journal_head: freeing b_committed_data
Aug 20 15:28:27 localhost last message repeated 3 times

Comment 23 Tomas Henzl 2008-09-05 15:19:22 UTC

Nick,Kevin,
I've just posted on http://people.redhat.com/thenzl/bz460789/areca.1.20.13/
which should be test version of RHEL5.2 with three latest Areca patches removed, this should be on the patchlevel as we have had in RHEL5.1.
Please test it if it makes any difference.

Comment 24 Nick Cheng 2008-09-09 06:08:57 UTC

Tomas,
I have verified your de-patch version.
It runs well so far.
The patched version would hang the system indeed while launching Areca AP, archttp.
Did you get system hanged with archttp or not?
Thank you,

Comment 25 Tomas Henzl 2008-09-09 09:42:05 UTC

Created attachment 316163 [details]
removed patches

Nick,
the best way to diagnose that problem is usually to bisect the kernel versions to find where it failed. Can you start at the last RHEL5.2  kernel version that you know is not working, and remove Areca patches to see what causes the problem ? 
The patches I removed are attached.

> The patched version would hang the system indeed while launching Areca AP,
> archttp.
> Did you get system hanged with archttp or not?
I didn't test that, I'll do it, but probably in a few days.

Comment 26 Nick Cheng 2008-09-09 10:46:21 UTC

Hi Tomas,
Yes, I am doing that.
I think I probably hit the nail on the head.
But I am not 100% sure.
Therefore, I have to verify by your experiments except mine.
In addition to that, Kevin ever said 3ware has the same trouble.
I will double-check to verify my hypothesis.
So far I use the driver which is reported to RHEL5.3 on RHEL5.2 and it works well.
If I am not clear enough or you still have questions about my doing, please let me know.
Thank you,

Comment 27 Tomas Henzl 2008-09-10 13:57:16 UTC

(In reply to comment #26)
Hi All,
> Therefore, I have to verify by your experiments except mine.
> In addition to that, Kevin ever said 3ware has the same trouble.
From what I've seen here it looks to me that this is probably a problem unrelated to this one with areca, not?. Please open a new bugzilla for this.
> I will double-check to verify my hypothesis.
> So far I use the driver which is reported to RHEL5.3 on RHEL5.2 and it works
> well.
Good news, so if somebody is willing to test it here - http://people.redhat.com/thenzl/bz460789/areca.1.20.15.RH1/
is the version with latest patch applied.
> If I am not clear enough or you still have questions about my doing, please let
> me know.
What exactly should I test ?
Thanks.

Comment 28 Nick Cheng 2008-09-11 00:58:46 UTC

Hi Tomas,
I can't open this hyperlink.
Would you please check it out?
BTW, what is the change inside?
Thanks,

Comment 29 Tomas Henzl 2008-09-11 08:42:40 UTC

(In reply to comment #28)
Hi Nick,
> I can't open this hyperlink.
I'm sorry it is -
http://people.redhat.com/thenzl/bz460789/areca.1.20.00.15.RH1/
> BTW, what is the change inside?
It has the latest patch applied (the patch from bz#436068). I did this in response
to this ->
>> So far I use the driver which is reported to RHEL5.3 on RHEL5.2 and it works
>> well.

Comment 30 Nick Cheng 2008-09-11 09:16:49 UTC

Hi Tomas,
In comment #27, you said the issue is not related to arcmsr.
How did you prove it?

Comment 31 Tomas Henzl 2008-09-11 09:44:49 UTC

(In reply to comment #30)
> Hi Tomas,
> In comment #27, you said the issue is not related to arcmsr.
> How did you prove it?
In comment #24 you stated :
......
I have verified your de-patch version.
It runs well so far.
......
From this I concluded(I only removed the areca patch not 3ware) that there probably is another problem with 3ware not related to areca.
Also this commnet from you:
..........
So far I use the driver which is reported to RHEL5.3 on RHEL5.2 and it works
well.
..........
On behalf of that I created a patched version with the newest drivers applied.

If I don't understand you comments well, please correct me.
Thanks.

Comment 32 Nick Cheng 2008-09-11 10:59:40 UTC

Hi Tomas,
I though you have any practice on your site to prove Areca driver is innocent.
I will make time to test your post.
Thank you,

Comment 33 Nick Cheng 2008-09-17 01:16:15 UTC

Tomas,
I have tested for two days long.
It has exceeded the period which former driver could last before.
FYI,

Comment 34 Tomas Henzl 2008-09-23 08:53:07 UTC

Nick,
how does it look now ? Can we close this issue ?

For the other problem with 3ware adapter if this still persist open a new bugzilla please.

Comment 35 Nick Cheng 2008-09-23 08:59:32 UTC

Tomas,
It is still working so far.
It seems to fix the issue.
I think we should close the issue.
Kevin,
Do you have any ideas?

Comment 36 Nick Cheng 2008-09-24 08:23:39 UTC

Tomas,
Kevin has no more comments.
Please close the issue.
Thanks for your time.

Comment 37 Tomas Henzl 2008-09-24 09:31:14 UTC

I'm closing this as a duplicate of "Bug 436068 - Update arcmsr to version 1.20.00.15.RH1"

*** This bug has been marked as a duplicate of bug 436068 ***