Bug 649705 - Buffer I/O errors on ghost path of multipath SCSI RDAC setup
Summary: Buffer I/O errors on ghost path of multipath SCSI RDAC setup
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel
Version: 5.5
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Red Hat Kernel Manager
QA Contact: Red Hat Kernel QE team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-11-04 11:06 UTC by Ben
Modified: 2018-12-02 16:34 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-06-03 12:49:21 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Ben 2010-11-04 11:06:01 UTC 
Description of problem:

We're running RHEL5.5 on a number of Sun X4600M2 servers with two Emulex FC cards in each of them.  For each server one port on each card is connected to each of the two controllers on a Sun 2540 storage array.  We are running the native multipathd on the servers set up in active/passive (ghost) mode.  At this time we are booting the kernel with the options "pci=noacpi irqpoll" and have rebuilt the initrd with "--preload=scsi_dh_rdac".

On every boot we see something similar to this:

[...]
Nov  4 07:14:42 foden kernel: Emulex LightPulse Fibre Channel SCSI driver 8.2.0.63.3p
Nov  4 07:14:42 foden kernel: Copyright(c) 2004-2009 Emulex.  All rights reserved.
Nov  4 07:14:42 foden kernel: scsi1 :  on PCI bus 03 device 00 irq 15
Nov  4 07:14:42 foden kernel: scsi2 :  on PCI bus 04 device 00 irq 7
Nov  4 07:14:42 foden kernel: lpfc 0000:03:00.0: 0:1303 Link Up Event x1 received Data: x1 x1 x10 x2 x0 x0 0
Nov  4 07:14:42 foden kernel:   Vendor: SUN       Model: LCSM100_F         Rev: 0617
Nov  4 07:14:42 foden kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Nov  4 07:14:42 foden kernel: SCSI device sdb: 3141768577 512-byte hdwr sectors (1608586 MB)
Nov  4 07:14:42 foden kernel: sdb: Write Protect is off
Nov  4 07:14:42 foden kernel: SCSI device sdb: drive cache: write back w/ FUA
Nov  4 07:14:42 foden kernel: SCSI device sdb: 3141768577 512-byte hdwr sectors (1608586 MB)
Nov  4 07:14:42 foden kernel: sdb: Write Protect is off
Nov  4 07:14:42 foden kernel: SCSI device sdb: drive cache: write back w/ FUA
Nov  4 07:14:42 foden kernel:  sdb:<6>device-mapper: uevent: version 1.0.3
Nov  4 07:14:42 foden kernel: device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel
Nov  4 07:14:42 foden kernel: device-mapper: multipath: version 1.0.5 loaded
Nov  4 07:14:42 foden kernel: rdac: device handler registered
Nov  4 07:14:42 foden kernel: device-mapper: dm-raid45: initialized v0.2594l
Nov  4 07:14:42 foden kernel: end_request: I/O error, dev sdb, sector 0
Nov  4 07:14:42 foden kernel: Buffer I/O error on device sdb, logical block 0
Nov  4 07:14:42 foden kernel: Buffer I/O error on device sdb, logical block 1
Nov  4 07:14:42 foden kernel: Buffer I/O error on device sdb, logical block 2
Nov  4 07:14:42 foden kernel: Buffer I/O error on device sdb, logical block 3
Nov  4 07:14:42 foden kernel: Buffer I/O error on device sdb, logical block 4
Nov  4 07:14:42 foden kernel: Buffer I/O error on device sdb, logical block 5
Nov  4 07:14:43 foden kernel: lpfc 0000:04:00.0: 1:1303 Link Up Event x1 received Data: x1 x1 x10 x2 x0 x0 0
Nov  4 07:14:43 foden kernel: Buffer I/O error on device sdb, logical block 6
Nov  4 07:14:43 foden kernel: Buffer I/O error on device sdb, logical block 7
Nov  4 07:14:43 foden kernel:   Vendor: SUN       Model: LCSM100_F         Rev: 0617
Nov  4 07:14:43 foden kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Nov  4 07:14:43 foden kernel: scsi 2:0:0:0: rdac: LUN 0 (owned)
Nov  4 07:14:43 foden kernel: SCSI device sdc: 3141768577 512-byte hdwr sectors (1608586 MB)
Nov  4 07:14:43 foden kernel: sdc: Write Protect is off
Nov  4 07:14:43 foden kernel: SCSI device sdc: drive cache: write back w/ FUA
Nov  4 07:14:43 foden kernel: SCSI device sdc: 3141768577 512-byte hdwr sectors (1608586 MB)
Nov  4 07:14:43 foden kernel: sdc: Write Protect is off
Nov  4 07:14:43 foden kernel: SCSI device sdc: drive cache: write back w/ FUA
Nov  4 07:14:43 foden kernel:  sdc: sdc1
Nov  4 07:14:43 foden kernel: sd 2:0:0:0: Attached scsi disk sdc
Nov  4 07:14:43 foden kernel: end_request: I/O error, dev sdb, sector 0
Nov  4 07:14:43 foden kernel: Buffer I/O error on device sdb, logical block 0
Nov  4 07:14:43 foden kernel: Buffer I/O error on device sdb, logical block 1
Nov  4 07:14:43 foden kernel: end_request: I/O error, dev sdb, sector 0
Nov  4 07:14:43 foden last message repeated 4 times
Nov  4 07:14:43 foden kernel: Dev sdb: unable to read RDB block 0
Nov  4 07:14:43 foden kernel: end_request: I/O error, dev sdb, sector 0
Nov  4 07:14:43 foden kernel: end_request: I/O error, dev sdb, sector 0
Nov  4 07:14:43 foden kernel:  unable to read partition table
Nov  4 07:14:43 foden kernel: sd 1:0:0:0: Attached scsi disk sdb
Nov  4 07:14:43 foden kernel: end_request: I/O error, dev sdb, sector 0
Nov  4 07:14:43 foden kernel: printk: 62 messages suppressed.
Nov  4 07:14:43 foden kernel: Buffer I/O error on device sdb, logical block 0
Nov  4 07:14:43 foden kernel: Buffer I/O error on device sdb, logical block 1
Nov  4 07:14:43 foden kernel: Buffer I/O error on device sdb, logical block 2
Nov  4 07:14:43 foden kernel: end_request: I/O error, dev sdb, sector 0
Nov  4 07:14:43 foden kernel: end_request: I/O error, dev sdb, sector 3141768448
Nov  4 07:14:43 foden kernel: end_request: I/O error, dev sdb, sector 3141768560
Nov  4 07:14:43 foden kernel: end_request: I/O error, dev sdb, sector 0
Nov  4 07:14:44 foden kernel: end_request: I/O error, dev sdb, sector 8
Nov  4 07:14:44 foden kernel: end_request: I/O error, dev sdb, sector 0
Nov  4 07:14:44 foden kernel: end_request: I/O error, dev sdb, sector 0
Nov  4 07:14:44 foden kernel: end_request: I/O error, dev sdb, sector 3141768448
Nov  4 07:14:44 foden kernel: end_request: I/O error, dev sdb, sector 3141768448
Nov  4 07:14:44 foden kernel: printk: 45 messages suppressed.
Nov  4 07:14:44 foden kernel: Buffer I/O error on device sdb, logical block 3141768448
Nov  4 07:14:44 foden kernel: end_request: I/O error, dev sdb, sector 0
Nov  4 07:14:44 foden kernel: end_request: I/O error, dev sdb, sector 0
Nov  4 07:14:44 foden kernel: end_request: I/O error, dev sdb, sector 3141768448
Nov  4 07:14:44 foden kernel: end_request: I/O error, dev sdb, sector 3141768448
Nov  4 07:14:44 foden kernel: end_request: I/O error, dev sdb, sector 0
Nov  4 07:14:44 foden kernel: end_request: I/O error, dev sdb, sector 0
Nov  4 07:14:44 foden kernel: end_request: I/O error, dev sdb, sector 3141768448
Nov  4 07:14:44 foden kernel: end_request: I/O error, dev sdb, sector 3141768448
Nov  4 07:14:44 foden kernel: end_request: I/O error, dev sdb, sector 0
Nov  4 07:14:44 foden kernel: printk: 183 messages suppressed.
Nov  4 07:14:44 foden kernel: Buffer I/O error on device sdb, logical block 0
Nov  4 07:14:44 foden kernel: end_request: I/O error, dev sdb, sector 0
[...]

which is obviously down to the kernel being unable to talk to the array via sdb as sdc is the active path:

# multipath -ll 
datavol (3600a0b800038b3e500000224477df1d2) dm-2 SUN,LCSM100_F
[size=1.5T][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=100][active]
 \_ 2:0:0:0 sdc 8:32  [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:0:0 sdb 8:16  [active][ghost]

The thing is, I'd assumed that adding the preload to the initrd would solve this (and stop these errors happening/appearing) by allowing the kernel to realise that sdb was part of an RDAC/multipath setup (with sdc).  Is this not the case?

I note that there was a suggestion to add a QUIET enhancement to the kernel in 2.6.29 for this kind of thing.  Can/did this get backported?


Version-Release number of selected component (if applicable):

Emulex LightPulse Fibre Channel SCSI driver 8.2.0.63.3p
device-mapper-multipath-0.4.7-34.el5_5.6


How reproducible:

Every boot, and repeatedly during normal operation


Steps to Reproduce:

1. Connect a two-controller array capable of active/failover
2. Set up multipath.conf and enable multipathd
3. Rebuild the initrd with scsi_dh_rdac preloaded
4. Reboot


Actual results:

Buffer I/O errors reported for the passive/ghost path.


Expected results:

Blessed silence, until the active path fails, of course.  Then messages about the ghost path coming up (and about the previously active path being broken).


Additional info:

Obviously everything is working on the active path, no complaints there.  But it would be much cleaner if the ghost path didn't cause the kernel to bleat about this.  It scares the DBAs...
Comment 1 Ben 2011-02-14 10:57:18 UTC 
Since logging this bug I've tried a new kernel or two (RHEL ones), rebuilding the initrd each time with the scsi_dh_rdac module preloaded.  Recent boots claim that the module cannot be inserted twice (meaning that it is already being preloaded).  I've modified /etc/lvm/lvm.conf with the line:

filter = [ "a/dev/sda[0-9]+/", "r/.*/" ]

and yet we still get many many I/O errors on the ghost path.  

Please can you either fix this or detail the solution which will make these errors go away?
Comment 2 Alvaro Fernandez 2011-02-14 11:27:22 UTC 
I too last night modified lvm.conf to exclude that way non-LVM managed paths and still errors. RHEL5.5 + device-mapper-multipath-0.4.7-42.el5). Using a IBM DS4300 SAN. 

the Buffer I/O errors over ghost paths we're having them on startup, but not on normal operation (only when we switch or start the cluster services and the volumes managed by dm-multipath.

Hope this get corrected too,
Comment 3 Alvaro Fernandez 2011-05-31 09:38:07 UTC 
Hello,

¿Any updates on this issue?
Comment 4 Ben 2011-11-17 11:57:41 UTC 
Just another data point given I've just updated a few of my boxes with ghost paths to the most recent kernel (2.6.18-274.7.1.el5).  Much of the data is the same:

# multipath -ll
datavol (3600a0b800038b3e500000224477df1d2) dm-2 SUN,LCSM100_F
[size=1.5T][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=100][active]
 \_ 2:0:0:0 sdc 8:32  [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:0:0 sdb 8:16  [active][ghost]


[...]
Nov 17 07:46:44 foden kernel: Emulex LightPulse Fibre Channel SCSI driver 8.2.0.96.2p
Nov 17 07:46:44 foden kernel: Copyright(c) 2004-2011 Emulex.  All rights reserved.
Nov 17 07:46:44 foden kernel: scsi1 :  on PCI bus 03 device 00 irq 15
Nov 17 07:46:44 foden kernel: scsi2 :  on PCI bus 04 device 00 irq 7
Nov 17 07:46:44 foden kernel: lpfc 0000:03:00.0: 0:1303 Link Up Event x1 received Data: x1 x1 x10 x2 x0 x0 0
Nov 17 07:46:44 foden kernel: lpfc 0000:03:00.0: 0:1310 Link Up Event npiv not supported in loop topology
Nov 17 07:46:44 foden kernel: lpfc 0000:03:00.0: 0:(0):2858 FLOGI failure Status:x3/x18 TMO:x0
Nov 17 07:46:44 foden last message repeated 2 times
Nov 17 07:46:44 foden kernel: lpfc 0000:03:00.0: 0:(0):0100 FLOGI failure Status:x3/x18 TMO:x0
Nov 17 07:46:45 foden kernel:   Vendor: SUN       Model: LCSM100_F         Rev: 0617
Nov 17 07:46:45 foden kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Nov 17 07:46:45 foden kernel: scsi 1:0:0:0: rdac: LUN 0 (unowned)
Nov 17 07:46:45 foden kernel: SCSI device sdb: 3141768577 512-byte hdwr sectors (1608586 MB)
Nov 17 07:46:45 foden kernel: sdb: Write Protect is off
Nov 17 07:46:45 foden kernel: SCSI device sdb: drive cache: write back w/ FUA
Nov 17 07:46:45 foden kernel: SCSI device sdb: 3141768577 512-byte hdwr sectors (1608586 MB)
Nov 17 07:46:45 foden kernel: sdb: Write Protect is off
Nov 17 07:46:45 foden kernel: SCSI device sdb: drive cache: write back w/ FUA
Nov 17 07:46:45 foden kernel:  sdb:<3>Buffer I/O error on device sdb, logical block 0
Nov 17 07:46:45 foden kernel: Buffer I/O error on device sdb, logical block 1
Nov 17 07:46:45 foden kernel: Buffer I/O error on device sdb, logical block 2
Nov 17 07:46:45 foden kernel: Buffer I/O error on device sdb, logical block 3
Nov 17 07:46:45 foden kernel: Buffer I/O error on device sdb, logical block 4
Nov 17 07:46:45 foden kernel: Buffer I/O error on device sdb, logical block 5
Nov 17 07:46:45 foden kernel: Buffer I/O error on device sdb, logical block 6
Nov 17 07:46:45 foden kernel: Buffer I/O error on device sdb, logical block 7
Nov 17 07:46:45 foden kernel: Buffer I/O error on device sdb, logical block 0
Nov 17 07:46:45 foden kernel: Buffer I/O error on device sdb, logical block 1
Nov 17 07:46:45 foden kernel: Dev sdb: unable to read RDB block 0
Nov 17 07:46:45 foden kernel:  unable to read partition table
Nov 17 07:46:45 foden kernel: sd 1:0:0:0: Attached scsi disk sdb
Nov 17 07:46:45 foden kernel: device-mapper: uevent: version 1.0.3
Nov 17 07:46:45 foden kernel: device-mapper: ioctl: 4.11.6-ioctl (2011-02-18) initialised: dm-devel
Nov 17 07:46:45 foden kernel: device-mapper: multipath: version 1.0.6 loaded
Nov 17 07:46:45 foden kernel: device-mapper: dm-raid45: initialized v0.2594l
Nov 17 07:46:45 foden kernel: lpfc 0000:04:00.0: 1:1303 Link Up Event x1 received Data: x1 x1 x10 x2 x0 x0 0
Nov 17 07:46:45 foden kernel: lpfc 0000:04:00.0: 1:1310 Link Up Event npiv not supported in loop topology
Nov 17 07:46:45 foden kernel: lpfc 0000:04:00.0: 1:(0):2858 FLOGI failure Status:x3/x18 TMO:x0
Nov 17 07:46:45 foden last message repeated 2 times
Nov 17 07:46:45 foden kernel: lpfc 0000:04:00.0: 1:(0):0100 FLOGI failure Status:x3/x18 TMO:x0
Nov 17 07:46:45 foden kernel:   Vendor: SUN       Model: LCSM100_F         Rev: 0617
Nov 17 07:46:45 foden kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Nov 17 07:46:45 foden kernel: scsi 2:0:0:0: rdac: LUN 0 (owned)
Nov 17 07:46:46 foden kernel: SCSI device sdc: 3141768577 512-byte hdwr sectors (1608586 MB)
Nov 17 07:46:46 foden kernel: sdc: Write Protect is off
Nov 17 07:46:46 foden kernel: SCSI device sdc: drive cache: write back w/ FUA
Nov 17 07:46:46 foden kernel: SCSI device sdc: 3141768577 512-byte hdwr sectors (1608586 MB)
Nov 17 07:46:46 foden kernel: sdc: Write Protect is off
Nov 17 07:46:46 foden kernel: SCSI device sdc: drive cache: write back w/ FUA
Nov 17 07:46:46 foden kernel:  sdc: sdc1
Nov 17 07:46:46 foden kernel: sd 2:0:0:0: Attached scsi disk sdc
Nov 17 07:46:46 foden kernel: printk: 62 messages suppressed.
Nov 17 07:46:46 foden kernel: Buffer I/O error on device sdb, logical block 3141768448
Nov 17 07:46:46 foden kernel: Buffer I/O error on device sdb, logical block 3141768449
Nov 17 07:46:46 foden kernel: kjournald starting.  Commit interval 5 seconds
Nov 17 07:46:46 foden kernel: EXT3-fs: mounted filesystem with ordered data mode.
Nov 17 07:46:46 foden kernel: SELinux:  Disabled at runtime.
Nov 17 07:46:46 foden kernel: EDAC MC: Ver: 2.0.1 Oct 17 2011
Nov 17 07:46:46 foden kernel: Intel(R) PRO/1000 Network Driver - version 7.3.21-k4-3-NAPI
Nov 17 07:46:46 foden kernel: Copyright (c) 1999-2006 Intel Corporation.
Nov 17 07:46:46 foden kernel: i2c_adapter i2c-0: nForce2 SMBus adapter at 0xed00
Nov 17 07:46:46 foden kernel: i2c_adapter i2c-1: nForce2 SMBus adapter at 0xed40
Nov 17 07:46:46 foden kernel: e1000: 0000:05:01.0: e1000_probe: (PCI-X:133MHz:64-bit) 00:14:4f:a8:38:b8
Nov 17 07:46:46 foden kernel: input: PC Speaker as /class/input/input2
Nov 17 07:46:46 foden kernel: e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
Nov 17 07:46:46 foden kernel: scsi 0:0:0:0: Attached scsi generic sg0 type 0
Nov 17 07:46:46 foden kernel: scsi 0:0:1:0: Attached scsi generic sg1 type 0
Nov 17 07:46:46 foden kernel: sd 0:1:0:0: Attached scsi generic sg2 type 0
Nov 17 07:46:46 foden kernel: sd 1:0:0:0: Attached scsi generic sg3 type 0
Nov 17 07:46:46 foden kernel: sd 2:0:0:0: Attached scsi generic sg4 type 0
Nov 17 07:46:46 foden kernel: e1000: 0000:05:01.1: e1000_probe: (PCI-X:133MHz:64-bit) 00:14:4f:a8:38:b9
Nov 17 07:46:46 foden kernel: e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
Nov 17 07:46:46 foden kernel: e1000: 0000:05:02.0: e1000_probe: (PCI-X:133MHz:64-bit) 00:14:4f:a8:38:ba
Nov 17 07:46:46 foden kernel: e1000: eth2: e1000_probe: Intel(R) PRO/1000 Network Connection
Nov 17 07:46:46 foden kernel: e1000: 0000:05:02.1: e1000_probe: (PCI-X:133MHz:64-bit) 00:14:4f:a8:38:bb
Nov 17 07:46:46 foden kernel: e1000: eth3: e1000_probe: Intel(R) PRO/1000 Network Connection
Nov 17 07:46:46 foden kernel: EDAC MC0: Giving out device to k8_edac Athlon64/Opteron: DEV 0000:00:18.2
Nov 17 07:46:46 foden kernel: EDAC MC1: Giving out device to k8_edac Athlon64/Opteron: DEV 0000:00:19.2
Nov 17 07:46:46 foden kernel: EDAC MC2: Giving out device to k8_edac Athlon64/Opteron: DEV 0000:00:1a.2
Nov 17 07:46:46 foden kernel: EDAC MC3: Giving out device to k8_edac Athlon64/Opteron: DEV 0000:00:1b.2
Nov 17 07:46:46 foden kernel: EDAC MC4: Giving out device to k8_edac Athlon64/Opteron: DEV 0000:00:1c.2
Nov 17 07:46:46 foden kernel: EDAC MC5: Giving out device to k8_edac Athlon64/Opteron: DEV 0000:00:1d.2
Nov 17 07:46:47 foden kernel: EDAC MC6: Giving out device to k8_edac Athlon64/Opteron: DEV 0000:00:1e.2
Nov 17 07:46:47 foden kernel: EDAC MC7: Giving out device to k8_edac Athlon64/Opteron: DEV 0000:00:1f.2
Nov 17 07:46:47 foden kernel: printk: 265 messages suppressed.
Nov 17 07:46:47 foden kernel: Buffer I/O error on device sdb, logical block 0
Nov 17 07:46:47 foden kernel: floppy0: no floppy controllers found
Nov 17 07:46:47 foden kernel: work still pending
Nov 17 07:46:47 foden kernel: lp: driver loaded but no devices found
Nov 17 07:46:47 foden kernel: ACPI: Power Button (FF) [PWRF]
Nov 17 07:46:47 foden kernel: ACPI: Power Button (CM) [PWRB]
Nov 17 07:46:47 foden kernel: ACPI: Mapper loaded
Nov 17 07:46:47 foden kernel: dell-wmi: No known WMI GUID found
Nov 17 07:46:47 foden kernel: md: Autodetecting RAID arrays.
Nov 17 07:46:47 foden kernel: md: autorun ...
Nov 17 07:46:47 foden kernel: md: ... autorun DONE.
Nov 17 07:46:47 foden kernel: device-mapper: multipath: Using scsi_dh module scsi_dh_rdac for failover/failback and device management.
Nov 17 07:46:47 foden kernel: device-mapper: multipath round-robin: version 1.0.0 loaded
Nov 17 07:46:47 foden kernel: printk: 39 messages suppressed.
Nov 17 07:46:47 foden kernel: Buffer I/O error on device sdb, logical block 3141768576
Nov 17 07:46:47 foden kernel: EXT3 FS on dm-0, internal journal
Nov 17 07:46:47 foden kernel: kjournald starting.  Commit interval 5 seconds
Nov 17 07:46:47 foden kernel: EXT3 FS on sda1, internal journal
Nov 17 07:46:47 foden kernel: EXT3-fs: mounted filesystem with ordered data mode.
Nov 17 07:46:47 foden kernel: kjournald starting.  Commit interval 5 seconds
Nov 17 07:46:47 foden kernel: EXT3 FS on dm-3, internal journal
Nov 17 07:46:47 foden kernel: EXT3-fs: mounted filesystem with ordered data mode.
Nov 17 07:46:47 foden kernel: Adding 33554424k swap on /dev/VolGroup00/LogVol01.  Priority:-1 extents:1 across:33554424k
Nov 17 07:46:47 foden kernel: NET: Registered protocol family 10
Nov 17 07:46:47 foden kernel: lo: Disabled Privacy Extensions
Nov 17 07:46:47 foden kernel: IPv6 over IPv4 tunneling driver
Nov 17 07:46:47 foden kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Nov 17 07:46:47 foden kernel: e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Nov 17 07:46:47 foden kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Nov 17 07:46:47 foden kernel: ipmi message handler version 39.1
Nov 17 07:46:47 foden kernel: IPMI System Interface driver.
Nov 17 07:46:48 foden kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca4, slave address 0x20, irq 0
Nov 17 07:46:48 foden kernel: ipmi: Found new BMC (man_id: 0x00002a,  prod_id: 0x4701, dev_id: 0x20)
Nov 17 07:46:48 foden kernel:  IPMI kcs interface initialized
Nov 17 07:46:48 foden kernel: ipmi_si: Trying ACPI-specified kcs state machine at mem address 0x0, slave address 0x0, irq 0
Nov 17 07:46:48 foden kernel:  Could not set up I/O space
Nov 17 07:46:48 foden kernel: ipmi device interface
Nov 17 07:46:48 foden kernel: Installing knfsd (copyright (C) 1996 okir.de).
Nov 17 07:46:48 foden kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
Nov 17 07:46:48 foden kernel: NFSD: starting 90-second grace period
Nov 17 07:46:48 foden kernel: printk: 758 messages suppressed.
Nov 17 07:46:48 foden kernel: Buffer I/O error on device sdb, logical block 0
Nov 17 07:46:48 foden kernel: Buffer I/O error on device sdb, logical block 1
Nov 17 07:46:48 foden kernel: Buffer I/O error on device sdb, logical block 2
Nov 17 07:46:48 foden kernel: Buffer I/O error on device sdb, logical block 3
Nov 17 07:46:48 foden kernel: Fusion MPT misc device (ioctl) driver 3.04.18rh
Nov 17 07:46:48 foden kernel: mptctl: Registered with Fusion MPT base driver
Nov 17 07:46:48 foden kernel: mptctl: /dev/mptctl @ (major,minor=10,220)

I've still been rebuilding the initrd with "--preload=scsi_dh_rdac", which hasn't helped.  And in fact since this series of kernels (274) during boot have been seeing lots of attempts to use other drivers too, with I/O error results, before it apparently settles on the right one.
Comment 5 Ant 2012-07-27 09:00:02 UTC 
Just a 'me too' on this. I've done the 'preload' thing and also unpacked the initramfs to make sure that the lvm filter line is set correctly (it is) . I get errors much like the above during boot, but not during normal operation. vgscan doesn't report any arrors since I filtered out the ghost paths.

I've got a pair of  Emulex LPe12000 1Gb cards in an IBM X series box, connected to an IBM 1746 disk array.

here's a sample:

Jul 26 18:39:45 mycomputer kernel: sd 0:0:0:0: [sda]  <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
Jul 26 18:39:45 mycomputer kernel: sd 0:0:0:0: [sda] CDB: Read(10): 28 00 00 00 00 04 00 00 04 00
Jul 26 18:39:45 mycomputer kernel: end_request: I/O error, dev sda, sector 4
Jul 26 18:39:45 mycomputer kernel: Buffer I/O error on device sda, logical block 1
Jul 26 18:39:45 mycomputer kernel: sd 1:0:0:1: [sde]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 26 18:39:45 mycomputer kernel: sd 1:0:0:1: [sde]  Sense Key : Illegal Request [current]
Jul 26 18:39:45 mycomputer kernel: sd 1:0:0:1: [sde]  <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
Jul 26 18:39:45 mycomputer kernel: sd 1:0:0:1: [sde] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Jul 26 18:39:45 mycomputer kernel: end_request: I/O error, dev sde, sector 0
Jul 26 18:39:45 mycomputer kernel: Buffer I/O error on device sde, logical block 0


sdb and d are the active paths; sdc is the root partition on an internal LSI controller without multipath:

# cat /proc/partitions
major minor  #blocks  name

   8       32  291991552 sdc
   8       33     204800 sdc1
   8       34     512000 sdc2
   8       35  291272704 sdc3
   8       16  524288000 sdb
   8       48 1181116006 sdd
   8        0 1181116006 sda
   8       64  524288000 sde
Comment 6 RHEL Program Management 2012-10-30 05:56:57 UTC 
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.
Comment 7 Frank Liu 2012-12-08 19:05:42 UTC 
Have you tried adding 'rdloaddriver=scsi_dh_rdac' to the kernel commandline in grub.conf ? 
That worked for me with IBM DS4700.
Comment 8 Frank Liu 2012-12-08 19:19:41 UTC 
Just realized this issue was for Redhat 5.
I am running Redhat 6, and same issues there:

https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/6.2_Technical_Notes/kernel_issues.html

...
 kernel component

    If the storage driver is loaded before multipathd is started, I/O errors occur. To work around this issue, use one of the following kernel command line parameters which are consumed by dracut:

    rdloaddriver=scsi_dh_emc

    or

    rdloaddriver=scsi_dh_rdac

    or

    rdloaddriver=scsi_dh_emc,scsi_dh_rdac

    The above command line parameters will cause the scsi_dh module to load before multipath is started.
...
Comment 9 RHEL Program Management 2014-03-07 12:14:43 UTC 
Thank you for submitting this request for inclusion in Red Hat Enterprise Linux 5. We've carefully evaluated the request, but are unable to include it in the  last planned RHEL5 minor release. This Bugzilla will soon be CLOSED as WONTFIX. To request that Red Hat re-consider this request, please re-open the bugzilla via  appropriate support channels and provide additional business and/or technical details about its importance to you.
Comment 10 RHEL Program Management 2014-06-03 12:49:21 UTC 
Thank you for submitting this request for inclusion in Red Hat Enterprise Linux 5. We've carefully evaluated the request, but are unable to include it in RHEL5 stream. If the issue is critical for your business, please provide additional business justification through the appropriate support channels (https://access.redhat.com/site/support).
Comment 11 Ben 2014-06-09 14:07:18 UTC 
I have no info to provide.  Given you're not going to fix it! (-:
Note You need to log in before you can comment on or make changes to this bug.