Created attachment 354287 [details] Bug fix for uninitialized variables in sat driver. Description of problem: After upgrading a server to F11, whenever I run smartmontools, I get an error, and then the following get logged to dmesg: mptscsih: ioc0: attempting task abort! (sc=ffff8800bd1b0d00) sd 7:0:2:0: [sdd] CDB: ATA command pass through(16): 85 08 0e 00 00 00 01 00 00 00 00 00 00 00 ec 00 mptbase: ioc0: LogInfo(0x31140000): Originator={PL}, Code={IO Executed}, SubCode(0x0000) mptscsih: ioc0: task abort: SUCCESS (sc=ffff8800bd1b0d00) mptbase: ioc0: LogInfo(0x31170000): Originator={PL}, Code={IO Device Missing Delay Retry}, SubCode(0x0000) mptscsih: ioc0: attempting target reset! (sc=ffff8800bd1b0d00) sd 7:0:2:0: [sdd] CDB: ATA command pass through(16): 85 08 0e 00 00 00 01 00 00 00 00 00 00 00 ec 00 mptsas: ioc0: removing sata device, channel 0, id 18, phy 2 port-7:2: mptsas: ioc0: delete port (2) I was not happy when smartd started up and the kernel offlined all eight drives in a RAID-5 configuration :-( After some investigations and looking at bug #452389 at https://bugzilla.redhat.com/show_bug.cgi?id=452389 I found the root cause of the problem. There are some uninitialized variables passed down to the kernel and the SCSI or libata layer gets confused and offline the drive. Please consider fixing this ASAP as SAT is mostly used with big disks array and the consequences of an unpatched smartmontools can be dire. I did not enjoy having to get my RAID array back on its feet after this problem.
Hold on there, I'm still seeing the problem with the patch.
So, this patch is useless. Let me restate the symptoms: # smartctl -d sat -a /dev/sde where sde is a SATA hard drive attached to a SAS controller (LSI SAS1068) gives the following output: smartctl version 5.38 [x86_64-unknown-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ [hangs there for 15-30 seconds] === START OF INFORMATION SECTION === Device Model: [No Information Found] Serial Number: [No Information Found] Firmware Version: [No Information Found] Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 1 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Sun Jul 19 16:30:26 2009 PDT SMART is only available in ATA Version 3 Revision 3 or greater. We will try to proceed in spite of this. SMART support is: Ambiguous - ATA IDENTIFY DEVICE words 82-83 don't show if SMART supported. Checking for SMART support by trying SMART ENABLE command. SMART ENABLE appeared to work! Continuing. SMART support is: Ambiguous - ATA IDENTIFY DEVICE words 85-87 don't show if SMART is enabled. A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options. At the same time, the kernel logs: mptscsih: ioc0: attempting task abort! (sc=ffff8800b0151e00) sd 7:0:3:0: [sde] CDB: ATA command pass through(16): 85 08 0e 00 00 00 01 00 00 00 00 00 00 00 ec 00 mptbase: ioc0: LogInfo(0x31140000): Originator={PL}, Code={IO Executed}, SubCode(0x0000) mptscsih: ioc0: task abort: SUCCESS (sc=ffff8800b0151e00) mptbase: ioc0: LogInfo(0x31170000): Originator={PL}, Code={IO Device Missing Delay Retry}, SubCode(0x0000) mptscsih: ioc0: attempting target reset! (sc=ffff8800b0151e00) sd 7:0:3:0: [sde] CDB: ATA command pass through(16): 85 08 0e 00 00 00 01 00 00 00 00 00 00 00 ec 00 mptsas: ioc0: removing sata device, channel 0, id 19, phy 3 port-7:3: mptsas: ioc0: delete port (3) sd 7:0:3:0: [sde] Synchronizing SCSI cache mptscsih: ioc0: target reset: SUCCESS (sc=ffff8800b0151e00) mptscsih: ioc0: attempting bus reset! (sc=ffff8800b0151e00) sd 7:0:3:0: [sde] CDB: ATA command pass through(16): 85 08 0e 00 00 00 01 00 00 00 00 00 00 00 ec 00 mptscsih: ioc0: bus reset: SUCCESS (sc=ffff8800b0151e00) mptscsih: ioc0: attempting host reset! (sc=ffff8800b0151e00) mptbase: ioc0: Initiating recovery mptscsih: ioc0: host reset: SUCCESS (sc=ffff8800b0151e00) sd 7:0:3:0: Device offlined - not ready after error recovery program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO sd 7:0:3:0: [sde] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
Ok, as pointed out in bug #452389 disabling -Wl,-z,relro,-z,now makes a working executable. I have no idea why it would though. I would consider disabling it as the consequences for people with SAS controllers are very annoying, and the security benefits of relro are negligible: smartmontools and smartd are only run by root and do not otherwise interact with the network. Phil.
I've prepared svn snapshot for different bug report. Smartmontools has changed a lot since last official release. Could you try if this package works for you? http://koji.fedoraproject.org/koji/taskinfo?taskID=1465244
Scuse my thickness, but how do I download the packages from there? The "Output" section is empty...
Sorry, my fault, it seems scratch builds are delete quite soon, I've prepared new package all architectures sub-pages now have links to packages
jeez... and this time completely without link... sorry :( here is that link: http://koji.fedoraproject.org/koji/taskinfo?taskID=1494352 x86_64 package: http://koji.fedoraproject.org/koji/getfile?taskID=1494354&name=smartmontools-5.39-0.fc11.svn20090721.x86_64.rpm i686 package: http://koji.fedoraproject.org/koji/getfile?taskID=1494356&name=smartmontools-5.39-0.fc11.svn20090721.i586.rpm if you want packages for different architecture or need debuginfo packages, use first link
This scratch build appears to have been deleted too. I don't see any links to the .src.rpm either. Can you prepare something new? Thanks, Pim
yes, scratch build lives only a few days before it's deleted build: http://koji.fedoraproject.org/koji/taskinfo?taskID=1583235 x86_64: https://koji.fedoraproject.org/koji/getfile?taskID=1583237&name=smartmontools-5.39-0.svn20090721.fc11.x86_64.rpm i586: https://koji.fedoraproject.org/koji/getfile?taskID=1583239&name=smartmontools-5.39-0.svn20090721.fc11.i586.rpm
I tried this scratch build on one of six systems I have seen this problem on. It looked to be OK for a while; smartctl did not immediately bring the drives down, but smartd would, after a while, randomly. Al of the systems I'm seeing this problem on have LSI 1068E SAS controllers, with host RAID disabled, using mdadm RAID arrays of 4 to 12 drives. Needless to say it hurts when several drives in an mdadm RAID array go down unexpectedly. On Fedora 9 and older, this problem did not exist. It might be that not smartmontools but the mptsas driver is the problem, with smartmontools just triggering the bug. Has anyone seen this problem on SAS controllers other than those of LSI? Thanks, Pim
(In reply to comment #10) > I tried this scratch build on one of six systems I have seen this problem on. > > It looked to be OK for a while; smartctl did not immediately bring the drives > down, but smartd would, after a while, randomly. are there any messages in logs related? > > Al of the systems I'm seeing this problem on have LSI 1068E SAS controllers, > with host RAID disabled, using mdadm RAID arrays of 4 to 12 drives. do you have any similar systems where it works? What are they? > On Fedora 9 and older, this problem did not exist. It might be that not > smartmontools but the mptsas driver is the problem, with smartmontools just > triggering the bug. > > Has anyone seen this problem on SAS controllers other than those of LSI? I don't know about anything related. I've found only bug #493697
Here is the part of the log where things go wrong. Like I said, with this scratch build it takes a while to cause the problem to appear. For this crash to happen, I started smartd on all drives (1 ata and 8 sat drive) and ran "smartctl -a /dev/sdX" on each drive. I had a hit on /dev/sdo: Aug 7 19:00:12 moog kernel: mptscsih: ioc0: attempting task abort! (sc=ffff8802255bac00) Aug 7 19:00:12 moog kernel: sd 8:0:6:0: [sdo] CDB: ATA command pass through(16): 85 08 0e 00 d5 00 01 00 09 00 4f 00 c2 00 b0 00 Aug 7 19:00:17 moog kernel: mptbase: ioc0: LogInfo(0x31140000): Originator={PL}, Code={IO Executed}, SubCode(0x0000) Aug 7 19:00:17 moog kernel: mptsas: ioc0: removing sata device, channel 0, id 6, phy 6 Aug 7 19:00:17 moog kernel: sd 8:0:6:0: [sdo] Synchronizing SCSI cache Aug 7 19:00:17 moog kernel: mptscsih: ioc0: task abort: SUCCESS (sc=ffff8802255bac00) Aug 7 19:00:17 moog kernel: mptscsih: ioc0: attempting task abort! (sc=ffff880225495e00) Aug 7 19:00:17 moog kernel: sd 8:0:6:0: [sdo] CDB: Read(10): 28 00 13 3e 75 d7 00 00 08 00 Aug 7 19:00:17 moog kernel: mptscsih: ioc0: task abort: SUCCESS (sc=ffff880225495e00) Aug 7 19:00:17 moog kernel: mptscsih: ioc0: attempting target reset! (sc=ffff8802255bac00) Aug 7 19:00:17 moog kernel: sd 8:0:6:0: [sdo] CDB: ATA command pass through(16): 85 08 0e 00 d5 00 01 00 09 00 4f 00 c2 00 b0 00 Aug 7 19:00:17 moog kernel: mptscsih: ioc0: target reset: SUCCESS (sc=ffff8802255bac00) Aug 7 19:00:17 moog kernel: mptscsih: ioc0: attempting bus reset! (sc=ffff8802255bac00) Aug 7 19:00:17 moog kernel: sd 8:0:6:0: [sdo] CDB: ATA command pass through(16): 85 08 0e 00 d5 00 01 00 09 00 4f 00 c2 00 b0 00 Aug 7 19:00:18 moog kernel: mptscsih: ioc0: bus reset: SUCCESS (sc=ffff8802255bac00) Aug 7 19:00:28 moog kernel: mptscsih: ioc0: attempting host reset! (sc=ffff8802255bac00) Aug 7 19:00:28 moog kernel: mptbase: ioc0: Initiating recovery Aug 7 19:00:39 moog kernel: mptscsih: ioc0: host reset: SUCCESS (sc=ffff8802255bac00) Aug 7 19:00:39 moog kernel: sd 8:0:6:0: Device offlined - not ready after error recovery Aug 7 19:00:39 moog kernel: sd 8:0:6:0: Device offlined - not ready after error recovery Aug 7 19:00:39 moog kernel: sd 8:0:6:0: [sdo] Unhandled error code Aug 7 19:00:39 moog kernel: sd 8:0:6:0: [sdo] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK Aug 7 19:00:39 moog kernel: end_request: I/O error, dev sdo, sector 322860503 Aug 7 19:00:39 moog kernel: sd 8:0:6:0: [sdo] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK Aug 7 19:00:39 moog kernel: raid5: Disk failure on sdo1, disabling device. Aug 7 19:00:39 moog kernel: raid5: Operation continuing on 6 devices. Aug 7 19:00:39 moog kernel: end_request: I/O error, dev sdm, sector 1953519935 Aug 7 19:00:39 moog kernel: md: super_written gets error=-5, uptodate=0 Aug 7 19:00:39 moog kernel: raid5: Disk failure on sdm1, disabling device. Aug 7 19:00:39 moog kernel: raid5: Operation continuing on 6 devices. Aug 7 19:00:39 moog kernel: RAID5 conf printout: Aug 7 19:00:39 moog kernel: --- rd:7 wd:6 Aug 7 19:00:39 moog kernel: disk 0, o:1, dev:sdj1 Aug 7 19:00:39 moog kernel: disk 1, o:1, dev:sdl1 Aug 7 19:00:39 moog kernel: disk 2, o:1, dev:sdp1 Aug 7 19:00:39 moog kernel: disk 3, o:1, dev:sdn1 Aug 7 19:00:39 moog kernel: disk 4, o:1, dev:sdk1 Aug 7 19:00:39 moog kernel: disk 5, o:1, dev:sdi1 Aug 7 19:00:39 moog kernel: disk 6, o:0, dev:sdo1 Aug 7 19:00:39 moog kernel: RAID5 conf printout: Aug 7 19:00:39 moog kernel: --- rd:7 wd:6 Aug 7 19:00:39 moog kernel: disk 0, o:1, dev:sdj1 Aug 7 19:00:39 moog kernel: disk 1, o:1, dev:sdl1 Aug 7 19:00:39 moog kernel: disk 2, o:1, dev:sdp1 Aug 7 19:00:39 moog kernel: disk 3, o:1, dev:sdn1 Aug 7 19:00:39 moog kernel: disk 4, o:1, dev:sdk1 Aug 7 19:00:39 moog kernel: disk 5, o:1, dev:sdi1 Aug 7 19:00:45 moog kernel: JBD: barrier-based sync failed on dm-1:8 - disabling barriers reloading mptsas kernel module brings all drives back, but the drives are marked as failed in mdadm. re-adding them starts a resync which takes three hours on this system. Pim
(In reply to comment #9) > yes, scratch build lives only a few days before it's deleted > > build: > > http://koji.fedoraproject.org/koji/taskinfo?taskID=1583235 Can you share the .src.rpm? I'd like to try building with different compiler flags as suggested by various sources http://markmail.org/message/zow4jsbfl2zxhimx Pim
ok, srpm is here: http://mhlavink.fedorapeople.org/smartmontools-5.39-0.svn20090721.fc11.src.rpm I'll rebuild this package without '-fpie' if you can confirm it helps
(In reply to comment #14) > ok, srpm is here: > > http://mhlavink.fedorapeople.org/smartmontools-5.39-0.svn20090721.fc11.src.rpm > > I'll rebuild this package without '-fpie' if you can confirm it helps I haven't been able to test this release built without -fpie, but I did build 5.38-11, the current fedora package, without -fpie. It did not make any difference. I don't think building this newer package without -fpie will make a difference.
(In reply to comment #11) > > do you have any similar systems where it works? What are they? > I am administering six systems with LSI 1068E based SAS controllers. 1 Fedora 9 with LSI SAS3800X, works 2 Fedora 10 with LSI SAS3081E-R, work 1 Fedora 10 with LSI SAS3081E-R, problems 1 Fedora 11 with IBM-branded LSI SAS3444E, problems 1 Fedora 11 with Fujitsu card similar to LSI SAS3081E-R, problems All controllers are controlling SATA disks. LSI's host-RAID bios is disabled or not used. All disks are used for mdadm software RAID. All systems are booting off a SATA flash disk controlled by an Intel ICHxxx SATA controller. The most peculiar thing is that two of these systems are exactly the same, yet only one has the problem. Exact same hardware, same age, same SAS controller, same firmware, same Fedora 10 release. Their only difference is their upgrade history. The system without problems was installed with Fedora 8, then upgraded to Fedora 10. The system with problems was installed with Fedora 9, then upgraded to Fedora 11 (where the problem started) then downgraded to Fedora 10 (for other reasons than smartmontools). The smartmontools problem persists after downgrading. The system with the Fujitsu card was the only one that started exposing this problem rightaway in Fedora 10. As this was the first one, I was suspecting the card itself then. This system now runs Fedora 11 and still has the problem.
Upgrading firmware does not help either. I have upgraded the firmware of the problematic systems from 1.26 to 1.28 for the LSI branded cards and from 1.24 to 1.27.82 for the IBM branded card. The Fujitsu card happily accepts LSI firmware. No difference in behaviour was noted after upgrading.
(In reply to comment #15) > (In reply to comment #14) > > ok, srpm is here: > > > > http://mhlavink.fedorapeople.org/smartmontools-5.39-0.svn20090721.fc11.src.rpm > > > > I'll rebuild this package without '-fpie' if you can confirm it helps > > I haven't been able to test this release built without -fpie, but I did build > 5.38-11, the current fedora package, without -fpie. > > It did not make any difference. I don't think building this newer package > without -fpie will make a difference. well, I've meant without '-pie' and '-fpie' as mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=452389#c1 and http://markmail.org/message/zow4jsbfl2zxhimx in comment #3 you've said: > disabling -Wl,-z,relro,-z,now makes a working executable. but in original comment it's said: >the problem is that Fedora builds with the following command: >make CXXFLAGS="$RPM_OPT_FLAGS -fpie" LDFLAGS="-pie -Wl,-z,relro,-z,now" > Rebuilding without the 'pie' flags makes the problem go away.... so it says keeping the flags you said you've removed (-Wl,-z,...) and removing just -fpie from CXXFLAGS and -pie from LDFLAGS so... what options did you use and was it working without the patch you've attached or only with the patch? Did you test combination: removed -fpie, -pie and without the patch? If not, could you test it? I'll try to get some answers from some kernel-storage guy meanwhile.
Sorry for the confusion. I will restate my previous findings: I built smartmontools-5.38-11.fc11.src.rpm using make CXXFLAGS="$RPM_OPT_FLAGS" LDFLAGS="-Wl,-z,relro,-z,now" It did not make a difference. I used smartmontools-5.39-0.svn20090721.fc11.x86_64.rpm as built by yourself. It appeared to take longer before the drives went offline. I have not yet tried custom builds of smartmontools-5.39-0.svn20090721.fc11.src.rpm It's not so easy to test this. If I deactivate my volume groups and stop the md arrays, nothing unusual happens when I use smartd or smartctl. I have to bring everything up for the problem to rise. But then when they rise, things get very scary. Multiple drives from an MD RAID go offline. The ones that stay online often are marked as failed, because they did not respond for a while. Repairing this means re-enabling failed drives and forcing re-assembly of broken RAID arrays, and resyncs that take hours. I did not lose any data yet, but wonder how long my luck will last. Apart from my home file server, all the other servers are production servers at work... Next time I will try to force the problem while using non-RAID related disk activity, like reading all drives simultaneously using dd. Thanks, Pim
(In reply to comment #19) > I built smartmontools-5.38-11.fc11.src.rpm using > make CXXFLAGS="$RPM_OPT_FLAGS" LDFLAGS="-Wl,-z,relro,-z,now" so, to be clear, there is no build option, that can be used. It simply does not work. Even build options mentioned in comment #3 do not help. right? > I have not yet tried custom builds of > smartmontools-5.39-0.svn20090721.fc11.src.rpm I think there is no need to do this for now. could you get me pci ids of that sas controllers? I'll look at our lab if I can find one. thanks
(In reply to comment #20) > so, to be clear, there is no build option, that can be used. It simply does not > work. Simply put: on those systems where the problem appeared there was nothing I could do to make it go away, other than disabling smartd and not using smartctl. Not using different versions of smartmontools. Not using different build options. Not by upgrading the firmware. Not by downgrading Fedora 11 to Fedora 10. > Even build options mentioned in comment #3 do not help. right? I never tried the build options suggested in comment #3. > could you get me pci ids of that sas controllers? Fujitsu card is 1000:0058 Subsystem: 1734:1130 LSI SAS3081E is 1000:0058 Subsystem: 1000:3140 LSI SAS3800X is 1000:0054 Subsystem: 1000:3070 IBM SAS3444E is 1000:0058 Subsystem: 1000:3100
Kashyap, this could be a problem with the controller, does it look familiar to you ? thanks, tomas
what are the values for OS level IO timeout and Firmware level IO missing delay and Device missing delay ? Can I have below values ? 1. cat /sys/class/scsi_host/hostX/device_delay 2. cat /sys/class/scsi_host/hostX/io_delay 2. cat /sys/class/scsi_device/B:T:L:C/device/timeout. Thanks, Kashyap
On every system, device_delay and io_delay are 00 Timeout values differ on the systems described in previous comments: Fujitsu card 30 (fedora 11, failing) LSI SAS3081E 60 (fedora 10, 2 working, 1 failing) LSI SAS3800X 60 (fedora 9, working) IBM SAS3444E 30 (fedora 11, failing) The conclusion could almost be that 60 is good and 30 is bad. Except that there's one system that's failing with 60. Thanks Pim
There might there be a relation with http://bugzilla.kernel.org/show_bug.cgi?id=13594 Pim
In comment #2 I am seeing this log mptbase: ioc0: LogInfo(0x31170000): Originator={PL}, Code={IO Device Missing Delay Retry}, SubCode(0x0000) It indicates device missing delay is set to non-zero. Thanks, Kashyap
Thanks for the answer, Kashyap. Could you explain me your last comment? Does it mean: a) controller is misconfigured? b) smartmontools are using wrong ioctl/something c) bug in the driver d) something else? Thanks, Michal
See my comments inline. a) controller is misconfigured? -> Any configuration is valid. Why I have asked for this is just to get some pointers for me. When you found device missing delay is set to 0, I would like you to cross check again, since Logs are indicating me that there are some device missing delay values set. b) smartmontools are using wrong ioctl/something c) bug in the driver d) something else? -> Need investigation for #b, #c, $d Please ON driver log level to 0x8188 (It can be set by changing value of /sys/class/scsih_host/hostX/logging_level) This can help me to see things from MPT fusion side. Thanks, Kashyap
For what it's worth, I'm seeing this issue on Ubuntu Jaunty, Kernel 2.6.28 although my drives and mdadm come up after boot 01:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1064ET PCI-Express Fusion-MPT SAS (rev 08) Subsystem: LSI Logic / Symbios Logic Device 3090 Flags: bus master, fast devsel, latency 0, IRQ 2294 I/O ports at b000 [size=256] Memory at d0610000 (64-bit, non-prefetchable) [size=16K] Memory at d0600000 (64-bit, non-prefetchable) [size=64K] Expansion ROM at d0400000 [disabled] [size=2M] Capabilities: [50] Power Management version 2 Capabilities: [68] Express Endpoint, MSI 00 Capabilities: [98] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable+ Capabilities: [b0] MSI-X: Enable- Mask- TabSize=1 Capabilities: [100] Advanced Error Reporting <?> Kernel driver in use: mptsas Kernel modules: mptsas with 4 disks ST3500630AS [ 13.259129] mptsas 0000:01:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 [ 13.259225] mptbase: ioc0: Initiating bringup [ 14.980070] mptsas 0000:01:00.0: irq 2294 for MSI/MSI-X [ 14.980083] mptbase: ioc0: PCI-MSI enabled [ 14.980105] mptsas 0000:01:00.0: setting latency timer to 64 [ 47.040043] mptscsih: ioc0: attempting task abort! (sc=ffff880238de5200) [ 47.514167] mptbase: ioc0: LogInfo(0x31140000): Originator={PL}, Code={IO Executed}, SubCode(0x0000) [ 47.563886] mptscsih: ioc0: task abort: SUCCESS (sc=ffff880238de5200) [ 56.040032] mptscsih: ioc0: attempting task abort! (sc=ffff880238de5500) [ 56.513902] mptbase: ioc0: LogInfo(0x31140000): Originator={PL}, Code={IO Executed}, SubCode(0x0000) [ 56.560024] mptscsih: ioc0: task abort: SUCCESS (sc=ffff880238de5500) [ 65.040048] mptscsih: ioc0: attempting task abort! (sc=ffff880238de5000) [ 65.509567] mptbase: ioc0: LogInfo(0x31140000): Originator={PL}, Code={IO Executed}, SubCode(0x0000) [ 65.560024] mptscsih: ioc0: task abort: SUCCESS (sc=ffff880238de5000) [ 74.011304] mptscsih: ioc0: attempting task abort! (sc=ffff8802391cea00) [ 74.505256] mptbase: ioc0: LogInfo(0x31140000): Originator={PL}, Code={IO Executed}, SubCode(0x0000) [ 74.530021] mptscsih: ioc0: task abort: SUCCESS (sc=ffff8802391cea00)
(In reply to comment #28) > When you found device missing delay is set to 0, I would like > you to cross check again, since Logs are indicating me that there are some > device missing delay values set. I think the confusion here is that the non-zero "device missing delay" values were seen in Philippe Troin's report. Right? I think my reports do not show this, and this would match the /sys/class/scsi_host/hostX/device_delay values that I reported. Thanks, Pim
I got your point. It means with and without Device missing delay problem is still there. Right ? We are targeting why Device is getting off lined by OS right ? Is this problem still Alive ? Since I am not seeing Device Offline prints in Comment #30. Can you try doing MSI disable and see what is behavior ?
I haven't been able to answer questions posed in previous comments, as I deemed it too dangerous to try more on production systems. I now have assembled a test machine, a desktop PC with an IBM SAS HBA (1000:0058 Subsystem: 1000:3100) in a secondary PEG slot. This works. I configured 4 (different) laptop drives in an mdadm RAID-5. But I now have trouble reproducing the problem. I can reproduce I/O errors, card resets, mdadm failed disks, but I have not been able to let drives actually go offline. These laptop drives are much slower than the drives from the production systems. Maybe I need faster drives, or maybe I need to simulate more disk activity. Or maybe the current symptopms (which are bad enough on their own) suffice for continuing this bug report. Pim
There is yet another report of this problem on the linux-scsi mailinglist http://kerneltrap.org/mailarchive/linux-scsi/2009/9/14/6409773 Pim
On my new test system, I see different values as described in comment #23 1. cat /sys/class/scsi_host/hostX/device_delay 144 2. cat /sys/class/scsi_host/hostX/io_delay 08 3. cat /sys/class/scsi_device/B:T:L:C/device/timeout. 30 30 30 30 This card has older firmware: FwRev=01175400h Should I upgrade firmware first?
I have upgraded my test system with yum update which means it now runs kernel 2.6.30.5-43.fc11.x86_64 I have upgraded the firmware to 011b5200h as on production system. I have found the delay values 144 and 8 in the BIOS setup of the adapter and have changed them to zero Before all these changes, the symptoms were the same as on the production systems, except that the drives would not disappear from /proc/scsi/scsi, although they would be inaccessible. Now after these changes, drives no longer fail. I do see errors in the syslog, but they are different: Sep 21 21:38:50 botero kernel: mptscsih: ioc0: attempting task abort! (sc=ffff88012e5feb00) Sep 21 21:38:50 botero kernel: sd 6:0:2:0: [sde] CDB: ATA command pass through(16): 85 08 0e 00 00 00 01 00 00 00 00 00 00 00 ec 00 Sep 21 21:38:51 botero kernel: mptbase: ioc0: LogInfo(0x31140000): Originator={PL}, Code={IO Executed}, SubCode(0x0000) Sep 21 21:38:51 botero kernel: mptscsih: ioc0: task abort: SUCCESS (sc=ffff88012e5feb00) Sep 21 21:38:52 botero kernel: mptscsih: ioc0: attempting task abort! (sc=ffff88012e439d00) Sep 21 21:38:52 botero kernel: sd 6:0:2:0: [sde] CDB: Read(10): 28 00 00 13 83 3f 00 00 08 00 Sep 21 21:38:52 botero kernel: mptscsih: ioc0: task abort: SUCCESS (sc=ffff88012e439d00) Sep 21 21:38:52 botero kernel: mptscsih: ioc0: attempting task abort! (sc=ffff88012e439600) Sep 21 21:38:52 botero kernel: sd 6:0:2:0: [sde] CDB: Read(10): 28 00 00 13 83 47 00 00 08 00 Sep 21 21:38:52 botero kernel: mptscsih: ioc0: task abort: SUCCESS (sc=ffff88012e439600) Sep 21 21:38:52 botero kernel: mptscsih: ioc0: attempting task abort! (sc=ffff8800bbd2ea00) Sep 21 21:38:52 botero kernel: sd 6:0:2:0: [sde] CDB: Read(10): 28 00 00 32 0c bf 00 00 08 00 Sep 21 21:38:52 botero kernel: mptscsih: ioc0: task abort: SUCCESS (sc=ffff8800bbd2ea00) Sep 21 21:38:52 botero kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO Sep 21 21:38:52 botero kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO Sep 21 21:38:52 botero kernel: sd 6:0:2:0: [sde] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:38:52 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:38:52 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 01 00 00 00 Sep 21 21:38:52 botero kernel: 00 4f 00 c2 00 50 Sep 21 21:38:52 botero kernel: sd 6:0:2:0: [sde] Add. Sense: ATA pass through information available Sep 21 21:38:52 botero kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO Sep 21 21:38:52 botero kernel: sd 6:0:2:0: [sde] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:38:52 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:38:52 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 01 00 00 00 Sep 21 21:38:52 botero kernel: 00 4f 00 c2 00 50 Sep 21 21:38:52 botero kernel: sd 6:0:2:0: [sde] Add. Sense: ATA pass through information available Sep 21 21:38:52 botero ata_id[3196]: HDIO_GET_IDENTITY failed for '/dev/block/8:64' Sep 21 21:38:52 botero kernel: sd 6:0:2:0: [sde] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:38:52 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:38:52 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 Sep 21 21:38:52 botero kernel: 00 00 00 00 00 00 Sep 21 21:38:52 botero kernel: sd 6:0:2:0: [sde] Add. Sense: ATA pass through information available Sep 21 21:38:52 botero kernel: sd 6:0:2:0: [sde] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:38:52 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:38:52 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 Sep 21 21:38:52 botero kernel: 00 00 00 00 00 00 Sep 21 21:38:52 botero kernel: sd 6:0:2:0: [sde] Add. Sense: ATA pass through information available Sep 21 21:38:52 botero kernel: sd 6:0:2:0: [sde] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:38:52 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:38:52 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 Sep 21 21:38:52 botero kernel: 00 00 00 00 00 00 Sep 21 21:38:52 botero kernel: sd 6:0:2:0: [sde] Add. Sense: ATA pass through information available Sep 21 21:39:49 botero ntpd[2063]: synchronized to 82.204.76.2, stratum 2 Sep 21 21:39:49 botero ntpd[2063]: kernel time sync status change 2001 Sep 21 21:40:08 botero kernel: mptscsih: ioc0: attempting task abort! (sc=ffff88012d9d1600) Sep 21 21:40:08 botero kernel: sd 6:0:1:0: [sdd] CDB: ATA command pass through(16): 85 08 0e 00 00 00 01 00 00 00 00 00 00 00 ec 00 Sep 21 21:40:09 botero kernel: mptbase: ioc0: LogInfo(0x31140000): Originator={PL}, Code={IO Executed}, SubCode(0x0000) Sep 21 21:40:09 botero kernel: mptscsih: ioc0: task abort: SUCCESS (sc=ffff88012d9d1600) Sep 21 21:40:10 botero kernel: mptscsih: ioc0: attempting task abort! (sc=ffff88012e439700) Sep 21 21:40:10 botero kernel: sd 6:0:1:0: [sdd] CDB: Write(10): 2a 00 00 21 71 3f 00 00 08 00 Sep 21 21:40:10 botero kernel: mptscsih: ioc0: task abort: SUCCESS (sc=ffff88012e439700) Sep 21 21:40:10 botero kernel: mptscsih: ioc0: attempting task abort! (sc=ffff88012e439e00) Sep 21 21:40:10 botero kernel: sd 6:0:1:0: [sdd] CDB: Write(10): 2a 00 00 21 86 87 00 00 08 00 Sep 21 21:40:10 botero kernel: mptscsih: ioc0: task abort: SUCCESS (sc=ffff88012e439e00) Sep 21 21:40:10 botero kernel: mptscsih: ioc0: attempting task abort! (sc=ffff88012ddf9400) Sep 21 21:40:10 botero kernel: sd 6:0:1:0: [sdd] CDB: Write(10): 2a 00 00 21 90 ef 00 00 08 00 Sep 21 21:40:10 botero kernel: mptscsih: ioc0: task abort: SUCCESS (sc=ffff88012ddf9400) Sep 21 21:40:10 botero kernel: mptscsih: ioc0: attempting task abort! (sc=ffff88012dd75d00) Sep 21 21:40:10 botero kernel: sd 6:0:1:0: [sdd] CDB: Write(10): 2a 00 00 21 bb 7f 00 00 08 00 Sep 21 21:40:10 botero kernel: mptscsih: ioc0: task abort: SUCCESS (sc=ffff88012dd75d00) Sep 21 21:40:10 botero kernel: mptscsih: ioc0: attempting task abort! (sc=ffff88012df84800) Sep 21 21:40:10 botero kernel: sd 6:0:1:0: [sdd] CDB: Read(10): 28 00 00 33 bd 77 00 00 08 00 Sep 21 21:40:10 botero kernel: mptscsih: ioc0: task abort: SUCCESS (sc=ffff88012df84800) Sep 21 21:40:10 botero kernel: mptscsih: ioc0: attempting task abort! (sc=ffff880117338900) Sep 21 21:40:10 botero kernel: sd 6:0:1:0: [sdd] CDB: Read(10): 28 00 00 33 bd 7f 00 00 08 00 Sep 21 21:40:10 botero kernel: mptscsih: ioc0: task abort: SUCCESS (sc=ffff880117338900) Sep 21 21:40:10 botero kernel: mptscsih: ioc0: attempting task abort! (sc=ffff880117338000) Sep 21 21:40:10 botero kernel: sd 6:0:1:0: [sdd] CDB: Read(10): 28 00 00 33 bd 87 00 00 08 00 Sep 21 21:40:10 botero kernel: mptscsih: ioc0: task abort: SUCCESS (sc=ffff880117338000) Sep 21 21:40:10 botero kernel: mptscsih: ioc0: attempting task abort! (sc=ffff88012b499000) Sep 21 21:40:10 botero kernel: sd 6:0:1:0: [sdd] CDB: Read(10): 28 00 00 33 bd 8f 00 00 08 00 Sep 21 21:40:10 botero kernel: mptscsih: ioc0: task abort: SUCCESS (sc=ffff88012b499000) Sep 21 21:40:10 botero kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO Sep 21 21:40:10 botero kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO Sep 21 21:40:10 botero kernel: sd 6:0:1:0: [sdd] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:40:10 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:40:10 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 01 00 00 00 Sep 21 21:40:10 botero kernel: 00 4f 00 c2 00 50 Sep 21 21:40:10 botero kernel: sd 6:0:1:0: [sdd] Add. Sense: ATA pass through information available Sep 21 21:40:10 botero kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO Sep 21 21:40:10 botero kernel: sd 6:0:1:0: [sdd] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:40:10 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:40:10 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 01 00 00 00 Sep 21 21:40:10 botero kernel: 00 4f 00 c2 00 50 Sep 21 21:40:10 botero kernel: sd 6:0:1:0: [sdd] Add. Sense: ATA pass through information available Sep 21 21:40:10 botero ata_id[3284]: HDIO_GET_IDENTITY failed for '/dev/block/8:48' Sep 21 21:40:10 botero kernel: sd 6:0:1:0: [sdd] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:40:10 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:40:10 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 Sep 21 21:40:10 botero kernel: 00 00 00 00 00 00 Sep 21 21:40:10 botero kernel: sd 6:0:1:0: [sdd] Add. Sense: ATA pass through information available Sep 21 21:40:10 botero kernel: sd 6:0:1:0: [sdd] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:40:10 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:40:10 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 Sep 21 21:40:10 botero kernel: 00 00 00 00 00 00 Sep 21 21:40:10 botero kernel: sd 6:0:1:0: [sdd] Add. Sense: ATA pass through information available Sep 21 21:41:07 botero smartd[2206]: smartd received signal 15: Terminated Sep 21 21:41:07 botero smartd[2206]: smartd is exiting (exit status 0) Sep 21 21:41:07 botero smartd[3365]: smartd version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen Sep 21 21:41:07 botero smartd[3365]: Home page is http://smartmontools.sourceforge.net/#012 Sep 21 21:41:07 botero smartd[3365]: Opened configuration file /etc/smartd.conf Sep 21 21:41:07 botero smartd[3365]: Configuration file /etc/smartd.conf parsed. Sep 21 21:41:07 botero smartd[3365]: Device: /dev/disk/by-id/ata-ST3500320AS_9QM48WNB, opened Sep 21 21:41:07 botero smartd[3365]: Device: /dev/disk/by-id/ata-ST3500320AS_9QM48WNB, found in smartd database. Sep 21 21:41:07 botero smartd[3365]: Device: /dev/disk/by-id/ata-ST3500320AS_9QM48WNB, is SMART capable. Adding to "monitor" list. Sep 21 21:41:07 botero smartd[3365]: Device: /dev/disk/by-id/ata-ST3500320AS_9QM48X07, opened Sep 21 21:41:07 botero smartd[3365]: Device: /dev/disk/by-id/ata-ST3500320AS_9QM48X07, found in smartd database. Sep 21 21:41:07 botero smartd[3365]: Device: /dev/disk/by-id/ata-ST3500320AS_9QM48X07, is SMART capable. Adding to "monitor" list. Sep 21 21:41:07 botero smartd[3365]: Device: /dev/disk/by-id/scsi-SATA_FUJITSU_MHV2100_NW04T6525F28, opened Sep 21 21:41:07 botero smartd[3365]: Device: /dev/disk/by-id/scsi-SATA_FUJITSU_MHV2100_NW04T6525F28, found in smartd database. Sep 21 21:41:07 botero kernel: sd 6:0:0:0: [sdc] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:41:07 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:41:07 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 Sep 21 21:41:07 botero kernel: 00 4f 00 c2 00 50 Sep 21 21:41:07 botero kernel: sd 6:0:0:0: [sdc] Add. Sense: ATA pass through information available Sep 21 21:41:08 botero smartd[3365]: Device: /dev/disk/by-id/scsi-SATA_FUJITSU_MHV2100_NW04T6525F28, is SMART capable. Adding to "monitor" list. Sep 21 21:41:08 botero smartd[3365]: Device: /dev/disk/by-id/scsi-SATA_HTS541010G9SA00_MP2ZX0X0K9WPBS, opened Sep 21 21:41:08 botero ata_id[3369]: HDIO_GET_IDENTITY failed for '/dev/block/8:32' Sep 21 21:41:08 botero kernel: sd 6:0:0:0: [sdc] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:41:08 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:41:08 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 Sep 21 21:41:08 botero kernel: 00 00 00 00 00 00 Sep 21 21:41:08 botero kernel: sd 6:0:0:0: [sdc] Add. Sense: ATA pass through information available Sep 21 21:41:08 botero kernel: sd 6:0:0:0: [sdc] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:41:08 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:41:08 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 Sep 21 21:41:08 botero kernel: 00 00 00 00 00 00 Sep 21 21:41:08 botero kernel: sd 6:0:0:0: [sdc] Add. Sense: ATA pass through information available Sep 21 21:41:08 botero smartd[3365]: Device: /dev/disk/by-id/scsi-SATA_HTS541010G9SA00_MP2ZX0X0K9WPBS, found in smartd database. Sep 21 21:41:09 botero kernel: sd 6:0:2:0: [sde] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:41:09 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:41:09 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 01 00 00 00 Sep 21 21:41:09 botero kernel: 00 4f 00 c2 00 50 Sep 21 21:41:09 botero kernel: sd 6:0:2:0: [sde] Add. Sense: ATA pass through information available Sep 21 21:41:09 botero smartd[3365]: Device: /dev/disk/by-id/scsi-SATA_HTS541010G9SA00_MP2ZX0X0K9WPBS, is SMART capable. Adding to "monitor" list. Sep 21 21:41:09 botero smartd[3365]: Device: /dev/disk/by-id/scsi-SATA_HTS541080G9SA00_MPBDPAXNGR2PVM, opened Sep 21 21:41:09 botero smartd[3365]: Device: /dev/disk/by-id/scsi-SATA_HTS541080G9SA00_MPBDPAXNGR2PVM, found in smartd database. Sep 21 21:41:09 botero kernel: sd 6:0:3:0: [sdf] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:41:09 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:41:09 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 01 00 00 00 Sep 21 21:41:09 botero kernel: 00 4f 00 c2 00 50 Sep 21 21:41:09 botero kernel: sd 6:0:3:0: [sdf] Add. Sense: ATA pass through information available Sep 21 21:41:09 botero ata_id[3382]: HDIO_GET_IDENTITY failed for '/dev/block/8:64' Sep 21 21:41:09 botero smartd[3365]: Device: /dev/disk/by-id/scsi-SATA_HTS541080G9SA00_MPBDPAXNGR2PVM, is SMART capable. Adding to "monitor" list. Sep 21 21:41:09 botero smartd[3365]: Device: /dev/disk/by-id/scsi-SATA_HTS721010G9SA00_MPCZN2Y0GPYPHH, opened Sep 21 21:41:09 botero smartd[3365]: Device: /dev/disk/by-id/scsi-SATA_HTS721010G9SA00_MPCZN2Y0GPYPHH, found in smartd database. Sep 21 21:41:09 botero ata_id[3391]: HDIO_GET_IDENTITY failed for '/dev/block/8:80' Sep 21 21:41:09 botero kernel: sd 6:0:2:0: [sde] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:41:09 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:41:09 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 Sep 21 21:41:09 botero kernel: 00 00 00 00 00 00 Sep 21 21:41:09 botero kernel: sd 6:0:2:0: [sde] Add. Sense: ATA pass through information available Sep 21 21:41:09 botero kernel: sd 6:0:1:0: [sdd] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:41:09 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:41:09 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 01 00 00 00 Sep 21 21:41:09 botero kernel: 00 4f 00 c2 00 50 Sep 21 21:41:09 botero kernel: sd 6:0:1:0: [sdd] Add. Sense: ATA pass through information available Sep 21 21:41:09 botero kernel: sd 6:0:3:0: [sdf] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:41:09 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:41:09 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 Sep 21 21:41:09 botero kernel: 00 00 00 00 00 00 Sep 21 21:41:09 botero kernel: sd 6:0:3:0: [sdf] Add. Sense: ATA pass through information available Sep 21 21:41:09 botero kernel: sd 6:0:2:0: [sde] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:41:09 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:41:09 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 Sep 21 21:41:09 botero kernel: 00 00 00 00 00 00 Sep 21 21:41:09 botero kernel: sd 6:0:2:0: [sde] Add. Sense: ATA pass through information available Sep 21 21:41:09 botero kernel: sd 6:0:3:0: [sdf] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:41:09 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:41:09 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 Sep 21 21:41:09 botero kernel: 00 00 00 00 00 00 Sep 21 21:41:09 botero kernel: sd 6:0:3:0: [sdf] Add. Sense: ATA pass through information available Sep 21 21:41:09 botero smartd[3365]: Device: /dev/disk/by-id/scsi-SATA_HTS721010G9SA00_MPCZN2Y0GPYPHH, is SMART capable. Adding to "monitor" list. Sep 21 21:41:09 botero smartd[3365]: Monitoring 6 ATA and 0 SCSI devices Sep 21 21:41:09 botero ata_id[3404]: HDIO_GET_IDENTITY failed for '/dev/block/8:48' Sep 21 21:41:10 botero kernel: sd 6:0:1:0: [sdd] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:41:10 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:41:10 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 Sep 21 21:41:10 botero kernel: 00 00 00 00 00 00 Sep 21 21:41:10 botero kernel: sd 6:0:1:0: [sdd] Add. Sense: ATA pass through information available Sep 21 21:41:10 botero kernel: sd 6:0:1:0: [sdd] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:41:10 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:41:10 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 Sep 21 21:41:10 botero kernel: 00 00 00 00 00 00 Sep 21 21:41:10 botero kernel: sd 6:0:1:0: [sdd] Add. Sense: ATA pass through information available Sep 21 21:41:10 botero kernel: sd 6:0:0:0: [sdc] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:41:10 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:41:10 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 Sep 21 21:41:10 botero kernel: 00 4f 00 c2 00 50 Sep 21 21:41:10 botero kernel: sd 6:0:0:0: [sdc] Add. Sense: ATA pass through information available Sep 21 21:41:10 botero kernel: sd 6:0:2:0: [sde] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:41:10 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:41:10 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 01 00 00 00 Sep 21 21:41:10 botero kernel: 00 4f 00 c2 00 50 Sep 21 21:41:10 botero kernel: sd 6:0:2:0: [sde] Add. Sense: ATA pass through information available Sep 21 21:41:10 botero kernel: sd 6:0:3:0: [sdf] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:41:10 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:41:10 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 01 00 00 00 Sep 21 21:41:10 botero kernel: 00 4f 00 c2 00 50 Sep 21 21:41:10 botero kernel: sd 6:0:3:0: [sdf] Add. Sense: ATA pass through information available Sep 21 21:41:10 botero kernel: sd 6:0:1:0: [sdd] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:41:10 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:41:10 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 01 00 00 00 Sep 21 21:41:10 botero kernel: 00 4f 00 c2 00 50 Sep 21 21:41:10 botero kernel: sd 6:0:1:0: [sdd] Add. Sense: ATA pass through information available Sep 21 21:41:10 botero smartd[3414]: smartd has fork()ed into background mode. New PID=3414. Sep 21 21:41:10 botero ata_id[3418]: HDIO_GET_IDENTITY failed for '/dev/block/8:48' Sep 21 21:41:10 botero kernel: sd 6:0:1:0: [sdd] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:41:10 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:41:10 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 Sep 21 21:41:10 botero kernel: 00 00 00 00 00 00 Sep 21 21:41:10 botero kernel: sd 6:0:1:0: [sdd] Add. Sense: ATA pass through information available Sep 21 21:41:10 botero kernel: sd 6:0:1:0: [sdd] Sense Key : Recovered Error [current] [descriptor] Sep 21 21:41:10 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 21:41:10 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 Sep 21 21:41:10 botero kernel: 00 00 00 00 00 00 Sep 21 21:41:10 botero kernel: sd 6:0:1:0: [sdd] Add. Sense: ATA pass through information available In spite of all these messages, this looks good. Except that smartctl output is incomplete. I will try this one of the production systems. Pim
smarctl -H -d scsi /dev/sdX returns SMART Health Status: OK immediately, with this in the syslog Sep 21 22:02:55 botero kernel: sd 6:0:2:0: [sde] Sense Key : Recovered Error [current] [descriptor] Sep 21 22:02:55 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 22:02:55 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 Sep 21 22:02:55 botero kernel: 00 00 00 00 00 00 Sep 21 22:02:55 botero kernel: sd 6:0:2:0: [sde] Add. Sense: ATA pass through information available Sep 21 22:02:55 botero kernel: sd 6:0:2:0: [sde] Sense Key : Recovered Error [current] [descriptor] Sep 21 22:02:55 botero kernel: Descriptor sense data with sense descriptors (in hex): Sep 21 22:02:55 botero kernel: 72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 Sep 21 22:02:55 botero kernel: 00 00 00 00 00 00 Sep 21 22:02:55 botero kernel: sd 6:0:2:0: [sde] Add. Sense: ATA pass through information available So, faking "-d scsi" in /etc/smartd.conf might give a working SMART monitoring without bringing the system down. Pim Pim
(In reply to comment #31) > Can you try doing MSI disable and see what is behavior ? Where and how do I do this? Thanks, Pim
I have upgraded one of the production systems using yum update, which upgrades the kernel to 2.6.30.5-43.fc11.x86_64 Everything seems to work now, both smartd and smartctl, using "-d sat". I do get syslog messages like in comment #35. It assume these messages are related to http://bugzilla.kernel.org/show_bug.cgi?id=13594 But that should probably be filed as a new bug. Thanks, Pim
(In reply to comment #37) > (In reply to comment #31) > > > Can you try doing MSI disable and see what is behavior ? > > Where and how do I do this? > > Thanks, > Pim There used to be mpt_msi_enable option, now replaced by (mpt_msi_enable_sas, mpt_msi_enable_spi, mpt_msi_enable_fc) - they do the same for the particular buses. Any module options specified in /etc/modprobe.conf are passed to the modules as they are loaded by the initial ramdisk. So change the options in modprobe.conf and recreate the initrd. I'm not sure, but I think that the default value is 0 (msi disabled) so this probably will not help much. You can try the option mpt_msi_enable_sas=0 to be sure. tomash
Do you really want me to try to disable MSI on an old kernel, now that I reported that the problem is fixed in kernel 2.6.30 ? I would suggest to close this bug. Pim
(In reply to comment #38) > I do get syslog messages like in comment #35. > It assume these messages are related to > http://bugzilla.kernel.org/show_bug.cgi?id=13594 > > But that should probably be filed as a new bug. I have filed this new problem as bug 533035 https://bugzilla.redhat.com/show_bug.cgi?id=533035 Pim
(In reply to comment #40) ... > the problem is fixed in kernel 2.6.30 ... > I would suggest to close this bug. agreed