|Summary:||linux readahead bug is back (causing mediacheck to fail)|
|Product:||[Fedora] Fedora||Reporter:||Andre Robatino <robatino>|
|Component:||kernel||Assignee:||Alan Cox <alan>|
|Status:||CLOSED DUPLICATE||QA Contact:||Fedora Extras Quality Assurance <extras-qa>|
|Version:||10||CC:||cebbert, davej, kmcmartin, mishu, paul|
|Fixed In Version:||Doc Type:||Bug Fix|
|Doc Text:||Story Points:||---|
|Last Closed:||2008-11-28 11:09:52 UTC||Type:||---|
|oVirt Team:||---||RHEL 7.3 requirements from Atomic Host:|
|Cloudforms Team:||---||Target Upstream Version:|
Description Andre Robatino 2007-11-23 18:04:12 UTC
Description of problem: My father bought a F8 DVD, and did the mediacheck, which failed. Then he used the rawread script at http://www.troubleshooters.com/linux/coasterless.htm to read off the ISO image to a file (this uses the isoinfo command to determine how big the ISO is, then runs a dd command to attempt reading off exactly that much). What it read off was correct, except for being truncated by about 40 KB. The end of the rawread output was this (it repeated many more times): 3424731136 bytes (3.4 GB) copied, 345.048 s, 9.9 MB/s dd: reading `/dev/dvd': Input/output error 1672232+0 records in 1672232+0 records out 3424731136 bytes (3.4 GB) copied, 350.265 s, 9.8 MB/s 1672232+0 records in 1672232+0 records out 3424731136 bytes (3.4 GB) copied, 350.266 s, 9.8 MB/s 60294a5a6efea17c1b13b50a8d83da6b2cc44c2f - By using wget -c, he was able to resume downloading the last 40 KB of the file and get a correct sha1sum for the result. He is currently running a fully updated 32-bit F7, so the bug is affecting that kernel. Then I started testing my stack of old, un-zero-padded, Fedora CDs, using the rawread script, but on my fully updated 32-bit F8. Although every disc I tested checked good about 6 months ago, using checkisomd5 from anaconda-runtime, just after F7 came out, now the first 3 of 3 discs all test bad (I stopped testing there). Running rawread on the first FC4 CD, which is unpadded, I was able to read off only 665305088 out of 665434112 bytes, but the part I could read off was correct (again, using wget -c to finish the file). Unlike my father, my rawread never stopped, but just kept repeating the same attempt to read the same part of the file: 665305088 bytes (665 MB) copied, 364.556 s, 1.8 MB/s dd: reading `/dev/cdrom': Input/output error 324856+0 records in 324856+0 records out 665305088 bytes (665 MB) copied, 364.752 s, 1.8 MB/s 324856+0 records in 324856+0 records out 665305088 bytes (665 MB) copied, 364.753 s, 1.8 MB/s etc., etc., so I finally Ctrl-C'd it. In dmesg, there were lots of the following messages: end_request: I/O error, dev sr0, sector 1299680 end_request: I/O error, dev sr0, sector 1299680 printk: 26 messages suppressed. Buffer I/O error on device sr0, logical block 324920 end_request: I/O error, dev sr0, sector 1299680 end_request: I/O error, dev sr0, sector 1299680 end_request: I/O error, dev sr0, sector 1299680 end_request: I/O error, dev sr0, sector 1299680 end_request: I/O error, dev sr0, sector 1299680 end_request: I/O error, dev sr0, sector 1299680 end_request: I/O error, dev sr0, sector 1299680 end_request: I/O error, dev sr0, sector 1299680 end_request: I/O error, dev sr0, sector 1299680 end_request: I/O error, dev sr0, sector 1299680 end_request: I/O error, dev sr0, sector 1299680 end_request: I/O error, dev sr0, sector 1299680 end_request: I/O error, dev sr0, sector 1299680 end_request: I/O error, dev sr0, sector 1299680 end_request: I/O error, dev sr0, sector 1299680 end_request: I/O error, dev sr0, sector 1299680 Version-Release number of selected component (if applicable): kernel-188.8.131.52-49.fc8 How reproducible: always
Comment 1 Andre Robatino 2007-11-23 18:15:29 UTC
Forgot to include the important info that both I and my father are using identical Sony DW-G120A DVD RW drives (firmware version MYR5, no firmware update available) purchased less than a year ago.
Comment 2 Andre Robatino 2007-11-23 20:13:04 UTC
Additionally, we were using these same drives 6 months ago, when F7 came out. So in my case, I was checking those old unpadded discs with the same drive, so the fact that they all passed back then, with this drive, but not now, indicates that there really is an OS regression, not just a failure to take this particular drive into account.
Comment 3 Andre Robatino 2007-11-24 02:35:15 UTC
I took my unpadded but otherwise good FC4 disc 1, and did a mediacheck on it after booting from each of several recent install DVDs. Fedora 7 DVD: passes Fedura Unity 20070912 7 DVD: passes Fedora 7.92 (F8t3) DVD: fails Fedora 8 DVD: fails So it appears the problem came back sometime in September. I don't happen to have a F8 test 1 or 2 DVD, but could download and test it if really necessary.
Comment 4 Andre Robatino 2007-11-24 08:16:11 UTC
One more data point: Fedora Unity 20071030 7 DVD: fails FU7 20070912 shipped with kernel-184.108.40.206-76.fc7, and FU7 20071030 shipped with kernel-220.127.116.11-10.fc7. Looking through fedora-package-announce for September and October, the only F7 kernel updates between these two were Sep. 25: kernel-18.104.22.168-85.fc7 Sep. 28: kernel-22.214.171.124-91.fc7 so the first F7 kernel to bring back the readahead bug was one of kernel-126.96.36.199-85.fc7, kernel-188.8.131.52-91.fc7, or kernel-184.108.40.206-10.fc7.
Comment 5 Andre Robatino 2007-11-24 20:42:49 UTC
I downloaded and installed the kernels for F7.90 and F7.91 (F8 test 1 and 2, resp.). The former has almost been wiped from the face of the earth, but I was able to find a copy at ftp.cse.yzu.edu.tw. That shouldn't happen - all the old versions should be available. Anyway, after booting from the F7.90 kernel-2.6.23-0.164.rc5.fc8, and then checking my unpadded disc with checkisomd5, the mediacheck fails. I couldn't check F7.91 since the kernel panicked while booting. But it seems that the readahead bug had already returned with F8t1, which was released on August 7.
Comment 6 Andre Robatino 2007-11-24 20:44:17 UTC
Sorry, I should have said "latter", not "former". The download.fedora.redhat.com server, and almost all of the mirrors, only have 7.90 and 7.92.
Comment 7 Andre Robatino 2007-11-24 23:01:09 UTC
I had my father, who is running a fully updated F7, install the anaconda-runtime package, and then use checkisomd5 to check his unpadded purchased F8 DVD, which he could not read near the end due to the readahead bug affecting the kernel he's currently running. Oddly, the mediacheck passed. I have no explanation for this.
Comment 8 Andre Robatino 2007-12-04 00:51:48 UTC
Still broken in kernel-220.127.116.11-63.fc8.
Comment 9 Andre Robatino 2007-12-20 18:55:13 UTC
Still broken in kernel-18.104.22.168-85.fc8. This time, though, when trying to use rawread to read my unpadded FC4 disc 1, it was able to read 665419776 out of 665434112 bytes, whereas the two previous F8 kernels both failed at 665305088 bytes. Has the relevant kernel code changed?
Comment 10 Andre Robatino 2008-01-24 23:02:03 UTC
Still broken in kernel-22.214.171.124-107.fc8. The number of bytes it can read from the unpadded FC4 disc 1 has gone back to 665305088.
Comment 11 Andre Robatino 2008-02-02 01:14:22 UTC
Could the priority on this be raised? There appears to be no interest currently in fixing this. Considering that it WAS fixed when F7 came out, it shouldn't be too hard for someone familiar with the relevant code to discover what went wrong, assuming it was around 2.6.23. And mediacheck failures won't be taken seriously until well after this is fixed, since the bug has existed for almost the entire time mediacheck has, and people have been trained to ignore failures and install anyway.
Comment 12 Alan Cox 2008-02-04 16:37:20 UTC
I have no other reports, no way to duplicate it and no plausible code changes to explain it so at the moment the priority for this bug is very low.
Comment 13 Andre Robatino 2008-02-05 02:08:29 UTC
Following the advice in the following post http://email@example.com/msg11444.html I ran the command [root@localhost ~]# blockdev --setra 0 /dev/dvd to change the default readahead from 256 to 0, and then ran the rawread command again on the same disc. This time, it read 665432064 bytes instead of 665305088 (the read is also much slower), but still failed near the end. Unfortunately, most of the discussion on that thread is over my head, so I don't know if this is helpful or not. I know the bug is reproducible with a Sony DRU-120C DVD drive and an unpadded Fedora CD burned in TAO mode, so I would be willing to buy one of these drives and have it shipped to someone qualified to diagnose the bug, if a listing turns up on Ebay (I bought mine in 12/2006 and I don't think they are sold anymore).
Comment 14 Andre Robatino 2008-02-06 21:38:30 UTC
I downloaded Fedora-9-Alpha-i386-DVD.iso and burned this ISO, unpadded, to both a DVD+R and a DVD+RW using the command growisofs -dvd-compat -speed=1 -Z /dev/dvd=Fedora-9-Alpha-i386-DVD.iso and then checked each using the command rawread /dev/dvd | sha1sum while running F8. To my surprise, both discs checked properly. Then I downloaded Fedora-9-Alpha-i386-rescuecd.iso and burned this ISO, unpadded, to a CD-RW using the command wodim -v dev=/dev/cdrom blank=fast driveropts=burnfree Fedora-9-Alpha-i386-rescuecd.iso and to a CD-R using the command wodim -v dev=/dev/cdrom driveropts=burnfree Fedora-9-Alpha-i386-rescuecd.iso and checked each using the command rawread /dev/cdrom | sha1sum In both cases, the read failed shortly before the end. With the CD-RW, it read 117452800 out of 117508096 bytes. With the CD-R, it read 117424128 bytes. In both cases, writing the partial reads to a file, using wget -c to complete them, and sha1sum to check them, shows that the partial reads are correct up to the cutoff point. So to reproduce this bug, one should burn an unpadded ISO to a CD using one of the above commands, then check it using rawread with a >= 2.6.23 kernel and a CD or DVD drive known to have been vulnerable to the readahead bug before F7. The bug doesn't seem to affect DVDs.
Comment 15 Andre Robatino 2008-02-06 21:50:35 UTC
Correction - I know it affects DVDs, or at least did until recently, since both unpadded F8 DVDs my father bought online failed mediacheck, but it seems to be much harder to reproduce, plus a rescue CD image is much smaller so makes it much easier to burn the disc and reproduce the bug.
Comment 16 Andre Robatino 2008-02-07 00:52:26 UTC
I tried reproducing my father's experience by burning the unpadded F8 DVD ISO to a DVD+RW, booting from it, and running mediacheck. It passes for me - which means that there must be something different about the mass-pressed discs which is what he probably got. I tried to do the same with Fedora 9 Alpha, but unfortunately due to bug #431138, it's impossible to see whether mediacheck passed or failed. So currently I can only reproduce the problem with CDs, and most people never install using these anymore, which may account for the lack of reports.
Comment 17 Andre Robatino 2008-02-07 10:22:07 UTC
Still broken with kernel-126.96.36.199-115.fc8. With the unpadded FC4 disc, can read 665419776 out of 665434112 bytes again. With the Fedora 9 Alpha rescuecd CD-RW, can read 117489664 out of 117508096 bytes, and with the CD-R, 117424128 bytes, same as the previous kernel.
Comment 18 Alan Cox 2008-02-07 20:44:49 UTC
The fact mass produced disks work is consistent with the bug report. Mass produced disks have a defined end, CD-R's don't.
Comment 19 Andre Robatino 2008-02-07 21:17:00 UTC
I think you misunderstood - the online-purchased F8 DVD my father bought is the one that failed mediacheck, and also had the failed read near the end. The unpadded DVD+R and DVD+RW discs I burned for myself work properly. I don't doubt that if I obtained an identical F8 DVD from the same vendor, it would fail for me as well, since I'm using exactly the same DVD drive as my father.
Comment 20 Andre Robatino 2008-02-09 19:49:37 UTC
I just had my father read the F8 DVD he purchased while running the latest F8 kernel, and the problem still exists for him. The rawread command fails after reading 3424710656 out of 3424749568 bytes which is 38912 bytes short of the end. If anyone else can reproduce the problem with CDs, then either they could purchase the same DVD from the same vendor, or since my father has 2 copies of the same DVD (the vendor sent a second one which failed in the same way), I could have him send one of the copies to the person's address.
Comment 21 Andre Robatino 2008-02-12 01:12:23 UTC
Still broken in kernel-188.8.131.52-137.fc8. FC4 i386 Disc 1 CD-R: 665419776/665434112 bytes Fedora 9 Alpha i386 rescuecd CD-RW: 117424128/117508096 bytes (changed) Fedora 9 Alpha i386 rescuecd CD-R: 117452800/117508096 bytes (changed)
Comment 22 Andre Robatino 2008-03-07 05:15:17 UTC
Still broken in kernel-184.108.40.206-12.fc8. FC4 i386 Disc 1 CD-R: 665305088/665434112 bytes Fedora 9 Alpha i386 rescuecd CD-RW: 117424128/117508096 bytes Fedora 9 Alpha i386 rescuecd CD-R: 117424128/117508096 bytes
Comment 23 Andre Robatino 2008-03-10 11:54:06 UTC
I noticed that Fedora 9 Alpha was released with CD as well as DVD ISOs. Will this also be true for F9 final? Since this bug manifests more easily with CDs, it would result in a lot more reports of mediacheck failures, making it more likely that this would get fixed. (I never understood why providing CD images was so difficult, and yes I know about Fedora Unity and jigdo, but those don't help people who don't already have a working Fedora box.)
Comment 24 Andre Robatino 2008-03-17 03:23:00 UTC
Still broken in kernel-220.127.116.11-34.fc8. FC4 i386 Disc 1 CD-R: 665427968/665434112 bytes Fedora 9 Alpha i386 rescuecd CD-RW: 117424128/117508096 bytes Fedora 9 Alpha i386 rescuecd CD-R: 117424128/117508096 bytes
Comment 25 Andre Robatino 2008-03-26 09:11:50 UTC
Burned the 6 F9-Beta CD images to CD-RWs in default TAO mode without padding, using wodim. Testing them using the original F7 mediacheck, the discs all failed. Previously I had only tested F7 using CD-Rs. So apparently this bug was not fixed even in F7. I then tried reburning one disc using DAO mode without padding, and another using TAO mode with padding. Both passed. So apparently either DAO mode or padding is enough to allow the discs to be read properly. I then used rawread to read each of these two discs while running F8, and both discs could be read properly.
Comment 27 Andre Robatino 2008-04-27 20:27:32 UTC
Using the rawread script mentioned above would be the way to tell - if it is, the read would fail about 100K or less from the end. (BTW, this method can be used instead of mediacheck to check burned Fedora discs, without having to reboot. Just pipe through sha1sum and compare to the expected value.)
Comment 28 Michael Schwendt 2008-04-27 21:43:54 UTC
For comparison, bug 440343 is not due to I/O errors. dmesg is free of such errors. "readcd" can read the burnt image just fine.
Comment 29 Andre Robatino 2008-05-01 17:36:46 UTC
I also find that I cannot use readcd to properly read off the ISO file from a DVD, even a properly padded one which rawread has no trouble with. What I get contains extra data after the end of the actual ISO. See comments 12 and 13 of bug #440343. Could this be related to the readahead bug?
Comment 30 Andre Robatino 2008-06-07 23:27:42 UTC
Still broken in F9 with kernel-18.104.22.168-30.fc9.i686. Changing version to 9.
Comment 31 Andre Robatino 2008-11-28 05:13:45 UTC
Still broken in F10 with kernel-22.214.171.124-117.fc10.i686. Changing Version to 10. (Should this be made a duplicate of bug #186512?)