Bug 498162 - kernel: bio too big device md3 (248 > 240)
kernel: bio too big device md3 (248 > 240)
Status: CLOSED WONTFIX
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
20
x86_64 Linux
low Severity low
: ---
: ---
Assigned To: Kernel Maintainer List
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-04-29 04:42 EDT by Jan Kratochvil
Modified: 2014-05-02 13:25 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-05-02 13:25:13 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jan Kratochvil 2009-04-29 04:42:26 EDT
Description of problem:
Getting 60000+ messages "kernel: bio too big device md3", rpm also complains, due to:
Running root filesystem on luks.
This luks is on md3.
This md3 is raid1 on sata sda3 and on USB sdb3.

Version-Release number of selected component (if applicable):
kernel-2.6.29.1-102.fc11.x86_64

How reproducible:
Always.

Steps to Reproduce:
1. Setup such system, it happens during writes.

Actual results:
60000 messages in my current /var/log/messages:
Apr 29 10:36:17 host0 kernel: bio too big device md3 (248 > 240)

rpmdb: fsync: Input/output error
error: db4 error(5) from db->sync: Input/output error

Expected results:
No warnings/errors.

Additional info:
Googled it is a known upstream problem with USB disks as kernel cannot split large I/O requests.

Still in such case curious how can a problem with only one of the raid1 disks propagate through raid1 to the application level.

Google says it corrupts data, it seems to me even the USB partition is perfectly valid (but did not do any artificial test besides boot/fsck for it).

It was happening even on F10, just filed it now for the fresh kernel.
Comment 1 Chuck Ebbert 2009-04-30 13:13:43 EDT
It looks like this is really a dm problem. Maybe dm should limit itself to 128 sectors no matter what the underlying device says it can handle?
Comment 2 Mikulas Patocka 2009-04-30 15:34:50 EDT
Was the USB-disk hot-added to md-raid1?

If so, it is known problem and cannot be fixed unless redesigning the bio interface.
Comment 3 Jan Kratochvil 2009-04-30 15:39:25 EDT
(In reply to comment #2)
> Was the USB-disk hot-added to md-raid1?

Yes, it occurs at least if the USB disk _was_ hot-added.

(Whether it does / does not also occur when it gets found while assembling the md device I do not know.)
Comment 4 Milan Broz 2009-04-30 16:37:56 EDT
See also upstream bug 
http://bugzilla.kernel.org/show_bug.cgi?id=9401
Comment 5 Chuck Ebbert 2009-04-30 21:56:19 EDT
(In reply to comment #2)
> Was the USB-disk hot-added to md-raid1?
> 
> If so, it is known problem and cannot be fixed unless redesigning the bio
> interface.  

How about never letting dm use more than 240 sectors even if the device will allow more?
Comment 6 Maciej Żenczykowski 2009-05-10 17:04:46 EDT
I get the exact same error message "kernel: bio too big device md2 (248 > 240)" on my fedora 10 box (kernel 2.6.29.2-52.fc10.x86_64).  This is a raid1 with 2 or 3 drives (2 sata partitions which are always present, plus one sometimes present usb partition used for occasional mirror backups).  This error shows up even after failing/removing the usb partition.  It seems the lower value sticks around somewhere in the md driver.  Thankfully it doesn't seem to cause any data corruption...
Comment 7 Maciej Żenczykowski 2009-05-10 17:29:28 EDT
Oh, I should also point out that the exact same setup of a raid1 md1 does not suffer this problem (direct ext3 on top of it, for /boot), while for raid1 md2, there is a luks-md2 layer, on top of which there is an ext3 partition (/).  So inline with the bug linked above this does seem to be specific to the layering.

Furthermore:
dmsetup reload luks-md2 --table "`dmsetup table --showkeys luks-md2`"
dmsetup suspend luks-md2; dmsetup resume luks-md2

Does appear to fix the issue - although it's pretty scary (md2 is the root fs, so this only works because of caching).
Comment 8 Bug Zapper 2009-06-09 10:45:41 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 11 development cycle.
Changing version to '11'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 9 Maciej Żenczykowski 2010-04-03 18:27:16 EDT
FYI still happens with Fedora 12
Comment 10 Mikulas Patocka 2010-04-06 02:20:22 EDT
There will be no fix. It is design problem and it can't be fixed unless bio interface is redesigned and rewritten (and I don't know if Jens Axboe allows it). If you want to avoid it, don't run raid1 on different device types.
Comment 11 Maciej Żenczykowski 2010-04-06 02:35:53 EDT
Well... even just plain hardcoding md to never return that it supports more than 240 would be a certain kind of fix.  Alternatively if dm on receiving an error (does it even see the error?) refetched the value from the base device, it would cause this to rarely happen...
Comment 12 Mikulas Patocka 2010-04-28 11:09:46 EDT
If we hardcode it for 240 sectors, it will fail with some device that has lower limits (i.e. raid1 on floppies :)

Refetching the values would make the error happen less likely, but not really fix it.
The problem is this:
1. some code builds bio, keeps on adding pages to it until it fills raid1 sector limits
2. some other code adds a new leg with smaller i/o limits to that raid1 array 
3. the bio is submitted to raid1
4. the bio is already built, it can't be shrunk, it can't be split and it will certainly fail when submitted to the new raid1 leg
--- this can't be just fixed, it needs redesign.

If we applied the workaround and refetched limits, we would change reproducible failure into a racy random failure as described in the previous paragraph. Racy failure is worse than reproducible failure.
Comment 13 Maciej Żenczykowski 2010-04-29 14:52:53 EDT
Ok, I was under the impression that the current behaviour printed a warning, but still continued to function correctly.  Is that not the case?  Does this end up corrupting something?


If it is the case, that this is just a warning that is 'harmless' and the kernel works around the problem, then refetching the limits sounds like it would just work.  You would get one error (well a couple if it raced on smp) when the device shrunk, and afterwards you would get no more errors.

Currently you get errors occasionally 'for ever more' when a raid device shrinks it's max bio size.

Am I misunderstanding something?
Comment 14 Mikulas Patocka 2010-05-18 18:00:17 EDT
Yes, it may corrupt a filesystem (i.e. lose writes to files, or less likely, metadata).

When raid device shrinks bio size, you also get error, it is similar case.

The I/O layer just needs redesign. dm-raid or md-raid can't do anything about it.
Comment 15 scott.m.mcdermott+rh-omnisys-bugs 2010-09-10 18:19:45 EDT
This is still a problem with F13.  I ran into it trying to
make a simple RAID copy of my laptop using another disk.  I
am using an encrypted LV for a home directory, which then
resides on an MD member.

If I understand right, once I start seeing these "bio too
big" I had better force-umount and reboot with sysrq because
if I let everything sync I will see filesystem corruption.

So this is a known issue that causes *silent data corruption*
in the Linux kernel for actually several years (since at
least FC9, I just upgraded from, hoping it would go away)?
Well, kernel tells you it's corrupting your data, but still
proceeds and user thinks everything is ok if he's not
watching kernel log.

This turns what would be a simple plug-in USB disk
auto-backup via udev/mdadm into data corruption.  And to
backup instead, have to boot with init=/bin/bash before LVM
starts and wait for the sync.

Is it really so uncommon and unimportant a thing to be
assigned low/low priority?
Comment 16 scott.m.mcdermott+rh-omnisys-bugs 2010-09-10 18:26:48 EDT
Perhaps a module option for MD telling it "never make bio
higher than this" until upstream feels like fixing this?

It sounds like the required "redesign" will essentialy never
happen.  Possibly the bug should be closed as a WONTFIX
then.
Comment 17 Bug Zapper 2010-11-04 07:18:16 EDT
This message is a reminder that Fedora 12 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 12.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '12'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 12's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 12 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 18 Maciej Żenczykowski 2010-11-04 16:07:06 EDT
This is still very much a problem.
I like the module option suggestion.
I know I will never put in devices that support less than 240.
Comment 19 scott.m.mcdermott+rh-omnisys-bugs 2010-11-04 16:25:18 EDT
Still bug for sure.

I keep forgetting about this until I plug in disk and it starts chewing
on my filesystems.  This seems like pretty severe bug, filesystem
corruption for years in linux kernel causing silent data corruption.
Once this happens I never know about the integrity of my data again.

I also would be happy with some way to just pass in a never-exceed
bio size for the device so we could at least put the max size in a file
somewhere and don't have to risk this silent data corruption if we do
not remember about this permanent limitation of Linux kernel.
Comment 20 Bug Zapper 2010-12-05 01:55:39 EST
Fedora 12 changed to end-of-life (EOL) status on 2010-12-02. Fedora 12 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.
Comment 21 Frank Ch. Eigler 2014-05-02 12:48:54 EDT
Problem still present on f20 kernel 3.13.7-200.fc20.i686.
Comment 22 Alasdair Kergon 2014-05-02 13:25:13 EDT
By all means keep adding instances of this to this bug, but this is an upstream design issue best pursued in a different forum (a kernel mailing list).

Note You need to log in before you can comment on or make changes to this bug.