Created attachment 413391 [details]
File: backtrace

things I found in /var/log/messages regarding the usb hdd:
May 12 10:38:10 t500 kernel: EXT3-fs error (device sdb1): ext3_find_entry: reading directory #2 offset 0

lots of these:
May 12 12:19:00 t500 kernel: sd 5:0:0:0: [sdc] Add. Sense: ATA pass through information available
May 12 12:19:00 t500 kernel: sd 5:0:0:0: [sdc] Sense Key : Recovered Error [current] [descriptor]
May 12 12:19:00 t500 kernel: Descriptor sense data with sense descriptors (in hex):
May 12 12:19:00 t500 kernel:        72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00·
May 12 12:19:00 t500 kernel:        00 4f 00 c2 e0 50·
May 12 12:19:00 t500 kernel: sd 5:0:0:0: [sdc] Add. Sense: ATA pass through information available
May 12 12:19:00 t500 kernel: sd 5:0:0:0: [sdc] Sense Key : Recovered Error [current] [descriptor]
May 12 12:19:00 t500 kernel: Descriptor sense data with sense descriptors (in hex):
May 12 12:19:00 t500 kernel:        72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00·
May 12 12:19:00 t500 kernel:        00 4f 00 c2 e0 50·
May 12 12:19:00 t500 kernel: sd 5:0:0:0: [sdc] Add. Sense: ATA pass through information available
May 12 12:19:00 t500 kernel: sd 5:0:0:0: [sdc] Sense Key : Recovered Error [current] [descriptor]
May 12 12:19:00 t500 kernel: Descriptor sense data with sense descriptors (in hex):
May 12 12:19:00 t500 kernel:        72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00·
May 12 12:19:00 t500 kernel:        00 4f 00 c2 e0 50·
May 12 12:19:00 t500 kernel: sd 5:0:0:0: [sdc] Add. Sense: ATA pass through information available

kernel warning happend at:
May 12 12:26:39 t500 kernel: WARNING: at fs/buffer.c:1159 mark_buffer_dirty+0x82/0xa0() (Not tainted)

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.

The warning is from:

void mark_buffer_dirty(struct buffer_head *bh)
{
        WARN_ON_ONCE(!buffer_uptodate(bh));

The buffer in question is that containing the superblock;

ext3_put_super():
                mark_buffer_dirty(sbi->s_sbh);

All of your scsi errors were on sdc, but you mentioned an ext3 error on sdb.  The warning was generated from the unmount task, but you said you had completed (?) unmount 1 minute prior to the trace?  Which device got unmounted, sdb or sdc - I'm guessing sdb?

> May 12 10:38:10 t500 kernel: EXT3-fs error (device sdb1): ext3_find_entry:
reading directory #2 offset 0

Anything else related to sdb in the logs?

Hi Eric,

sorry I didn't see that there's sdb and sdc. I'm using only 1 external drive so what I'm guessing that might have happened is: I've unmounted (and unplugged) sdb1 which caused the first "EXT3-fs error (device sdb1): ext3_find_entry:". Didn't really notice that at that time. Replugged the device later and it appeared as sdc. Later I unmounted that again and approx 1 minute after that I got the abrt notification.

No other messages regarding sd* in the log except lots of these
sd 4:0:0:0: [sdb] Sense Key : Recovered Error [current] [descriptor]
Descriptor sense data with sense descriptors (in hex):
        72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00 
        00 4f 00 c2 e0 50
messages after I attach the device.

Are you certain that you got the error -after- you unmounted?  It doesn't make any sense to get an ext3_find_entry message on an unmounted filesystem; nothing could get to that point in the code...

Is there any chance there was user error here? :)

This issue has been proposed when we are only considering blocker
issues in the current Red Hat Enterprise Linux release. It has
been denied for the current Red Hat Enterprise Linux release.

** If you would still like this issue considered for the current
release, ask your support representative to file as a blocker on
your behalf. Otherwise ask that it be considered for the next
Red Hat Enterprise Linux release. **

Ok, this looks similar to Bug 614206 - WARNING: at fs/buffer.c:1159 mark_buffer_dirty+0x82/0xa0() after device removal

it seems that after an IO error -> ext3 error -> superblock write we sometimes have a not-uptodate superblock buffer ... I'll have to look into this one.

It's not super-critical because it's only on error paths...

*** Bug 614206 has been marked as a duplicate of this bug. ***

This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unfortunately unable to
address this request at this time. Red Hat invites you to
ask your support representative to propose this request, if
appropriate and relevant, in the next release of Red Hat
Enterprise Linux. If you would like it considered as an
exception in the current release, please ask your support
representative.

This request was erroneously denied for the current release of Red Hat
Enterprise Linux.  The error has been fixed and this request has been
re-proposed for the current release.

There was an upstream commit for this issue:
dff6825e9fde93891e60751e01480337a991235e

I've sent a backport of this commit as well as a backport of dependent commit
4cf46b67eb6de94532c1bea11d2479d085229d0e to rhkernel-list

Edward.

This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux maintenance release. Product Management has 
requested further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed 
products. This request is not yet committed for inclusion in an Update release.

*** Bug 640419 has been marked as a duplicate of this bug. ***

Patch(es) available on kernel-2.6.32-112.el6

*** Bug 682209 has been marked as a duplicate of this bug. ***

Package: kernel
Architecture: x86_64
OS Release: Red Hat Enterprise Linux Server release 6.0 (Santiago)


How to reproduce
-----
Not sure aout this exact sequence, but something like the following:

1. Plug-in a USB storage device with an Ext2 partition (this device also has an Ext4 partition)
2. Hibernate the machine
3. Wake the machine up
4. Greetings from ABRT

Comment
-----
I've got some other funky behavior going on as well, so have the following /proc/cmdline (note the iommu)igfx parm):

[pmcdonou@patredhat /]$ cat /proc/cmdline 
ro root=/dev/mapper/vg_thinkpad--01-lv_root rd_LVM_LV=vg_thinkpad-01/lv_root rd_LVM_LV=vg_thinkpad-01/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb quiet intel_iommu=igfx_off

Bug for ext2 seems to be bug #679930

Comment #26: Yes, at least in IBM's opinion.

------- Comment From djwong.com 2011-02-03 12:41 EDT-------
There's some confusion going on in LTC64882 as to whether or not this bug (LTC65071) is a duplicate.

To clarify, 65071 is a fix for the badness warning when ext3 produces it.  64882 is a fix for the badness warning when ext2 produces it.  These two bugs are not the same, they are not duplicates of each other, and I hope this is true on the RH end too, though I don't know how to confirm that.  In the past, the two LTC bugs were marked as duplicates of each other, but this should no longer be the case.

Created attachment 487779 [details]
sosreport

Created attachment 487780 [details]
patch against rc-1 (2.6.32-71.el6)

RH Bug 679930 for ext2 is still valid & open, it will be fixed.

The patch attached in comment #30 for ext3 is already integrated into RHEL6, see comment #17.

------- Comment From afox.com 2011-03-29 06:39 EDT-------
fix verified with kernel-2.6.32-118.el6.s390x

Created attachment 489309 [details]
File: backtrace

Reproduced on -71 kernel by umounting a USB disk which was already removed physically after some kinds of buffered IO. (though it's not so easy to hit this issue)

dd if=/dev/zero of=/mnt/testfile bs=1M count=100
# unplug USB disk
umount /mnt

usb 1-4: USB disconnect, address 10
------------[ cut here ]------------
WARNING: at fs/buffer.c:1159 mark_buffer_dirty+0x6a/0x80() (Not tainted)
Hardware name: 26681BC
Modules linked in: ext3 jbd usb_storage xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat bridge stp llc autofs4 sunrpc cpufreq_ondemand acpi_cpufreq ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log ppdev thinkpad_acpi hwmon rfkill parport_pc parport ipw2200 libipw lib80211 sg i2c_i801 iTCO_wdt iTCO_vendor_support snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc tg3 ext4 mbcache jbd2 video output yenta_socket rsrc_nonstatic sd_mod crc_t10dif sr_mod cdrom pata_acpi ata_generic ahci ata_piix radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mod [last unloaded: microcode]
Pid: 3089, comm: umount Not tainted 2.6.32-71.el6.i686 #1
Call Trace:
 [<c04501c1>] ? warn_slowpath_common+0x81/0xc0
 [<c054719a>] ? mark_buffer_dirty+0x6a/0x80
 [<c054719a>] ? mark_buffer_dirty+0x6a/0x80
 [<c045021b>] ? warn_slowpath_null+0x1b/0x20
 [<c054719a>] ? mark_buffer_dirty+0x6a/0x80
 [<f98b48fd>] ? ext3_put_super+0x15d/0x230 [ext3]
 [<c053469d>] ? invalidate_inodes+0xbd/0x120
 [<c05647c0>] ? vfs_quota_off+0x0/0x10
 [<c051fee5>] ? generic_shutdown_super+0x45/0xc0
 [<c05150b6>] ? free_percpu+0x66/0x100
 [<c051ff82>] ? kill_block_super+0x22/0x40
 [<c0520d0b>] ? deactivate_super+0x5b/0x90
 [<c0537d8a>] ? sys_umount+0x6a/0x350
 [<c0538087>] ? sys_oldumount+0x17/0x20
 [<c04099fb>] ? sysenter_do_call+0x12/0x28
---[ end trace 2f8381ad354ffc4c ]---

On -128 kernel, I tried 30 times or so, no such message found. Set it to VERIFIED.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0542.html