Bug 699151 - ext4_lookup: deleted inode referenced
Summary: ext4_lookup: deleted inode referenced
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel (Show other bugs)
(Show other bugs)
Version: 6.0
Hardware: x86_64 Linux
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Lukáš Czerner
QA Contact: Eryu Guan
URL:
Whiteboard:
Keywords:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-04-23 16:51 UTC by colyli
Modified: 2011-12-06 13:19 UTC (History)
6 users (show)

Fixed In Version: kernel-2.6.32-157.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-12-06 13:19:24 UTC
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2011:1530 normal SHIPPED_LIVE Moderate: Red Hat Enterprise Linux 6 kernel security, bug fix and enhancement update 2011-12-06 01:45:35 UTC

Description colyli 2011-04-23 16:51:58 UTC
Description of problem:
The kernel version is 2.6.32-71.24.1.el6 kernel, file system is Ext4.
The problematic file system acts as squid AUFS and running for 1 day.
From dmesg following error messages are observed:
[215639.957529] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 14942465
[215640.050837] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 12845313
[215640.151467] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 3145985
[215640.331108] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 524545
Is seems a dentry of a deleted inode is looked up.

Version-Release number of selected component (if applicable):
The kernel version is 2.6.32-71.24.1.el6 kernel, file system is Ext4.

How reproducible:

The issue is observed on machines running for days, not easy to reproduce, but the error is observed on multiple servers.

Formated Ext4 file system with (no journal mode here):
mkfs.ext4 -m 0 -O ^has_journal,^resize_inode,^uninit_bg,extent,meta_bg /dev/sda5

The file system is mounted by:
mount -t ext4 -o noatime /dev/sda5 /bigcache1

The squid workload is from network compressing test, which use the server running squid as a http cache proxy.

Steps to Reproduce:
1. Format file system and mount it
2. Running squid service, use this Ext4 file system as AUFS of squid
3. Run squid with workload for more than 1 day, and check dmesg output.
  
Actual results:
[215639.957529] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 14942465
[215640.050837] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 12845313
[215640.151467] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 3145985
[215640.331108] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 524545
[215640.429561] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 27787521
[215640.515848] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 27918593
[215640.610949] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 4718849
[215640.708957] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 18350337
[215640.807029] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 13893889
[215640.980314] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 786689
[215641.077811] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 25166081
[215641.169594] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 13107457
[215641.283940] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 9175297
[215641.368250] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 18612481
[215641.449704] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 17826049
[215641.609611] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 3277057
[215641.692728] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 262401
[215641.784362] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 24117505
[215641.909296] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 13369601
[215642.001024] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 17301761
[215642.090829] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 25428225
[215642.248619] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 6291713
[215642.345757] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 13631745
[215642.485334] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 917761
[215642.573151] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 15859969
[215642.662059] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 27263233
[215642.748798] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 4849921
[215642.896418] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 5767425
[215642.986730] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 15466753
[215643.084979] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 15991041
[215643.175590] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 4980993
[215643.266024] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 11272449
[215643.355879] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 27525377
[215643.511513] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 23855361
[215643.599642] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 27656449
[215643.687801] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 12189953
[215643.771853] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 4456705
[215643.861584] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 17563905
[215643.951737] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 22937857
[215644.101868] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 27394305
[215644.192511] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 9961729
[215644.313424] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 7078145
[215644.400823] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 17039617
[215644.487774] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 18743553
[215644.575162] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 5243137
[215644.732499] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 13762817
[215644.815265] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 3670273
[215644.904751] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 16122113
[215644.990540] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 14024961
[215645.072729] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 10748161
[215645.158292] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 3932417
[215645.314966] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 10879233
[215645.406217] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 3408129
[215645.489714] EXT4-fs error (device sda5): ext4_lookup: deleted inode referenced: 655617
[215645.765836] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 26214657
[215645.865617] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 2359553
[215646.004500] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 29098241
[215646.097573] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 1573121
[215646.203451] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 28311809
[215646.407719] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 8651009
[215646.503106] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 4980993
[215646.598040] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 13369601
[215646.775104] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 15204609
[215646.874277] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 4718849
[215646.969719] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 8388865
[215649.947628] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 10354945
[248747.307090] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 26214657
[248747.362515] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 26214657
[248762.300038] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 2359553
[248762.322957] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 2359553
[248777.299720] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 29098241
[248777.322228] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 29098241
[248792.307786] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 1573121
[248792.330839] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 1573121
[248807.304950] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 28311809
[248807.327543] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 28311809
[248837.298812] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 8651009
[248837.322004] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 8651009
[248852.300108] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 4980993
[248852.323088] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 4980993
[248867.300279] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 13369601
[248867.323433] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 13369601
[248897.300973] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 15204609
[248897.322804] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 15204609
[248912.305299] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 4718849
[248912.327136] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 4718849
[248927.305523] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 8388865
[248927.329041] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 8388865
[249512.310062] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 10354945
[249512.331954] EXT4-fs error (device sda7): ext4_lookup: deleted inode referenced: 10354945

Expected results:
No such information.

Additional info:

Comment 1 colyli 2011-04-23 16:57:15 UTC
I ran a vanilla 2.6.36 kernel for 4 months, didn't observe this error. After running RHEL6 2.6.32 kernel for around 1 week, I first time observed this error.

Now I observe this error on two machines with 2.6.32-71.18.2 and 2.6.32-71.24.1.

Comment 3 RHEL Product and Program Management 2011-04-23 17:17:34 UTC
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unfortunately unable to
address this request at this time. Red Hat invites you to
ask your support representative to propose this request, if
appropriate and relevant, in the next release of Red Hat
Enterprise Linux. If you would like it considered as an
exception in the current release, please ask your support
representative.

Comment 4 colyli 2011-04-24 12:06:58 UTC
I collected meta data images of two corrupted file systems, them can be downloaded temporarily from:
http://blog.coly.li/tmp/sda5.img.bz2
http://blog.coly.li/tmp/sda7.img.bz2

I try to run read-only fsck.ext4 on sda5 image, fsck.ext4 reports such info:
# fsck.ext4 -n ./sda5-img 
e2fsck 1.41.14 (22-Dec-2010)
./sda5-img contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Entry 'FF' in /14 (262145) has deleted/unused inode 262401.  Clear? no

Entry 'FF' in /14 (262145) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /05 (524289) has deleted/unused inode 524545.  Clear? no

Entry 'FF' in /05 (524289) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /3F (655361) has deleted/unused inode 655617.  Clear? no

Entry 'FF' in /3F (655361) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /0C (786433) has deleted/unused inode 786689.  Clear? no

Entry 'FF' in /0C (786433) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /1C (917505) has deleted/unused inode 917761.  Clear? no

Entry 'FF' in /1C (917505) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /03 (3145729) has deleted/unused inode 3145985.  Clear? no

Entry 'FF' in /03 (3145729) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /13 (3276801) has deleted/unused inode 3277057.  Clear? no

Entry 'FF' in /13 (3276801) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /3E (3407873) has deleted/unused inode 3408129.  Clear? no

Entry 'FF' in /3E (3407873) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /37 (3670017) has deleted/unused inode 3670273.  Clear? no

Entry 'FF' in /37 (3670017) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /3B (3932161) has deleted/unused inode 3932417.  Clear? no

Entry 'FF' in /3B (3932161) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /2B (4456449) has deleted/unused inode 4456705.  Clear? no

Entry 'FF' in /2B (4456449) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /08 (4718593) has deleted/unused inode 4718849.  Clear? no

Entry 'FF' in /08 (4718593) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /1F (4849665) has deleted/unused inode 4849921.  Clear? no

Entry 'FF' in /1F (4849665) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /24 (4980737) has deleted/unused inode 4980993.  Clear? no

Entry 'FF' in /24 (4980737) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /34 (5242881) has deleted/unused inode 5243137.  Clear? no

Entry 'FF' in /34 (5242881) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /21 (5767169) has deleted/unused inode 5767425.  Clear? no

Entry 'FF' in /21 (5767169) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /1A (6291457) has deleted/unused inode 6291713.  Clear? no

Entry 'FF' in /1A (6291457) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /31 (7077889) has deleted/unused inode 7078145.  Clear? no

Entry 'FF' in /31 (7077889) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /0F (9175041) has deleted/unused inode 9175297.  Clear? no

Entry 'FF' in /0F (9175041) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /30 (9961473) has deleted/unused inode 9961729.  Clear? no

Entry 'FF' in /30 (9961473) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /3A (10747905) has deleted/unused inode 10748161.  Clear? no

Entry 'FF' in /3A (10747905) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /3D (10878977) has deleted/unused inode 10879233.  Clear? no

Entry 'FF' in /3D (10878977) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /25 (11272193) has deleted/unused inode 11272449.  Clear? no

Entry 'FF' in /25 (11272193) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /2A (12189697) has deleted/unused inode 12189953.  Clear? no

Entry 'FF' in /2A (12189697) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /02 (12845057) has deleted/unused inode 12845313.  Clear? no

Entry 'FF' in /02 (12845057) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /0E (13107201) has deleted/unused inode 13107457.  Clear? no

Entry 'FF' in /0E (13107201) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /16 (13369345) has deleted/unused inode 13369601.  Clear? no

Entry 'FF' in /16 (13369345) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /1B (13631489) has deleted/unused inode 13631745.  Clear? no

Entry 'FF' in /1B (13631489) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /36 (13762561) has deleted/unused inode 13762817.  Clear? no

Entry 'FF' in /36 (13762561) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /0A (13893633) has deleted/unused inode 13893889.  Clear? no

Entry 'FF' in /0A (13893633) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /39 (14024705) has deleted/unused inode 14024961.  Clear? no

Entry 'FF' in /39 (14024705) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /01 (14942209) has deleted/unused inode 14942465.  Clear? no

Entry 'FF' in /01 (14942209) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /22 (15466497) has deleted/unused inode 15466753.  Clear? no

Entry 'FF' in /22 (15466497) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /1D (15859713) has deleted/unused inode 15859969.  Clear? no

Entry 'FF' in /1D (15859713) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /23 (15990785) has deleted/unused inode 15991041.  Clear? no

Entry 'FF' in /23 (15990785) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /38 (16121857) has deleted/unused inode 16122113.  Clear? no

Entry 'FF' in /38 (16121857) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /32 (17039361) has deleted/unused inode 17039617.  Clear? no

Entry 'FF' in /32 (17039361) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /17 (17301505) has deleted/unused inode 17301761.  Clear? no

Entry 'FF' in /17 (17301505) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /2C (17563649) has deleted/unused inode 17563905.  Clear? no

Entry 'FF' in /2C (17563649) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /11 (17825793) has deleted/unused inode 17826049.  Clear? no

Entry 'FF' in /11 (17825793) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /09 (18350081) has deleted/unused inode 18350337.  Clear? no

Entry 'FF' in /09 (18350081) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /10 (18612225) has deleted/unused inode 18612481.  Clear? no

Entry 'FF' in /10 (18612225) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /33 (18743297) has deleted/unused inode 18743553.  Clear? no

Entry 'FF' in /33 (18743297) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /2D (22937601) has deleted/unused inode 22937857.  Clear? no

Entry 'FF' in /2D (22937601) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /28 (23855105) has deleted/unused inode 23855361.  Clear? no

Entry 'FF' in /28 (23855105) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /15 (24117249) has deleted/unused inode 24117505.  Clear? no

Entry 'FF' in /15 (24117249) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /0D (25165825) has deleted/unused inode 25166081.  Clear? no

Entry 'FF' in /0D (25165825) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /18 (25427969) has deleted/unused inode 25428225.  Clear? no

Entry 'FF' in /18 (25427969) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /1E (27262977) has deleted/unused inode 27263233.  Clear? no

Entry 'FF' in /1E (27262977) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /2F (27394049) has deleted/unused inode 27394305.  Clear? no

Entry 'FF' in /2F (27394049) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /26 (27525121) has deleted/unused inode 27525377.  Clear? no

Entry 'FF' in /26 (27525121) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /29 (27656193) has deleted/unused inode 27656449.  Clear? no

Entry 'FF' in /29 (27656193) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /06 (27787265) has deleted/unused inode 27787521.  Clear? no

Entry 'FF' in /06 (27787265) has an incorrect filetype (was 2, should be 0).
Fix? no

Entry 'FF' in /07 (27918337) has deleted/unused inode 27918593.  Clear? no

Entry 'FF' in /07 (27918337) has an incorrect filetype (was 2, should be 0).
Fix? no

Directory inode 14942308, block #1, offset 0: directory corrupted
Salvage? no

e2fsck: aborted

Comment 5 Lukáš Czerner 2011-04-26 11:01:35 UTC
Thanks for reporting, but I have couple of questions. Does it happen with default mkfs options as well ? Does it happen with journal enabled ? Unfortunately I am not sure what is the squid work load, do you have any reliable reproducer that does not require setting up working squid for a day ?

Comment 6 Lukáš Czerner 2011-04-26 13:49:18 UTC
I found the reliable reproducer, it looks like directory htree corruption and can be reproduced with:

for i in $(seq 1 300); do touch $i; done

when number of entries in directory grows above 258 we hit the problem.

EXT4-fs error (device loop0): ext4_lookup: deleted inode referenced: 27918593
EXT4-fs error (device loop0): ext4_lookup: deleted inode referenced: 27918593
EXT4-fs error (device loop0): ext4_lookup: deleted inode referenced: 27918593
EXT4-fs error (device loop0): ext4_lookup: deleted inode referenced: 27918593
EXT4-fs error (device loop0): ext4_lookup: deleted inode referenced: 27918593

hopefully I will have a fix soon.

Thanks!
-Lukas

Comment 7 Lukáš Czerner 2011-05-04 08:16:16 UTC
Ok, I was too hasty :-/. The above is not reliable reproducer at all, sorry for the noise. I am still trying to reproduce it and more information about the workload would help very much!

Thanks!
-Lukas

Comment 8 Lukáš Czerner 2011-05-04 15:53:53 UTC
Will you be willing to try to reproduce it with default mkfs.ext4 options and if it does not happen, again with ^has_journal since this is the most suspicious option. It would be really helpful in narrowing what is causing the problem.

Thanks!
-Lukas

Comment 9 colyli 2011-05-04 16:08:39 UTC
Hi Lukas,

The connection to bugzilla.redhat.com was not stable, I was not able to log in and reply you :-(

Now I run more servers with a little more printk info to reproduce the error. Also I will run more server with journal enabled.

Once I have information available to update, I will post what I find here.

Thanks for looking into this :-)

Coly

Comment 10 RHEL Product and Program Management 2011-05-13 15:26:46 UTC
This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux maintenance release. Product Management has 
requested further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed 
products. This request is not yet committed for inclusion in an Update release.

Comment 11 colyli 2011-05-18 04:32:00 UTC
We (Taobao kernel team) identified the problem is in ext4_write_inode() in no journal mode. Indeed, google fixed this bug in upstream commit 8b472d739b2ddd8ab7fb278874f696cd95b25a5e, a more detailed description can also be found in the commit log. It seems RHEL6 2.6.32-71.29.1 kernel does not have this patch.

Now we are testing the fix on 2.6.32-71.29.1 kernel. Just FYI, hope this patch will be in RHEL6.1 kernel soon.

Comment 12 Lukáš Czerner 2011-05-18 10:22:08 UTC
Hi,

thanks for tracking this down! I'll backport the patch and do some more testing. Please let me know if it solves your issue, since I am still not able to find reliable reproducer.

Thanks!
-Lukas

Comment 15 Aristeu Rozanski 2011-06-14 14:58:56 UTC
Patch(es) available on kernel-2.6.32-157.el6

Comment 19 errata-xmlrpc 2011-12-06 13:19:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2011-1530.html


Note You need to log in before you can comment on or make changes to this bug.