Bug 699151
| Summary: | ext4_lookup: deleted inode referenced | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | colyli |
| Component: | kernel | Assignee: | Lukáš Czerner <lczerner> |
| Status: | CLOSED ERRATA | QA Contact: | Eryu Guan <eguan> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 6.0 | CC: | eguan, esandeen, kzhang, lczerner, rwheeler, syeghiay |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | kernel-2.6.32-157.el6 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2011-12-06 13:19:24 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
colyli
2011-04-23 16:51:58 UTC
I ran a vanilla 2.6.36 kernel for 4 months, didn't observe this error. After running RHEL6 2.6.32 kernel for around 1 week, I first time observed this error. Now I observe this error on two machines with 2.6.32-71.18.2 and 2.6.32-71.24.1. This request was evaluated by Red Hat Product Management for inclusion in the current release of Red Hat Enterprise Linux. Because the affected component is not scheduled to be updated in the current release, Red Hat is unfortunately unable to address this request at this time. Red Hat invites you to ask your support representative to propose this request, if appropriate and relevant, in the next release of Red Hat Enterprise Linux. If you would like it considered as an exception in the current release, please ask your support representative. I collected meta data images of two corrupted file systems, them can be downloaded temporarily from: http://blog.coly.li/tmp/sda5.img.bz2 http://blog.coly.li/tmp/sda7.img.bz2 I try to run read-only fsck.ext4 on sda5 image, fsck.ext4 reports such info: # fsck.ext4 -n ./sda5-img e2fsck 1.41.14 (22-Dec-2010) ./sda5-img contains a file system with errors, check forced. Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Entry 'FF' in /14 (262145) has deleted/unused inode 262401. Clear? no Entry 'FF' in /14 (262145) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /05 (524289) has deleted/unused inode 524545. Clear? no Entry 'FF' in /05 (524289) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /3F (655361) has deleted/unused inode 655617. Clear? no Entry 'FF' in /3F (655361) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /0C (786433) has deleted/unused inode 786689. Clear? no Entry 'FF' in /0C (786433) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /1C (917505) has deleted/unused inode 917761. Clear? no Entry 'FF' in /1C (917505) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /03 (3145729) has deleted/unused inode 3145985. Clear? no Entry 'FF' in /03 (3145729) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /13 (3276801) has deleted/unused inode 3277057. Clear? no Entry 'FF' in /13 (3276801) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /3E (3407873) has deleted/unused inode 3408129. Clear? no Entry 'FF' in /3E (3407873) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /37 (3670017) has deleted/unused inode 3670273. Clear? no Entry 'FF' in /37 (3670017) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /3B (3932161) has deleted/unused inode 3932417. Clear? no Entry 'FF' in /3B (3932161) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /2B (4456449) has deleted/unused inode 4456705. Clear? no Entry 'FF' in /2B (4456449) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /08 (4718593) has deleted/unused inode 4718849. Clear? no Entry 'FF' in /08 (4718593) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /1F (4849665) has deleted/unused inode 4849921. Clear? no Entry 'FF' in /1F (4849665) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /24 (4980737) has deleted/unused inode 4980993. Clear? no Entry 'FF' in /24 (4980737) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /34 (5242881) has deleted/unused inode 5243137. Clear? no Entry 'FF' in /34 (5242881) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /21 (5767169) has deleted/unused inode 5767425. Clear? no Entry 'FF' in /21 (5767169) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /1A (6291457) has deleted/unused inode 6291713. Clear? no Entry 'FF' in /1A (6291457) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /31 (7077889) has deleted/unused inode 7078145. Clear? no Entry 'FF' in /31 (7077889) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /0F (9175041) has deleted/unused inode 9175297. Clear? no Entry 'FF' in /0F (9175041) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /30 (9961473) has deleted/unused inode 9961729. Clear? no Entry 'FF' in /30 (9961473) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /3A (10747905) has deleted/unused inode 10748161. Clear? no Entry 'FF' in /3A (10747905) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /3D (10878977) has deleted/unused inode 10879233. Clear? no Entry 'FF' in /3D (10878977) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /25 (11272193) has deleted/unused inode 11272449. Clear? no Entry 'FF' in /25 (11272193) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /2A (12189697) has deleted/unused inode 12189953. Clear? no Entry 'FF' in /2A (12189697) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /02 (12845057) has deleted/unused inode 12845313. Clear? no Entry 'FF' in /02 (12845057) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /0E (13107201) has deleted/unused inode 13107457. Clear? no Entry 'FF' in /0E (13107201) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /16 (13369345) has deleted/unused inode 13369601. Clear? no Entry 'FF' in /16 (13369345) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /1B (13631489) has deleted/unused inode 13631745. Clear? no Entry 'FF' in /1B (13631489) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /36 (13762561) has deleted/unused inode 13762817. Clear? no Entry 'FF' in /36 (13762561) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /0A (13893633) has deleted/unused inode 13893889. Clear? no Entry 'FF' in /0A (13893633) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /39 (14024705) has deleted/unused inode 14024961. Clear? no Entry 'FF' in /39 (14024705) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /01 (14942209) has deleted/unused inode 14942465. Clear? no Entry 'FF' in /01 (14942209) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /22 (15466497) has deleted/unused inode 15466753. Clear? no Entry 'FF' in /22 (15466497) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /1D (15859713) has deleted/unused inode 15859969. Clear? no Entry 'FF' in /1D (15859713) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /23 (15990785) has deleted/unused inode 15991041. Clear? no Entry 'FF' in /23 (15990785) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /38 (16121857) has deleted/unused inode 16122113. Clear? no Entry 'FF' in /38 (16121857) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /32 (17039361) has deleted/unused inode 17039617. Clear? no Entry 'FF' in /32 (17039361) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /17 (17301505) has deleted/unused inode 17301761. Clear? no Entry 'FF' in /17 (17301505) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /2C (17563649) has deleted/unused inode 17563905. Clear? no Entry 'FF' in /2C (17563649) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /11 (17825793) has deleted/unused inode 17826049. Clear? no Entry 'FF' in /11 (17825793) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /09 (18350081) has deleted/unused inode 18350337. Clear? no Entry 'FF' in /09 (18350081) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /10 (18612225) has deleted/unused inode 18612481. Clear? no Entry 'FF' in /10 (18612225) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /33 (18743297) has deleted/unused inode 18743553. Clear? no Entry 'FF' in /33 (18743297) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /2D (22937601) has deleted/unused inode 22937857. Clear? no Entry 'FF' in /2D (22937601) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /28 (23855105) has deleted/unused inode 23855361. Clear? no Entry 'FF' in /28 (23855105) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /15 (24117249) has deleted/unused inode 24117505. Clear? no Entry 'FF' in /15 (24117249) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /0D (25165825) has deleted/unused inode 25166081. Clear? no Entry 'FF' in /0D (25165825) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /18 (25427969) has deleted/unused inode 25428225. Clear? no Entry 'FF' in /18 (25427969) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /1E (27262977) has deleted/unused inode 27263233. Clear? no Entry 'FF' in /1E (27262977) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /2F (27394049) has deleted/unused inode 27394305. Clear? no Entry 'FF' in /2F (27394049) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /26 (27525121) has deleted/unused inode 27525377. Clear? no Entry 'FF' in /26 (27525121) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /29 (27656193) has deleted/unused inode 27656449. Clear? no Entry 'FF' in /29 (27656193) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /06 (27787265) has deleted/unused inode 27787521. Clear? no Entry 'FF' in /06 (27787265) has an incorrect filetype (was 2, should be 0). Fix? no Entry 'FF' in /07 (27918337) has deleted/unused inode 27918593. Clear? no Entry 'FF' in /07 (27918337) has an incorrect filetype (was 2, should be 0). Fix? no Directory inode 14942308, block #1, offset 0: directory corrupted Salvage? no e2fsck: aborted Thanks for reporting, but I have couple of questions. Does it happen with default mkfs options as well ? Does it happen with journal enabled ? Unfortunately I am not sure what is the squid work load, do you have any reliable reproducer that does not require setting up working squid for a day ? I found the reliable reproducer, it looks like directory htree corruption and can be reproduced with: for i in $(seq 1 300); do touch $i; done when number of entries in directory grows above 258 we hit the problem. EXT4-fs error (device loop0): ext4_lookup: deleted inode referenced: 27918593 EXT4-fs error (device loop0): ext4_lookup: deleted inode referenced: 27918593 EXT4-fs error (device loop0): ext4_lookup: deleted inode referenced: 27918593 EXT4-fs error (device loop0): ext4_lookup: deleted inode referenced: 27918593 EXT4-fs error (device loop0): ext4_lookup: deleted inode referenced: 27918593 hopefully I will have a fix soon. Thanks! -Lukas Ok, I was too hasty :-/. The above is not reliable reproducer at all, sorry for the noise. I am still trying to reproduce it and more information about the workload would help very much! Thanks! -Lukas Will you be willing to try to reproduce it with default mkfs.ext4 options and if it does not happen, again with ^has_journal since this is the most suspicious option. It would be really helpful in narrowing what is causing the problem. Thanks! -Lukas Hi Lukas, The connection to bugzilla.redhat.com was not stable, I was not able to log in and reply you :-( Now I run more servers with a little more printk info to reproduce the error. Also I will run more server with journal enabled. Once I have information available to update, I will post what I find here. Thanks for looking into this :-) Coly This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release. We (Taobao kernel team) identified the problem is in ext4_write_inode() in no journal mode. Indeed, google fixed this bug in upstream commit 8b472d739b2ddd8ab7fb278874f696cd95b25a5e, a more detailed description can also be found in the commit log. It seems RHEL6 2.6.32-71.29.1 kernel does not have this patch. Now we are testing the fix on 2.6.32-71.29.1 kernel. Just FYI, hope this patch will be in RHEL6.1 kernel soon. Hi, thanks for tracking this down! I'll backport the patch and do some more testing. Please let me know if it solves your issue, since I am still not able to find reliable reproducer. Thanks! -Lukas Patch(es) available on kernel-2.6.32-157.el6 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2011-1530.html |