Bug 598150

Summary:	GFS2: Number of entries corrupt in dir 19266892
Product:	[Fedora] Fedora	Reporter:	Paulo Castro <pauloedgarcastro>
Component:	kernel	Assignee:	Robert Peterson <rpeterso>
Status:	CLOSED WORKSFORME	QA Contact:	Cluster QE <mspqa-list>
Severity:	medium	Docs Contact:
Priority:	low
Version:	11	CC:	adas, anton, bmarzins, dougsland, edamato, gansalmon, itamar, jonathan, kernel-maint, swhiteho
Target Milestone:	---
Target Release:	---
Hardware:	i686
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2010-06-28 15:45:03 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Paulo Castro 2010-05-31 15:07:53 UTC

Description of problem:

I have this message:

GFS2: fsid=mycluster:sharedlv.0: Number of entries corrupt in dir 19266892, ip->i_entries (10) != g.offset (2)

appearing in my /var/log/messages file whenever I try to ls a specific dir in my GFS2 mount.
Running fsck.gfs2 -yv /device always also spits out:
"Directory '..' and treewalk connections disagree for inode 19266892 (0x125fd4c)" 


Version-Release number of selected component (if applicable):

gfs2-utils 3.0.10

How reproducible:

Always.

Steps to Reproduce:
1. Mount drive
2. ls -altr specific dir where the problem is known to happen.
3.
  
Actual results:

ls -altr /gfs/foo/bar
ls: reading directory /gfs/foo/bar: Input/output error
total 0

rm -rf /gfs/foo/bar
rm: cannot remove directory `/gfs/foo/bar': Directory not empty



Expected results:


Additional info:

Fedora 11,
Kernel 2.6.30.10-105.2.23.fc11.i586

Comment 1 Robert Peterson 2010-06-01 13:45:43 UTC

Can you save off a copy of your file system metadata with a
command similar to this:

gfs2_edit savemeta /dev/device /tmp/598150.meta
bzip2 /tmp/598150.meta

Then put the resulting .bz2 file somewhere I can download it?
I want to see what's going on in there.

Comment 2 Steve Whitehouse 2010-06-01 15:43:38 UTC

Can you confirm which kernel version you are using?

Comment 3 Steve Whitehouse 2010-06-01 15:46:24 UTC

Oops, I missed that at the end of the report. sorry :( It would be worthwhile upgrading the kernel I think, although I don't know of any bugs which might affect directory structure. A number of other bugs have been fixed in the mean time.

Comment 4 Paulo Castro 2010-06-01 21:54:36 UTC

Robert,

Before even thinking much about it, I started the process only to find out that it wouldn't be able to complete.
On one of the boxes it segfaults with:

Reading resource groups...Done.

16556979 metadata blocks (85%) processed, 774264 metadata blocks (754MB) saved.    Segmentation fault

/var/log/messages:
kernel: gfs2_edit[16226]: segfault at 14 ip 0805bf76 sp bfc418a0 error 4 in gfs2_edit[8047000+23000]

And on the other ( this is a two node cluster ) it get's killed by the oom watchdog.

There's some sensitive information that we don't feel like disclosing, if there's any other way we can provide information please let me know.

Comment 5 Robert Peterson 2010-06-11 18:57:24 UTC

If I can get ssh access to the machine, I can figure out what's
wrong with that dinode with an interactive session of gfs2_edit.
That would be the fastest way.  It would also tell me why the
gfs2_edit savemeta is failing, and why fsck.gfs2 isn't fixing
the corruption.

Without access to the machine or its metadata, the only way to
tell what's going on is for me to give you individual commands,
have you execute them and report the results.  That will be a
very long, slow, tedious process due to the fact that we will
likely only exchange one email per day.

Here are a few commands you can do to get you started:

gfs2_edit -p sb /dev/device
(prints the superblock)

   This will tell me the block size and if the damaged dinode
corresponds to a system dinode or a user dinode.

gfs2_edit -x -p root /dev/device
(prints the root dinode)

   This will tell me if the corrupt dinode at block 0x125fd4c
is for the lost+found directory or something else in the root.

gfs2_edit -p 0x125fd4c /dev/device

This will print the dinode that has a problem.  It's a starting
point to tell me where its indirect blocks are located and
what values it has for the number of connections and such.

Comment 6 Paulo Castro 2010-06-15 19:06:36 UTC

Block #16    (0x10) of 19274154 (0x12619AA) (superblock)

Superblock:
  mh_magic              0x01161970(hex)
  mh_type               1                   0x1
  mh_format             100                 0x64
  sb_fs_format          1801                0x709
  sb_multihost_format   1900                0x76c
  sb_bsize              4096                0x1000
  sb_bsize_shift        12                  0xc
  master dir:           2                   0x2
        addr:           23                  0x17
  root dir  :           1                   0x1
        addr:           22                  0x16
  sb_lockproto          lock_dlm
  sb_locktable          mycluster:sharedlv
  sb_uuid               00000000-0000-0000-0000-000000000000

The superblock has 2 directories
     1. (1). 22 (0x16): Dir     root
     2. (2). 23 (0x17): Dir     master
------------------------------------------------------
Block #22    (0x16) of 19274154 (0x12619AA) (disk inode)
--------------- Root directory ------------------
00016000 01161970 00000004 00000000 00000000 [...p............] 
00016010 00000190 00000000 00000000 00000001 [................] 
00016020 00000000 00000016 000041ED 00000000 [..........A.....] 
00016030 00000000 0000000A 00000000 00000F18 [................] 
00016040 00000000 00000001 00000000 4C17CD77 [............L..w] 
00016050 00000000 4BE481A1 00000000 4BE481A1 [....K.......K...] 
00016060 00000000 00000000 00000000 00000016 [................] 
00016070 00000000 00000016 00000000 00000000 [................] 
00016080 00000001 000004B0 00000000 00000000 [................] 
00016090 00000000 0000000A 00000000 00000000 [................] 
000160A0 00000000 00000000 00000000 00000000 [................] 
000160B0 0A1F64B0 0207801C 0207801C 00000000 [..d.............] 
000160C0 00000000 00000000 00000000 00000000 [................] 
000160D0 00000000 00000000 00000000 00000000 [................] 
000160E0 00000000 00000000 00000000 00000001 [................] 
000160F0 00000000 00000016 0ED4E242 00300001 [...........B.0..] 
00016100 00040000 00000000 00000000 00000000 [................] 
00016110 2E000000 00000000 00000000 00000001 [................] 
00016120 00000000 00000016 9608161C 00300002 [.............0..] 
00016130 00040000 00000000 00000000 00000000 [................] 
00016140 2E2E0000 00000000 00000000 00000011 [................] 
00016150 00000000 000102B4 90530056 0038000B [.........S.V.8..] 
00016160 00040000 00000000 00000000 00000000 [................] 
00016170 6261636B 75705F74 656D7000 00000000 [backup_temp.....] 
00016180 00000000 000147A5 00000000 009FB4AD [......G.........] 
00016190 518E617C 00300003 00040000 00000000 [Q.a|.0..........] 
000161A0 00000000 00000000 6769746B 6D61726B [........gitkmark] 
000161B0 00000000 000084EC 00000000 00010389 [................] 
000161C0 3F2B07AB 00300002 00040000 00000000 [?+...0..........] 
000161D0 00000000 00000000 68670000 00000000 [........hg......] 
000161E0 00000000 0027D27E 00000000 00010381 [.....'.~........] 
000161F0 ED46B67C 00300006 00040000 00000000 [.F.|.0..........] 
00016200 00000000 00000000 77656264 61766500 [........webdave.] 
00016210 00000000 00100D63 00000000 0064FF80 [.......c.....d..] 
00016220 E9DCE904 0038000C 00040000 00000000 [.....8..........] 
00016230 00000000 00000000 65787472 615F6261 [........extra_ba] 
00016240 636B7570 00000000 00000000 00000016 [ckup............] 
00016250 00000000 000102B9 6AB47928 0148000A [........j.y(.H..] 
00016260 00040000 00000000 00000000 00000000 [................] 
00016270 6C6F7374 2B666F75 6E640000 00000000 [lost+found......] 
00016280 00000000 00129E1A 00000000 001A4D84 [..............M.] 
00016290 2D59E83F 00300004 00080000 00000000 [-Y.?.0..........] 
000162A0 00000000 00000000 6C69786F 69652E31 [........lixoie.1] 
000162B0 00000000 00013E20 00000000 00191951 [......> .......Q] 
000162C0 ED46B67C 00E00006 00040000 00000000 [.F.|............] 
000162D0 00000000 00000000 77656264 61760000 [........webdav..] 
000162E0 00000000 00014793 00000000 0090FF04 [......G.........] 
000162F0 7F85435D 00B00018 00040000 00000000 [..C]............] 
00016300 00000000 00000000 70656361 7374726F [........pecastro] 
00016310 40736869 66747261 636B6572 2E636F6D [@shiftracker.com] 
00016320 00000000 00063FCD 01444876 0078000C [......?..DHv.x..] 
00016330 00040000 00000000 00000000 00000000 [................] 
00016340 6261636B 75705F74 656D7031 00000000 [backup_temp1....] 
00016350 70726F6A 65637477 77770000 00000000 [projectwww......] 
00016360 00000000 0000001B 00000000 000102BE [................] 
00016370 9AFC5D97 00300007 00040000 00000000 [..]..0..........] 
00016380 00000000 00000000 7265646D 696E6500 [........redmine.] 
00016390 00000000 0000001C 00000000 000102BF [................] 
000163A0 CF8ADBBC 00680009 00040000 00000000 [.....h..........] 
000163B0 00000000 00000000 73797362 61636B75 [........sysbacku] 
000163C0 70000000 00000000 00000000 0000001D [p...............] 
000163D0 00000000 000102C0 22CDDC06 00300004 [........"....0..] 
000163E0 00040000 00000000 00000000 00000000 [................] 
000163F0 77696B69 00000000 00000000 0000001E [wiki............] 
00016400 00000000 000102C1 C6434ABB 0C080008 [.........CJ.....] 
00016410 00040000 00000000 00000000 00000000 [................] 
00016420 77696E73 68617265 00000000 0000835C [winshare.......\] 
00016430 00000000 00C4DF4F 04D1E886 0BD80006 [.......O........] 
00016440 00080000 00000000 00000000 00000000 [................] 
00016450 7A65726F 73326C6F 677E2E53 66485459 [zeros2log~.SfHTY] 
00016460 00000000 00008351 00000000 000200A9 [.......Q........] 
00016470 0065AD0B 0038000E 00080000 00000000 [.e...8..........] 
00016480 00000000 00000000 2E657272 6F722E6C [.........error.l] 
00016490 6F672E73 77700000 00000000 00008353 [og.swp.........S] 
000164A0 00000000 001B000B 2F5CE050 0B680009 [......../\.P.h..] 
000164B0 00080000 00000000 00000000 00000000 [................] 
000164C0 6572726F 722E6C6F 67672E73 77780000 [error.logg.swx..] 
000164D0 00000000 00000000 00000000 00000000 [................] 
000164E0 00000000 00000000 00000000 00000000 [................] 
000164F0 00000000 00000000 00000000 00000000 [................] 
00016500 00000000 00000000 00000000 00000000 [................] 
00016510 00000000 00000000 00000000 00000000 [................] 
00016520 00000000 00000000 00000000 00000000 [................] 
00016530 00000000 00000000 00000000 00000000 [................] 
00016540 00000000 00000000 00000000 00000000 [................] 
00016550 00000000 00000000 00000000 00000000 [................] 
00016560 00000000 00000000 00000000 00000000 [................] 
00016570 00000000 00000000 00000000 00000000 [................] 
00016580 00000000 00000000 00000000 00000000 [................] 
00016590 00000000 00000000 00000000 00000000 [................] 
000165A0 00000000 00000000 00000000 00000000 [................] 
000165B0 00000000 00000000 00000000 00000000 [................] 
000165C0 00000000 00000000 00000000 00000000 [................] 
000165D0 00000000 00000000 00000000 00000000 [................] 
000165E0 00000000 00000000 00000000 00000000 [................] 
000165F0 00000000 00000000 00000000 00000000 [................] 
00016600 00000000 00000000 00000000 00000000 [................] 
00016610 00000000 00000000 00000000 00000000 [................] 
00016620 00000000 00000000 00000000 00000000 [................] 
00016630 00000000 00000000 00000000 00000000 [................] 
00016640 00000000 00000000 00000000 00000000 [................] 
00016650 00000000 00000000 00000000 00000000 [................] 
00016660 00000000 00000000 00000000 00000000 [................] 
00016670 00000000 00000000 00000000 00000000 [................] 
00016680 00000000 00000000 00000000 00000000 [................] 
00016690 00000000 00000000 00000000 00000000 [................] 
000166A0 00000000 00000000 00000000 00000000 [................] 
000166B0 00000000 00000000 00000000 00000000 [................] 
000166C0 00000000 00000000 00000000 00000000 [................] 
000166D0 00000000 00000000 00000000 00000000 [................] 
000166E0 00000000 00000000 00000000 00000000 [................] 
000166F0 00000000 00000000 00000000 00000000 [................] 
00016700 00000000 00000000 00000000 00000000 [................] 
00016710 00000000 00000000 00000000 00000000 [................] 
00016720 00000000 00000000 00000000 00000000 [................] 
00016730 00000000 00000000 00000000 00000000 [................] 
00016740 00000000 00000000 00000000 00000000 [................] 
00016750 00000000 00000000 00000000 00000000 [................] 
00016760 00000000 00000000 00000000 00000000 [................] 
00016770 00000000 00000000 00000000 00000000 [................] 
00016780 00000000 00000000 00000000 00000000 [................] 
00016790 00000000 00000000 00000000 00000000 [................] 
000167A0 00000000 00000000 00000000 00000000 [................] 
000167B0 00000000 00000000 00000000 00000000 [................] 
000167C0 00000000 00000000 00000000 00000000 [................] 
000167D0 00000000 00000000 00000000 00000000 [................] 
000167E0 00000000 00000000 00000000 00000000 [................] 
000167F0 00000000 00000000 00000000 00000000 [................] 
00016800 00000000 00000000 00000000 00000000 [................] 
00016810 00000000 00000000 00000000 00000000 [................] 
00016820 00000000 00000000 00000000 00000000 [................] 
00016830 00000000 00000000 00000000 00000000 [................] 
00016840 00000000 00000000 00000000 00000000 [................] 
00016850 00000000 00000000 00000000 00000000 [................] 
00016860 00000000 00000000 00000000 00000000 [................] 
00016870 00000000 00000000 00000000 00000000 [................] 
00016880 00000000 00000000 00000000 00000000 [................] 
00016890 00000000 00000000 00000000 00000000 [................] 
000168A0 00000000 00000000 00000000 00000000 [................] 
000168B0 00000000 00000000 00000000 00000000 [................] 
000168C0 00000000 00000000 00000000 00000000 [................] 
000168D0 00000000 00000000 00000000 00000000 [................] 
000168E0 00000000 00000000 00000000 00000000 [................] 
000168F0 00000000 00000000 00000000 00000000 [................] 
00016900 00000000 00000000 00000000 00000000 [................] 
00016910 00000000 00000000 00000000 00000000 [................] 
00016920 00000000 00000000 00000000 00000000 [................] 
00016930 00000000 00000000 00000000 00000000 [................] 
00016940 00000000 00000000 00000000 00000000 [................] 
00016950 00000000 00000000 00000000 00000000 [................] 
00016960 00000000 00000000 00000000 00000000 [................] 
00016970 00000000 00000000 00000000 00000000 [................] 
00016980 00000000 00000000 00000000 00000000 [................] 
00016990 00000000 00000000 00000000 00000000 [................] 
000169A0 00000000 00000000 00000000 00000000 [................] 
000169B0 00000000 00000000 00000000 00000000 [................] 
000169C0 00000000 00000000 00000000 00000000 [................] 
000169D0 00000000 00000000 00000000 00000000 [................] 
000169E0 00000000 00000000 00000000 00000000 [................] 
000169F0 00000000 00000000 00000000 00000000 [................] 
00016A00 00000000 00000000 00000000 00000000 [................] 
00016A10 00000000 00000000 00000000 00000000 [................] 
00016A20 00000000 00000000 00000000 00000000 [................] 
00016A30 00000000 00000000 00000000 00000000 [................] 
00016A40 00000000 00000000 00000000 00000000 [................] 
00016A50 00000000 00000000 00000000 00000000 [................] 
00016A60 00000000 00000000 00000000 00000000 [................] 
00016A70 00000000 00000000 00000000 00000000 [................] 
00016A80 00000000 00000000 00000000 00000000 [................] 
00016A90 00000000 00000000 00000000 00000000 [................] 
00016AA0 00000000 00000000 00000000 00000000 [................] 
00016AB0 00000000 00000000 00000000 00000000 [................] 
00016AC0 00000000 00000000 00000000 00000000 [................] 
00016AD0 00000000 00000000 00000000 00000000 [................] 
00016AE0 00000000 00000000 00000000 00000000 [................] 
00016AF0 00000000 00000000 00000000 00000000 [................] 
00016B00 00000000 00000000 00000000 00000000 [................] 
00016B10 00000000 00000000 00000000 00000000 [................] 
00016B20 00000000 00000000 00000000 00000000 [................] 
00016B30 00000000 00000000 00000000 00000000 [................] 
00016B40 00000000 00000000 00000000 00000000 [................] 
00016B50 00000000 00000000 00000000 00000000 [................] 
00016B60 00000000 00000000 00000000 00000000 [................] 
00016B70 00000000 00000000 00000000 00000000 [................] 
00016B80 00000000 00000000 00000000 00000000 [................] 
00016B90 00000000 00000000 00000000 00000000 [................] 
00016BA0 00000000 00000000 00000000 00000000 [................] 
00016BB0 00000000 00000000 00000000 00000000 [................] 
00016BC0 00000000 00000000 00000000 00000000 [................] 
00016BD0 00000000 00000000 00000000 00000000 [................] 
00016BE0 00000000 00000000 00000000 00000000 [................] 
00016BF0 00000000 00000000 00000000 00000000 [................] 
00016C00 00000000 00000000 00000000 00000000 [................] 
00016C10 00000000 00000000 00000000 00000000 [................] 
00016C20 00000000 00000000 00000000 00000000 [................] 
00016C30 00000000 00000000 00000000 00000000 [................] 
00016C40 00000000 00000000 00000000 00000000 [................] 
00016C50 00000000 00000000 00000000 00000000 [................] 
00016C60 00000000 00000000 00000000 00000000 [................] 
00016C70 00000000 00000000 00000000 00000000 [................] 
00016C80 00000000 00000000 00000000 00000000 [................] 
00016C90 00000000 00000000 00000000 00000000 [................] 
00016CA0 00000000 00000000 00000000 00000000 [................] 
00016CB0 00000000 00000000 00000000 00000000 [................] 
00016CC0 00000000 00000000 00000000 00000000 [................] 
00016CD0 00000000 00000000 00000000 00000000 [................] 
00016CE0 00000000 00000000 00000000 00000000 [................] 
00016CF0 00000000 00000000 00000000 00000000 [................] 
00016D00 00000000 00000000 00000000 00000000 [................] 
00016D10 00000000 00000000 00000000 00000000 [................] 
00016D20 00000000 00000000 00000000 00000000 [................] 
00016D30 00000000 00000000 00000000 00000000 [................] 
00016D40 00000000 00000000 00000000 00000000 [................] 
00016D50 00000000 00000000 00000000 00000000 [................] 
00016D60 00000000 00000000 00000000 00000000 [................] 
00016D70 00000000 00000000 00000000 00000000 [................] 
00016D80 00000000 00000000 00000000 00000000 [................] 
00016D90 00000000 00000000 00000000 00000000 [................] 
00016DA0 00000000 00000000 00000000 00000000 [................] 
00016DB0 00000000 00000000 00000000 00000000 [................] 
00016DC0 00000000 00000000 00000000 00000000 [................] 
00016DD0 00000000 00000000 00000000 00000000 [................] 
00016DE0 00000000 00000000 00000000 00000000 [................] 
00016DF0 00000000 00000000 00000000 00000000 [................] 
00016E00 00000000 00000000 00000000 00000000 [................] 
00016E10 00000000 00000000 00000000 00000000 [................] 
00016E20 00000000 00000000 00000000 00000000 [................] 
00016E30 00000000 00000000 00000000 00000000 [................] 
00016E40 00000000 00000000 00000000 00000000 [................] 
00016E50 00000000 00000000 00000000 00000000 [................] 
00016E60 00000000 00000000 00000000 00000000 [................] 
00016E70 00000000 00000000 00000000 00000000 [................] 
00016E80 00000000 00000000 00000000 00000000 [................] 
00016E90 00000000 00000000 00000000 00000000 [................] 
00016EA0 00000000 00000000 00000000 00000000 [................] 
00016EB0 00000000 00000000 00000000 00000000 [................] 
00016EC0 00000000 00000000 00000000 00000000 [................] 
00016ED0 00000000 00000000 00000000 00000000 [................] 
00016EE0 00000000 00000000 00000000 00000000 [................] 
00016EF0 00000000 00000000 00000000 00000000 [................] 
00016F00 00000000 00000000 00000000 00000000 [................] 
00016F10 00000000 00000000 00000000 00000000 [................] 
00016F20 00000000 00000000 00000000 00000000 [................] 
00016F30 00000000 00000000 00000000 00000000 [................] 
00016F40 00000000 00000000 00000000 00000000 [................] 
00016F50 00000000 00000000 00000000 00000000 [................] 
00016F60 00000000 00000000 00000000 00000000 [................] 
00016F70 00000000 00000000 00000000 00000000 [................] 
00016F80 00000000 00000000 00000000 00000000 [................] 
00016F90 00000000 00000000 00000000 00000000 [................] 
00016FA0 00000000 00000000 00000000 00000000 [................] 
00016FB0 00000000 00000000 00000000 00000000 [................] 
00016FC0 00000000 00000000 00000000 00000000 [................] 
00016FD0 00000000 00000000 00000000 00000000 [................] 
00016FE0 00000000 00000000 00000000 00000000 [................] 
00016FF0 00000000 00000000 00000000 00000000 [................] 

This directory block contains 10 directory entries.
     1. (1). 22 (0x16): Dir     .
     2. (2). 22 (0x16): Dir     ..
     3. (3). 66228 (0x102b4): Dir     backup_temp
     4. (4). 10466477 (0x9fb4ad): Dir     git
     5. (5). 66441 (0x10389): Dir     hg
     6. (6). 66433 (0x10381): Dir     webdav
     7. (7). 6619008 (0x64ff80): Dir     extra_backup
     8. (8). 66233 (0x102b9): Dir     lost+found
     9. (9). 66239 (0x102bf): Dir     sysbackup
     10. (10). 66241 (0x102c1): Dir     winshare
------------------------------------------------------
Block #19266892    (0x125fd4c) of 19274154 (0x12619AA) (disk inode)

Dinode:
  mh_magic              0x01161970(hex)
  mh_type               4                   0x4
  mh_format             400                 0x190
  no_formal_ino         2567057             0x272b91
  no_addr               19266892            0x125fd4c
  di_mode               040700(decimal)
  di_uid                500                 0x1f4
  di_gid                500                 0x1f4
  di_nlink              1                   0x1
  di_size               3864                0xf18
  di_blocks             1                   0x1
  di_atime              1275508758          0x4c06b816
  di_mtime              1254703036          0x4ac93fbc
  di_ctime              1275264546          0x4c02fe22
  di_major              0                   0x0
  di_minor              0                   0x0
  di_goal_meta          19266892            0x125fd4c
  di_goal_data          19266892            0x125fd4c
  di_flags              0x00000001(hex)
  di_payload_format     1200                0x4b0
  di_height             0                   0x0
  di_depth              0                   0x0
  di_entries            10                  0xa
  di_eattr              0                   0x0

This directory block contains 2 directory entries.
     1. (1). 19266892 (0x125fd4c): Dir     .
     2. (2). 19266875 (0x125fd3b): Dir     ..
------------------------------------------------------

Comment 7 Robert Peterson 2010-06-15 21:44:48 UTC

I don't see any apparent corruption in the entries you posted.
However, I did notice that your GFS2 root dinode has a
lost+found directory.  That's almost always created by fsck.gfs2.
Not long ago, I discovered some major problems with how fsck.gfs2
manages the contents of the lost+found directory.  Specifically,
it was not always marking down blocks allocated to lost+found.
For example, it did this in cases where a file was moved to
lost+found, if the new directory entry inside lost+found didn't
fit and therefore it needed a new block allocation.
Therefore, blocks allocated to lost+found were considered to be
still free blocks and further use of the file system could allocate
those blocks for a second purpose.  Of course, this led to file
system corruption.

I can't guarantee that the lost+found problem I described is
what's wrong with your file system, but I'd say it's likely.

I highly recommend you download an experimental new fsck.gfs2 from
this link and run it:

http://people.redhat.com/rpeterso/Experimental/RHEL5.x/gfs2/fsck.gfs2

This version of fsck.gfs2 is not only much faster than its
predecessors, it also is much better at identifying problems and
fixing them.  It's also less destructive when fixing problems.
It will hopefully fix the corruption in your file system.

Please let me know if it doesn't solve the problem.

Comment 8 Robert Peterson 2010-06-15 21:56:23 UTC

Incidentally, the aforementioned problem is documented by, and
will be fixed via bug #455300. It is tentatively queued to go out
in RHEL5.6.  That version includes seventy (70) fixes to fsck.gfs2,
many of which are performance related.  Since that bug record was
generated internally by Red Hat, I don't know if the permission
flags will allow you to view it.  If you can't, I'm sorry but
I can't do anything about that.  At any rate, all of the
patches that went into the experimental version of fsck.gfs2 are
documented in the upstream for gfs2-utils, which can be accessed
publicly here:

http://git.fedoraproject.org/git/?p=cluster.git

Also, the experimental version has been tested extensively on
dozens of severely damaged file systems, and by several actual
gfs2 users.

Comment 9 Bug Zapper 2010-06-28 15:45:03 UTC

Fedora 11 changed to end-of-life (EOL) status on 2010-06-25. Fedora 11 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 10 Paulo Castro 2014-06-16 12:44:47 UTC

This has stopped being an issue.