Bug 232019 - gfs2_fsck doesn't fix an ea problem.
Summary: gfs2_fsck doesn't fix an ea problem.
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: gfs2-utils
Version: 5.1
Hardware: All
OS: Linux
Target Milestone: ---
: ---
Assignee: Robert Peterson
QA Contact: Dean Jansa
Depends On:
TreeView+ depends on / blocked
Reported: 2007-03-13 16:07 UTC by Josef Bacik
Modified: 2010-01-12 03:37 UTC (History)
3 users (show)

Fixed In Version: RHBA-2007-0579
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2007-11-07 18:04:40 UTC
Target Upstream Version:

Attachments (Terms of Use)
patch to fix the problem (550 bytes, patch)
2007-03-13 17:55 UTC, Robert Peterson
no flags Details | Diff
patch to fix the problem [try 2] (857 bytes, patch)
2007-03-16 18:24 UTC, Robert Peterson
no flags Details | Diff

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2007:0579 0 normal SHIPPED_LIVE gfs2-utils bug fix update 2007-11-08 14:13:05 UTC

Description Josef Bacik 2007-03-13 16:07:59 UTC
i got an assert in bz 232010 and I went to run gfs2_fsck on the filesystem and 
I got this

[root@rh5cluster1 ~]# gfs2_fsck /dev/gfs2vg/gfs2lv1 
Initializing fsck
Clearing journals (this may take a while)..
Journals cleared.
Starting pass1
EA leaf block has incorrect type.
gfs2_fsck: buffer count underflow for block 1753066 (0x1abfea)

Since I can't do anything until this is fixed I'll try and figure out whats 
wrong with gfs2_fsck.

Comment 1 Robert Peterson 2007-03-13 17:55:31 UTC
Created attachment 149962 [details]
patch to fix the problem

This was a simple case of the buffer release code getting called twice.
I should probably try to walk through the code and determine if there
are any others like it.

Comment 2 Josef Bacik 2007-03-13 20:44:43 UTC
ok after i run the fsck, i'm still getting an invalid metadata assertion when 
doing an rm -rf on the directory (without anything happening on the other 

GFS2: fsid=rhel5cluster:gfs2lv1.0: fatal: invalid metadata block
GFS2: fsid=rhel5cluster:gfs2lv1.0:   bh = 1753066 (type: exp=10, found=8)
GFS2: fsid=rhel5cluster:gfs2lv1.0:   function = ea_foreach_i, file = 
fs/gfs2/eattr.c, line = 80
GFS2: fsid=rhel5cluster:gfs2lv1.0: about to withdraw this file system
GFS2: fsid=rhel5cluster:gfs2lv1.0: telling LM to withdraw
GFS2: fsid=rhel5cluster:gfs2lv1.0: withdrawn
 [<f8beeb4b>] gfs2_lm_withdraw+0x82/0x8d [gfs2]
 [<f8bffe6e>] gfs2_metatype_check_ii+0x5e/0x6b [gfs2]
 [<f8be7cfe>] ea_foreach_i+0x86/0x120 [gfs2]
 [<f8be7dfc>] ea_foreach+0x64/0x1ad [gfs2]
 [<f8be85ff>] ea_dealloc_unstuffed+0x0/0x362 [gfs2]
 [<f8be7fdc>] gfs2_ea_dealloc+0x60/0x683 [gfs2]
 [<c04224d7>] __cond_resched+0x16/0x34
 [<f8bebd37>] glock_wait_internal+0x1e5/0x1f7 [gfs2]
 [<f8bebebb>] gfs2_glock_nq+0x172/0x1a6 [gfs2]
 [<f8bf8776>] gfs2_delete_inode+0xdc/0x190 [gfs2]
 [<c0490cbb>] inotify_inode_is_dead+0x18/0x6c
 [<f8bf86de>] gfs2_delete_inode+0x44/0x190 [gfs2]
 [<f8bf869a>] gfs2_delete_inode+0x0/0x190 [gfs2]
 [<c047ecbd>] generic_delete_inode+0xa1/0x107
 [<c047e4a6>] iput+0x60/0x62
 [<c0477108>] do_unlinkat+0xae/0x119
 [<c0407975>] do_syscall_trace+0x124/0x16b
 [<c0404e4c>] syscall_call+0x7/0xb
 [<c0610000>] rt_mutex_slowunlock+0x89/0x197
GFS2: fsid=rhel5cluster:gfs2lv1.0: gfs2_delete_inode: -5

Comment 3 Steve Whitehouse 2007-03-14 09:20:31 UTC
The type 10 is an EA block, so its looking for EAs which are unstuffed from the
inode. Type 8 is a log header, which is really odd as that shouldn't be seen
anywhere on the disk outside of the log itself.

Comment 4 Robert Peterson 2007-03-14 21:42:59 UTC
For some reason, dinode 0x1ac558 has a non-zero di_eattr pointer to
block 0x1ac559.  However, block 0x1ac559 is not an eattr block.
It looks like a journal log entry.  The inode's di_flags are == 0,
so it doesn't have the GFS2_DIF_EA_INDIRECT attribute, which is odd.
There was a second inode, 0x1ac560, in the same shape, pointing to
0x1ac561, which was also a log header and not an ea.

I compared gfs2's eattr.c to gfs1's eattr.c.  There were lots of
differences, but I didn't find any major problems.  Most were cosmetic.
The fsck code is very close to the original gfs1 version.

Josef said he just used cp -r to copy a kernel tree out there, then he
did an rm -fR * to delete it all, which is where he had problems.

I partially ran an gfs2_fsck on another fs that josef had, with his 
new copy of the kernel source tree.  There were no eattr problems 
reported there by fsck.

Comment 5 Robert Peterson 2007-03-14 22:16:29 UTC
I verified that there are no more occurrences of double-brelse as
promised in comment #1.

Comment 6 Josef Bacik 2007-03-14 22:23:38 UTC
well if you do an rm -rf and then on another node do a du you get a withdrawl,
and then the next time around you have this fs corruption.

Comment 7 Robert Peterson 2007-03-16 18:24:30 UTC
Created attachment 150267 [details]
patch to fix the problem [try 2]

Josef had several problems: (1) There was the original assert from
gfs2, which is out of the scope of this bugzilla record.  (2) As 
stated in comment #4, Josef's file system had some disk inodes that
had a di_eattr pointer to a block that wasn't an eattr.  That caused
gfs2_fsck to enter a little-used code path that had a double-free,
which caused fsck to abort abnormally, as noted in comment #1.
The original patch fixed that problem, allowing the code to get through
the file system completely, but as the description states, (3) gfs2_fsck
still didn't fix the corruption.

With the previous patch in place, when the bad blocks were encountered,
the code was setting the block type to "invalid" then sending back a 
return code of 1.  Later, it was  zeroing the di_eattr block number only
if the return code < 0.  Later still, in pass1c, it was running through
all the extended attributes, trying to fix what it could.  However, this
block wasn't getting processed by pass1c because it was no longer marked
as an eattr block (by that time, it was an "invalid" block).  Therefore, 
the di_eattr pointer was never fixed, and therefore, the corruption 

This version of the patch returns a -1 return code instead.  That
causes the di_eattr block to be set to 0, which is about all we can do
in this case.

I ran this on Josef's file system and it correctly set the di_eattr
pointer to zero.  A subsequent fsck had no issues with the inode then.

I have two additional concerns: (1) gfs v1's gfs_fsck has this same
problem.  Since no one has reported it until gfs2, I have to believe
this is a rare kind of corruption for gfs1.  So do we crosswrite the fix?
If so, where?  RHEL4.6?  RHEL5.1?  It's an easy fix, but we'd
have to open another bugzilla record to do it.
(2) When the EA damage is fixed, gfs2_fsck doesn't ask the user for
permission to repair; it just goes ahead and does it.  For most kinds of
corruption, fsck doesn't mess with your file system unless it first asks
nicely for permission.	However, I'm seeing a bunch of cases now where
it doesn't ask.  It would be easy enough to add a user query for 
permission.  But what is the current user's expectations in this regard?
I always thought that all the fsck's would leave your file system alone
unless you answered a yes/no question (unless overridden by -y or -n).
So what's the "right" thing to do here?

Comment 8 Robert Peterson 2007-03-26 19:17:42 UTC
Fix committed to HEAD and RHEL5 branches in cvs.  Setting status to

Comment 11 errata-xmlrpc 2007-11-07 18:04:40 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.