Bug 232019

Created attachment 149962 [details]
patch to fix the problem

This was a simple case of the buffer release code getting called twice.
I should probably try to walk through the code and determine if there
are any others like it.

ok after i run the fsck, i'm still getting an invalid metadata assertion when 
doing an rm -rf on the directory (without anything happening on the other 
node)

GFS2: fsid=rhel5cluster:gfs2lv1.0: fatal: invalid metadata block
GFS2: fsid=rhel5cluster:gfs2lv1.0:   bh = 1753066 (type: exp=10, found=8)
GFS2: fsid=rhel5cluster:gfs2lv1.0:   function = ea_foreach_i, file = 
fs/gfs2/eattr.c, line = 80
GFS2: fsid=rhel5cluster:gfs2lv1.0: about to withdraw this file system
GFS2: fsid=rhel5cluster:gfs2lv1.0: telling LM to withdraw
GFS2: fsid=rhel5cluster:gfs2lv1.0: withdrawn
 [<f8beeb4b>] gfs2_lm_withdraw+0x82/0x8d [gfs2]
 [<f8bffe6e>] gfs2_metatype_check_ii+0x5e/0x6b [gfs2]
 [<f8be7cfe>] ea_foreach_i+0x86/0x120 [gfs2]
 [<f8be7dfc>] ea_foreach+0x64/0x1ad [gfs2]
 [<f8be85ff>] ea_dealloc_unstuffed+0x0/0x362 [gfs2]
 [<f8be7fdc>] gfs2_ea_dealloc+0x60/0x683 [gfs2]
 [<c04224d7>] __cond_resched+0x16/0x34
 [<f8bebd37>] glock_wait_internal+0x1e5/0x1f7 [gfs2]
 [<f8bebebb>] gfs2_glock_nq+0x172/0x1a6 [gfs2]
 [<f8bf8776>] gfs2_delete_inode+0xdc/0x190 [gfs2]
 [<c0490cbb>] inotify_inode_is_dead+0x18/0x6c
 [<f8bf86de>] gfs2_delete_inode+0x44/0x190 [gfs2]
 [<f8bf869a>] gfs2_delete_inode+0x0/0x190 [gfs2]
 [<c047ecbd>] generic_delete_inode+0xa1/0x107
 [<c047e4a6>] iput+0x60/0x62
 [<c0477108>] do_unlinkat+0xae/0x119
 [<c0407975>] do_syscall_trace+0x124/0x16b
 [<c0404e4c>] syscall_call+0x7/0xb
 [<c0610000>] rt_mutex_slowunlock+0x89/0x197
 =======================
GFS2: fsid=rhel5cluster:gfs2lv1.0: gfs2_delete_inode: -5

The type 10 is an EA block, so its looking for EAs which are unstuffed from the
inode. Type 8 is a log header, which is really odd as that shouldn't be seen
anywhere on the disk outside of the log itself.

For some reason, dinode 0x1ac558 has a non-zero di_eattr pointer to
block 0x1ac559.  However, block 0x1ac559 is not an eattr block.
It looks like a journal log entry.  The inode's di_flags are == 0,
so it doesn't have the GFS2_DIF_EA_INDIRECT attribute, which is odd.
There was a second inode, 0x1ac560, in the same shape, pointing to
0x1ac561, which was also a log header and not an ea.

I compared gfs2's eattr.c to gfs1's eattr.c.  There were lots of
differences, but I didn't find any major problems.  Most were cosmetic.
The fsck code is very close to the original gfs1 version.

Josef said he just used cp -r to copy a kernel tree out there, then he
did an rm -fR * to delete it all, which is where he had problems.

I partially ran an gfs2_fsck on another fs that josef had, with his 
new copy of the kernel source tree.  There were no eattr problems 
reported there by fsck.

I verified that there are no more occurrences of double-brelse as
promised in comment #1.

well if you do an rm -rf and then on another node do a du you get a withdrawl,
and then the next time around you have this fs corruption.

Created attachment 150267 [details]
patch to fix the problem [try 2]

Josef had several problems: (1) There was the original assert from
gfs2, which is out of the scope of this bugzilla record.  (2) As 
stated in comment #4, Josef's file system had some disk inodes that
had a di_eattr pointer to a block that wasn't an eattr.  That caused
gfs2_fsck to enter a little-used code path that had a double-free,
which caused fsck to abort abnormally, as noted in comment #1.
The original patch fixed that problem, allowing the code to get through
the file system completely, but as the description states, (3) gfs2_fsck
still didn't fix the corruption.

With the previous patch in place, when the bad blocks were encountered,
the code was setting the block type to "invalid" then sending back a 
return code of 1.  Later, it was  zeroing the di_eattr block number only
if the return code < 0.  Later still, in pass1c, it was running through
all the extended attributes, trying to fix what it could.  However, this
block wasn't getting processed by pass1c because it was no longer marked
as an eattr block (by that time, it was an "invalid" block).  Therefore, 
the di_eattr pointer was never fixed, and therefore, the corruption 
remained.

This version of the patch returns a -1 return code instead.  That
causes the di_eattr block to be set to 0, which is about all we can do
in this case.

I ran this on Josef's file system and it correctly set the di_eattr
pointer to zero.  A subsequent fsck had no issues with the inode then.

I have two additional concerns: (1) gfs v1's gfs_fsck has this same
problem.  Since no one has reported it until gfs2, I have to believe
this is a rare kind of corruption for gfs1.  So do we crosswrite the fix?
If so, where?  RHEL4.6?  RHEL5.1?  It's an easy fix, but we'd
have to open another bugzilla record to do it.
(2) When the EA damage is fixed, gfs2_fsck doesn't ask the user for
permission to repair; it just goes ahead and does it.  For most kinds of
corruption, fsck doesn't mess with your file system unless it first asks
nicely for permission.	However, I'm seeing a bunch of cases now where
it doesn't ask.  It would be easy enough to add a user query for 
permission.  But what is the current user's expectations in this regard?
I always thought that all the fsck's would leave your file system alone
unless you answered a yes/no question (unless overridden by -y or -n).
So what's the "right" thing to do here?

Fix committed to HEAD and RHEL5 branches in cvs.  Setting status to
modified.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2007-0579.html