Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
+++ This bug was initially created as a clone of Bug #624691 +++
--- Additional comment from nstraz on 2010-08-25 18:37:02 EDT ---
Created attachment 441066[details]
fsck.gfs2 core dump
I'm trying to verify this but I'm think I'm missing something from Bob's description. I did the mkfs w/ 512 byte block size:
mkfs -t gfs2 -O -b 512 -j 1 -p lock_dlm -t west:brawl0 /dev/brawl/brawl0
/dev/brawl/brawl0 is a 1G LV.
Then I ran a loop to fill a directory with entries
while touch bigdir/a_long_filename_to_make_quick_use_of_dentry_blocks_$i; do
let i=i+1
if ((i % 10000 == 0)); then
echo "$i"
height=`gfs2_edit -p $bigdir_ino field di_height /dev/brawl/brawl0`
if [ "$height" -eq 4 ]; then
break
fi
fi
done
This loop ran until the file system was filled and did not reach a di_height of 4.
...
760000
touch: cannot touch `bigdir/a_long_filename_to_make_quick_use_of_dentry_blocks_767252': No space left on device
[root@west-01 ~]# fsck.gfs2 -y /dev/brawl/brawl0; echo $?
Initializing fsck
Validating Resource Group index.
Level 1 RG check.
(level 1 passed)
Starting pass1
Pass1 complete
Starting pass1b
Pass1b complete
Starting pass1c
Pass1c completelete.
Starting pass2
Entries is 767253 - should be 442972 for inode block 269009 (0x41ad1)
Pass2 complete
Starting pass3
Pass3 complete
Starting pass4
Found unlinked inode at 508636 (0x7c2dc)
Added inode #508636 (0x7c2dc) to lost+found dir
Found unlinked inode at 508665 (0x7c2f9)
Added inode #508665 (0x7c2f9) to lost+found dir
Found unlinked inode at 519865 (0x7eeb9)
Added inode #519865 (0x7eeb9) to lost+found dir
Found unlinked inode at 520069 (0x7ef85)
Segmentation fault (core dumped)
139
<abrt deleted core dump>
rerunning fsck.gfs2
[root@west-01 ~]# fsck.gfs2 -y /dev/brawl/brawl0; echo $?
Initializing fsck
Validating Resource Group index.
Level 1 RG check.
(level 1 passed)
Starting pass1
Pass1 complete
Starting pass1b
Pass1b complete
Starting pass1c
Pass1c completelete.
Starting pass2
Pass2 complete
Starting pass3
Pass3 complete
Starting pass4
Found unlinked inode at 508636 (0x7c2dc)
Added inode #508636 (0x7c2dc) to lost+found dir
Found unlinked inode at 508665 (0x7c2f9)
Added inode #508665 (0x7c2f9) to lost+found dir
Found unlinked inode at 519865 (0x7eeb9)
Added inode #519865 (0x7eeb9) to lost+found dir
Found unlinked inode at 520069 (0x7ef85)
Segmentation fault (core dumped)
#0 bwrite (bh=0xcc03070000000000)
at /usr/src/debug/cluster-3.0.12/gfs2/libgfs2/buf.c:63
#1 0x000000000041d80d in dir_make_exhash (dip=0x534c1d0,
filename=0x7fff14898640 "lost_file_520069", len=16, inum=0xd0d2e0, type=8)
at /usr/src/debug/cluster-3.0.12/gfs2/libgfs2/fs_ops.c:1193
#2 dir_l_add (dip=0x534c1d0, filename=0x7fff14898640 "lost_file_520069",
len=16, inum=0xd0d2e0, type=8)
at /usr/src/debug/cluster-3.0.12/gfs2/libgfs2/fs_ops.c:1202
#3 dir_add (dip=0x534c1d0, filename=0x7fff14898640 "lost_file_520069",
len=16, inum=0xd0d2e0, type=8)
at /usr/src/debug/cluster-3.0.12/gfs2/libgfs2/fs_ops.c:1220
#4 0x0000000000406c3d in add_inode_to_lf (ip=0xd0d2c0)
at /usr/src/debug/cluster-3.0.12/gfs2/fsck/lost_n_found.c:168
#5 0x000000000041549c in scan_inode_list (sbp=0x7fff14898820)
at /usr/src/debug/cluster-3.0.12/gfs2/fsck/pass4.c:130
#6 pass4 (sbp=0x7fff14898820)
at /usr/src/debug/cluster-3.0.12/gfs2/fsck/pass4.c:195
#7 0x0000000000407a27 in main (argc=<value optimized out>,
argv=<value optimized out>)
at /usr/src/debug/cluster-3.0.12/gfs2/fsck/main.c:311
--- Additional comment from nstraz on 2010-08-27 11:20:59 EDT ---
I reproduced this again w/o running out of space on the device. Once I get above 500,000 entries it truncates the directory and segfaults while moving stuff into lost+found.
[root@morph-04 ~]# fsck.gfs2 -y /dev/morph-cluster/bigdir
Initializing fsck
Validating Resource Group index.
Level 1 RG check.
(level 1 passed)
Starting pass1
Pass1 complete
Starting pass1b
Pass1b complete
Starting pass1c
Pass1c completeete..
Starting pass2
root inode 398 (0x18e): Entries is 516033 - should be 408582
Entries updated
Pass2 complete
Starting pass3
Pass3 complete
Starting pass4
Found unlinked inode at 377457 (0x5c271)
Added inode #377457 (0x5c271) to lost+found dir
Found unlinked inode at 377486 (0x5c28e)
Added inode #377486 (0x5c28e) to lost+found dir
Found unlinked inode at 388686 (0x5ee4e)
Added inode #388686 (0x5ee4e) to lost+found dir
Found unlinked inode at 388890 (0x5ef1a)
Segmentation fault (core dumped)
I re-ran the test running fsck.gfs2 every 10,000 entries added and hit this after passing 110,000 entries:
110000 11:38:41
Initializing fsck
Validating Resource Group index.
Level 1 RG check.
(level 1 passed)
Starting pass1
Pass1 complete
Starting pass1b
Pass1b complete
Starting pass1c
Pass1c completeete.
Starting pass2
Entries is 110002 - should be 109985 for inode block 137843 (0x21a73)
Pass2 complete
Starting pass3
Pass3 complete
Starting pass4
Found unlinked inode at 377458 (0x5c272)
Added inode #377458 (0x5c272) to lost+found dir
Found unlinked inode at 377487 (0x5c28f)
Added inode #377487 (0x5c28f) to lost+found dir
Found unlinked inode at 388687 (0x5ee4f)
Added inode #388687 (0x5ee4f) to lost+found dir
Found unlinked inode at 388891 (0x5ef1b)
/tmp/bz624691.sh: line 40: 17491 Segmentation fault (core dumped) fsck.gfs2 -y /dev/morph-cluster/bigdir
fsck.gfs2 exitted w/ code 0
Created attachment 442037[details]
gfs2_edit enhancement (continuation blocks)
I recreated this problem and figured out what was going on.
I had to enhance gfs2_edit a bit to figure this out,
otherwise it was searching for a needle in a haystack:
trying to track down 18 missing dirents in an sea of 110,000
dirents. This patch is the enhancement to gfs2_edit that
allowed me to figure out what's going on.
What's going on is this: Apparently fsck.gfs2 is ignoring
continuation leaf blocks (aka lf_next) in pass2 where it's
counting directory references. The gfs2_edit tool was also
ignoring them, thus the need for the enhancement.
Now I have to figure out why fsck.gfs2 is ignoring them and
what we can do about it. The good news is that the kernel
code is apparently doing things correctly.
This obviously has to be fixed, so I'm changing the status
to ASSIGNED and requesting ack flags. I should investigate
whether gfs1's gfs_fsck has the same issue. I bet it does,
but that will become apparent soon enough. I don't have a
fix yet, but I expect to have a fix tomorrow.
Created attachment 442201[details]
first go at a patch
This patch fixes the failing scenario. However, it needs a
lot more testing. I may need to move the statement
"ref_count = 1;" inside the loop as well. I won't find that
out until I look at the code some more and run the patch
through a couple of the complex metadata sets. So this may
not be the final version.
Created attachment 442432[details]
Revised patch
This is a revised patch. It fixes the failing scenario, and
I'm in the process of regression testing the RHEL5.6 crosswrite
against my collection of metadata.
This patch was tested on RHEL6 system west-10. The patch was
pushed to the master branch of the gfs2-utils git tree, and
the STABLE3 and RHEL6 branches of the cluster git tree for
inclusion into RHEL6.1. Changing status to POST until it
gets built into a 6.1 cluster package.
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.
http://rhn.redhat.com/errata/RHBA-2011-0537.html