RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 628013 - fsck.gfs2 truncates directories with more than 100,000 entries
Summary: fsck.gfs2 truncates directories with more than 100,000 entries
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: cluster
Version: 6.0
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: 6.0
Assignee: Fabio Massimo Di Nitto
QA Contact: Cluster QE
URL:
Whiteboard:
Depends On: 676626 724921
Blocks: 629010
TreeView+ depends on / blocked
 
Reported: 2010-08-27 16:40 UTC by Nate Straz
Modified: 2011-07-22 12:33 UTC (History)
14 users (show)

Fixed In Version: cluster-3.0.12-24.el6
Doc Type: Bug Fix
Doc Text:
Clone Of: 624691
: 629010 (view as bug list)
Environment:
Last Closed: 2011-05-19 12:53:50 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
fsck.gfs2 core dump (52 bytes, text/plain)
2010-08-27 16:47 UTC, Nate Straz 		no flags Details
gfs2_edit enhancement (continuation blocks) (8.28 KB, patch)
2010-08-30 21:06 UTC, Robert Peterson 		no flags Details | Diff
first go at a patch (1.28 KB, patch)
2010-08-31 14:38 UTC, Robert Peterson 		no flags Details | Diff
Revised patch (6.05 KB, patch)
2010-09-01 15:06 UTC, Robert Peterson 		no flags Details | Diff
Final patch for 6.1 (11.64 KB, patch)
2010-09-17 18:33 UTC, Robert Peterson 		no flags Details | Diff
Show Obsolete (2) View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2011:0537 0 normal SHIPPED_LIVE cluster and gfs2-utils bug fix update 2011-05-18 17:57:40 UTC

Description Nate Straz 2010-08-27 16:40:23 UTC 
+++ This bug was initially created as a clone of Bug #624691 +++

--- Additional comment from nstraz on 2010-08-25 18:37:02 EDT ---

Created attachment 441066 [details]
fsck.gfs2 core dump

I'm trying to verify this but I'm think I'm missing something from Bob's description.  I did the mkfs w/ 512 byte block size:

mkfs -t gfs2 -O -b 512 -j 1 -p lock_dlm -t west:brawl0 /dev/brawl/brawl0

/dev/brawl/brawl0 is a 1G LV.

Then I ran a loop to fill a directory with entries

while touch bigdir/a_long_filename_to_make_quick_use_of_dentry_blocks_$i; do 
	let i=i+1
	if ((i % 10000 == 0)); then 
		echo "$i"
		height=`gfs2_edit -p $bigdir_ino field di_height /dev/brawl/brawl0`
		if [ "$height" -eq 4 ]; then 
			break
		fi
	fi
done

This loop ran until the file system was filled and did not reach a di_height of 4.
...
760000
touch: cannot touch `bigdir/a_long_filename_to_make_quick_use_of_dentry_blocks_767252': No space left on device

[root@west-01 ~]# fsck.gfs2 -y /dev/brawl/brawl0; echo $?
Initializing fsck
Validating Resource Group index.
Level 1 RG check.
(level 1 passed)
Starting pass1
Pass1 complete      
Starting pass1b
Pass1b complete
Starting pass1c
Pass1c completelete.
Starting pass2
Entries is 767253 - should be 442972 for inode block 269009 (0x41ad1)
Pass2 complete      
Starting pass3
Pass3 complete      
Starting pass4
Found unlinked inode at 508636 (0x7c2dc)
Added inode #508636 (0x7c2dc) to lost+found dir
Found unlinked inode at 508665 (0x7c2f9)
Added inode #508665 (0x7c2f9) to lost+found dir
Found unlinked inode at 519865 (0x7eeb9)
Added inode #519865 (0x7eeb9) to lost+found dir
Found unlinked inode at 520069 (0x7ef85)
Segmentation fault (core dumped)
139

<abrt deleted core dump>

rerunning fsck.gfs2
[root@west-01 ~]# fsck.gfs2 -y /dev/brawl/brawl0; echo $?
Initializing fsck
Validating Resource Group index.
Level 1 RG check.
(level 1 passed)
Starting pass1
Pass1 complete      
Starting pass1b
Pass1b complete
Starting pass1c
Pass1c completelete.
Starting pass2
Pass2 complete      
Starting pass3
Pass3 complete      
Starting pass4
Found unlinked inode at 508636 (0x7c2dc)
Added inode #508636 (0x7c2dc) to lost+found dir
Found unlinked inode at 508665 (0x7c2f9)
Added inode #508665 (0x7c2f9) to lost+found dir
Found unlinked inode at 519865 (0x7eeb9)
Added inode #519865 (0x7eeb9) to lost+found dir
Found unlinked inode at 520069 (0x7ef85)
Segmentation fault (core dumped)

#0  bwrite (bh=0xcc03070000000000)
    at /usr/src/debug/cluster-3.0.12/gfs2/libgfs2/buf.c:63
#1  0x000000000041d80d in dir_make_exhash (dip=0x534c1d0, 
    filename=0x7fff14898640 "lost_file_520069", len=16, inum=0xd0d2e0, type=8)
    at /usr/src/debug/cluster-3.0.12/gfs2/libgfs2/fs_ops.c:1193
#2  dir_l_add (dip=0x534c1d0, filename=0x7fff14898640 "lost_file_520069", 
    len=16, inum=0xd0d2e0, type=8)
    at /usr/src/debug/cluster-3.0.12/gfs2/libgfs2/fs_ops.c:1202
#3  dir_add (dip=0x534c1d0, filename=0x7fff14898640 "lost_file_520069", 
    len=16, inum=0xd0d2e0, type=8)
    at /usr/src/debug/cluster-3.0.12/gfs2/libgfs2/fs_ops.c:1220
#4  0x0000000000406c3d in add_inode_to_lf (ip=0xd0d2c0)
    at /usr/src/debug/cluster-3.0.12/gfs2/fsck/lost_n_found.c:168
#5  0x000000000041549c in scan_inode_list (sbp=0x7fff14898820)
    at /usr/src/debug/cluster-3.0.12/gfs2/fsck/pass4.c:130
#6  pass4 (sbp=0x7fff14898820)
    at /usr/src/debug/cluster-3.0.12/gfs2/fsck/pass4.c:195
#7  0x0000000000407a27 in main (argc=<value optimized out>, 
    argv=<value optimized out>)
    at /usr/src/debug/cluster-3.0.12/gfs2/fsck/main.c:311

--- Additional comment from nstraz on 2010-08-27 11:20:59 EDT ---

I reproduced this again w/o running out of space on the device.  Once I get above 500,000 entries it truncates the directory and segfaults while moving stuff into lost+found.

[root@morph-04 ~]# fsck.gfs2 -y /dev/morph-cluster/bigdir
Initializing fsck
Validating Resource Group index.
Level 1 RG check.
(level 1 passed)
Starting pass1
Pass1 complete
Starting pass1b
Pass1b complete
Starting pass1c
Pass1c completeete..
Starting pass2
root inode 398 (0x18e): Entries is 516033 - should be 408582
Entries updated
Pass2 complete
Starting pass3
Pass3 complete
Starting pass4
Found unlinked inode at 377457 (0x5c271)
Added inode #377457 (0x5c271) to lost+found dir
Found unlinked inode at 377486 (0x5c28e)
Added inode #377486 (0x5c28e) to lost+found dir
Found unlinked inode at 388686 (0x5ee4e)
Added inode #388686 (0x5ee4e) to lost+found dir
Found unlinked inode at 388890 (0x5ef1a)
Segmentation fault (core dumped)
Comment 1 Nate Straz 2010-08-27 16:42:59 UTC 
I re-ran the test running fsck.gfs2 every 10,000 entries added and hit this after passing 110,000 entries:

110000 11:38:41
Initializing fsck
Validating Resource Group index.
Level 1 RG check.
(level 1 passed)
Starting pass1
Pass1 complete
Starting pass1b
Pass1b complete
Starting pass1c
Pass1c completeete.
Starting pass2
Entries is 110002 - should be 109985 for inode block 137843 (0x21a73)
Pass2 complete
Starting pass3
Pass3 complete
Starting pass4
Found unlinked inode at 377458 (0x5c272)
Added inode #377458 (0x5c272) to lost+found dir
Found unlinked inode at 377487 (0x5c28f)
Added inode #377487 (0x5c28f) to lost+found dir
Found unlinked inode at 388687 (0x5ee4f)
Added inode #388687 (0x5ee4f) to lost+found dir
Found unlinked inode at 388891 (0x5ef1b)
/tmp/bz624691.sh: line 40: 17491 Segmentation fault      (core dumped) fsck.gfs2 -y /dev/morph-cluster/bigdir
fsck.gfs2 exitted w/ code 0
Comment 2 Nate Straz 2010-08-27 16:47:24 UTC 
Created attachment 441565 [details]
fsck.gfs2 core dump
Comment 4 Robert Peterson 2010-08-30 21:06:25 UTC 
Created attachment 442037 [details]
gfs2_edit enhancement (continuation blocks)

I recreated this problem and figured out what was going on.

I had to enhance gfs2_edit a bit to figure this out,
otherwise it was searching for a needle in a haystack:
trying to track down 18 missing dirents in an sea of 110,000
dirents.  This patch is the enhancement to gfs2_edit that
allowed me to figure out what's going on.

What's going on is this: Apparently fsck.gfs2 is ignoring
continuation leaf blocks (aka lf_next) in pass2 where it's
counting directory references.  The gfs2_edit tool was also
ignoring them, thus the need for the enhancement.

Now I have to figure out why fsck.gfs2 is ignoring them and
what we can do about it.  The good news is that the kernel
code is apparently doing things correctly.
Comment 5 Robert Peterson 2010-08-30 21:09:13 UTC 
This obviously has to be fixed, so I'm changing the status
to ASSIGNED and requesting ack flags.  I should investigate
whether gfs1's gfs_fsck has the same issue.  I bet it does,
but that will become apparent soon enough.  I don't have a
fix yet, but I expect to have a fix tomorrow.
Comment 6 Robert Peterson 2010-08-31 14:38:45 UTC 
Created attachment 442201 [details]
first go at a patch

This patch fixes the failing scenario.  However, it needs a
lot more testing.  I may need to move the statement
"ref_count = 1;" inside the loop as well.  I won't find that
out until I look at the code some more and run the patch
through a couple of the complex metadata sets.  So this may
not be the final version.
Comment 7 Robert Peterson 2010-09-01 15:06:33 UTC 
Created attachment 442432 [details]
Revised patch

This is a revised patch.  It fixes the failing scenario, and
I'm in the process of regression testing the RHEL5.6 crosswrite
against my collection of metadata.
Comment 8 Robert Peterson 2010-09-17 18:33:36 UTC 
Created attachment 448082 [details]
Final patch for 6.1
Comment 9 Robert Peterson 2010-09-17 18:46:31 UTC 
This patch was tested on RHEL6 system west-10.  The patch was
pushed to the master branch of the gfs2-utils git tree, and
the STABLE3 and RHEL6 branches of the cluster git tree for
inclusion into RHEL6.1.  Changing status to POST until it
gets built into a 6.1 cluster package.
Comment 11 Nate Straz 2011-03-30 14:43:31 UTC 
Ran into 676626 while trying to verify this bug.
Comment 12 Nate Straz 2011-04-06 19:10:00 UTC 
Verified with gfs2-utils-3.0.12-41.el6.x86_64 using a new regression test with 150,000 files and by hand with 500,000 files.
Comment 13 errata-xmlrpc 2011-05-19 12:53:50 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0537.html
Note You need to log in before you can comment on or make changes to this bug.