Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1206149

Summary: fsck.gfs2: Duplicate refs inside one dinode are not fixed
Product: Red Hat Enterprise Linux 6 Reporter: Robert Peterson <rpeterso>
Component: clusterAssignee: Robert Peterson <rpeterso>
Status: CLOSED ERRATA QA Contact: cluster-qe <cluster-qe>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.7CC: ccaulfie, cluster-maint, dwysocha, gfs2-maint, jpayne, phracek, rpeterso, sbradley, teigland
Target Milestone: rcKeywords: Patch, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: cluster-3.0.12.1-74.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1236669 1277030 (view as bug list) Environment:
Last Closed: 2016-05-10 19:06:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1172231, 1236669, 1268411, 1277030    
Attachments:
Description Flags
Patch 1 of 2: fsck.gfs2: Change duptree structure to have generic flags
none
Patch 2 of 2: fsck.gfs2: Detect, fix and clone duplicate block refs within a dinode
none
Patch 2 of 2: fsck.gfs2: Detect, fix and clone duplicate block refs within a dinode none

Description Robert Peterson 2015-03-26 12:51:05 UTC
Description of problem:
A customer somehow managed to get a GFS2 file system in which
some dinodes that had the same data block listed twice within
the same dinode. The double reference eventually resulted in
the kernel trying to free the block twice when the file was
deleted, which resulted in a file system withdraw. Subsequent
attempts to mount the file system also failed because of it.

The fsck.gfs2 tool correctly identified that the block was
referenced twice, but because it couldn't find another dinode
referencing it, it ignored the problem. Thus, the file system
was not fixed. Thus, the file system was rendered unusable.

Version-Release number of selected component (if applicable):
RHEL6.6

How reproducible:
Always

Steps to Reproduce:
1. gfs2_edit restoremeta T9479dsb01a_mqm_data_metadata <dev>
2. fsck.gfs2 -y <dev> &> /tmp/fsck.out
3. fsck.gfs2 <dev>

Actual results:
Found duplicate data block 70519672 (0x4340b78) referenced as data by dinode 69995390 (0x42c0b7e) 
Found duplicate data block 70519687 (0x4340b87) referenced as data by dinode 69995390 (0x42c0b7e) 
etc.

Expected results:
The first fsck.gfs2 should identify the duplicate references
and fix them. The second fsck.gfs2 should find no errors.

Additional info:
There are really two problems here. The first problem was directly
caused by this patch:

https://git.fedorahosted.org/cgit/gfs2-utils.git/commit/gfs2/fsck/util.c?id=37480679e413e503de72d873021ce2c808f0d8cc

The purpose of the patch is to prevent situations where a
duplicate block reference is found, but the original reference
is deleted, thus leaving one single reference which is good.
However, it goes too far and filters out situations like this
where the duplicate reference is contained within a single dinode.

The second problem is: Even if you revert the patch above, the
problem is still not fixed by fsck.gfs2. In most cases of
duplicate references, fsck.gfs2 chooses between the two dinodes
and deletes one of them. In this case, we can either delete the
corrupt dinode entirely or just zero the bad pointer, possibly
creating a hole and turning the file into a sparse file.
Right now it doesn't do either.

Comment 1 Robert Peterson 2015-03-26 12:53:15 UTC
This section of fsck.gfs2, pass1b, is very delicate and prone
to breaking. For now I'm bumping it to 6.8 to give us more time
for testing and such. We do need to fix this, so I'm setting it
to assigned and devel_ack+.

Comment 3 Petr Hracek 2015-04-10 11:59:54 UTC
Did you have a time to look at the problem?
Is there any progress?
Do we have a patch for this problem?

Comment 4 Robert Peterson 2015-04-10 12:18:00 UTC
I know what the problem is, but it's not easy to fix without
severely impacting fsck.gfs2 performance. Progress has been slow
because of other high-priority problems. The fsck issue is
serious, but this condition should be extremely rare. This is
the first time I've seen it in 9 years, and I have other issues
that are impacting customers on a daily basis. Sorry, I don't
have a patch for the problem at this time.

Comment 5 Robert Peterson 2015-06-29 16:14:07 UTC
Created attachment 1044413 [details]
Patch 1 of 2: fsck.gfs2: Change duptree structure to have generic flags

This is the first of two patches for bug #1206149. It does not
change functionality. It merely changes the duptree structure
to use a generic flag (dup_flags) rather than a highly specialize
flag (first_ref_found). This will allow for a future patch that
adds a new flag for marking situations where duplicates are found
within one dinode, without adding more memory to the structure.

Comment 6 Robert Peterson 2015-06-29 16:15:39 UTC
Created attachment 1044414 [details]
Patch 2 of 2: fsck.gfs2: Detect, fix and clone duplicate block refs within a dinode

Prior to this patch, fsck.gfs2 was unable to detect and fix duplicate
block references within the same file. This patch detects when data
blocks are duplicated within a dinode, then tries to clone the data
to a new block.

Comment 8 Robert Peterson 2015-10-09 15:47:51 UTC
Created attachment 1081380 [details]
Patch 2 of 2: fsck.gfs2: Detect, fix and clone duplicate block refs within a dinode

This version fixes some obvious mistakes in the first version,
but they're related to differences between rhel7 and rhel6
gfs2-utils. This version is tested and fixes the failing case
as well as several other metadata sets.

Comment 9 Robert Peterson 2015-10-09 15:54:45 UTC
I have no problem shipping these latest two patches for RHEL6.8.
They are tested and fix the failing test case plus some other
tough metadata sets, as I mentioned in comment #8.

However, the resulting fsck.gfs2 will not pass my latest
version of the fsck.gfs2.nightmare2.sh test because I've added
several metadata sets that have unique corruption. Those new
cases of metadata corruption are fixed by the 20 new patches to
fsck.gfs2 in RHEL7 bug #1257625. I'm debating whether I want to
close that bug to RHEL6 so I can port those 20 as well.

Comment 10 Robert Peterson 2015-10-09 16:16:35 UTC
The two latest patches were pushed to the rhel6 branch of the
cluster.git repo for inclusion into rhel6.8. They are already
in the upstream gfs2-utils repo. They were tested on system
gfs-a16c-04. I tested the failing case and also several of
the toughest metadata sets from my fsck.gfs2.nightmare2.sh
test. I was unable to complete the entire test as explained
in comment #9. Changing status to POST until the patches are
built into a new release of the cluster rpms.

Comment 14 Justin Payne 2016-01-25 18:47:32 UTC
Verified in gfs2-utils-3.0.12.1-77.el6

[root@host-002 ~]# rpm -q gfs2-utils
gfs2-utils-3.0.12.1-77.el6.x86_64

[root@host-002 ~]# gfs2_edit restoremeta T9479dsb01a_mqm_data_metadata /dev/sda
Savemeta file format 1
Created Wed Mar 25 16:14:54 2015

File system size 999.1023G
There are 314572800 blocks of 4096 bytes in the destination device.

314572800 inodes processed, 152638 blocks saved (100%) processed,
File T9479dsb01a_mqm_data_metadata restore successful.
[root@host-002 ~]# fsck.gfs2 -y /dev/sda &> /tmp/fsck.out
[root@host-002 ~]# fsck.gfs2 /dev/sda
Initializing fsck
Validating Resource Group index.
Level 1 rgrp check: Checking if all rgrp and rindex values are good.
(level 1 passed)
Starting pass1
Pass1 complete      
Starting pass1b
Pass1b complete
Starting pass1c
Pass1c complete
Starting pass2
Pass2 complete      
Starting pass3
Pass3 complete      
Starting pass4
Pass4 complete      
Starting pass5
Pass5 complete      
gfs2_fsck complete

Comment 16 errata-xmlrpc 2016-05-10 19:06:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0729.html