Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1670153

Summary:	xfs_repair may rebuild free inode btree with wrong magic number in some nodes
Product:	Red Hat Enterprise Linux 8	Reporter:	Eric Sandeen <esandeen>
Component:	xfsprogs	Assignee:	Eric Sandeen <esandeen>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Zorro Lang <zlang>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	8.0	CC:	bfoster, dchinner, jwboyer, xzhou
Target Milestone:	rc	Flags:	rule-engine: mirror+
Target Release:	8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1670154 (view as bug list)		Environment:
Last Closed:	2019-06-14 01:31:11 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1670154

Description Eric Sandeen 2019-01-28 18:00:49 UTC

When xfs_repair rebuilds the free inode btree, it may write some nodes with the wrong magic number.  Newer versions of kernel verifiers now flag this as corruption, and will shut down the filesytem.

Brian Foster has sent a patch upstream to fix this and it is reviewed; we should get this change into RHEL8.0 prior to GA so that customers do not run into this problem.

[PATCH] xfs_repair: initialize non-leaf finobt blocks with correct magic

The free inode btree construction code in xfs_repair has a bug where
any non-leaf nodes outside of the leftmost block at the associated
level in the tree are incorrectly initialized with the inobt magic
value. Update the prop_ino_cursor() path responsible for growing the
non-leaf portion of the inode btrees to use the btnum of the
specific tree being generated rather than the hardcoded inode btree
type.

Signed-off-by: Brian Foster <bfoster>
Reported-by: Lucas Stach <l.stach>
Root-caused-by: Lucas Stach <l.stach>
Reviewed-by: Eric Sandeen <sandeen>

Comment 1 Eric Sandeen 2019-01-28 18:02:29 UTC

Asking for exception on this one - this is a late-discovered issue with the xfs_repair code which has a strong liklihood of corrupting on-disk metadata under certain circumstances.

Comment 5 Zorro Lang 2019-01-29 09:43:14 UTC

Hi Eric,

Besides the xfsprogs fix, will merge kernel part improvement patches in RHEL-8.0?
[PATCH RFC v2 0/3] xfs: fix [f]inobt magic value verification

Thanks,
Zorro

Comment 6 Eric Sandeen 2019-01-29 16:05:27 UTC

We don't know yet, that's still RFC.

Comment 8 Eric Sandeen 2019-02-04 20:41:11 UTC

Built in xfsprogs-4.19.0-2.el8

Comment 10 Zorro Lang 2019-02-21 09:22:49 UTC

Reproduce this bug on xfsprogs-4.19.0-1.el8:
# rpm -q xfsprogs
xfsprogs-4.19.0-1.el8.x86_64
# mkfs.xfs -f -m finobt=1 $dev
# mount $dev $mnt
# mkdir $mnt/dir
# pushd $mnt/dir
# seq -w 0 20000000| xargs  touch; sync; echo "create files done"; seq -w 0 32 20000000 | xargs rm
# xfs_db -r -c "agi 0" -c "p" $dev  (make sure free_level >= 3)
magicnum = 0x58414749
versionnum = 1
seqno = 0
length = 8192000
count = 14737984
root = 1090814
level = 3
freecount = 460569
newino = 15775168
dirino = null
unlinked[0-63] =
uuid = e808acb6-361b-4d39-b4f5-40de81b43608
crc = 0x88a9ce1b (correct)
lsn = 0x1b00064dc1
free_root = 1972411
free_level = 3
# umount $dev
# xfs_repair $dev (old xfs_repair will break all level blocks after the leftmost block)
# xfs_repair -n $dev (reproduced)
Phase 1 - find and verify superblock...
        - reporting progress in intervals of 15 minutes
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
bad magic # 0x49414233 in inobt block 0/1972356
        - 04:12:54: scanning filesystem freespace - 16 of 16 allocation groups done
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - 04:12:54: scanning agi unlinked lists - 16 of 16 allocation groups done
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 15
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - 04:14:18: process known inodes and inode discovery - 14737984 of 14737984 inodes done
        - process newly discovered inodes...
        - 04:14:18: process newly discovered inodes - 16 of 16 allocation groups done
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - 04:14:18: setting up duplicate extent list - 16 of 16 allocation groups done
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 2
        - agno = 3
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 13
        - agno = 4
        - agno = 1
        - agno = 15
        - agno = 12
        - agno = 14
        - 04:14:51: check for inodes claiming duplicate blocks - 14737984 of 14737984 inodes done
No modify flag set, skipping phase 5
Inode allocation btrees are too corrupted, skipping phases 6 and 7
No modify flag set, skipping filesystem flush and exiting.

Comment 11 Zorro Lang 2019-02-21 09:56:58 UTC

(In reply to Zorro Lang from comment #10)
> Reproduce this bug on xfsprogs-4.19.0-1.el8:
> # rpm -q xfsprogs
> xfsprogs-4.19.0-1.el8.x86_64
> # mkfs.xfs -f -m finobt=1 $dev
> # mount $dev $mnt
> # mkdir $mnt/dir
> # pushd $mnt/dir
> # seq -w 0 20000000| xargs  touch; sync; echo "create files done"; seq -w 0
> 32 20000000 | xargs rm
> # xfs_db -r -c "agi 0" -c "p" $dev  (make sure free_level >= 3)
> magicnum = 0x58414749
> versionnum = 1
> seqno = 0
> length = 8192000
> count = 14737984
> root = 1090814
> level = 3
> freecount = 460569
> newino = 15775168
> dirino = null
> unlinked[0-63] =
> uuid = e808acb6-361b-4d39-b4f5-40de81b43608
> crc = 0x88a9ce1b (correct)
> lsn = 0x1b00064dc1
> free_root = 1972411
> free_level = 3
> # umount $dev
> # xfs_repair $dev (old xfs_repair will break all level blocks after the
> leftmost block)
> # xfs_repair -n $dev (reproduced)
> Phase 1 - find and verify superblock...
>         - reporting progress in intervals of 15 minutes
> Phase 2 - using internal log
>         - zero log...
>         - scan filesystem freespace and inode maps...
> bad magic # 0x49414233 in inobt block 0/1972356
>         - 04:12:54: scanning filesystem freespace - 16 of 16 allocation
> groups done
>         - found root inode chunk
> Phase 3 - for each AG...
>         - scan (but don't clear) agi unlinked lists...
>         - 04:12:54: scanning agi unlinked lists - 16 of 16 allocation groups
> done
>         - process known inodes and perform inode discovery...
>         - agno = 0
>         - agno = 15
>         - agno = 1
>         - agno = 2
>         - agno = 3
>         - agno = 4
>         - agno = 5
>         - agno = 6
>         - agno = 7
>         - agno = 8
>         - agno = 9
>         - agno = 10
>         - agno = 11
>         - agno = 12
>         - agno = 13
>         - agno = 14
>         - 04:14:18: process known inodes and inode discovery - 14737984 of
> 14737984 inodes done
>         - process newly discovered inodes...
>         - 04:14:18: process newly discovered inodes - 16 of 16 allocation
> groups done
> Phase 4 - check for duplicate blocks...
>         - setting up duplicate extent list...
>         - 04:14:18: setting up duplicate extent list - 16 of 16 allocation
> groups done
>         - check for inodes claiming duplicate blocks...
>         - agno = 0
>         - agno = 2
>         - agno = 3
>         - agno = 5
>         - agno = 6
>         - agno = 7
>         - agno = 8
>         - agno = 9
>         - agno = 10
>         - agno = 11
>         - agno = 13
>         - agno = 4
>         - agno = 1
>         - agno = 15
>         - agno = 12
>         - agno = 14
>         - 04:14:51: check for inodes claiming duplicate blocks - 14737984 of
> 14737984 inodes done
> No modify flag set, skipping phase 5
> Inode allocation btrees are too corrupted, skipping phases 6 and 7
> No modify flag set, skipping filesystem flush and exiting.

Can't reproduce this bug on xfsprogs-4.19.0-2.el8
# rpm -q xfsprogs
xfsprogs-4.19.0-2.el8.x86_64
# xfs_repair $dev
# xfs_repair -n $dev
Phase 1 - find and verify superblock...
        - reporting progress in intervals of 15 minutes
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
        - 04:45:56: scanning filesystem freespace - 16 of 16 allocation groups done
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - 04:45:56: scanning agi unlinked lists - 16 of 16 allocation groups done
        - process known inodes and perform inode discovery...
        - agno = 15
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - 04:47:27: process known inodes and inode discovery - 14737984 of 14737984 inodes done
        - process newly discovered inodes...
        - 04:47:27: process newly discovered inodes - 16 of 16 allocation groups done
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - 04:47:27: setting up duplicate extent list - 16 of 16 allocation groups done
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 11
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 7
        - agno = 8
        - agno = 12
        - agno = 10
        - agno = 6
        - agno = 9
        - agno = 15
        - agno = 2
        - agno = 13
        - agno = 14
        - agno = 1
        - 04:47:58: check for inodes claiming duplicate blocks - 14737984 of 14737984 inodes done
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
        - 04:54:16: verify and correct link counts - 16 of 16 allocation groups done
No modify flag set, skipping filesystem flush and exiting.
# echo $?
0