Bug 1892166 - btrfsck segfaults on root 5 missing its root dir, recreating
Summary: btrfsck segfaults on root 5 missing its root dir, recreating
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: btrfs-progs
Version: 33
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: Josef Bacik
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-10-28 04:59 UTC by Sandino Araico Sánchez
Modified: 2021-01-27 02:38 UTC (History)
5 users (show)

Fixed In Version: btrfs-progs-5.10-1.fc33 btrfs-progs-5.10-1.fc32
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-01-22 01:33:03 UTC
Type: Bug


Attachments (Terms of Use)
btrfsck log (127.79 KB, text/plain)
2020-10-28 05:10 UTC, Sandino Araico Sánchez
no flags Details
btrfs check --readonly /dev/sdXY (328.85 KB, text/plain)
2020-10-30 09:50 UTC, Sandino Araico Sánchez
no flags Details
mount -o ro,usebackuproot,degraded /dev/sdXY (79.08 KB, text/plain)
2020-10-30 19:29 UTC, Sandino Araico Sánchez
no flags Details
btrfs insp dump-s -Ffa /dev/each (30.21 KB, text/plain)
2020-11-06 22:27 UTC, Sandino Araico Sánchez
no flags Details

Description Sandino Araico Sánchez 2020-10-28 04:59:50 UTC
Description of problem: 
btrfsck segfaults while attempting to fix a corrupted filesystem


Version-Release number of selected component (if applicable):
btrfs-progs-5.7-5.fc33.x86_64


How reproducible:
On the corrupted filesystem
btrfsck --init-extent-tree --backup --progress /dev/sdc4



Steps to Reproduce:
1. btrfsck --init-extent-tree --backup --progress /dev/sdc4


Actual results:
[3/7] checking free space cache                (0:00:01 elapsed, 1774 items checked)
root 5 missing its root dir, recreating        (0:00:00 elapsed)
Segmentation fault (core dumped)



Expected results:
Filesystem repaired


Additional info:
Oct 27 22:39:22 fedora systemd[1]: Started Process Core Dump (PID 1673/UID 0).
Oct 27 22:39:22 fedora audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-coredump@5-1673-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 27 22:39:31 fedora systemd-coredump[1674]: [🡕] Process 1663 (btrfsck) of user 0 dumped core.
                                               
                                               Stack trace of thread 1663:
                                               #0  0x00005567c4efcda5 btrfs_search_slot (btrfs + 0x73da5)
                                               #1  0x00005567c4f13f53 btrfs_alloc_chunk (btrfs + 0x8af53)
                                               #2  0x00005567c4f070b9 do_chunk_alloc (btrfs + 0x7e0b9)
                                               #3  0x00005567c4f072b1 btrfs_reserve_extent (btrfs + 0x7e2b1)
                                               #4  0x00005567c4f07f45 btrfs_alloc_free_block (btrfs + 0x7ef45)
                                               #5  0x00005567c4efa6d3 __btrfs_cow_block (btrfs + 0x716d3)
                                               #6  0x00005567c4efaf42 btrfs_cow_block (btrfs + 0x71f42)
                                               #7  0x00005567c4efd049 btrfs_search_slot (btrfs + 0x74049)
                                               #8  0x00005567c4eff159 btrfs_insert_empty_items (btrfs + 0x76159)
                                               #9  0x00005567c4eff578 btrfs_insert_item (btrfs + 0x76578)
                                               #10 0x00005567c4f1f464 btrfs_make_root_dir (btrfs + 0x96464)
                                               #11 0x00005567c4ec5441 check_inode_recs.lto_priv.0 (btrfs + 0x3c441)
                                               #12 0x00005567c4ecf81e cmd_check.lto_priv.0 (btrfs + 0x4681e)
                                               #13 0x00005567c4e9edd0 main (btrfs + 0x15dd0)
                                               #14 0x00007fb0d68e01a2 __libc_start_main (libc.so.6 + 0x281a2)
                                               #15 0x00005567c4e9f10e _start (btrfs + 0x1610e)
                                               
                                               Stack trace of thread 1672:
                                               #0  0x00007fb0d6a990fc read (libpthread.so.0 + 0x130fc)
                                               #1  0x00005567c4ec22dc print_status_check.lto_priv.0 (btrfs + 0x392dc)
                                               #2  0x00007fb0d6a8f3f9 start_thread (libpthread.so.0 + 0x93f9)
                                               #3  0x00007fb0d69b9b03 __clone (libc.so.6 + 0x101b03)
Oct 27 22:39:31 fedora systemd[1]: systemd-coredump@5-1673-0.service: Succeeded.

Comment 1 Sandino Araico Sánchez 2020-10-28 05:10:02 UTC
Created attachment 1724696 [details]
btrfsck log

btrfsck --init-extent-tree --backup --progress /dev/sdc4 > btrfsck.log 2>&1

Comment 3 Sandino Araico Sánchez 2020-10-28 07:01:11 UTC
214 [root@fedora src] gdb /usr/sbin/btrfsck core.btrfsck.0.980c24f22eca4a508bd80e5975729f24.1663.1603859962000000
GNU gdb (GDB) Fedora 9.2-7.fc33
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/btrfsck...
Reading symbols from /usr/lib/debug/usr/sbin/btrfs-5.7-5.fc33.x86_64.debug...

warning: core file may not match specified executable file.
[New LWP 1663]
[New LWP 1672]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `btrfsck --init-extent-tree --backup --progress /dev/sdc4'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  btrfs_search_slot (trans=0x5567c68fba70, root=0x0, key=0x7ffc3af03480, p=0x5567d96a20a0, ins_len=0, cow=0) at ctree.c:1275
1275            struct btrfs_fs_info *fs_info = root->fs_info;
[Current thread is 1 (Thread 0x7fb0d68b58c0 (LWP 1663))]
(gdb) bt
#0  btrfs_search_slot (trans=0x5567c68fba70, root=0x0, key=0x7ffc3af03480, p=0x5567d96a20a0, ins_len=0, cow=0) at ctree.c:1275
#1  0x00005567c4f13f53 in btrfs_device_avail_bytes (avail_bytes=<synthetic pointer>, device=0x5567c5fa3d80, trans=0x5567c68fba70) at volumes.c:936
#2  btrfs_alloc_chunk (trans=0x5567c68fba70, info=0x5567c5fabc40, start=0x7ffc3af03518, num_bytes=0x7ffc3af03510, type=20) at volumes.c:1122
#3  0x00005567c4f070b9 in do_chunk_alloc (trans=trans@entry=0x5567c68fba70, fs_info=fs_info@entry=0x5567c5fabc40, alloc_bytes=alloc_bytes@entry=2113536, 
    flags=flags@entry=20) at extent-tree.c:1728
#4  0x00005567c4f072b1 in btrfs_reserve_extent (trans=trans@entry=0x5567c68fba70, root=root@entry=0x5567c60466d0, num_bytes=16384, empty_size=empty_size@entry=0, 
    hint_byte=hint_byte@entry=5443871047680, search_end=search_end@entry=18446744073709551615, ins=0x7ffc3af03730, is_data=false) at extent-tree.c:2337
#5  0x00005567c4f07f45 in alloc_tree_block (flags=0, search_end=18446744073709551615, generation=<optimized out>, ins=0x7ffc3af03730, hint_byte=5443871047680, 
    empty_size=0, level=0, key=0x7ffc3af037d0, root_objectid=5, num_bytes=<optimized out>, root=0x5567c60466d0, trans=0x5567c68fba70) at extent-tree.c:2451
#6  btrfs_alloc_free_block (trans=0x5567c68fba70, root=0x5567c60466d0, blocksize=<optimized out>, root_objectid=5, key=0x7ffc3af037d0, level=0, hint=5443871047680, 
    empty_size=0) at extent-tree.c:2529
#7  0x00005567c4efa6d3 in __btrfs_cow_block (trans=trans@entry=0x5567c68fba70, root=root@entry=0x5567c60466d0, buf=buf@entry=0x5567c6046920, parent=parent@entry=0x0, 
    parent_slot=0, cow_ret=cow_ret@entry=0x7ffc3af03a48, search_start=5443871047680, empty_size=0) at ctree.c:428
#8  0x00005567c4efaf42 in btrfs_cow_block (trans=trans@entry=0x5567c68fba70, root=root@entry=0x5567c60466d0, buf=0x5567c6046920, parent=0x0, 
    parent_slot=<optimized out>, cow_ret=cow_ret@entry=0x7ffc3af03a48) at ctree.c:521
#9  0x00005567c4efd049 in btrfs_search_slot (trans=<optimized out>, root=root@entry=0x5567c60466d0, key=key@entry=0x7ffc3af03c20, p=p@entry=0x5567c5fa5480, 
    ins_len=ins_len@entry=185, cow=cow@entry=1) at ctree.c:1288
#10 0x00005567c4eff159 in btrfs_insert_empty_items (trans=trans@entry=0x5567c68fba70, root=root@entry=0x5567c60466d0, path=path@entry=0x5567c5fa5480, 
    cpu_key=cpu_key@entry=0x7ffc3af03c20, data_size=data_size@entry=0x7ffc3af03bd4, nr=nr@entry=1) at ctree.c:2742
#11 0x00005567c4eff578 in btrfs_insert_empty_item (data_size=<optimized out>, key=0x7ffc3af03c20, path=0x5567c5fa5480, root=0x5567c60466d0, trans=0x5567c68fba70)
    at ctree.h:2695
#12 btrfs_insert_item (trans=0x5567c68fba70, root=0x5567c60466d0, cpu_key=0x7ffc3af03c20, data=0x7ffc3af03c40, data_size=160) at ctree.c:2841
#13 0x00005567c4f1f464 in btrfs_insert_inode (inode_item=0x7ffc3af03c40, objectid=256, root=0x5567c60466d0, trans=0x5567c68fba70) at kernel-shared/inode-item.c:143
#14 btrfs_make_root_dir (trans=0x5567c68fba70, root=0x5567c60466d0, objectid=256) at common/utils.c:89
#15 0x00005567c4ec5441 in check_inode_recs (root=0x5567c60466d0, inode_cache=<optimized out>) at check/main.c:3015
#16 0x00005567c4ecf81e in check_fs_root (wc=0x7ffc3af04020, root_cache=0x7ffc3af03ed8, root=<optimized out>) at check/main.c:3692
#17 check_fs_roots (root_cache=0x7ffc3af03ed8, fs_info=<optimized out>) at check/main.c:3771
#18 do_check_fs_roots (root_cache=0x7ffc3af03ed8, fs_info=<optimized out>) at check/main.c:3888
#19 cmd_check (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at check/main.c:10393
#20 0x00005567c4e9edd0 in cmd_execute (argv=0x7ffc3af04458, argc=5, cmd=0x5567c4f761e0 <cmd_struct_check>) at cmds/commands.h:125
#21 main (argc=5, argv=0x7ffc3af04458) at btrfs.c:402

Comment 4 Chris Murphy 2020-10-29 20:37:50 UTC
I've reported this upstream. It's a bug and shouldn't crash, at least it should fail safe and gracefully. However, I can't tell what the history of the file system is, what happened, what was attempted prior to --init-extent-tree which is a big hammer.

>parent transid verify failed on 7715066789888 wanted 101433 found 102169

These are usually the result of a device failing to honor btrfs write ordering. There's over 700 commits separating them. Was there a crash or power failure preceding the need to run this command? It's also possible these errors can happen if one device is missing writes, and needs to be caught up with a scrub.

>warning, device 6 is missing

Is this device in a degraded state now? It's possible this will prevent self healing since there may only be one copy of metadata.

'btrfs dev stats' for each device (or just the mount point if it will at least mount -o ro,degraded) would be useful; as well as dmesg for any failed mount attempt.

If this file system mounts successfully with -o degraded (i.e. read-write), my suggestion is to 'btrfs replace start 6 /dev/newdevice /mnt' and get it replaced. Then once it's finished rebuilding the missing devices, scrub the file system. After that's done, it can be unmounted and checked with 'btrfs check --readonly' but I do not recommend --repair until it's known what the problems are first.

Comment 6 Sandino Araico Sánchez 2020-10-29 22:29:55 UTC
It's a very damage filesystem. Power failures in the middle of a device remove rebalance, repeated device failures in the middle of attempts ro revert the rebalance, superblock damaged, the filesystem does not mount any more. The failed disk image was recovered partialy so I will not include it for now, however I would like to see the btrfs scrub doing it's magic on such a damaged scenario.

The filesystem contains backups. Some of them are too old I would like to get them back, but I can live without them.

I have no problem with the fsck missing unfixable parts of the filesystem as long as it gets me to a consistent (mountable) state.

Comment 7 Chris Murphy 2020-10-30 00:03:52 UTC
When you get a chance, please attach the output from three things:

1. btrfs check --readonly /dev/sdXY
## any of the devices for this volume, only one needs checked, the check tool finds all the others

2. btrfs check --mode lowmem --readonly /dev/sdXY
## this will take a long time, possibly more than a day with a file system of this size; but it's a different implementation than the normal mode

3. mount -o ro,usebackuproot,degraded /dev/sdXY
   dmesg
## Disconnect the faulty device that was being removed; I guess that's devid 6 that's already missing. And try to mount with these options; and if it fails, attach dmesg for the failure.

The best chance for recovery in the near future is 'btrfs restore'. It's quite tedious to use, but has a high success rate. This amounts to offline scrape of your data, and once satisfied, start over with a new file system. There's often help on #fedora and #btrfs on irc.freenode. Future feature tentatively planned for kernel 5.11 is a more tolerant read-only rescue mount option for getting data off with normal tools. Significantly damaged Btrfs file systems are difficult to repair so I couldn't count on that happening soon. Of course, crashing is still a bug.

Comment 8 Sandino Araico Sánchez 2020-10-30 09:50:30 UTC
Created attachment 1725257 [details]
btrfs check --readonly /dev/sdXY

Comment 9 Sandino Araico Sánchez 2020-10-30 19:24:48 UTC
2. btrfs check --mode lowmem --readonly /dev/sdXY

4.9 GB file uploaded to upload.1101.mx/btrfs-check--mode-lowmem--readonly-sdc4

Comment 10 Sandino Araico Sánchez 2020-10-30 19:29:18 UTC
Created attachment 1725393 [details]
mount -o ro,usebackuproot,degraded /dev/sdXY

Comment 11 Chris Murphy 2020-10-30 21:14:01 UTC
Ok so ~12000+ snapshots it looks like. That complicates the repair, in particular init-extent-tree since all of those file btrees have to be walked to do the extent tree reconstruction. Depending on the RAM and metadata (block groups) size, it could take a long time indeed.

What's the status of the missing drive? Have you tried cloning it to a new drive? Can you then 'mount' without any options? If not, I'd like to see dmesg for that failure. And also try 'mount -o ro,usebackuproot', and see dmesg for that too if it doesn't work.

Thing is, so long as there aren't two bad copies of the same fs metadata, Btrfs can recover and self-heal. There might be something useful on the other drive still. Also, when was the last time the file system was scrubbed? It should get a scrub after each power fail or crash to make sure any corruption or stale metadata is fixed up.

Comment 12 Chris Murphy 2020-10-30 21:52:45 UTC
>[69325.537853] BTRFS error (device sdd4): chunk 7716140482560 has missing dev extent, have 0 expect 2

That's a problem. It's possible the only copy of this missing dev extent is on the missing drive.

Can you post the output from:

$ btrfs rescue super -v /dev/any
## only needs to be run on one device for this file system, it'll find the others

$ btrfs insp dump-s -Ffa /dev/each
## this needs to be run one time on every device in the file system; they can all get dumped in one text file though and attached to the bug

FWIW these are all read-only commands and do not make changes to the file system. Changes can make things worse.

Comment 13 Sandino Araico Sánchez 2020-11-06 22:24:55 UTC
 btrfs rescue super -v /dev/sdc4
All Devices:
        Device: id = 1, name = /dev/sdd4
        Device: id = 7, name = /dev/sdb4
        Device: id = 8, name = /dev/sdc4

Before Recovering:
        [All good supers]:
                device name = /dev/sdd4
                superblock bytenr = 65536

                device name = /dev/sdd4
                superblock bytenr = 67108864

                device name = /dev/sdd4
                superblock bytenr = 274877906944

                device name = /dev/sdb4
                superblock bytenr = 65536

                device name = /dev/sdb4
                superblock bytenr = 67108864

                device name = /dev/sdb4
                superblock bytenr = 274877906944

                device name = /dev/sdc4
                superblock bytenr = 65536

                device name = /dev/sdc4
                superblock bytenr = 67108864

                device name = /dev/sdc4
                superblock bytenr = 274877906944

        [All bad supers]:

All supers are valid, no need to recover

Comment 14 Sandino Araico Sánchez 2020-11-06 22:27:42 UTC
Created attachment 1727280 [details]
btrfs insp dump-s -Ffa /dev/each

Comment 15 Fedora Update System 2021-01-20 12:20:11 UTC
FEDORA-2021-3c30d1d273 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-3c30d1d273

Comment 16 Fedora Update System 2021-01-21 01:25:13 UTC
FEDORA-2021-3c30d1d273 has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-3c30d1d273`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-3c30d1d273

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 17 Fedora Update System 2021-01-21 02:02:32 UTC
FEDORA-2021-002891b7f1 has been pushed to the Fedora 32 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-002891b7f1`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-002891b7f1

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 18 Fedora Update System 2021-01-22 01:33:03 UTC
FEDORA-2021-3c30d1d273 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 19 Sandino Araico Sánchez 2021-01-22 20:32:28 UTC
I am sorry, I had to use the disks with the corrupted filesystem for another project. I am no longer able to verify the fixed version against the corrupted filesystem.
I will try to reproduce the corruption when I get the disks back.

Comment 20 Fedora Update System 2021-01-27 02:38:41 UTC
FEDORA-2021-002891b7f1 has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.