Bug 1960738
|
Description
a.trubitsyn
2021-05-14 17:34:25 UTC
The problem happened before the call trace above. It would be best to have a complete dmesg for this. Use 'journalctl -b $bootnum' or --since=, along with -k option to extract kernel messages from the journal for prior boots.
>scrub gets 148867 uncorrectable errors
Scrub reports details of these errors in dmesg as well, so without that we're not sure whether the problem is metadata or data related or some combination of it. Best to attache this as a file to the bug report.
Please do 'btrfs check --readonly' while booted from a Fedora installation image/USB stick, and attach it as a file to the bug report. Thanks.
Created attachment 1784169 [details]
result of sudo btrfs check --readonly /dev/sda3 after booting from live usb
1. Before you answer i did yesterday "sudo mount /dev/sda3", then several "sudo btrfs scrub start /mnt", then extracted and deleted files from dmesg after "(path: " to ")". Now i have 48327 uncorrectable errors according to btrfs scrub check.
2. According to "Use 'journalctl -b $bootnum'". I don't know how to do it on current system before repair btrfs filesystem. If you know how to do it after boot from usb stick, please tell me. My main btrfs partition is on /dev/sda3.
i reconnect ssd with broken btrfs to second computer with fedora 34 & 5.12.6 kernel after backup all needed information i perform 30 or 40 "sudo btrfs scrub start" command, then delete all files from dmesg and try again & again until i get only 2 unrecovererrors then i try sudo btrfs check --repair /dev/sdb3 > btrfs_check_repair_sdb3.txt but it can't repair all errors. Is it possible to repair btrfs file system or i need to reformat partition to get btrfs without errors? Created attachment 1787624 [details]
after several btrfs scrub start & delete all files with errors from dmesg output i get only 2 uncorrectable errors
Created attachment 1787625 [details]
dmesg with 2 errors after last btrfs scrub
Created attachment 1787626 [details]
btrfs check --repair can't repair all errors from 1st run, errors exists
Created attachment 1787627 [details]
brtfs check --repair can't repair all errors from 2nd run, errors still exists
in dmesg now i only have this BTRFS info (device sdb3): disk space caching is enabled BTRFS info (device sdb3): has skinny extents BTRFS info (device sdb3): bdev /dev/sdb3 errs: wr 0, rd 0, flush 0, corrupt 32152762, gen 0 BTRFS info (device sdb3): enabling ssd optimizations BTRFS info (device sdb3): checking UUID tree From the last btrfs check, I think the real problems are fixed, but I'm not sure about the many isize messages. Could you update to btrfs-progs 5.12.1 and do: btrfs check --readonly /dev/sdb3 btrfs check --readonly --mode=lowmem /dev/sdb3 And report the output of both? These are currently different check implementations, so we might get more information about residual issues. If you do still see "reset isize for dir" messages, please take a btrfs-image of the file system: btrfs-image -c9 -t4 --ss /dev/sdb3 /path/to/bug1960738-btrfs.image The -ss command will hash filenames; short filenames result in messages about hash collisions, don't worry about it. This is a metadata only image, it doesn't contain any file contents, and is just used by developers for debugging. Created attachment 1787824 [details]
txt file after execute "sudo btrfs check --readonly /dev/sdb3 > btrfs_check_readonly_sdb3.txt"
Created attachment 1787825 [details]
output of "sudo btrfs check --readonly /dev/sdb3 > btrfs_check_readonly_sdb3.txt"
Created attachment 1787826 [details]
txt file after execute "sudo btrfs check --readonly --mode=lowmem /dev/sdb3 > btrfs_check_readonly_lowmem_sdb3.txt"
Created attachment 1787827 [details]
output of "sudo btrfs check --readonly --mode=lowmem /dev/sdb3 > btrfs_check_readonly_lowmem_sdb3.txt"
(In reply to Chris Murphy from comment #9) > From the last btrfs check, I think the real problems are fixed, but I'm not > sure about the many isize messages. Could you update to btrfs-progs 5.12.1 Thank you for advice. 1.btrfs-progs already updates to 5.12.1 sudo dnf install btrfs-progs Последняя проверка окончания срока действия метаданных: 4:03:15 назад, Чт 27 мая 2021 20:12:22. Пакет btrfs-progs-5.12.1-1.fc34.x86_64 уже установлен. Зависимости разрешены. Отсутствуют действия для выполнения Выполнено! > and do: > > btrfs check --readonly /dev/sdb3 > btrfs check --readonly --mode=lowmem /dev/sdb3 2. I update kernel to kernel-core-5.12.7-300.fc34.x86_64 and execute sudo btrfs check --readonly /dev/sdb3 sudo btrfs check --readonly --mode=lowmem /dev/sdb3 Partial logs & txt files in attachments. They say "cache and super generation don't match, space cache will be invalidated" sudo btrfs scrub start /dev/sdb3 say "no errors found" scrub started on /dev/sdb3, fsid 3b47ef70-9dd5-4e57-8003-570eaa7f514a (pid=2229) [xx@yyyy 20210514_bad_btrfs]$ sudo btrfs scrub status /dev/sdb3 [sudo] пароль для al: UUID: 3b47ef70-9dd5-4e57-8003-570eaa7f514a Scrub started: Fri May 28 14:48:30 2021 Status: finished Duration: 0:01:16 Total to scrub: 19.69GiB Rate: 265.41MiB/s Error summary: no errors found Is it ok or i must do some repair??? How to do it? > And report the output of both? These are currently different check > implementations, so we might get more information about residual issues. If > you do still see "reset isize for dir" messages, please take a btrfs-image > of the file system: There are not "reset isize for dir" messages in the logs any more. Only "cache and super generation don't match, space cache will be invalidated" messages exists. OK my mistake. I thought that "reset isize" messages were persisting after --repair, but they are gone now. You file system is OK, nothing more needs to be done. You can optionally reset the statistics by using the -z option with 'btrfs device stats' command; 'man btrfs device' for more info.
>cache and super generation don't match, space cache will be invalidated
This message is bogus, it's a known bug that will be fixed soon.
OK hold on, I'm completely confused by the two attachments with "partial_log" in their file names that seemed to have come from the btrfs check command being redirected to std out? I am unfamiliar with this usage that results in two outputs for one command. But it looks like the partial_log files show problems still, even after you did 'btrfs check --repair' in comment 3. So it does seem there are unfixed problems still. Please create a btrfs image per comment 9. And we'll go from there. I've started an upstream thread. https://lore.kernel.org/linux-btrfs/CAJCQCtRnxq2mKOkjQzOedjnh9oxsNOFKoP92pjxDGwuUw1AOYg@mail.gmail.com/T/#u (In reply to Chris Murphy from comment #9) > btrfs-image -c9 -t4 --ss /dev/sdb3 /path/to/bug1960738-btrfs.image Ok. Thank you for your time. It's typo in "--ss". i do sudo btrfs-image -c9 -t4 -ss /dev/sdb3 /home/xx/bug/20210514_bad_btrfs/bug1960738-btrfs.image and get some warnings like WARNING: cannot find a hash collision for '8618', generating garbage, it won't match indexes WARNING: cannot find a hash collision for '1843', generating garbage, it won't match indexes Please try to upload image from https://cloud.mail.ru/public/Uf3R/paBECvbN2 . It's 82.3MB >WARNING: cannot find a hash collision for '8618', generating garbage, it won't match indexes
These are expected for short filenames. There's nothing wrong with the image.
The remaining errors following --repair take this form for original mode: unresolved ref dir 81792 index 0 namelen 33 name sd_bus_message_get_signature.3.gz filetype 1 errors 6, no dir index, no inode ref root 257 inode 1637367 errors 2001, no inode item, link count wrong And take this form for lowmem mode: ERROR: root 257 DIR INODE [69956] size 56 not equal to 39 Honestly, I can't make heads or tails of this. Since the btrfs image is taken, the developers have all the info they need to discover the problem; and you can just backup, mkfs, and restore. I don't think your data is at risk so it probably isn't urgent. But I think you'd rather have a clean file system at some point sooner than later. You can backup/restore however convenient, including using btrfs send/receive. Btrfs doesn't propagate corruption, but I do recommend doing `journalctl -fk` to follow kernel messages during the backup so you can see if there are any btrfs related errors, it'll indicate what files *aren't* backed up if they happen to be corrupt. > Honestly, I can't make heads or tails of this. Since the btrfs image is > taken, the developers have all the info they need to discover the problem; > and you can just backup, mkfs, and restore. Thank you very much, Chris! > I don't think your data is at risk so it probably isn't urgent. Yes, there are no any urgent on this. I only want to get usb-based iso of fedora 34 with fresh kernel & btrfs-progs at the moment of resolve this issue. > system at some point sooner than later. You can backup/restore however > convenient, including using btrfs send/receive. Btrfs doesn't propagate > corruption, but I do recommend doing `journalctl -fk` to follow kernel > messages during the backup Thank you for tip & tricks. This message is a reminder that Fedora Linux 34 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 34 on 2022-06-07. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '34'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 34 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed. Fedora Linux 34 entered end-of-life (EOL) status on 2022-06-07. Fedora Linux 34 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. Thank you for reporting this bug and we are sorry it could not be fixed. |