Bug 1997477
| Summary: | Inconsistent results from repeated runs of 'btrfs check' command | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Thomas Köller <thomas> |
| Component: | btrfs-progs | Assignee: | Josef Bacik <josef> |
| Status: | CLOSED NOTABUG | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 34 | CC: | adam900710, bugzilla, davide, esandeen, igor.raits, josef, ngompa13 |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-09-02 20:38:46 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Thomas Köller
2021-08-25 10:31:15 UTC
Offline scrub finds no problems 1 in 5 attempts. What happens with multiple online scrubs? Transient errors like this suggest a hardware issue: it could be bad RAM, it could be prefailing device. HDD tend not to have transient checksum failures, but SSDs do and it is one of the earliest indications of a device that will eventually fail. You can use the following and check to see if there are any unexpected complaints, or attached to the bug: journalctl --no-hostname | grep "ata\|Btrfs\|BTRFS\|] hd\| scsi\| sd\| sdhci\| mmc\| nvme" > journal.log sudo dnf install smartmontools sudo smartctl -x /dev/sdX > smartctl.log (In reply to Chris Murphy from comment #1) > Offline scrub finds no problems 1 in 5 attempts. What happens with multiple > online scrubs? These invariably do not show any errors. > > Transient errors like this suggest a hardware issue: it could be bad RAM, it > could be prefailing device. HDD tend not to have transient checksum > failures, but SSDs do and it is one of the earliest indications of a device > that will eventually fail. > > You can use the following and check to see if there are any unexpected > complaints, or attached to the bug: > > journalctl --no-hostname | grep "ata\|Btrfs\|BTRFS\|] hd\| scsi\| sd\| > sdhci\| mmc\| nvme" > journal.log Looking at the journal did not reveal anything suspicious. > > sudo dnf install smartmontools > sudo smartctl -x /dev/sdX > smartctl.log After running 'smartctl -t long' on devices /dev/sda and /dev/sdb, the constituent devices of the file system, no errors are reported: [root@sarkovy workspace]# smartctl -l selftest /dev/sda smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.13.12-200.fc34.x86_64] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF READ SMART DATA SECTION === SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 12005 - # 2 Extended offline Interrupted (host reset) 50% 12003 - # 3 Extended offline Completed without error 00% 11903 - # 4 Extended offline Completed without error 00% 9242 - [root@sarkovy workspace]# smartctl -l xerror /dev/sda smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.13.12-200.fc34.x86_64] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF READ SMART DATA SECTION === SMART Extended Comprehensive Error Log Version: 1 (1 sectors) No Errors Logged [root@sarkovy workspace]# smartctl -l selftest /dev/sdb smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.13.12-200.fc34.x86_64] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF READ SMART DATA SECTION === SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 12005 - # 2 Extended offline Interrupted (host reset) 50% 12003 - # 3 Extended offline Completed without error 00% 11903 - # 4 Extended offline Completed without error 00% 9242 - [root@sarkovy workspace]# smartctl -l xerror /dev/sdb smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.13.12-200.fc34.x86_64] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF READ SMART DATA SECTION === SMART Extended Comprehensive Error Log Version: 1 (1 sectors) No Errors Logged Also note that the detected checksum errors are on both mirror devices, which would mean that both these devices had started to fail simultaneously. That is quite unlikely. >btrfs check --check-data-csum /dev/disk/by-label/SysStorage
Is the file system entirely umounted? You say it's unmounted in the description but because btrfs supports mounting it multiple times, and anywhere, it can often seem to be umounted but there's a straggler somewhere that makes the fs still mounted.
Firstly, btrfs check will reject the device if it's already mounted. And the reporter is not using --force option, it should not be the case. Since the reporter is using two devices, and two data copies, I guess it's RAID1. Can you provide "btrfs fi usage" of the fs when mounted? (In reply to Qu Wenruo from comment #4) > Firstly, btrfs check will reject the device if it's already mounted. > > And the reporter is not using --force option, it should not be the case. > Exactly. > > Since the reporter is using two devices, and two data copies, I guess it's > RAID1. It is raid1 for both data and metadata. > > Can you provide "btrfs fi usage" of the fs when mounted? [root@sarkovy ~]# btrfs fi usage / Overall: Device size: 1.82TiB Device allocated: 586.02GiB Device unallocated: 1.25TiB Device missing: 0.00B Used: 575.18GiB Free (estimated): 642.63GiB (min: 642.63GiB) Free (statfs, df): 642.63GiB Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 382.77MiB (used: 0.00B) Multiple profiles: no Data,RAID1: Size:291.00GiB, Used:286.37GiB (98.41%) /dev/sda2 291.00GiB /dev/sdb2 291.00GiB Metadata,RAID1: Size:2.00GiB, Used:1.22GiB (61.06%) /dev/sda2 2.00GiB /dev/sdb2 2.00GiB System,RAID1: Size:8.00MiB, Used:64.00KiB (0.78%) /dev/sda2 8.00MiB /dev/sdb2 8.00MiB Unallocated: /dev/sda2 638.00GiB /dev/sdb2 638.00GiB Can you fetch the following btrfs-progs branch? https://github.com/adam900710/btrfs-progs/tree/csum_enhance This branch has proper csum error output message instead of the meaningless one. Then could you run "btrfs check --check-data-csum" on the unmount fs for around 256~512 runs with all output dumped? The 5 runs provided has some interesting results: Pass #1 ======= mirror 2 bytenr 146126761984 csum 231 expected csum 14 mirror 1 bytenr 289154838528 csum 101 expected csum 55 Pass #2 ======= mirror 2 bytenr 116899885056 csum 189 expected csum 242 mirror 1 bytenr 243106209792 csum 146 expected csum 123 Pass #3 ======= NULL Pass #4 ======= mirror 1 bytenr 273130061824 csum 55 expected csum 222 Pass #5 ======= mirror 2 bytenr 14546903040 csum 224 expected csum 125 mirror 2 bytenr 272884711424 csum 65 expected csum 226 Notice that all bytenrs are unique, thus I guess it may be some unexpected memory corruption. Not sure if it's btrfs-progs' code, or hardware. If you can accumulate more runs, maybe we can find some pattern just from the result. (In reply to Qu Wenruo from comment #6) > Can you fetch the following btrfs-progs branch? > > https://github.com/adam900710/btrfs-progs/tree/csum_enhance > > This branch has proper csum error output message instead of the meaningless > one. I tried that, but the results were basically the same as before. Here's the output: Pass #1: ======== [1/7] checking root items [2/7] checking extents [3/7] checking free space cache [4/7] checking fs roots [5/7] checking csums against data mirror 2 bytenr 55823724544 csum 0xf3c56b8e expected csum 0x9e996d9c ERROR: errors found in csum tree [6/7] checking root refs [7/7] checking quota groups skipped (not enabled on this FS) Opening filesystem to check... Checking filesystem on /dev/disk/by-label/SysStorage UUID: f827adea-4448-493c-9ae9-3154458bc814 cache and super generation don't match, space cache will be invalidated found 309086564352 bytes used, error(s) found total csum bytes: 300122908 total tree bytes: 1311391744 total fs tree bytes: 909393920 total extent tree bytes: 47431680 btree space waste bytes: 198618602 file data blocks allocated: 1411201572864 referenced 307423453184 Pass #2: ======== [1/7] checking root items [2/7] checking extents [3/7] checking free space cache [4/7] checking fs roots [5/7] checking csums against data mirror 2 bytenr 112425172992 csum 0x3c2ff1d1 expected csum 0xcb5122e0 ERROR: errors found in csum tree [6/7] checking root refs [7/7] checking quota groups skipped (not enabled on this FS) Opening filesystem to check... Checking filesystem on /dev/disk/by-label/SysStorage UUID: f827adea-4448-493c-9ae9-3154458bc814 cache and super generation don't match, space cache will be invalidated found 309086564352 bytes used, error(s) found total csum bytes: 300122908 total tree bytes: 1311391744 total fs tree bytes: 909393920 total extent tree bytes: 47431680 btree space waste bytes: 198618602 file data blocks allocated: 1411201572864 referenced 307423453184 Pass #3: ======== [1/7] checking root items [2/7] checking extents [3/7] checking free space cache [4/7] checking fs roots [5/7] checking csums against data mirror 1 bytenr 15479435264 csum 0xa7d11656 expected csum 0x543711f8 ERROR: errors found in csum tree [6/7] checking root refs [7/7] checking quota groups skipped (not enabled on this FS) Opening filesystem to check... Checking filesystem on /dev/disk/by-label/SysStorage UUID: f827adea-4448-493c-9ae9-3154458bc814 cache and super generation don't match, space cache will be invalidated found 309086564352 bytes used, error(s) found total csum bytes: 300122908 total tree bytes: 1311391744 total fs tree bytes: 909393920 total extent tree bytes: 47431680 btree space waste bytes: 198618602 file data blocks allocated: 1411201572864 referenced 307423453184 Pass #4: ======== [1/7] checking root items [2/7] checking extents [3/7] checking free space cache [4/7] checking fs roots [5/7] checking csums against data [6/7] checking root refs [7/7] checking quota groups skipped (not enabled on this FS) Opening filesystem to check... Checking filesystem on /dev/disk/by-label/SysStorage UUID: f827adea-4448-493c-9ae9-3154458bc814 cache and super generation don't match, space cache will be invalidated found 309086564352 bytes used, no error found total csum bytes: 300122908 total tree bytes: 1311391744 total fs tree bytes: 909393920 total extent tree bytes: 47431680 btree space waste bytes: 198618602 file data blocks allocated: 1411201572864 referenced 307423453184 Pass #5: ======== [1/7] checking root items [2/7] checking extents [3/7] checking free space cache [4/7] checking fs roots [5/7] checking csums against data mirror 1 bytenr 15634317312 csum 0x5b21923e expected csum 0xb2809f9e ERROR: errors found in csum tree [6/7] checking root refs [7/7] checking quota groups skipped (not enabled on this FS) Opening filesystem to check... Checking filesystem on /dev/disk/by-label/SysStorage UUID: f827adea-4448-493c-9ae9-3154458bc814 cache and super generation don't match, space cache will be invalidated found 309086564352 bytes used, error(s) found total csum bytes: 300122908 total tree bytes: 1311391744 total fs tree bytes: 909393920 total extent tree bytes: 47431680 btree space waste bytes: 198618602 file data blocks allocated: 1411201572864 referenced 307423453184 > > > Then could you run "btrfs check --check-data-csum" on the unmount fs for > around 256~512 runs with all output dumped? > Unfortunately I cannot do that for practical reasons. As you may have noticed, the file system contains my machine's root file system, so in order to have it unmounted I have to take the machine down (boot into initramfs and run the tests from there). Doing this for longer periods of time is out of the question, as it which would leave my entire network without internet connectivity for the entire duration of this test. I am currently running a memory test, it has been executing for about an hour now and did not detect any issues so far. I'll leave it running overnight. I also find it interesting that only the offline scrub seems to be affected; online scrubbing apparently works just fine. Memtest is always a good idea to rule out the hardware problem. Can't wait for DDR5 to bring ECC to mainstream systems. For the possible reason why online scrub not exposing the same behavior, it can be possible that the offending pages are only allocated to user space progcesses. But for now I'm not that into the idea of RAM corruption. As normally btrfs kernel module should be enough to detect basic bitflips (yeah, btrfs can detect bad RAM!) If you can still test, I could craft some debug patches to dump the corrupted sectors to see if it's a bitflip, or we're getting garbage in user space. (In reply to Qu Wenruo from comment #8) > If you can still test, I could craft some debug patches to dump the > corrupted sectors to see if it's a bitflip, or we're getting garbage in user > space. I have a second btrfs file system on the same machine; it lives on a set of old hard disks I did not want to dispose off: [root@sarkovy ~]# btrfs filesystem usage /workspace Overall: Device size: 1.82TiB Device allocated: 17.02GiB Device unallocated: 1.80TiB Device missing: 0.00B Used: 12.35GiB Free (estimated): 1.80TiB (min: 925.71GiB) Free (statfs, df): 1.35TiB Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 29.08MiB (used: 0.00B) Multiple profiles: no Data,RAID0: Size:15.00GiB, Used:12.30GiB (81.98%) /dev/sdc 5.00GiB /dev/sdd 5.00GiB /dev/sde 5.00GiB Metadata,RAID1: Size:1.00GiB, Used:29.59MiB (2.89%) /dev/sdd 1.00GiB /dev/sde 1.00GiB System,RAID1: Size:8.00MiB, Used:16.00KiB (0.20%) /dev/sdd 8.00MiB /dev/sde 8.00MiB Unallocated: /dev/sdc 460.76GiB /dev/sdd 459.75GiB /dev/sde 925.50GiB This file system can be unmounted while the system is up and running. It is not used very much and contains much less data than the other one. I tried to compensate for this by increasing the number of test passes to 100. And indeed, one of the 100 passes actually reported a data checksum error. I could also move the contents to some other place, so that it is completely empty, and then create a number of large files containing either random garbage (from /dev/urandom) or, maybe even better, all zeroes (from /dev/zero) so that the contents of all file data blocks are known. What do you think? I ran a more thorough memory test and found that my RAM is failing, so that's probably the reason of the problem. Sorry about that, but typically 3rd party RAM is covered by lifetime warranty (and I'd put up a huge fuss if I discovered it in soldered RAM). What memory tester did you use? (In reply to Chris Murphy from comment #11) > Sorry about that, but typically 3rd party RAM is covered by lifetime > warranty (and I'd put up a huge fuss if I discovered it in soldered RAM). > What memory tester did you use? I used memtester: http://pyropus.ca/software/memtester/old-versions/memtester-4.5.1.tar.gz. My machine is equipped with 16GB RAM. I disabled the upper 8GB by using the 'mem=8G' kernel argument, disabled kernel lockdown in order to get access to /dev/mem (mokutil --disable-validation) and then ran the memory test on the upper 8GB of RAM. The test detected a considerable amount of single-bit failures at various addresses and bit positions. The machine is quite old, I thing I am going to replace it. Too bad. I'm not aware of memtester producing false positives for bad RAM, so it should be reliable and by reports seems to find problem quicker than other testers. We've been discussing pcmemtest on devel@ list, it's not yet built in Fedora repos but you can build it for BIOS or UEFI: https://github.com/martinwhitaker/pcmemtest - a drawback of this method is that with UEFI Secure Boot enabled you need to self-sign it or disable UEFI Secure Boot and I won't recommend doing that (it's almost certainly safe to do so, but it's just not a good idea for person A to recommend that person B take an unqualified risk). I'll close this bug for now as notabug, if you learn something new that makes you suspect a bug in btrfs-progs please reopen. (I intend to do a bunch of offline scrubs on all my btrfs's to see if I can reproduce.) |