Bug 1997477 - Inconsistent results from repeated runs of 'btrfs check' command
Summary: Inconsistent results from repeated runs of 'btrfs check' command
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: btrfs-progs
Version: 34
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: Josef Bacik
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-25 10:31 UTC by Thomas Köller
Modified: 2021-09-02 20:38 UTC (History)
7 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2021-09-02 20:38:46 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Thomas Köller 2021-08-25 10:31:15 UTC
Description of problem:

I am checking an unmounted btrfs file system for checksum errors by executing the command 'btrfs check --check-data-csum /dev/disk/by-label/SysStorage'. Doing this in a loop several times over, the results of the individual passes are different:

Pass #1
=======
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking csums against data
mirror 2 bytenr 146126761984 csum 231 expected csum 14
mirror 1 bytenr 289154838528 csum 101 expected csum 55
ERROR: errors found in csum tree
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
Opening filesystem to check...
Checking filesystem on /dev/disk/by-label/SysStorage
UUID: f827adea-4448-493c-9ae9-3154458bc814
found 308654841856 bytes used, error(s) found
total csum bytes: 299925284
total tree bytes: 1311211520
total fs tree bytes: 910000128
total extent tree bytes: 47087616
btree space waste bytes: 196926976
file data blocks allocated: 1409247297536
 referenced 307116109824

Pass #2
=======
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking csums against data
mirror 2 bytenr 116899885056 csum 189 expected csum 242
mirror 1 bytenr 243106209792 csum 146 expected csum 123
ERROR: errors found in csum tree
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
Opening filesystem to check...
Checking filesystem on /dev/disk/by-label/SysStorage
UUID: f827adea-4448-493c-9ae9-3154458bc814
found 308654841856 bytes used, error(s) found
total csum bytes: 299925284
total tree bytes: 1311211520
total fs tree bytes: 910000128
total extent tree bytes: 47087616
btree space waste bytes: 196926976
file data blocks allocated: 1409247297536
 referenced 307116109824

Pass #3
=======
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking csums against data
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
Opening filesystem to check...
Checking filesystem on /dev/disk/by-label/SysStorage
UUID: f827adea-4448-493c-9ae9-3154458bc814
found 308654841856 bytes used, no error found
total csum bytes: 299925284
total tree bytes: 1311211520
total fs tree bytes: 910000128
total extent tree bytes: 47087616
btree space waste bytes: 196926976
file data blocks allocated: 1409247297536
 referenced 307116109824

Pass #4
=======
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking csums against data
mirror 1 bytenr 273130061824 csum 55 expected csum 222
ERROR: errors found in csum tree
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
Opening filesystem to check...
Checking filesystem on /dev/disk/by-label/SysStorage
UUID: f827adea-4448-493c-9ae9-3154458bc814
found 308654841856 bytes used, error(s) found
total csum bytes: 299925284
total tree bytes: 1311211520
total fs tree bytes: 910000128
total extent tree bytes: 47087616
btree space waste bytes: 196926976
file data blocks allocated: 1409247297536
 referenced 307116109824

Pass #5
=======
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking csums against data
mirror 2 bytenr 14546903040 csum 224 expected csum 125
mirror 2 bytenr 272884711424 csum 65 expected csum 226
ERROR: errors found in csum tree
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
Opening filesystem to check...
Checking filesystem on /dev/disk/by-label/SysStorage
UUID: f827adea-4448-493c-9ae9-3154458bc814
found 308654841856 bytes used, error(s) found
total csum bytes: 299925284
total tree bytes: 1311211520
total fs tree bytes: 910000128
total extent tree bytes: 47087616
btree space waste bytes: 196926976
file data blocks allocated: 1409247297536
 referenced 307116109824



Version-Release number of selected component (if applicable):

[root@sarkovy workspace]# btrfs --version
btrfs-progs v5.13.1 


How reproducible:

always

Steps to Reproduce:

Run the above command repeatedly

Actual results:

Results are different for every run


Expected results:

Results should be consistent.


Additional info:

Mounting the file system and then scrubbing it doues not detect any problems with mismatching checksums:

[root@sarkovy workspace]# btrfs scrub start -B /
scrub done for f827adea-4448-493c-9ae9-3154458bc814
Scrub started:    Wed Aug 25 09:49:02 2021
Status:           finished
Duration:         0:10:31
Total to scrub:   586.02GiB
Rate:             933.36MiB/s
Error summary:    no errors found

Comment 1 Chris Murphy 2021-08-25 18:25:21 UTC
Offline scrub finds no problems 1 in 5 attempts. What happens with multiple online scrubs?

Transient errors like this suggest a hardware issue: it could be bad RAM, it could be prefailing device. HDD tend not to have transient checksum failures, but SSDs do and it is one of the earliest indications of a device that will eventually fail.

You can use the following and check to see if there are any unexpected complaints, or attached to the bug:

journalctl --no-hostname | grep "ata\|Btrfs\|BTRFS\|] hd\| scsi\| sd\| sdhci\| mmc\| nvme" > journal.log

sudo dnf install smartmontools
sudo smartctl -x /dev/sdX > smartctl.log

Comment 2 Thomas Köller 2021-08-25 21:23:38 UTC
(In reply to Chris Murphy from comment #1)
> Offline scrub finds no problems 1 in 5 attempts. What happens with multiple
> online scrubs?

These invariably do not show any errors.

> 
> Transient errors like this suggest a hardware issue: it could be bad RAM, it
> could be prefailing device. HDD tend not to have transient checksum
> failures, but SSDs do and it is one of the earliest indications of a device
> that will eventually fail.
> 
> You can use the following and check to see if there are any unexpected
> complaints, or attached to the bug:
> 
> journalctl --no-hostname | grep "ata\|Btrfs\|BTRFS\|] hd\| scsi\| sd\|
> sdhci\| mmc\| nvme" > journal.log

Looking at the journal did not reveal anything suspicious.

> 
> sudo dnf install smartmontools
> sudo smartctl -x /dev/sdX > smartctl.log

After running 'smartctl -t long' on devices /dev/sda and /dev/sdb, the constituent devices of the file system, no errors are reported:

[root@sarkovy workspace]# smartctl -l selftest /dev/sda
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.13.12-200.fc34.x86_64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     12005         -
# 2  Extended offline    Interrupted (host reset)      50%     12003         -
# 3  Extended offline    Completed without error       00%     11903         -
# 4  Extended offline    Completed without error       00%      9242         -
[root@sarkovy workspace]# smartctl -l xerror /dev/sda
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.13.12-200.fc34.x86_64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Extended Comprehensive Error Log Version: 1 (1 sectors)
No Errors Logged

[root@sarkovy workspace]# smartctl -l selftest /dev/sdb
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.13.12-200.fc34.x86_64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     12005         -
# 2  Extended offline    Interrupted (host reset)      50%     12003         -
# 3  Extended offline    Completed without error       00%     11903         -
# 4  Extended offline    Completed without error       00%      9242         -
[root@sarkovy workspace]# smartctl -l xerror /dev/sdb
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.13.12-200.fc34.x86_64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Extended Comprehensive Error Log Version: 1 (1 sectors)
No Errors Logged

Also note that the detected checksum errors are on both mirror devices, which would mean that both these devices had started to fail simultaneously. That is quite unlikely.

Comment 3 Chris Murphy 2021-08-26 03:14:10 UTC
>btrfs check --check-data-csum /dev/disk/by-label/SysStorage

Is the file system entirely umounted? You say it's unmounted in the description but because btrfs supports mounting it multiple times, and anywhere, it can often seem to be umounted but there's a straggler somewhere that makes the fs still mounted.

Comment 4 Qu Wenruo 2021-08-26 05:42:46 UTC
Firstly, btrfs check will reject the device if it's already mounted.

And the reporter is not using --force option, it should not be the case.


Since the reporter is using two devices, and two data copies, I guess it's RAID1.

Can you provide "btrfs fi usage" of the fs when mounted?

Comment 5 Thomas Köller 2021-08-26 09:30:59 UTC
(In reply to Qu Wenruo from comment #4)
> Firstly, btrfs check will reject the device if it's already mounted.
> 
> And the reporter is not using --force option, it should not be the case.
> 

Exactly.

> 
> Since the reporter is using two devices, and two data copies, I guess it's
> RAID1.

It is raid1 for both data and metadata.

> 
> Can you provide "btrfs fi usage" of the fs when mounted?

[root@sarkovy ~]# btrfs fi usage /
Overall:
    Device size:		   1.82TiB
    Device allocated:		 586.02GiB
    Device unallocated:		   1.25TiB
    Device missing:		     0.00B
    Used:			 575.18GiB
    Free (estimated):		 642.63GiB	(min: 642.63GiB)
    Free (statfs, df):		 642.63GiB
    Data ratio:			      2.00
    Metadata ratio:		      2.00
    Global reserve:		 382.77MiB	(used: 0.00B)
    Multiple profiles:		        no

Data,RAID1: Size:291.00GiB, Used:286.37GiB (98.41%)
   /dev/sda2	 291.00GiB
   /dev/sdb2	 291.00GiB

Metadata,RAID1: Size:2.00GiB, Used:1.22GiB (61.06%)
   /dev/sda2	   2.00GiB
   /dev/sdb2	   2.00GiB

System,RAID1: Size:8.00MiB, Used:64.00KiB (0.78%)
   /dev/sda2	   8.00MiB
   /dev/sdb2	   8.00MiB

Unallocated:
   /dev/sda2	 638.00GiB
   /dev/sdb2	 638.00GiB

Comment 6 Qu Wenruo 2021-08-26 10:11:25 UTC
Can you fetch the following btrfs-progs branch?

https://github.com/adam900710/btrfs-progs/tree/csum_enhance

This branch has proper csum error output message instead of the meaningless one.


Then could you run "btrfs check --check-data-csum" on the unmount fs for around 256~512 runs with all output dumped?


The 5 runs provided has some interesting results:
Pass #1
=======
mirror 2 bytenr 146126761984 csum 231 expected csum 14
mirror 1 bytenr 289154838528 csum 101 expected csum 55

Pass #2
=======
mirror 2 bytenr 116899885056 csum 189 expected csum 242
mirror 1 bytenr 243106209792 csum 146 expected csum 123

Pass #3
=======
NULL

Pass #4
=======
mirror 1 bytenr 273130061824 csum 55 expected csum 222

Pass #5
=======
mirror 2 bytenr 14546903040 csum 224 expected csum 125
mirror 2 bytenr 272884711424 csum 65 expected csum 226

Notice that all bytenrs are unique, thus I guess it may be some unexpected memory corruption.

Not sure if it's btrfs-progs' code, or hardware.

If you can accumulate more runs, maybe we can find some pattern just from the result.

Comment 7 Thomas Köller 2021-08-26 22:14:30 UTC
(In reply to Qu Wenruo from comment #6)
> Can you fetch the following btrfs-progs branch?
> 
> https://github.com/adam900710/btrfs-progs/tree/csum_enhance
> 
> This branch has proper csum error output message instead of the meaningless
> one.

I tried that, but the results were basically the same as before. Here's the output:

Pass #1:
========
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking csums against data
mirror 2 bytenr 55823724544 csum 0xf3c56b8e expected csum 0x9e996d9c
ERROR: errors found in csum tree
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
Opening filesystem to check...
Checking filesystem on /dev/disk/by-label/SysStorage
UUID: f827adea-4448-493c-9ae9-3154458bc814
cache and super generation don't match, space cache will be invalidated
found 309086564352 bytes used, error(s) found
total csum bytes: 300122908
total tree bytes: 1311391744
total fs tree bytes: 909393920
total extent tree bytes: 47431680
btree space waste bytes: 198618602
file data blocks allocated: 1411201572864
 referenced 307423453184

Pass #2:
========
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking csums against data
mirror 2 bytenr 112425172992 csum 0x3c2ff1d1 expected csum 0xcb5122e0
ERROR: errors found in csum tree
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
Opening filesystem to check...
Checking filesystem on /dev/disk/by-label/SysStorage
UUID: f827adea-4448-493c-9ae9-3154458bc814
cache and super generation don't match, space cache will be invalidated
found 309086564352 bytes used, error(s) found
total csum bytes: 300122908
total tree bytes: 1311391744
total fs tree bytes: 909393920
total extent tree bytes: 47431680
btree space waste bytes: 198618602
file data blocks allocated: 1411201572864
 referenced 307423453184

Pass #3:
========
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking csums against data
mirror 1 bytenr 15479435264 csum 0xa7d11656 expected csum 0x543711f8
ERROR: errors found in csum tree
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
Opening filesystem to check...
Checking filesystem on /dev/disk/by-label/SysStorage
UUID: f827adea-4448-493c-9ae9-3154458bc814
cache and super generation don't match, space cache will be invalidated
found 309086564352 bytes used, error(s) found
total csum bytes: 300122908
total tree bytes: 1311391744
total fs tree bytes: 909393920
total extent tree bytes: 47431680
btree space waste bytes: 198618602
file data blocks allocated: 1411201572864
 referenced 307423453184

Pass #4:
========
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking csums against data
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
Opening filesystem to check...
Checking filesystem on /dev/disk/by-label/SysStorage
UUID: f827adea-4448-493c-9ae9-3154458bc814
cache and super generation don't match, space cache will be invalidated
found 309086564352 bytes used, no error found
total csum bytes: 300122908
total tree bytes: 1311391744
total fs tree bytes: 909393920
total extent tree bytes: 47431680
btree space waste bytes: 198618602
file data blocks allocated: 1411201572864
 referenced 307423453184

Pass #5:
========
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking csums against data
mirror 1 bytenr 15634317312 csum 0x5b21923e expected csum 0xb2809f9e
ERROR: errors found in csum tree
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
Opening filesystem to check...
Checking filesystem on /dev/disk/by-label/SysStorage
UUID: f827adea-4448-493c-9ae9-3154458bc814
cache and super generation don't match, space cache will be invalidated
found 309086564352 bytes used, error(s) found
total csum bytes: 300122908
total tree bytes: 1311391744
total fs tree bytes: 909393920
total extent tree bytes: 47431680
btree space waste bytes: 198618602
file data blocks allocated: 1411201572864
 referenced 307423453184


> 
> 
> Then could you run "btrfs check --check-data-csum" on the unmount fs for
> around 256~512 runs with all output dumped?
> 

Unfortunately I cannot do that for practical reasons. As you may have noticed, the file system contains my machine's root file system, so in order to have it unmounted I have to take the machine down (boot into initramfs and run the tests from there). Doing this for longer periods of time is out of the question, as it which would leave my entire network without internet connectivity for the entire duration of this test. 

I am currently running a memory test, it has been executing for about an hour now and did not detect any issues so far. I'll leave it running overnight.

I also find it interesting that only the offline scrub seems to be affected; online scrubbing apparently works just fine.

Comment 8 Qu Wenruo 2021-08-27 03:00:08 UTC
Memtest is always a good idea to rule out the hardware problem.
Can't wait for DDR5 to bring ECC to mainstream systems.

For the possible reason why online scrub not exposing the same behavior, it can be possible that the offending pages are only allocated to user space progcesses.

But for now I'm not that into the idea of RAM corruption.
As normally btrfs kernel module should be enough to detect basic bitflips (yeah, btrfs can detect bad RAM!)


If you can still test, I could craft some debug patches to dump the corrupted sectors to see if it's a bitflip, or we're getting garbage in user space.

Comment 9 Thomas Köller 2021-08-27 11:34:51 UTC
(In reply to Qu Wenruo from comment #8)
> If you can still test, I could craft some debug patches to dump the
> corrupted sectors to see if it's a bitflip, or we're getting garbage in user
> space.

I have a second btrfs file system on the same machine; it lives on a set of old hard disks I did not want to dispose off:

[root@sarkovy ~]# btrfs filesystem usage /workspace
Overall:
    Device size:		   1.82TiB
    Device allocated:		  17.02GiB
    Device unallocated:		   1.80TiB
    Device missing:		     0.00B
    Used:			  12.35GiB
    Free (estimated):		   1.80TiB	(min: 925.71GiB)
    Free (statfs, df):		   1.35TiB
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		  29.08MiB	(used: 0.00B)
    Multiple profiles:		        no

Data,RAID0: Size:15.00GiB, Used:12.30GiB (81.98%)
   /dev/sdc	   5.00GiB
   /dev/sdd	   5.00GiB
   /dev/sde	   5.00GiB

Metadata,RAID1: Size:1.00GiB, Used:29.59MiB (2.89%)
   /dev/sdd	   1.00GiB
   /dev/sde	   1.00GiB

System,RAID1: Size:8.00MiB, Used:16.00KiB (0.20%)
   /dev/sdd	   8.00MiB
   /dev/sde	   8.00MiB

Unallocated:
   /dev/sdc	 460.76GiB
   /dev/sdd	 459.75GiB
   /dev/sde	 925.50GiB

This file system can be unmounted while the system is up and running. It is not used very much and contains much less data than the other one. I tried to compensate for this by increasing the number of test passes to 100. And indeed, one of the 100 passes actually reported a data checksum error.

I could also move the contents to some other place, so that it is completely empty, and then create a number of large files containing either random garbage (from /dev/urandom) or, maybe even better, all zeroes (from /dev/zero) so that the contents of all file data blocks are known. What do you think?

Comment 10 Thomas Köller 2021-08-29 23:35:24 UTC
I ran a more thorough memory test and found that my RAM is failing, so that's probably the reason of the problem.

Comment 11 Chris Murphy 2021-08-30 16:14:56 UTC
Sorry about that, but typically 3rd party RAM is covered by lifetime warranty (and I'd put up a huge fuss if I discovered it in soldered RAM). What memory tester did you use?

Comment 12 Thomas Köller 2021-08-30 20:32:43 UTC
(In reply to Chris Murphy from comment #11)
> Sorry about that, but typically 3rd party RAM is covered by lifetime
> warranty (and I'd put up a huge fuss if I discovered it in soldered RAM).
> What memory tester did you use?

I used memtester: http://pyropus.ca/software/memtester/old-versions/memtester-4.5.1.tar.gz. My machine is equipped with 16GB RAM. I disabled the upper 8GB by using the 'mem=8G' kernel argument, disabled kernel lockdown in order to get access to /dev/mem (mokutil --disable-validation) and then ran the memory test on the upper 8GB of RAM. The test detected a considerable amount of single-bit failures at various addresses and bit positions.

The machine is quite old, I thing I am going to replace it.

Comment 13 Chris Murphy 2021-09-02 20:38:46 UTC
Too bad. I'm not aware of memtester producing false positives for bad RAM, so it should be reliable and by reports seems to find problem quicker than other testers. We've been discussing pcmemtest on devel@ list, it's not yet built in Fedora repos but you can build it for BIOS or UEFI: https://github.com/martinwhitaker/pcmemtest - a drawback of this method is that with UEFI Secure Boot enabled you need to self-sign it or disable UEFI Secure Boot and I won't recommend doing that (it's almost certainly safe to do so, but it's just not a good idea for person A to recommend that person B take an unqualified risk).

I'll close this bug for now as notabug, if you learn something new that makes you suspect a bug in btrfs-progs please reopen. (I intend to do a bunch of offline scrubs on all my btrfs's to see if I can reproduce.)


Note You need to log in before you can comment on or make changes to this bug.