Bug 1011714 - btrfs scrub finds csum corruption after btrfs balance
Summary: btrfs scrub finds csum corruption after btrfs balance
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 20
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Josef Bacik
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedFreezeException
Depends On:
Blocks: F20BetaFreezeException F20FinalBlocker
TreeView+ depends on / blocked
 
Reported: 2013-09-24 23:06 UTC by Chris Murphy
Modified: 2014-08-31 16:47 UTC (History)
16 users (show)

Fixed In Version: kernel-3.11.7-100.fc18
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-11-06 20:19:17 UTC


Attachments (Terms of Use)
dmesg on reboot after balance (239.70 KB, text/plain)
2013-09-24 23:07 UTC, Chris Murphy
no flags Details
dmesg 3.12.0-0.rc2.git0.1 (126.52 KB, text/plain)
2013-09-25 01:51 UTC, Chris Murphy
no flags Details

Description Chris Murphy 2013-09-24 23:06:03 UTC
Description of problem: btrfs scrub finds no problems prior to balance, after balance it finds csum corruption and doesn't fix it.


Version-Release number of selected component (if applicable):
File system created with Fedora 20 alpha:
kernel-3.11.0-300.fc20.x86_64
btrfs-progs-0.20.rc1.20130501git7854c8b-4.fc20.x86_64


File system scrubbed, balanced, scrubbed again with:
kernel-3.11.1-300.fc20.x86_64
btrfs-progs-0.20.rc1.20130917git194aa4a-1.fc20.x86_64



How reproducible:
Always


Steps to Reproduce:
1. wipefs -a all preexisting partitions
2. Boot Fedora 20 alpha installer, netinst.iso
3. Layout on GPT, three partitions: swap, EFI system partition, btrfs with subvolumes boot, root, home.
4. Reboot after successful install.
5. btrfs scrub /
(no errors reported)
6. btrfs balance start /
(no errors reported)
7. btrfs scrub start /
csum errors reported


Actual results:
csum errors reported by scrub, and also on all subsequent reboots in dmesg


Expected results:
No balance induced corruption.


Additional info:
reproducible on the follow two completely different hardware booted in EFI mode
Apple Inc. MacBookPro4,1 with WDC Scorpio Blue SATA WD5000BEVT-22ZAT0
Apple Inc. MacBookPro8,2 with SAMSUNG SSD 830 Series

Of course in the first case metadata profile is DUP, yet not repairable. And the second case metadata is single, by default.

btrfs-image of the file system is available

Comment 1 Chris Murphy 2013-09-24 23:07:18 UTC
Created attachment 802521 [details]
dmesg on reboot after balance

full dmesg, after reboot following balance and scrub indicating csum corruption

Comment 2 Chris Murphy 2013-09-24 23:10:53 UTC
Starting at step 6 in the Description above:

[root@oldlaptop ~]# btrfs balance start /
Done, had to relocate 5 out of 5 chunks

[root@oldlaptop ~]# dmesg (snippet)
[  390.770699] btrfs: relocating block group 1103101952 flags 1
[  406.639113] btrfs: found 10341 extents
[  414.172873] btrfs: found 10331 extents
[  414.530059] btrfs: relocating block group 29360128 flags 36
[  418.761208] btrfs: found 9281 extents
[  419.136338] btrfs: relocating block group 20971520 flags 34
[  419.536539] btrfs: found 1 extents
[  419.880757] btrfs: relocating block group 12582912 flags 1
[  420.380511] btrfs: found 282 extents
[  421.080667] btrfs: found 282 extents
[  421.426891] btrfs: relocating block group 4194304 flags 4



[root@oldlaptop ~]# btrfs scrub start /
scrub started on /, fsid 1463a31b-472a-47cd-a8c8-86bf09f978fa (pid=894)

[root@oldlaptop ~]# dmesg (snippet)
[  460.533990] btrfs: checksum error at logical 2607853568 on dev /dev/sda5, sector 7207000, root 256, inode 24622, offset 4247552, length 4096, links 1 (path: var/log/journal/d212cf4a840f4e78a33781c56189a7da/system.journal)
[  460.534045] btrfs: bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[  460.534082] btrfs: unable to fixup (regular) error at logical 2607853568 on dev /dev/sda5
[  460.534581] btrfs: checksum error at logical 2607869952 on dev /dev/sda5, sector 7207032, root 256, inode 24622, offset 4263936, length 4096, links 1 (path: var/log/journal/d212cf4a840f4e78a33781c56189a7da/system.journal)
[  460.534594] btrfs: bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[  460.534614] btrfs: unable to fixup (regular) error at logical 2607869952 on dev /dev/sda5
[  460.535128] btrfs: checksum error at logical 2607886336 on dev /dev/sda5, sector 7207064, root 256, inode 24622, offset 4280320, length 4096, links 1 (path: var/log/journal/d212cf4a840f4e78a33781c56189a7da/system.journal)
[  460.535140] btrfs: bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
[  460.535161] btrfs: unable to fixup (regular) error at logical 2607886336 on dev /dev/sda5
[  460.535607] btrfs: checksum error at logical 2607902720 on dev /dev/sda5, sector 7207096, root 256, inode 24622, offset 4296704, length 4096, links 1 (path: var/log/journal/d212cf4a840f4e78a33781c56189a7da/system.journal)
[  460.535619] btrfs: bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
[  460.535639] btrfs: unable to fixup (regular) error at logical 2607902720 on dev /dev/sda5
[  460.536421] btrfs: checksum error at logical 2608025600 on dev /dev/sda5, sector 7207336, root 256, inode 24622, offset 4313088, length 4096, links 1 (path: var/log/journal/d212cf4a840f4e78a33781c56189a7da/system.journal)
[  460.536437] btrfs: bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0
[  460.536457] btrfs: unable to fixup (regular) error at logical 2608025600 on dev /dev/sda5
[  460.779192] btrfs: checksum error at logical 2626674688 on dev /dev/sda5, sector 7243760, root 256, inode 24622, offset 4595712, length 4096, links 1 (path: var/log/journal/d212cf4a840f4e78a33781c56189a7da/system.journal)
[  460.779210] btrfs: bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 6, gen 0
[  460.779245] btrfs: unable to fixup (regular) error at logical 2626674688 on dev /dev/sda5
[  460.779822] btrfs: checksum error at logical 2626715648 on dev /dev/sda5, sector 7243840, root 256, inode 24622, offset 4231168, length 4096, links 1 (path: var/log/journal/d212cf4a840f4e78a33781c56189a7da/system.journal)
[  460.779834] btrfs: bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0
[  460.779854] btrfs: unable to fixup (regular) error at logical 2626715648 on dev /dev/sda5


First reboot after the above scrub completes:

And now on reboot:
[root@f20s ~]# dmesg | grep -i btrfs
[    1.725224] Btrfs loaded
[    1.980491] btrfs: disk space caching is enabled
[    2.001684] btrfs: bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0
[    3.011628] SELinux: initialized (dev sda5, type btrfs), uses xattr
[    5.092593] btrfs: disk space caching is enabled
[    8.703883] btrfs no csum found for inode 24622 start 4235264
[    8.844562] btrfs no csum found for inode 24622 start 4251648
[    8.844589] btrfs no csum found for inode 24622 start 4272128
[    8.844611] btrfs no csum found for inode 24622 start 4288512
[    8.844632] btrfs no csum found for inode 24622 start 4304896
[    8.844658] btrfs no csum found for inode 24622 start 4321280
[    8.856069] BTRFS info (device sda5): csum failed ino 24622 off 4251648 csum 1113579642 private 0
[    8.856084] BTRFS info (device sda5): csum failed ino 24622 off 4272128 csum 2433646103 private 0
[    8.856092] BTRFS info (device sda5): csum failed ino 24622 off 4288512 csum 2276263411 private 0
[    8.857248] BTRFS info (device sda5): csum failed ino 24622 off 4304896 csum 1156822344 private 0
[    8.857424] BTRFS info (device sda5): csum failed ino 24622 off 4321280 csum 3967991073 private 0
[    8.867242] BTRFS info (device sda5): csum failed ino 24622 off 4235264 csum 172180530 private 0

Comment 3 Chris Murphy 2013-09-25 01:05:47 UTC
Starting with F20 alpha RC4, this is reproducible when the btrfs volume is created with, and system clean installed using:
kernel-3.11.0-300.fc20.x86_64
btrfs-progs-0.20.rc1.20130917git194aa4a-1.fc20.x86_64

Then reboot, scrub (comes up clean), balance (no errors), scrub (csum errors unfixable) using:
kernel-3.12.0-0.rc2.git0.1.fc21
btrfs-progs-0.20.rc1.20130917git194aa4a-1.fc20.x86_64

Will post a complete dmesg shortly. For some reason none of my emails today are going to the btrfs list although I am receiving.

Comment 4 Chris Murphy 2013-09-25 01:51:30 UTC
Created attachment 802528 [details]
dmesg 3.12.0-0.rc2.git0.1

3.12.0-0.rc2.git0.1.fc21.x86_64+debug
btrfs-progs-0.20.rc1.20130917git194aa4a-1.fc20.x86_64

Reboot followed by scrub with no errors, followed by balance, followed by scrub which has many csum corruption errors. With this debug kernel, I get a deadlock warning right before the csum errors but I'm uncertain if it's related to the ensuing csum errors.

Comment 5 Chris Murphy 2013-09-25 04:35:09 UTC
Looks like this is systemd-journal related. If I change from persistent to volatile storage of the journal, erase the corrupt logs, the problem is not reproducible. The corruption seems to be limited to journald logs so far.

Further, this is not a corruption of btrfs metadata, but data itself:

[   19.354354] systemd-journald[210]: /var/log/journal/8e4cbfea404512ae70096c6202c9a3bf/system.journal: Journal file corrupted, rotating.

When changing systemd-journal to use volatile instead of persistent storage, the problem is not reproducible.

However, even after deleting all corrupt journal files, and a subsequent scrub reporting no errors, on each reboot (and mount of the filesystem) I get:

[    3.646448] btrfs: bdev /dev/sda6 errs: wr 0, rd 0, flush 0, corrupt 17, gen 0

Comment 6 Chris Murphy 2013-09-25 04:50:32 UTC
chattr +C /var/log/journal

then deleting existing journals, rebooting, appears to solve this problem.

Comment 7 Josef Bacik 2013-09-27 13:38:36 UTC
Just posted the patch upstream and I've marked it for stable so it should make its way back to fedora soon.  The patch is

[PATCH] Btrfs: relocate csums properly with prealloc extents

Thanks.

Comment 8 Chris Murphy 2013-10-05 00:08:15 UTC
Nominating as Fedora 20 beta freeze exception *and* final blocker: "All known bugs that can cause corruption of user data must be fixed or documented at Common F20 bugs."

Workaround above, and permanent fix on the way, but figure it's best to nominate so we don't lose track of it.

Comment 9 Mike Ruckman 2013-10-09 21:13:27 UTC
Discussed this in 2013-10-09 Blocker Review Meeting [1]. We'd like to see more information on how big/invasive/risky this patch is before making a decision on FE, will ask relevant devs and revisit at the next review meeting.

[1] http://meetbot.fedoraproject.org/fedora-blocker-review/2013-10-09/

Comment 10 Josh Boyer 2013-10-10 13:11:00 UTC
Link to the actual patch:

http://www.spinics.net/lists/linux-btrfs/msg27836.html

It hasn't been pulled into the btrfs tree or Linus' tree yet.  That means we haven't picked it up via stable, and likely won't until at least 3.11.6.  There are about 4 other btrfs commits that should be making their way to stable in that timeframe as well.

The patch looks fairly straight forward to me, but Josef can probably chime in on the "riskiness" of it better than I could.

(Moving this bug to POST until something is actually committed to the Fedora tree)

Comment 11 Adam Williamson 2013-10-16 17:37:32 UTC
Discussed at 2013-10-16 freeze exception review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2013-10-16/f20beta-blocker-review-4.2013-10-16-16.02.log.txt . Accepted as a freeze exception, but we'd rather the patch go in ASAP, so we have time to do further testing after it lands and back out if it turns out to be a bad idea.

Comment 12 Josh Boyer 2013-10-16 17:52:12 UTC
Patch added and build started.

Comment 13 Fedora Update System 2013-10-17 02:13:03 UTC
kernel-3.11.5-302.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/kernel-3.11.5-302.fc20

Comment 14 Fedora Update System 2013-10-17 20:26:52 UTC
Package kernel-3.11.5-302.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing kernel-3.11.5-302.fc20'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-19165/kernel-3.11.5-302.fc20
then log in and leave karma (feedback).

Comment 15 Fedora Update System 2013-10-20 19:14:25 UTC
kernel-3.11.6-200.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/kernel-3.11.6-200.fc19

Comment 16 Fedora Update System 2013-10-20 19:15:52 UTC
kernel-3.11.6-100.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/kernel-3.11.6-100.fc18

Comment 17 Chris Murphy 2013-10-22 02:01:18 UTC
Appears fixed in kernel-3.11.5-302.fc20 with no regressions, will continue testing with kernel-3.11.6-300.fc20.

Comment 18 Adam Williamson 2013-10-23 00:23:36 UTC
3.11.6-300 went stable, this bug just wasn't marked as being fixed by it; closing.

Comment 19 Fedora Update System 2013-10-23 03:36:22 UTC
kernel-3.11.6-200.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 20 Fedora Update System 2013-11-02 19:25:10 UTC
kernel-3.11.6-101.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/kernel-3.11.6-101.fc18

Comment 21 Fedora Update System 2013-11-03 04:36:21 UTC
Package kernel-3.11.6-101.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing kernel-3.11.6-101.fc18'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-20545/kernel-3.11.6-101.fc18
then log in and leave karma (feedback).

Comment 22 Fedora Update System 2013-11-04 20:21:08 UTC
kernel-3.11.7-100.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/kernel-3.11.7-100.fc18

Comment 23 Adam Williamson 2013-11-05 05:09:40 UTC
It's still fixed, Bodhi. Quit it.

Comment 24 Fedora Update System 2013-11-06 07:42:42 UTC
Package kernel-3.11.7-100.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing kernel-3.11.7-100.fc18'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-20748/kernel-3.11.7-100.fc18
then log in and leave karma (feedback).

Comment 25 Adam Williamson 2013-11-06 20:19:17 UTC
FFS, Bodhi.

Comment 26 Fedora Update System 2013-11-10 08:04:47 UTC
kernel-3.11.5-302.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 27 Fedora Update System 2013-11-13 02:15:53 UTC
kernel-3.11.7-100.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 28 James Patterson 2014-03-11 12:52:32 UTC
I had this problem after using btrfs-convert: after running a balance, then btrfsck failed with csum failed errors. Latest updates as of today.

Comment 29 D. Charles Pyle 2014-05-11 20:07:57 UTC
Got hit with the same problem with a 3.14.0-1 kernel just yesterday, after conversion but not until after balancing the drive.  Dozens of csum errors.  Filesystem got trashed when I tried to repair it using latest btrfs-progs.  Crashed caja when trying to mount in a live-CD X environment.  Had to do a complete reinstall of Fesora 21 Rawhide with fresh btrfs filesystem because system would no longer boot, including a number of kernel panics involving btrfs.  Saved snapshot also was damaged, and image of prior conversion also was no longer usable for an attempted restore even though stored on drive.

Comment 30 Chris Murphy 2014-05-11 20:45:05 UTC
(In reply to D. Charles Pyle from comment #29)
No, this bug is specifically doing back to back scrub-balance-scrub, with no corruption with the first scrub, and with corruption for the 2nd scrub, specifically with fallocated files (such as systemd logs). It was fixed quite a while ago and I see no regressions in 3.14 or 3.15.

Your bug is sufficiently different it needs a separate bug report: you say conversion, but do you mean ext4->btrfs conversion? That's a completely different state and scenario than this bug so it needs to go in the reproduce steps. And when you did the scrub and had csum errors, dmesg should report what files were affected, so we need to know what those files are. Also Rawhide hasn't been on kernel 3.14 for over a month, it's on 3.15 so the new bug report also needs to specify what kernel version and btrfs progs version is being used. Thanks.

Comment 31 D. Charles Pyle 2014-05-12 23:41:47 UTC
(In reply to Chris Murphy from comment #30)
> (In reply to D. Charles Pyle from comment #29)
> No, this bug is specifically doing back to back scrub-balance-scrub, with no
> corruption with the first scrub, and with corruption for the 2nd scrub,
> specifically with fallocated files (such as systemd logs). It was fixed
> quite a while ago and I see no regressions in 3.14 or 3.15.
> 
> Your bug is sufficiently different it needs a separate bug report: you say
> conversion, but do you mean ext4->btrfs conversion? That's a completely
> different state and scenario than this bug so it needs to go in the
> reproduce steps. And when you did the scrub and had csum errors, dmesg
> should report what files were affected, so we need to know what those files
> are. Also Rawhide hasn't been on kernel 3.14 for over a month, it's on 3.15
> so the new bug report also needs to specify what kernel version and btrfs
> progs version is being used. Thanks.

I also did a balance directly after the conversion from ext4 to btrfs.  I checked for errors after the conversion and there were no errors.  After balancing there were errors so I ran scrubbing and then balanced again.  It was after the first balancing that the csum errors began and after the second that the hard drive started to become unbootable and then unmountable.

On the 3.14.0 kernel, I was, until most recently, forced to stick with the 3.14.0 kernel because any 3.15 kernel would take down my system in various nasty ways, as well as causing pulseaudio problems and cause mate-settings-daemon to go into do_wait and crash on reboot.  The latest 3.15 did not do that so I am back on track there in rawhide.

There is no way to reproduce now because I had to wipe the system and start over.  This time I used btrfs on install rather than convcerting.

Comment 32 Chris Murphy 2014-05-13 04:42:00 UTC
(In reply to D. Charles Pyle from comment #31)
While off-topic for this bug, I just installed Fedora 20 to ext4, did a btrfs conversion using btrfs-progs 3.14 and kernel 3.15rc5, scrubbed it, balanced it, scrubbed it again, and there were no errors. It also boots fine. If you can reproduce, please file a new bug. Thanks.

Comment 33 Matthias Scholz 2014-08-29 13:19:35 UTC
(In reply to D. Charles Pyle from comment #31)
I could reproduce problem doing exact the same thing:
Doing btrfs-convert and than calling btrfs balance on this device results in csum errors, too.

"""
[13462.740058] BTRFS: checksum error at logical 65601536 on dev /dev/sdb5, sector 128128, root 256, inode 257, offset 16384, length 4096, links 1 (path: image)
"""
So it seems not to be related to the mentioned journal corruption.

Using kernel 3.15.10-200.fc20.x86_64

Should the bug be reopened?

Comment 34 Chris Murphy 2014-08-31 16:47:35 UTC
I suggest not reopening it. Comments 28, 29, 33 are all related to btrfs-convert as the means of creating the Btrfs filesystem, so a new bug should be filed. Ideally, first try to reproduce this with a current kernel and btrfs-progs: btrfs-progs-3.16 is available in koji, and I'd suggest using a 3.17rc2 or rc3 kernel also in koji.


Note You need to log in before you can comment on or make changes to this bug.