Description of problem: btrfs scrub finds no problems prior to balance, after balance it finds csum corruption and doesn't fix it. Version-Release number of selected component (if applicable): File system created with Fedora 20 alpha: kernel-3.11.0-300.fc20.x86_64 btrfs-progs-0.20.rc1.20130501git7854c8b-4.fc20.x86_64 File system scrubbed, balanced, scrubbed again with: kernel-3.11.1-300.fc20.x86_64 btrfs-progs-0.20.rc1.20130917git194aa4a-1.fc20.x86_64 How reproducible: Always Steps to Reproduce: 1. wipefs -a all preexisting partitions 2. Boot Fedora 20 alpha installer, netinst.iso 3. Layout on GPT, three partitions: swap, EFI system partition, btrfs with subvolumes boot, root, home. 4. Reboot after successful install. 5. btrfs scrub / (no errors reported) 6. btrfs balance start / (no errors reported) 7. btrfs scrub start / csum errors reported Actual results: csum errors reported by scrub, and also on all subsequent reboots in dmesg Expected results: No balance induced corruption. Additional info: reproducible on the follow two completely different hardware booted in EFI mode Apple Inc. MacBookPro4,1 with WDC Scorpio Blue SATA WD5000BEVT-22ZAT0 Apple Inc. MacBookPro8,2 with SAMSUNG SSD 830 Series Of course in the first case metadata profile is DUP, yet not repairable. And the second case metadata is single, by default. btrfs-image of the file system is available
Created attachment 802521 [details] dmesg on reboot after balance full dmesg, after reboot following balance and scrub indicating csum corruption
Starting at step 6 in the Description above: [root@oldlaptop ~]# btrfs balance start / Done, had to relocate 5 out of 5 chunks [root@oldlaptop ~]# dmesg (snippet) [ 390.770699] btrfs: relocating block group 1103101952 flags 1 [ 406.639113] btrfs: found 10341 extents [ 414.172873] btrfs: found 10331 extents [ 414.530059] btrfs: relocating block group 29360128 flags 36 [ 418.761208] btrfs: found 9281 extents [ 419.136338] btrfs: relocating block group 20971520 flags 34 [ 419.536539] btrfs: found 1 extents [ 419.880757] btrfs: relocating block group 12582912 flags 1 [ 420.380511] btrfs: found 282 extents [ 421.080667] btrfs: found 282 extents [ 421.426891] btrfs: relocating block group 4194304 flags 4 [root@oldlaptop ~]# btrfs scrub start / scrub started on /, fsid 1463a31b-472a-47cd-a8c8-86bf09f978fa (pid=894) [root@oldlaptop ~]# dmesg (snippet) [ 460.533990] btrfs: checksum error at logical 2607853568 on dev /dev/sda5, sector 7207000, root 256, inode 24622, offset 4247552, length 4096, links 1 (path: var/log/journal/d212cf4a840f4e78a33781c56189a7da/system.journal) [ 460.534045] btrfs: bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0 [ 460.534082] btrfs: unable to fixup (regular) error at logical 2607853568 on dev /dev/sda5 [ 460.534581] btrfs: checksum error at logical 2607869952 on dev /dev/sda5, sector 7207032, root 256, inode 24622, offset 4263936, length 4096, links 1 (path: var/log/journal/d212cf4a840f4e78a33781c56189a7da/system.journal) [ 460.534594] btrfs: bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 [ 460.534614] btrfs: unable to fixup (regular) error at logical 2607869952 on dev /dev/sda5 [ 460.535128] btrfs: checksum error at logical 2607886336 on dev /dev/sda5, sector 7207064, root 256, inode 24622, offset 4280320, length 4096, links 1 (path: var/log/journal/d212cf4a840f4e78a33781c56189a7da/system.journal) [ 460.535140] btrfs: bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0 [ 460.535161] btrfs: unable to fixup (regular) error at logical 2607886336 on dev /dev/sda5 [ 460.535607] btrfs: checksum error at logical 2607902720 on dev /dev/sda5, sector 7207096, root 256, inode 24622, offset 4296704, length 4096, links 1 (path: var/log/journal/d212cf4a840f4e78a33781c56189a7da/system.journal) [ 460.535619] btrfs: bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0 [ 460.535639] btrfs: unable to fixup (regular) error at logical 2607902720 on dev /dev/sda5 [ 460.536421] btrfs: checksum error at logical 2608025600 on dev /dev/sda5, sector 7207336, root 256, inode 24622, offset 4313088, length 4096, links 1 (path: var/log/journal/d212cf4a840f4e78a33781c56189a7da/system.journal) [ 460.536437] btrfs: bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0 [ 460.536457] btrfs: unable to fixup (regular) error at logical 2608025600 on dev /dev/sda5 [ 460.779192] btrfs: checksum error at logical 2626674688 on dev /dev/sda5, sector 7243760, root 256, inode 24622, offset 4595712, length 4096, links 1 (path: var/log/journal/d212cf4a840f4e78a33781c56189a7da/system.journal) [ 460.779210] btrfs: bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 6, gen 0 [ 460.779245] btrfs: unable to fixup (regular) error at logical 2626674688 on dev /dev/sda5 [ 460.779822] btrfs: checksum error at logical 2626715648 on dev /dev/sda5, sector 7243840, root 256, inode 24622, offset 4231168, length 4096, links 1 (path: var/log/journal/d212cf4a840f4e78a33781c56189a7da/system.journal) [ 460.779834] btrfs: bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 [ 460.779854] btrfs: unable to fixup (regular) error at logical 2626715648 on dev /dev/sda5 First reboot after the above scrub completes: And now on reboot: [root@f20s ~]# dmesg | grep -i btrfs [ 1.725224] Btrfs loaded [ 1.980491] btrfs: disk space caching is enabled [ 2.001684] btrfs: bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 [ 3.011628] SELinux: initialized (dev sda5, type btrfs), uses xattr [ 5.092593] btrfs: disk space caching is enabled [ 8.703883] btrfs no csum found for inode 24622 start 4235264 [ 8.844562] btrfs no csum found for inode 24622 start 4251648 [ 8.844589] btrfs no csum found for inode 24622 start 4272128 [ 8.844611] btrfs no csum found for inode 24622 start 4288512 [ 8.844632] btrfs no csum found for inode 24622 start 4304896 [ 8.844658] btrfs no csum found for inode 24622 start 4321280 [ 8.856069] BTRFS info (device sda5): csum failed ino 24622 off 4251648 csum 1113579642 private 0 [ 8.856084] BTRFS info (device sda5): csum failed ino 24622 off 4272128 csum 2433646103 private 0 [ 8.856092] BTRFS info (device sda5): csum failed ino 24622 off 4288512 csum 2276263411 private 0 [ 8.857248] BTRFS info (device sda5): csum failed ino 24622 off 4304896 csum 1156822344 private 0 [ 8.857424] BTRFS info (device sda5): csum failed ino 24622 off 4321280 csum 3967991073 private 0 [ 8.867242] BTRFS info (device sda5): csum failed ino 24622 off 4235264 csum 172180530 private 0
Starting with F20 alpha RC4, this is reproducible when the btrfs volume is created with, and system clean installed using: kernel-3.11.0-300.fc20.x86_64 btrfs-progs-0.20.rc1.20130917git194aa4a-1.fc20.x86_64 Then reboot, scrub (comes up clean), balance (no errors), scrub (csum errors unfixable) using: kernel-3.12.0-0.rc2.git0.1.fc21 btrfs-progs-0.20.rc1.20130917git194aa4a-1.fc20.x86_64 Will post a complete dmesg shortly. For some reason none of my emails today are going to the btrfs list although I am receiving.
Created attachment 802528 [details] dmesg 3.12.0-0.rc2.git0.1 3.12.0-0.rc2.git0.1.fc21.x86_64+debug btrfs-progs-0.20.rc1.20130917git194aa4a-1.fc20.x86_64 Reboot followed by scrub with no errors, followed by balance, followed by scrub which has many csum corruption errors. With this debug kernel, I get a deadlock warning right before the csum errors but I'm uncertain if it's related to the ensuing csum errors.
Looks like this is systemd-journal related. If I change from persistent to volatile storage of the journal, erase the corrupt logs, the problem is not reproducible. The corruption seems to be limited to journald logs so far. Further, this is not a corruption of btrfs metadata, but data itself: [ 19.354354] systemd-journald[210]: /var/log/journal/8e4cbfea404512ae70096c6202c9a3bf/system.journal: Journal file corrupted, rotating. When changing systemd-journal to use volatile instead of persistent storage, the problem is not reproducible. However, even after deleting all corrupt journal files, and a subsequent scrub reporting no errors, on each reboot (and mount of the filesystem) I get: [ 3.646448] btrfs: bdev /dev/sda6 errs: wr 0, rd 0, flush 0, corrupt 17, gen 0
chattr +C /var/log/journal then deleting existing journals, rebooting, appears to solve this problem.
Just posted the patch upstream and I've marked it for stable so it should make its way back to fedora soon. The patch is [PATCH] Btrfs: relocate csums properly with prealloc extents Thanks.
Nominating as Fedora 20 beta freeze exception *and* final blocker: "All known bugs that can cause corruption of user data must be fixed or documented at Common F20 bugs." Workaround above, and permanent fix on the way, but figure it's best to nominate so we don't lose track of it.
Discussed this in 2013-10-09 Blocker Review Meeting [1]. We'd like to see more information on how big/invasive/risky this patch is before making a decision on FE, will ask relevant devs and revisit at the next review meeting. [1] http://meetbot.fedoraproject.org/fedora-blocker-review/2013-10-09/
Link to the actual patch: http://www.spinics.net/lists/linux-btrfs/msg27836.html It hasn't been pulled into the btrfs tree or Linus' tree yet. That means we haven't picked it up via stable, and likely won't until at least 3.11.6. There are about 4 other btrfs commits that should be making their way to stable in that timeframe as well. The patch looks fairly straight forward to me, but Josef can probably chime in on the "riskiness" of it better than I could. (Moving this bug to POST until something is actually committed to the Fedora tree)
Discussed at 2013-10-16 freeze exception review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2013-10-16/f20beta-blocker-review-4.2013-10-16-16.02.log.txt . Accepted as a freeze exception, but we'd rather the patch go in ASAP, so we have time to do further testing after it lands and back out if it turns out to be a bad idea.
Patch added and build started.
kernel-3.11.5-302.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/kernel-3.11.5-302.fc20
Package kernel-3.11.5-302.fc20: * should fix your issue, * was pushed to the Fedora 20 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing kernel-3.11.5-302.fc20' as soon as you are able to, then reboot. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2013-19165/kernel-3.11.5-302.fc20 then log in and leave karma (feedback).
kernel-3.11.6-200.fc19 has been submitted as an update for Fedora 19. https://admin.fedoraproject.org/updates/kernel-3.11.6-200.fc19
kernel-3.11.6-100.fc18 has been submitted as an update for Fedora 18. https://admin.fedoraproject.org/updates/kernel-3.11.6-100.fc18
Appears fixed in kernel-3.11.5-302.fc20 with no regressions, will continue testing with kernel-3.11.6-300.fc20.
3.11.6-300 went stable, this bug just wasn't marked as being fixed by it; closing.
kernel-3.11.6-200.fc19 has been pushed to the Fedora 19 stable repository. If problems still persist, please make note of it in this bug report.
kernel-3.11.6-101.fc18 has been submitted as an update for Fedora 18. https://admin.fedoraproject.org/updates/kernel-3.11.6-101.fc18
Package kernel-3.11.6-101.fc18: * should fix your issue, * was pushed to the Fedora 18 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing kernel-3.11.6-101.fc18' as soon as you are able to, then reboot. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2013-20545/kernel-3.11.6-101.fc18 then log in and leave karma (feedback).
kernel-3.11.7-100.fc18 has been submitted as an update for Fedora 18. https://admin.fedoraproject.org/updates/kernel-3.11.7-100.fc18
It's still fixed, Bodhi. Quit it.
Package kernel-3.11.7-100.fc18: * should fix your issue, * was pushed to the Fedora 18 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing kernel-3.11.7-100.fc18' as soon as you are able to, then reboot. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2013-20748/kernel-3.11.7-100.fc18 then log in and leave karma (feedback).
FFS, Bodhi.
kernel-3.11.5-302.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.
kernel-3.11.7-100.fc18 has been pushed to the Fedora 18 stable repository. If problems still persist, please make note of it in this bug report.
I had this problem after using btrfs-convert: after running a balance, then btrfsck failed with csum failed errors. Latest updates as of today.
Got hit with the same problem with a 3.14.0-1 kernel just yesterday, after conversion but not until after balancing the drive. Dozens of csum errors. Filesystem got trashed when I tried to repair it using latest btrfs-progs. Crashed caja when trying to mount in a live-CD X environment. Had to do a complete reinstall of Fesora 21 Rawhide with fresh btrfs filesystem because system would no longer boot, including a number of kernel panics involving btrfs. Saved snapshot also was damaged, and image of prior conversion also was no longer usable for an attempted restore even though stored on drive.
(In reply to D. Charles Pyle from comment #29) No, this bug is specifically doing back to back scrub-balance-scrub, with no corruption with the first scrub, and with corruption for the 2nd scrub, specifically with fallocated files (such as systemd logs). It was fixed quite a while ago and I see no regressions in 3.14 or 3.15. Your bug is sufficiently different it needs a separate bug report: you say conversion, but do you mean ext4->btrfs conversion? That's a completely different state and scenario than this bug so it needs to go in the reproduce steps. And when you did the scrub and had csum errors, dmesg should report what files were affected, so we need to know what those files are. Also Rawhide hasn't been on kernel 3.14 for over a month, it's on 3.15 so the new bug report also needs to specify what kernel version and btrfs progs version is being used. Thanks.
(In reply to Chris Murphy from comment #30) > (In reply to D. Charles Pyle from comment #29) > No, this bug is specifically doing back to back scrub-balance-scrub, with no > corruption with the first scrub, and with corruption for the 2nd scrub, > specifically with fallocated files (such as systemd logs). It was fixed > quite a while ago and I see no regressions in 3.14 or 3.15. > > Your bug is sufficiently different it needs a separate bug report: you say > conversion, but do you mean ext4->btrfs conversion? That's a completely > different state and scenario than this bug so it needs to go in the > reproduce steps. And when you did the scrub and had csum errors, dmesg > should report what files were affected, so we need to know what those files > are. Also Rawhide hasn't been on kernel 3.14 for over a month, it's on 3.15 > so the new bug report also needs to specify what kernel version and btrfs > progs version is being used. Thanks. I also did a balance directly after the conversion from ext4 to btrfs. I checked for errors after the conversion and there were no errors. After balancing there were errors so I ran scrubbing and then balanced again. It was after the first balancing that the csum errors began and after the second that the hard drive started to become unbootable and then unmountable. On the 3.14.0 kernel, I was, until most recently, forced to stick with the 3.14.0 kernel because any 3.15 kernel would take down my system in various nasty ways, as well as causing pulseaudio problems and cause mate-settings-daemon to go into do_wait and crash on reboot. The latest 3.15 did not do that so I am back on track there in rawhide. There is no way to reproduce now because I had to wipe the system and start over. This time I used btrfs on install rather than convcerting.
(In reply to D. Charles Pyle from comment #31) While off-topic for this bug, I just installed Fedora 20 to ext4, did a btrfs conversion using btrfs-progs 3.14 and kernel 3.15rc5, scrubbed it, balanced it, scrubbed it again, and there were no errors. It also boots fine. If you can reproduce, please file a new bug. Thanks.
(In reply to D. Charles Pyle from comment #31) I could reproduce problem doing exact the same thing: Doing btrfs-convert and than calling btrfs balance on this device results in csum errors, too. """ [13462.740058] BTRFS: checksum error at logical 65601536 on dev /dev/sdb5, sector 128128, root 256, inode 257, offset 16384, length 4096, links 1 (path: image) """ So it seems not to be related to the mentioned journal corruption. Using kernel 3.15.10-200.fc20.x86_64 Should the bug be reopened?
I suggest not reopening it. Comments 28, 29, 33 are all related to btrfs-convert as the means of creating the Btrfs filesystem, so a new bug should be filed. Ideally, first try to reproduce this with a current kernel and btrfs-progs: btrfs-progs-3.16 is available in koji, and I'd suggest using a 3.17rc2 or rc3 kernel also in koji.