Bug 2399898
| Summary: | fs-verity "FILE CORRUPTED!" errors on ostree files after upgrading to kernel 6.16.8 | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Max Chernoff <redhat> | ||||||||||
| Component: | kernel | Assignee: | fedora-kernel-btrfs | ||||||||||
| Status: | NEW --- | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||||||
| Severity: | unspecified | Docs Contact: | |||||||||||
| Priority: | unspecified | ||||||||||||
| Version: | 42 | CC: | acaringi, adscvr, airlied, bugzilla, hdegoede, hpa, josef, kernel-maint, linville, masami256, mchehab, ptalbert, steved, suraj.ghimire7, walters | ||||||||||
| Target Milestone: | --- | ||||||||||||
| Target Release: | --- | ||||||||||||
| Hardware: | x86_64 | ||||||||||||
| OS: | Unspecified | ||||||||||||
| Whiteboard: | |||||||||||||
| Fixed In Version: | Doc Type: | --- | |||||||||||
| Doc Text: | Story Points: | --- | |||||||||||
| Clone Of: | Environment: | ||||||||||||
| Last Closed: | Type: | Bug | |||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||
| Documentation: | --- | CRM: | |||||||||||
| Verified Versions: | Category: | --- | |||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
| Embargoed: | |||||||||||||
| Attachments: |
|
||||||||||||
Created attachment 2107770 [details]
dmesg-6.16.7.log
Created attachment 2107771 [details]
dmesg-6.16.8.log
Max reports the only difference between the deployments is the kernel. And the only btrfs changes between 6.16.7 and 6.16.8 are:
commit 8193ddffd50d2c298469db7de103efa3c382b59f
btrfs: fix corruption reading compressed range when block size is smaller than page size
commit d50721cbc9d6c68790caa3eb92a9190e62d9da72
btrfs: use readahead_expand() on compressed extents
But also, while the lower file system on vda4 (btrfs) supports fs-verity, Max reports composefs.enabled = yes, not composefs.enabled = verity.
The first error looks like this, the immediately preceding message is unrelated and 3 minutes earlier.
> [ 221.844071] kernel: fs-verity (vda4, inode 9108895): FILE CORRUPTED! pos=40960, level=0, want_hash=sha256:76f0ce8a0517fd59ad7e063a9a2be910655f55d5707872183f013b9ea08562dd, real_hash=sha256:ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7
This looks like in-kernel signature verification, but is that expected? I'm wonder if the messages themselves are bogus, but then how is that possible?
No config changes or new patches: https://src.fedoraproject.org/rpms/kernel/c/5ccfa7c145d189accf6c27c25a05367235c77fbe?branch=f42 I was able to reproduce this with the debug kernel. With the debug kernel booted, I also tried setting SELinux to permissive after the system booted, but that didn't make any difference. I've attached the full-ish system logs (the exact command that I ran was `journalctl -b-3 --output=short-monotonic --no-hostname _UID=0 + _TRANSPORT=kernel | grep -Ev "staff_sudo_t|sshd-session"`). If it's any help, the full server configuration is available on GitHub: https://github.com/gucci-on-fleek/maxchernoff.ca > And the only btrfs changes between 6.16.7 and 6.16.8 are: > > commit 8193ddffd50d2c298469db7de103efa3c382b59f > btrfs: fix corruption reading compressed range when block size is smaller than page size > > commit d50721cbc9d6c68790caa3eb92a9190e62d9da72 > btrfs: use readahead_expand() on compressed extents I've set "compress=zstd:3" in /etc/fstab, so I probably have compression enabled (I'm not 100% certain since ostree might ignore the mount options set in /etc/fstab). Created attachment 2107871 [details]
system-logs-6.16.8-debug.log
Max confirms mount shows compression isn't being used for this Btrfs.
> /dev/vda4 on /sysroot type btrfs (ro,relatime,seclabel,discard=async,space_cache=v2,subvolid=5,subvol=/)
Offhand this isn't likely somehow specific to ostree/composefs - it smells like a generic page cache + fsverity regression and is likely reproducible by just enabling fsverity on any file I would guess. > This looks like in-kernel signature verification, but is that expected? I'm wonder if the messages themselves are bogus, but then how is that possible? It's not verifying a signature, it's verifying a digest (more in https://www.kernel.org/doc/html/latest/filesystems/fsverity.html ) Missed these in 6.16.8 somehow, but still don't seem related. And no verity changes in this kernel update. commit f1498abaf74f8d7b1e7001f16ed77818d8ae6a59 btrfs: fix subvolume deletion lockup caused by inodes xarray race commit 203cee72cf98b536ff7479deed29d1f7fd6d8e3b btrfs: fix squota compressed stats leak Fedora kernel config doesn't enable built-in signature verification. # CONFIG_FS_VERITY_BUILTIN_SIGNATURES is not set The error above is in fs/verity/verify.c:231 Max reports using bees (dedup) but although this seems like an obvious explanation, why would one kernel complain about deduped files but not another? And why initially allow access following boot, with the file corrupted message coming after a delay? The output of running `btrfs inspect-internal dump-tree /dev/vda4`: https://www.maxchernoff.ca/files/OQmNrq_6axXhwnrqkzu8D7Qsftf3T_Vv-fs-verity-tree.zstd Ok, since I was able to reproduce the issue by mounting the disk image on my desktop, I was able to bisect it to the following:
d50721cbc9d6c68790caa3eb92a9190e62d9da72 is the first bad commit
commit d50721cbc9d6c68790caa3eb92a9190e62d9da72
Author: Boris Burkov <boris>
Date: Sat Sep 13 10:12:32 2025 -0400
btrfs: use readahead_expand() on compressed extents
[ Upstream commit 9e9ff875e4174be939371667d2cc81244e31232f ]
[...]
Created attachment 2108079 [details]
0001-Revert-btrfs-use-readahead_expand-on-compressed-exte.patch
Reverting d50721cbc9d6 on v6.16.8 also fixes the problem, so that's for sure the problematic commit.
|
1. Please describe the problem: After upgrading to kernel 6.16.8, my Fedora IoT 42 server began issuing lots of "fs-verity: FILE CORRUPTED!" errors on ostree-managed files. Some files were "corrupted" always, while other files were fine after rebooting, and only became "corrupted" after running lots of file read operations. Reverting to the previous ostree deployment with kernel 6.16.7 made the problem go away. "btrfs scrub" and "ostree fsck --all" found no errors, and the exact same files are okay with the earlier kernel, so I don't think that the files are actually corrupted. The root file system is composefs on top of btrfs. I haven't explicitly enabled fs-verity, and the system logs with both 6.16.7 and 6.16.8 claim that it isn't enabled: kernel: Btrfs loaded, zoned=yes, fsverity=yes kernel: fs-verity: sha256 using implementation "sha256-x86_64" ostree-prepare-root[450]: composefs: mounted successfully (verity=false) Some more information is in my Ask Fedora post: https://discussion.fedoraproject.org/t/fs-verity-file-corrupted-errors-on-ostree-file-despite-btrfs-scrub-showing-no-errors/165159 2. What is the Version-Release number of the kernel: 6.16.8 3. Did it work previously in Fedora? If so, what kernel version did the issue *first* appear? Old kernels are available for download at https://koji.fedoraproject.org/koji/packageinfo?packageID=8 : Yes, I had no issues with 6.16.7 (and with all other kernels in the past 12 months); 6.16.8 is the first kernel with the issue. 4. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: On this specific computer, yes. I haven't tried reproducing it on another computer, but the exact container image that I'm using is available from "maxchernoff.ca/fedora-iot:fs-verity-errors". 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: I haven't tested this yet, but I can if you want. 6. Are you running any modules that not shipped with directly Fedora's kernel?: No. 7. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag. See the attachments. I've removed any "audit" entries since there are lots of them and I think that they're irrelevant, but let me know if they're useful and I can upload a version containing them. Also, I'm comfortable building a kernel from source and bisecting it, but I have no idea how to do that with ostree/bootc. I'm also comfortable with plain text mailing lists if you want me to report this somewhere upstream. And please let me know if you need more information or want me to test anything out. Thanks!