Bug 2374607 - kernel-6.16.0-0.rc3.31.fc43 maybe broke BTRFS root file system in F42 test: error btrfs_replay_log errno=-5 IO failure after 6.16rc3 freeze
Summary: kernel-6.16.0-0.rc3.31.fc43 maybe broke BTRFS root file system in F42 test: e...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 42
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2025-06-24 15:30 UTC by Christopher Klooz
Modified: 2025-08-06 01:55 UTC (History)
20 users (show)

Fixed In Version: kernel-6.15.9-201.fc42 kernel-6.15.9-101.fc41
Clone Of:
Environment:
Last Closed: 2025-08-06 01:36:16 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
dmesg-log-after-mount-attempt-in-livesystem.txt (8.03 KB, text/plain)
2025-06-24 15:31 UTC, Christopher Klooz
no flags Details
Emergency-mode-after-boot-of-6.16rc3-with-external-hdmi-monitor.jpg (400.47 KB, image/jpeg)
2025-06-24 15:31 UTC, Christopher Klooz
no flags Details
Emergency-mode-after-boot-of-6.16rc3-without-external-hdmi-monitor.jpg (142.04 KB, image/jpeg)
2025-06-24 15:32 UTC, Christopher Klooz
no flags Details
journalctl-emergency-boot-0(overlapping).jpg (1012.08 KB, image/jpeg)
2025-06-24 15:32 UTC, Christopher Klooz
no flags Details
journalctl-emergency-boot-3(overlapping).jpg (910.31 KB, image/jpeg)
2025-06-24 15:33 UTC, Christopher Klooz
no flags Details
journalctl-emergency-boot-2(overlapping).jpg (979.96 KB, image/jpeg)
2025-06-24 15:33 UTC, Christopher Klooz
no flags Details
journalctl-emergency-boot-1(overlapping).jpg (1.05 MB, image/jpeg)
2025-06-24 15:33 UTC, Christopher Klooz
no flags Details
journalctl-emergency-boot-3_lines-on-right.jpg (302.93 KB, image/jpeg)
2025-06-24 15:34 UTC, Christopher Klooz
no flags Details

Description Christopher Klooz 2025-06-24 15:30:36 UTC
1. Please describe the problem:

In order to test if I can reproduce AMD issue #4141 [1] with 6.16.0-0.rc3.31.fc43, I installed kernel-6.16.0-0.rc3.31.fc43 through dnf from koji [2]. While it is the first kernel in which I cannot reproduce #4141 actively/intentionally (unlike 6.15.3, in which I still can do so), I booted 6.16.0-0.rc3.31.fc43 twice: the first time, I could shutdown properly. The second time of 6.16.0-0.rc3.31.fc43, I could boot properly, but the screen froze when I clicked on shutdown (it already jumped to the terminal when it froze, but just three lines shown, I unfortunately did not save them).

However, after I did hard reset, I always ended up in the emergency mode, whatever kernel I was booting (6.14.*, 6.15.*): the first two times I reset I had a HDMI screen attached, the outcome was the emergency screen "Emergency-mode-after-boot-of-6.16rc3-with-external-hdmi-monitor.jpg", though I do not believe the warning on the emergency screen is related to the problem. In subsequent attempts (including the same kernels of the first two attempts but also other kernels too), I removed the HDMI screen and then had the emergency screen "Emergency-mode-after-boot-of-6.16rc3-without-external-hdmi-monitor.jpg".

Unfortunately, in the emergency mode, it was not possible to mount anything, including USB devices, so I could not save the massively long rdsosreport file but only make some screenshots of the journalctl of the emergency mode's boots: see the files journalctl-emergency-boot-* attached. In order to provide all possible information: I later booted a live system (f42 kde live) and tried to manually mount the btrfs root file system some times -> the mount attempts led to errors, and I also attached the very dmesg lines of a mount attempt in the live system (dmesg-log-after-mount-attempt-in-livesystem.txt), although I think this is only data about the broken btrfs and not relevant for the 6.16 evaluation.

The emergency mode had a kernel taint value of 128.

I CANNOT say for sure that the kernel broke btrfs, as theoretically this can happen at each hard reset. But I have had a lot of hard resets especially in the recent months due to AMD bugs and also in the recent years: given that the broken btrfs occurred when testing a 6.16.0-0.rc3.31.fc43 (I usually avoid rc on production systems for obvious reasons but I am maybe the only one who can currently intentionally reproduce the #4141 bug, so its worth it), and given that this impact is very high, I assume it is worth to be forwarded for your consideration: feel free to close the report if you think this occurred just on coincident or if the data does not suffice reproduction.

Important to note: the btrfs root fs is encrypted (dmcrypt). Beyond the btrfs, my system also has an XFS file system (not a system partition, only for data), but the XFS was not affected.

Hardware AMD Ryzen 7 PRO 6850U with Radeon Graphics, only internal graphics.

I am aware that this kernel is experimental, I had my backups in place.


2. What is the Version-Release number of the kernel:
6.16.0-0.rc3.31.fc43 running on a F42 KDE Spin (only stable repos), up to date as of today at the time of the occurrence. My system uses SELinux confined user accounts.

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :
See above. I could once boot 6.16rc3 normally, and at the second time, btrfs root fs broke. Earlier, all kernels worked. After, none, given that btrfs was damaged.

4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:
Not sure: I installed it using dnf (`dnf update <https links of koji packages>`), booted it, and then I tried to reproduce the AMD #4141 issue by adjusting the brightness with the mouse by putting brightness up and down in an analogue way. Then I shutdown.


6. Are you running any modules that not shipped with directly Fedora's kernel?:
If there are no kernel warnings or errors, my taint level is 0. Except the mesa drivers from rpmfusion, I use only software from the default repos of Fedora (I ensure to only contain the mesa packages and their dependencies from rpmfusion using the "includepkgs" option of dnf). I limit this system to what is installed through dnf. I use only stable repos (except I manually install and test a kernel using the koji packages, which I install through dnf). At the time of the incident, my system was up to date as of today based on dnf.

7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.

[1] https://gitlab.freedesktop.org/drm/amd/-/issues/4141
[2] https://koji.fedoraproject.org/koji/buildinfo?buildID=2737973


Reproducible: Always

Comment 1 Christopher Klooz 2025-06-24 15:31:23 UTC
Created attachment 2094982 [details]
dmesg-log-after-mount-attempt-in-livesystem.txt

Comment 2 Christopher Klooz 2025-06-24 15:31:47 UTC
Created attachment 2094983 [details]
Emergency-mode-after-boot-of-6.16rc3-with-external-hdmi-monitor.jpg

Comment 3 Christopher Klooz 2025-06-24 15:32:09 UTC
Created attachment 2094984 [details]
Emergency-mode-after-boot-of-6.16rc3-without-external-hdmi-monitor.jpg

Comment 4 Christopher Klooz 2025-06-24 15:32:31 UTC
Created attachment 2094985 [details]
journalctl-emergency-boot-0(overlapping).jpg

Comment 5 Christopher Klooz 2025-06-24 15:33:10 UTC
Created attachment 2094986 [details]
journalctl-emergency-boot-3(overlapping).jpg

Comment 6 Christopher Klooz 2025-06-24 15:33:27 UTC
Created attachment 2094987 [details]
journalctl-emergency-boot-2(overlapping).jpg

Comment 7 Christopher Klooz 2025-06-24 15:33:46 UTC
Created attachment 2094988 [details]
journalctl-emergency-boot-1(overlapping).jpg

Comment 8 Christopher Klooz 2025-06-24 15:34:06 UTC
Created attachment 2094989 [details]
journalctl-emergency-boot-3_lines-on-right.jpg

Comment 9 Justin M. Forbes 2025-06-24 15:53:38 UTC
There was a btrfs pull yesterday, so that may be resolved with kernel-6.16.0-0.rc3.250624g78f4e737a53e.32.fc43 (building now)

Comment 10 alex 2025-07-04 16:07:21 UTC
I experienced what appears to be the same issue after upgrading to 6.15.4-200.fc42. btrfs check didn't detect anything wrong, but btrfs rescue zero-log fixed it.

Comment 11 Christopher Klooz 2025-07-17 20:14:22 UTC
Ok, I now just got the same issue again with 6.15.6-200.fc42.x86_64 when I experienced an occurrence of AMD #4141 [1]. So the system froze with the errors of AMD #4141 (verified log entries after I could repair the btrfs), and then btrfs was broken. Just like above. So not sure if its just the btrfs that occasionally breaks when the system freezes (force reboot without umount etc), or if there is an error that causes this (which would mean it has been backported to any 6.15 kernel). I also mentioned it at the AMD ticket [2], just in case. Anyway, my tendency would be to assume it's just an fs that occasionally breaks when it regularly experiences system freezes without proper umounts and such (though the likelihood indeed makes me question that myself), but Mario Limonciello of AMD just mentioned in a post [3] that he does not exclude a relation of #4141 to a current btrfs issue.

[1] https://gitlab.freedesktop.org/drm/amd/-/issues/4141
[2] https://gitlab.freedesktop.org/drm/amd/-/issues/4141#note_3013701
[3] https://gitlab.freedesktop.org/drm/amd/-/issues/4141#note_3013469

Comment 12 Adam Williamson 2025-08-02 15:22:07 UTC
This is likely https://lore.kernel.org/linux-btrfs/fce139db-4458-4788-bb97-c29acf6cb1df@cachyos.org/ , there seem to be various reports of this issue with recent 6.16 and 6.15 kernels.

Comment 13 Fedora Update System 2025-08-02 20:45:25 UTC
FEDORA-2025-a3bae4266e (kernel-6.15.9-101.fc41) has been submitted as an update to Fedora 41.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-a3bae4266e

Comment 14 Fedora Update System 2025-08-02 20:47:33 UTC
FEDORA-2025-09b1545c7a (kernel-6.15.9-201.fc42) has been submitted as an update to Fedora 42.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-09b1545c7a

Comment 15 Fedora Update System 2025-08-03 00:57:14 UTC
FEDORA-2025-09b1545c7a has been pushed to the Fedora 42 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-09b1545c7a`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-09b1545c7a

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 16 Fedora Update System 2025-08-03 01:33:05 UTC
FEDORA-2025-a3bae4266e has been pushed to the Fedora 41 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-a3bae4266e`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-a3bae4266e

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 17 Christopher Klooz 2025-08-04 10:28:17 UTC
I added a **careful** "thumbs up" in bodhi for BZ#2374607 with the possibility to reopen the ticket here: I cannot verify if the issue is solved given that it so far only occurred when I intentionally provoked the [AMD issue #4141](https://gitlab.freedesktop.org/drm/amd/-/issues/4141), and I can no longer intentionally provoke the AMD issue #4141. In the AMD ticket, the idea was put forward some time ago that the AMD issue #4141 maybe is just a manifest of the btrfs issue. This assumption is based only on some correlations, but because I have seen something like that before (my wifi driver's discovery function once caused freezes with amdgpu errors), and because I can no longer reproduce the AMD issue with this kernel (though I tried last time to provoke it on 6.15.6 that was still affected), I hope for the best and assume the btrfs issue is solved in this kernel. I re-report in bodhi & re-open the bug ticket if the issue reoccurs. Thanks for taking care :)

Comment 18 Fedora Update System 2025-08-06 01:36:16 UTC
FEDORA-2025-09b1545c7a (kernel-6.15.9-201.fc42) has been pushed to the Fedora 42 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 19 Fedora Update System 2025-08-06 01:55:48 UTC
FEDORA-2025-a3bae4266e (kernel-6.15.9-101.fc41) has been pushed to the Fedora 41 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.