1. Please describe the problem: kernel-5.19.7-200.fc36.x86_64 fails to boot. It hangs midst of the boot process. Last messages, I am seeing are related to initializing amdgpu (Typed off a photograph taken from : ... kernel: [drm] amdgpu kernel modesetting enabled. kernel: amdgpu: Virtual CRAT table created for CPU kernel: amdgpu: Topology: Add CPU node ... Comparing this to the log of a successfull boot up with kernel-5.19.6-200.fc36.x86_64, this message "kernel: AMD-Vi: AMD IOMMUv2 loaded and initialized" seems to be missing. 2. What is the Version-Release number of the kernel: kernel-5.19.7-200.fc36.x86_64 3. Did it work previously in Fedora? If so, what kernel version did the issue *first* appear? Old kernels are available for download at https://koji.fedoraproject.org/koji/packageinfo?packageID=8 : kernel-5.19.7-200.fc36.x86_64 4. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: Just boot 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: Not tried 6. Are you running any modules that not shipped with directly Fedora's kernel?: No. 7. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag. There ain't no logs - Apparently nothing is being logged.
Same happenned to me with 5.19.7 on Thinkpad T495s. In my case boot ends during initramfs stage on: [drm] amdgpu kernel modesetting enabled. Previous kernel 5.19.6 was booting fine. I am writing "was", as it is probably not direct problem of kernel. I initially thought it is this problem https://ask.fedoraproject.org/t/can-not-boot-from-new-kernels/26002/2 but it did not help so i've tried to run dracut which rebuild also initramfs for 5.19.6 and now it ends on exactly the same spot. Therefore i'd also like to ask for help how to recover, as i am not able to boot at all now. I know this is not the right place to ask, but i am desperate and hope here i'll get attention sooner. I've tried f36 live image in which i've installed kernel 5.19.7 and have rewritten original files in /boot with ones created in live image. Now i am able to bypass initramfs stage and system starts to load (another hint that it is not caused by kernel version). However i've hit another problem and that is i've luks encrypted disk and using dracut image copied from live media does not show me password prompt so the boot hangs later but hangs anyway :-(
I've managed to fix it. In case anyone looks for solution in case he is not able to boot older kernel and thus boot at all, boot using live image, chroot into existing system (i've used this guide https://docs.fedoraproject.org/en-US/quick-docs/bootloading-with-grub2/#restoring-bootloader-using-live-disk). Now what i've done was try to find which packages were upgraded between 5.19.6 which was working and installation of 5.19.7 (via dnf history). I found one transaction containing only firmware upgrade (which i thought could be related as the boot hangs somewhere around GPU). It was these packages: Upgrade erlang-compiler-24.3.4.4-1.fc36.x86_64 @updates Upgraded erlang-compiler-24.3.4.3-1.fc36.x86_64 @@System Upgrade erlang-crypto-24.3.4.4-1.fc36.x86_64 @updates Upgraded erlang-crypto-24.3.4.3-1.fc36.x86_64 @@System Upgrade erlang-erts-24.3.4.4-1.fc36.x86_64 @updates Upgraded erlang-erts-24.3.4.3-1.fc36.x86_64 @@System Upgrade erlang-kernel-24.3.4.4-1.fc36.x86_64 @updates Upgraded erlang-kernel-24.3.4.3-1.fc36.x86_64 @@System Upgrade erlang-stdlib-24.3.4.4-1.fc36.x86_64 @updates Upgraded erlang-stdlib-24.3.4.3-1.fc36.x86_64 @@System Upgrade erlang-syntax_tools-24.3.4.4-1.fc36.x86_64 @updates Upgraded erlang-syntax_tools-24.3.4.3-1.fc36.x86_64 @@System Upgrade iwl100-firmware-39.31.5.1-138.fc36.noarch @updates Upgraded iwl100-firmware-39.31.5.1-136.fc36.noarch @@System Upgrade iwl1000-firmware-1:39.31.5.1-138.fc36.noarch @updates Upgraded iwl1000-firmware-1:39.31.5.1-136.fc36.noarch @@System Upgrade iwl105-firmware-18.168.6.1-138.fc36.noarch @updates Upgraded iwl105-firmware-18.168.6.1-136.fc36.noarch @@System Upgrade iwl135-firmware-18.168.6.1-138.fc36.noarch @updates Upgraded iwl135-firmware-18.168.6.1-136.fc36.noarch @@System Upgrade iwl2000-firmware-18.168.6.1-138.fc36.noarch @updates Upgraded iwl2000-firmware-18.168.6.1-136.fc36.noarch @@System Upgrade iwl2030-firmware-18.168.6.1-138.fc36.noarch @updates Upgraded iwl2030-firmware-18.168.6.1-136.fc36.noarch @@System Upgrade iwl3160-firmware-1:25.30.13.0-138.fc36.noarch @updates Upgraded iwl3160-firmware-1:25.30.13.0-136.fc36.noarch @@System Upgrade iwl3945-firmware-15.32.2.9-138.fc36.noarch @updates Upgraded iwl3945-firmware-15.32.2.9-136.fc36.noarch @@System Upgrade iwl4965-firmware-228.61.2.24-138.fc36.noarch @updates Upgraded iwl4965-firmware-228.61.2.24-136.fc36.noarch @@System Upgrade iwl5000-firmware-8.83.5.1_1-138.fc36.noarch @updates Upgraded iwl5000-firmware-8.83.5.1_1-136.fc36.noarch @@System Upgrade iwl5150-firmware-8.24.2.2-138.fc36.noarch @updates Upgraded iwl5150-firmware-8.24.2.2-136.fc36.noarch @@System Upgrade iwl6000-firmware-9.221.4.1-138.fc36.noarch @updates Upgraded iwl6000-firmware-9.221.4.1-136.fc36.noarch @@System Upgrade iwl6000g2a-firmware-18.168.6.1-138.fc36.noarch @updates Upgraded iwl6000g2a-firmware-18.168.6.1-136.fc36.noarch @@System Upgrade iwl6000g2b-firmware-18.168.6.1-138.fc36.noarch @updates Upgraded iwl6000g2b-firmware-18.168.6.1-136.fc36.noarch @@System Upgrade iwl6050-firmware-41.28.5.1-138.fc36.noarch @updates Upgraded iwl6050-firmware-41.28.5.1-136.fc36.noarch @@System Upgrade iwl7260-firmware-1:25.30.13.0-138.fc36.noarch @updates Upgraded iwl7260-firmware-1:25.30.13.0-136.fc36.noarch @@System Upgrade iwlax2xx-firmware-20220815-138.fc36.noarch @updates Upgraded iwlax2xx-firmware-20220708-136.fc36.noarch @@System Upgrade libertas-usb8388-firmware-2:20220815-138.fc36.noarch @updates Upgraded libertas-usb8388-firmware-2:20220708-136.fc36.noarch @@System Upgrade linux-firmware-20220815-138.fc36.noarch @updates Upgraded linux-firmware-20220708-136.fc36.noarch @@System Upgrade linux-firmware-whence-20220815-138.fc36.noarch @updates Upgraded linux-firmware-whence-20220708-136.fc36.noarch @@System After undoing this transaction, i've reinstalled kernel 5.19.7 (so new initramfs image was generated), rebooted, and everything works :-) True is i am not sure whether it was that dowgrade which helped, or whether it was running dracut in chrooted environment (as live image also runs on older firmware)...but i definitely think it is not problem of kernel itself.
(In reply to Petr Bartos from comment #2) Interesting observation ;) Of those packages you list, I only have linux-firmware and linux-firmware-whence installed. This made me try as follows (All with kernel-5.19.6-200 running): - Install kernel-5.19.8 with linux-firmware*-20220708-136 installed => system boots. - Install kernel-5.19.8 with linux-firmware*-20220815-138 installed => boot hangs. [N.B.: I used kernel-5.19.8 from koji, because it exposes the same symptoms as kernel-5.19.7] Now, I am wondering whether there might be an ABI/API breakage between kernel and the firmware packages or if this is a bug in one of the firmware blobs (May-be only affecting AMD-processors/GPUs)?, [N.B.: The system I am using here is an AMD Ryzen 5 PRO 4650G (Renoir iGPU), but I am observing probably related problems on 2 other, older AMD systems]
Is there a way to get the older linux-firmware packages to test? Unfortunately I get: # dnf history undo 1322 Last metadata expiration check: 1:46:36 ago on Sat 10 Sep 2022 10:42:54 AM EDT. Error: The following problems occurred while running a transaction: Cannot find rpm nevra "iwl100-firmware-39.31.5.1-136.fc36.noarch". Cannot find rpm nevra "iwl1000-firmware-1:39.31.5.1-136.fc36.noarch". Cannot find rpm nevra "iwl105-firmware-18.168.6.1-136.fc36.noarch". Cannot find rpm nevra "iwl135-firmware-18.168.6.1-136.fc36.noarch". Cannot find rpm nevra "iwl2000-firmware-18.168.6.1-136.fc36.noarch". Cannot find rpm nevra "iwl2030-firmware-18.168.6.1-136.fc36.noarch". Cannot find rpm nevra "iwl3160-firmware-1:25.30.13.0-136.fc36.noarch". Cannot find rpm nevra "iwl3945-firmware-15.32.2.9-136.fc36.noarch". Cannot find rpm nevra "iwl4965-firmware-228.61.2.24-136.fc36.noarch". Cannot find rpm nevra "iwl5000-firmware-8.83.5.1_1-136.fc36.noarch". Cannot find rpm nevra "iwl5150-firmware-8.24.2.2-136.fc36.noarch". Cannot find rpm nevra "iwl6000-firmware-9.221.4.1-136.fc36.noarch". Cannot find rpm nevra "iwl6000g2a-firmware-18.168.6.1-136.fc36.noarch". Cannot find rpm nevra "iwl6000g2b-firmware-18.168.6.1-136.fc36.noarch". Cannot find rpm nevra "iwl6050-firmware-41.28.5.1-136.fc36.noarch". Cannot find rpm nevra "iwl7260-firmware-1:25.30.13.0-136.fc36.noarch". Cannot find rpm nevra "libertas-usb8388-firmware-2:20220708-136.fc36.noarch". Cannot find rpm nevra "linux-firmware-20220708-136.fc36.noarch". Cannot find rpm nevra "linux-firmware-whence-20220708-136.fc36.noarch".
(In reply to Zing from comment #4) > Is there a way to get the older linux-firmware packages to test? > Unfortunately I get: > > # dnf history undo 1322 > Last metadata expiration check: 1:46:36 ago on Sat 10 Sep 2022 10:42:54 AM > EDT. > Error: The following problems occurred while running a transaction: > Cannot find rpm nevra "iwl100-firmware-39.31.5.1-136.fc36.noarch". > Cannot find rpm nevra "iwl1000-firmware-1:39.31.5.1-136.fc36.noarch". > Cannot find rpm nevra "iwl105-firmware-18.168.6.1-136.fc36.noarch". > Cannot find rpm nevra "iwl135-firmware-18.168.6.1-136.fc36.noarch". > Cannot find rpm nevra "iwl2000-firmware-18.168.6.1-136.fc36.noarch". > Cannot find rpm nevra "iwl2030-firmware-18.168.6.1-136.fc36.noarch". > Cannot find rpm nevra "iwl3160-firmware-1:25.30.13.0-136.fc36.noarch". > Cannot find rpm nevra "iwl3945-firmware-15.32.2.9-136.fc36.noarch". > Cannot find rpm nevra "iwl4965-firmware-228.61.2.24-136.fc36.noarch". > Cannot find rpm nevra "iwl5000-firmware-8.83.5.1_1-136.fc36.noarch". > Cannot find rpm nevra "iwl5150-firmware-8.24.2.2-136.fc36.noarch". > Cannot find rpm nevra "iwl6000-firmware-9.221.4.1-136.fc36.noarch". > Cannot find rpm nevra "iwl6000g2a-firmware-18.168.6.1-136.fc36.noarch". > Cannot find rpm nevra "iwl6000g2b-firmware-18.168.6.1-136.fc36.noarch". > Cannot find rpm nevra "iwl6050-firmware-41.28.5.1-136.fc36.noarch". > Cannot find rpm nevra "iwl7260-firmware-1:25.30.13.0-136.fc36.noarch". > Cannot find rpm nevra > "libertas-usb8388-firmware-2:20220708-136.fc36.noarch". > Cannot find rpm nevra "linux-firmware-20220708-136.fc36.noarch". > Cannot find rpm nevra "linux-firmware-whence-20220708-136.fc36.noarch". You can run: dnf downgrade *firmware* that will install previous available version of all firmware packages you have installed (i've done it like this also myself)
I downloaded them from: https://koji.fedoraproject.org/koji/buildinfo?buildID=2003340
Same observation as in Comment#3 on an other, ancient AMD system. AMD Turion(tm) II Neo N40L with RS880M [Mobility Radeon HD 4225/4250] GPU, which is using xorg-x11-drv-ati.
*** Bug 2125620 has been marked as a duplicate of this bug. ***
Thanks, after downgrade of: dnf downgrade: linux-firmware noarch 20220310-130.fc36 fedora 187 M linux-firmware-whence noarch 20220310-130.fc36 fedora 48 k I am able to boot 5.19.7-200.fc36.x86_64 In case this helps cpu is: model name : AMD Athlon(tm) 64 X2 Dual Core Processor 5000+
And another instance, where https://bugzilla.redhat.com/show_bug.cgi?id=2125536#c3 helped: AMD Opteron(tm) X3216 APU, with AMD/ATI Wani [Radeon R5/R6/R7 Graphics] GPU and amdgpu.
I think, I've found the cause: With linux-firmware-20220815-138 the gpu-firmware blobs, so far having been contained in the main linux-firmware rpm have been split out into separate packages: - amd-gpu-firmware - nvidia-gpu-firmware - intel-gpu-firmware I.e. all systems which so far only had the linux-firmware.rpm installed and pulled gpu firmware from it, will loose their gpu-firmware when updating to linux-firmware-20220815-138. This will cause kernel updates (dracut) to loose the gpu firmware and thus cause these systems to raise boot failures, when something tries to access something requiring the GPU. I.e. the actual work-around to this issue is to make sure to have {amd|nvidia|intel}-gpu-firmware installed _before_ upgrading a kernel, with linux-firmware > 138. This also explains, why downgrading to linux-firmware < 138 helped. linux-firmware < 138 packages still contain the gpu-firmware blobs, so dracut picks them up on kernel-upgrades.
*** Bug 2125774 has been marked as a duplicate of this bug. ***
I'm copy and pasting my comment on the bodhi update here because this is probably a better place to discuss this: Was the split of the linux-firmware package not supposed to be a Fedora 37 feature? Since this is breaking peoples system left and right IMHO a new update reverting the split should be pushed for F36 and this should be made a F37 only thing? : https://fedoraproject.org/wiki/Changes/Linux_Firmware_Minimization For examples of the issues caused by this see: https://bugzilla.redhat.com/show_bug.cgi?id=2125536 https://bugzilla.redhat.com/show_bug.cgi?id=2125774
Seconded, breaking F36 installs seems like a very bad idea. Instead of reverting maybe we could also push an update where linux-firmware requires "*-gpu-firmware" on F36? Obviously users won't enjoy the reduced storage requirements but it would ensure systems still boot correctly and once users upgrade to F37 they can uninstall gpu firmware they don't need.
I'd like to add that a basic "Recommends: amd-gpu-firmware" in F36 does not cut it: IMHO "install_weak_deps=False" is a supported setting in Fedora so you can not rely dnf installing recommended packages.
Well, i am not any authority at all to contradict split of firmware, but isn't it little problematic in case of e.g. replacing gpu in desktop? Because it seems when i replace some nvidia card with amd one, i will not be able to boot at all. One thing is to not be able to boot into graphic mode due to some missing driver which can be fixed from command line afterwards, but in this case it will not boot at all and i have no chance to fix it without booting some rescue media, chrooting etc. And in case you know this, the preparation process for gpu swap will not be so simple as you will not only need to install firmware package but also rebuild initramfs images. Which may be fine in case of planned gpu swap, but will be problem e.g. in case of replacement due to hardware failure.
*** Bug 2125784 has been marked as a duplicate of this bug. ***
I had similar issues using the kernel-5.19.8-200.fc36: My Fedora 36 doesn't detect my second Monitor and reduced the screen refresh rate down to 30Hz. I do not have the issues using an earlier kernel such as: kernel-5.19.6-200.fc36 I had the latest linux-firmware installed, but missed the amd-gpu-firmware. I have the following hardware: CPU: AMD Ryzen 9 5950X GPU: AMD ATI Radeon RX 6900 XT here are the errors from my boot log, running `journalctl -p 3 -b`: ``` Sep 12 19:21:28 metis kernel: amdgpu 0000:07:00.0: amdgpu: failed to init sos firmware Sep 12 19:21:28 metis kernel: [drm:psp_sw_init [amdgpu]] *ERROR* Failed to load psp firmware! Sep 12 19:21:28 metis kernel: [drm:amdgpu_device_init.cold [amdgpu]] *ERROR* sw_init of IP block <psp> failed -2 Sep 12 19:21:28 metis kernel: amdgpu 0000:07:00.0: amdgpu: amdgpu_device_ip_init failed Sep 12 19:21:28 metis kernel: amdgpu 0000:07:00.0: amdgpu: Fatal error during GPU init ``` After installing the amd-gpu-firmware and !regenerated! InitRAMFS (took me a while that fedora doesn't do that automatically when installing firmware) my system runs flawless again. Furthermore, I agree with @bartos.petr in comment #16.
> Instead of reverting maybe we could also push an update where linux-firmware requires "*-gpu-firmware" on F36? Obviously users won't enjoy the reduced storage requirements but it would ensure systems still boot correctly and once users upgrade to F37 they can uninstall gpu firmware they don't need. That is a good idea. I will discuss this with the Fedora linux-firmware maintainer. Changing component to linux-firmware.
> Well, i am not any authority at all to contradict split of firmware, but isn't it little problematic in case of e.g. replacing gpu in desktop? Because it seems when i replace some nvidia card with amd one, i will not be able to boot at all. The default Fedora 37 (and later) workstation install will still include all the GPU firmwares. This is only intended to help make more lightweight images for special cases. For a user to loose the ability to swap the GPU they would need to both set install_weak_deps=False and then manually remove the GPU specific firmwares. After which they get exactly what they have asked for. As for the split breaking F36 installs with install_weak_deps=False that is a bug which we need to fix; and after that fix upgrades from F36 should always have all the GPU firmwares installed just like fresh F37 installs.
FEDORA-2022-9fcd50526b has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-9fcd50526b
FEDORA-2022-ceec02d76d has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-ceec02d76d
FEDORA-2022-67de9a27be has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-67de9a27be
(In reply to Hans de Goede from comment #20) > For a user to loose the ability to swap the GPU they would need to both set > install_weak_deps=False and then manually remove the GPU specific firmwares. > After which they get exactly what they have asked for. The problem with this is, it's an abuse of weak_deps, or to rephrase it "PC": It's more a semi-functional migitation of this issue, but a fix. Weak_deps purpose is to pull-in _optional_ packages. However, the gpu-firmware package are not optional. They are mandatory/necessary, depending on a system's HWs. That's something current rpm can't handle. > As for the split breaking F36 installs with install_weak_deps=False that is > a bug which we need to fix; and after that fix upgrades from F36 should > always have all the GPU firmwares installed just like fresh F37 installs. Depending on which packages you have installed, this may pull-in trees of further packages. It's only due to the fact, "Recommends:" aren't widely used in Fedora, which hides away this issue from most users.
FEDORA-2022-ceec02d76d has been pushed to the Fedora 37 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-ceec02d76d` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-ceec02d76d See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2022-67de9a27be has been pushed to the Fedora 36 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-67de9a27be` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-67de9a27be See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2022-9fcd50526b has been pushed to the Fedora 35 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-9fcd50526b` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-9fcd50526b See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2022-67de9a27be has been pushed to the Fedora 36 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2022-ceec02d76d has been pushed to the Fedora 37 stable repository. If problem still persists, please make note of it in this bug report.
Confirming everything works for me after update.
FEDORA-2022-9fcd50526b has been pushed to the Fedora 35 stable repository. If problem still persists, please make note of it in this bug report.