Bug 2122271 - Early microcode loading not present due to compressed microcode files in linux-firmware (AMD, microcode_amd.bin.xz)
Summary: Early microcode loading not present due to compressed microcode files in linu...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: linux-firmware
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: dracut-maint-list
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-08-29 16:11 UTC by samoht0
Modified: 2023-07-07 01:54 UTC (History)
12 users (show)

Fixed In Version: linux-firmware-20230625-151.fc38 linux-firmware-20230625-151.fc37
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-07-06 02:19:31 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
linux-firmware AMD ucode patch (1.65 KB, patch)
2022-11-13 13:17 UTC, samoht0
no flags Details | Diff

Description samoht0 2022-08-29 16:11:16 UTC
Description of problem:
-----------------------
Early microcode loading not present due to missing microcode files.

### CPU ###

Old AMD APU:

processor       : 1
vendor_id       : AuthenticAMD
cpu family      : 20
model           : 2
model name      : AMD E-450 APU with Radeon(tm) HD Graphics

Detection using dracut-functions.sh is OK:

grep -E "cpu family" /proc/cpuinfo | head -1 | sed "s/.*:\ //"
20
grep -E "model" /proc/cpuinfo | grep -v name | head -1 | sed "s/.*:\ //"
2
grep -E "stepping" /proc/cpuinfo | head -1 | sed "s/.*:\ //"
0

Bobcat-family, microcode in:
/usr/lib/firmware/amd-ucode/microcode_amd.bin.xz

https://wiki.gentoo.org/wiki/AMD_microcode

Version-Release number of selected component:
---------------------------------------------
dracut-056-1.fc36.x86_64
dracut-config-rescue-056-1.fc36.x86_64

How reproducible:
-----------------
Just install/update a kernel.

Actual results:
---------------
No Early CPIO image present. Early microcode loading won't do.

[    2.699481] microcode: CPU0: patch_level=0x05000101
[    2.699609] microcode: CPU1: patch_level=0x05000101
[    2.699629] microcode: Microcode Update Driver: v2.2.

### lsinitrd ###

lsinitrd /media/ramdisk/initramfs-5.19.4-200.fc36.x86_64.img | grep -i "amd-"

drwxr-xr-x   2 root     root            0 Mar 30 10:53 usr/lib/modules/5.19.4-200.fc36.x86_64/kernel/drivers/hid/amd-sfh-hid
-rw-r--r--   1 root     root        13788 Mar 30 10:53 usr/lib/modules/5.19.4-200.fc36.x86_64/kernel/drivers/hid/amd-sfh-hid/amd_sfh.ko.xz

So there's not a single file from /usr/lib/firmware/amd-ucode/ in initramfs! Should be like this, I think:

/usr/lib/firmware/amd-ucode/microcode_amd.bin.xz (required)
/usr/lib/firmware/amd-ucode/microcode_amd_fam15h.bin.xz
/usr/lib/firmware/amd-ucode/microcode_amd_fam16h.bin.xz
/usr/lib/firmware/amd-ucode/microcode_amd_fam17h.bin.xz
/usr/lib/firmware/amd-ucode/microcode_amd_fam19h.bin.xz

### Late loading ###

Late loading with kernel 5.18.19 works, as /usr/lib/firmware/amd-ucode/microcode_amd.bin.xz is present:

echo 1 > /sys/devices/system/cpu/microcode/reload

[ 1599.831178] microcode: CPU1: patch_level=0x05000101
[ 2466.130720] microcode: CPU1: patch_level=0x05000101
[ 2576.016514] microcode: CPU1: new patch_level=0x05000119
[ 2576.016514] microcode: CPU0: new patch_level=0x05000119
[ 2576.017239] x86/CPU: CPU features have changed after loading microcode, but might not take effect.
[ 2576.017254] microcode: Reload completed, microcode revision: 0x5000119

Late loading is not compiled in since 5.19.x in Fedora (DEFAULT)! So there's no work around any more.

https://lore.kernel.org/lkml/20220813223825.3164861-2-ashok.raj@intel.com/

### dracut-config-generic ### 

With dracut-config-generic installed there's a Early CPIO image, but microcode won't load (wrong data in AuthenticAMD.bin, I guess).

Expected results:
-----------------
Early CPIO image present and early microcode loading.

Comment 1 samoht0 2022-09-25 07:11:44 UTC
{ping}

Comment 2 samoht0 2022-10-07 14:50:43 UTC
{update}
No change in behaviour with new version:
dracut-057-3.fc36.x86_64
dracut-config-rescue-057-3.fc36.x86_64

Comment 3 samoht0 2022-10-30 12:49:14 UTC
{ping2}
Why isn't there any feedback?

Comment 4 Sergey Kozharinov 2022-11-06 00:19:22 UTC
The cause of the issue is that AMD microcode is compressed in linux-firmware package, and that dracut does not support compressed microcode binaries. You can workaround the issue by executing the following command inside /lib/firmware/amd-ucode:

sudo xz -d *

...and regenerating initramfs.

This likely belongs to linux-firmware, not dracut. Fedora's linux-firmware has a patch that compressed everything listed in WHENCE. Since dracut has never supported compressed microcode and WHENCE has listed AMD microcode files at least for as long as compression patch exists in Fedora, AMD probably didn't have early microcode in Fedora since February 2021 (when the patch was introduced).

Comment 5 samoht0 2022-11-06 12:03:35 UTC
(In reply to Sergey Kozharinov from comment #4)
> The cause of the issue is that AMD microcode is compressed in linux-firmware
> package, and that dracut does not support compressed microcode binaries. You
> can workaround the issue by executing the following command inside
> /lib/firmware/amd-ucode:
> 
> sudo xz -d *
> 
> ...and regenerating initramfs.

Yes, works this way.👍
At least a new workaround.

> This likely belongs to linux-firmware, not dracut. Fedora's linux-firmware
> has a patch that compressed everything listed in WHENCE. Since dracut has
> never supported compressed microcode and WHENCE has listed AMD microcode
> files at least for as long as compression patch exists in Fedora, AMD
> probably didn't have early microcode in Fedora since February 2021 (when the
> patch was introduced).

I see. Install with compression.

'make DESTDIR=%{buildroot}/ FIRMWAREDIR=%{_firmwarepath} installcompress'

Would be easy fix to just uncompress /lib/firmware/amd-ucode after the install line in the spec file. Same as removal of unwanted files and moving some files/folders.

Comment 6 samoht0 2022-11-06 12:07:21 UTC
Assigning to linux-firmware. Suggestion to make early microcode loading work again on AMD:

Uncompress /lib/firmware/amd-ucode after the install with installcompress in spec file.

See comments #4 and #5.

Comment 7 Peter Robinson 2022-11-06 12:17:40 UTC
This is a bug in dracut, please do not move it to the wrong component.

Comment 8 samoht0 2022-11-06 12:34:02 UTC
(In reply to Peter Robinson from comment #7)
> This is a bug in dracut, please do not move it to the wrong component.

Dracut misses a feature OK, but I don't consider a missing feature (compression support) a bug.

But let's say, it's a bug in dracut. As this discussion leads nowhere.
A fix in linux-firmware really simple. Why not just do it for now, as the user experience is the real thing that matters?

Comment 9 Sergey Kozharinov 2022-11-06 22:36:57 UTC
Just looked at the linux-firmware-20221012-r1.ebuild from Gentoo. They compress every file in the linux-firmware *except* amd-ucode directory.

Comment 10 samoht0 2022-11-07 20:47:18 UTC
(In reply to Sergey Kozharinov from comment #9)
> Just looked at the linux-firmware-20221012-r1.ebuild from Gentoo. They
> compress every file in the linux-firmware *except* amd-ucode directory.

In archlinux amd-ucode/microcode_amd*.bin is also handled separately. They put a combined "amd-ucode.img" into a sub package.
Plenty of options. And no need to have uncompressed ucode files in the main package. If that really matters.

I'm curious, if a kernel maintainer from cc: picks this up and puts the few lines into the linux-firmware spec...

Comment 11 samoht0 2022-11-12 17:21:12 UTC
So it seems, there are currently three distributions compressing firmware (Fedora, Gentoo, Arch and derivatives).

Conclusion:
Two distributions have working AMD microcode patching and handled it in linux-firmware packaging. Both are using dracut. 
One distribution with especially one linux-firmware maintainer claims a bug in dracut and leaves AMD microcode broken without activity and/or explanation. 

Perfectly fine to fake check me. Comments welcome.

Comment 12 Peter Robinson 2022-11-13 10:22:27 UTC
(In reply to samoht0 from comment #11)
> So it seems, there are currently three distributions compressing firmware
> (Fedora, Gentoo, Arch and derivatives).
> 
> Conclusion:
> Two distributions have working AMD microcode patching and handled it in
> linux-firmware packaging. Both are using dracut. 
> One distribution with especially one linux-firmware maintainer claims a bug
> in dracut and leaves AMD microcode broken without activity and/or
> explanation. 

There has been explanations, just because you choose to ignore them is not my problem.

The kernel handles compressed firmware just fine for literally 1000s of firmwares, if dracut can't handle is for one specific set of firmware it's a bug in dracut. From a 2 second grep it handles it here:
https://github.com/dracutdevs/dracut/blob/master/dracut.sh#L2253

If two other distros handle it "correctly", and no special casing what firmware is compressed or not is not fixing it, it's a hack to work around a bug elsewhere, why haven't you references how they do it in their packaging.

> Perfectly fine to fake check me. Comments welcome.

Patches are even more welcome. Comments are easy, fixing the problem seemingly less so. If you're so pent up with getting it fixed references to those other distros and more details rather than pointless "comments welcome" may actually get you closer to a solution to fixing your bug. Pointless unhelpful comments and "+1"s or "me too"s definitely will not.

Comment 13 samoht0 2022-11-13 13:17:47 UTC
Created attachment 1924015 [details]
linux-firmware AMD ucode patch

Comment 14 samoht0 2022-11-13 13:19:33 UTC
(In reply to Peter Robinson from comment #12)
> There has been explanations, just because you choose to ignore them is not
> my problem.
> 
> The kernel handles compressed firmware just fine for literally 1000s of
> firmwares, if dracut can't handle is for one specific set of firmware it's a
> bug in dracut. From a 2 second grep it handles it here:
> https://github.com/dracutdevs/dracut/blob/master/dracut.sh#L2253

True, but as noted above "dracut does not support compressed microcode binaries". Spicing this in at distribution level means: Amateurs are hacking around a complex script, that is vital for system boot-up. This not my department and I'd strongly recommend to never do such things. As there's no upstream patch, there's no backport, too.

> If two other distros handle it "correctly", and no special casing what
> firmware is compressed or not is not fixing it, it's a hack to work around a
> bug elsewhere, why haven't you references how they do it in their packaging.

Compressing firmware is unsupported by kernel upstream. Considering fixing AMD ucode a hack, compressing itself would be a hack even more clearly.
Arch and Gentoo have public git repos. My point was, they handle it in linux-firmware. The solution is isn't fully clear to me. Maybe it's boot config dependent. So I didn't recommend to copy it. I strongly discourage to do so.

Just for the record, this are the current package definitions:
https://github.com/archlinux/svntogit-packages/tree/packages/linux-firmware/trunk
https://gitweb.gentoo.org/repo/gentoo.git/tree/sys-kernel/linux-firmware/linux-firmware-20221109.ebuild

> Patches are even more welcome. Comments are easy, fixing the problem
> seemingly less so. If you're so pent up with getting it fixed references to
> those other distros and more details rather than pointless "comments
> welcome" may actually get you closer to a solution to fixing your bug.
> Pointless unhelpful comments and "+1"s or "me too"s definitely will not.

Ok, if this is really the matter: I've done those tiny changes and package builds fine.

Comment 15 samoht0 2022-11-13 14:00:27 UTC
(In reply to samoht0 from comment #14)
> Compressing firmware is unsupported by kernel upstream.

This references the copy/install script provided by kernel.org firmware package.

Comment 16 Peter Robinson 2022-11-13 15:35:10 UTC
(In reply to samoht0 from comment #15)
> (In reply to samoht0 from comment #14)
> > Compressing firmware is unsupported by kernel upstream.
> 
> This references the copy/install script provided by kernel.org firmware
> package.

The kernel itself supports compressed firmware just fine

Comment 17 samoht0 2022-11-22 12:03:47 UTC
What's the state of checking/applying the proposed packaging patch?

Comment 18 Peter Robinson 2022-11-22 17:19:32 UTC
(In reply to samoht0 from comment #17)
> What's the state of checking/applying the proposed packaging patch?

I will not be applying any patch to linux-firmware, and I've not seen any proposed fix for dracut, so there is no state

Comment 19 Sergey Kozharinov 2022-11-22 20:17:20 UTC
> The kernel handles compressed firmware just fine for literally 1000s of firmwares

CPU microcode isn't firmware. And Linux kernel (6.1-rc6) handles it differently:

x86_64_start_kernel(...)
-> load_ucode_bsp()
  -> load_ucode_amd_bsp(cpuid_1_eax)
    -> __load_ucode_amd(cpuid_1_eax, ret)
      -> find_microcode_in_initrd(path="kernel/x86/microcode/AuthenticAMD.bin", use_pa=false)
        -> find_cpio_data(path, start, size, NULL)
    -> apply_microcode_early_amd(cpuid_1_eax, ucode, size, save_patch=true)
      -> scan_containers(ucode, size, desc)
        -> parse_container(ucode, size, desc)

None of these functions try to do anything even remotely close to decompression.
I am also unsure if kernel can allocate memory this early, if not - decompression is probably impossible.

All of the above makes me wonder if AMD CPU microcode belogns to a subpackage.

Comment 20 samoht0 2022-11-23 10:17:21 UTC
(In reply to Peter Robinson from comment #18)
> I will not be applying any patch to linux-firmware, and I've not seen any
> proposed fix for dracut, so there is no state

This is my understanding: You're refusing to apply a patch, that is tested to fix the bug and has no packaging issues. Because dracut needs a patch, even all other distributions that use compression handle the problem in firmware packaging.
This isn't some package lottery. There's a obvious reason to do it that way. It just works.
It's not like I'm depending on your cooperation. Was a try to follow the advertised fedora principle:

"It is built and used by people across the globe who work together as a community."

Well, mostly it's like that. But as shown here: Sometimes totally the opposite, because of ... (some reason here).

I'll put the uncompressed files into a separate package. Works for me. Fedora stays broken. Perfectly fine by me.

(In reply to Sergey Kozharinov from comment #19)
> > The kernel handles compressed firmware just fine for literally 1000s of firmwares
> 
> CPU microcode isn't firmware. And Linux kernel (6.1-rc6) handles it
> differently:
> 
> x86_64_start_kernel(...)
> -> load_ucode_bsp()
>   -> load_ucode_amd_bsp(cpuid_1_eax)
>     -> __load_ucode_amd(cpuid_1_eax, ret)
>       ->
> find_microcode_in_initrd(path="kernel/x86/microcode/AuthenticAMD.bin",
> use_pa=false)
>         -> find_cpio_data(path, start, size, NULL)
>     -> apply_microcode_early_amd(cpuid_1_eax, ucode, size, save_patch=true)
>       -> scan_containers(ucode, size, desc)
>         -> parse_container(ucode, size, desc)
> 
> None of these functions try to do anything even remotely close to
> decompression.
> I am also unsure if kernel can allocate memory this early, if not -
> decompression is probably impossible.
> 
> All of the above makes me wonder if AMD CPU microcode belogns to a
> subpackage.

You've got a good point. Arch uses the package "amd-ucode". But I wanted the changes to be as minor as possible, to make changes nicely reviewable and acceptable.

As I need to fix it myself, I'll use an extra package alongside. A bit wonky, but better than file management by hand ("We're not on Windows.").

Thanks for your helpful hints and comments.

Comment 21 David Woodhouse 2022-11-24 11:46:16 UTC
Hm, am I understanding this correctly?

 • Upstream firmware is not compressed.
 • dracut does not handle compressed firmware.
 • In Fedora, we made a change to the linux-firmware package to compress the files on disk.
 • AMD early microcode loading no longer works until/unless dracut learns to handle compression.

Typically, we try to make sure that we don't just make changes in one package, that depend on features that haven't yet been implemented in another package and then "break" things until that new feature has been implemented.

That's kind of the point of a Linux distribution — to curate versions and configurations of each downstream software component, that work together correctly. 

I understand that the best answer is to fix dracut to handle the compression, but that's not the point. From the point of view of the distribution overall, this is wrong. We expect package maintainers to care about the whole distribution working together properly, not just consider their own packages in isolation.

I feel very strongly that the right thing for the distribution is to revert the compression (at least for the AMD microcode) until dracut can cope.

I'm going to reopen this ticket; if you feel equally strongly that you want it closed, let's consult FESCo for adjudication? That's the kind of thing they're there for.

Comment 22 Sergey Kozharinov 2022-11-24 13:42:19 UTC
(In reply to David Woodhouse from comment #21)
> Hm, am I understanding this correctly?
> 
>  • Upstream firmware is not compressed.
>  • dracut does not handle compressed firmware.
>  • In Fedora, we made a change to the linux-firmware package to compress the
> files on disk.
>  • AMD early microcode loading no longer works until/unless dracut learns to
> handle compression.
> 
> Typically, we try to make sure that we don't just make changes in one
> package, that depend on features that haven't yet been implemented in
> another package and then "break" things until that new feature has been
> implemented.
> 
> That's kind of the point of a Linux distribution — to curate versions and
> configurations of each downstream software component, that work together
> correctly. 
> 
> I understand that the best answer is to fix dracut to handle the
> compression, but that's not the point. From the point of view of the
> distribution overall, this is wrong. We expect package maintainers to care
> about the whole distribution working together properly, not just consider
> their own packages in isolation.
> 
> I feel very strongly that the right thing for the distribution is to revert
> the compression (at least for the AMD microcode) until dracut can cope.
> 
> I'm going to reopen this ticket; if you feel equally strongly that you want
> it closed, let's consult FESCo for adjudication? That's the kind of thing
> they're there for.

Dracut can handle compressed firmware. But microcode is handled differently both in dracut and in kernel. If dracut were to support compressed microcode, it would have to decompress it before combining it with initrd. Seems like a waste of computational resources with no real benefits since packages are usually compressed regardless.

Comment 23 samoht0 2022-11-24 14:20:10 UTC
(In reply to David Woodhouse from comment #21)
> Hm, am I understanding this correctly?
> 
>  • Upstream firmware is not compressed.
>  • dracut does not handle compressed firmware.
>  • In Fedora, we made a change to the linux-firmware package to compress the
> files on disk.
>  • AMD early microcode loading no longer works until/unless dracut learns to
> handle compression.

You're right, David. That are the key points. Sergey already noted, that CPU microcode is a speciality within the upstream kernel firmware files.
And we're speaking of tiny files. Compressing AMD microcode .bin files shrinks them to 31,6 KiB coming from 49,4 KiB.

Comment 24 Ben Cotton 2023-04-25 18:25:12 UTC
This message is a reminder that Fedora Linux 36 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 36 on 2023-05-16.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '36'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 36 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 25 Fedora Update System 2023-07-02 16:41:58 UTC
FEDORA-2023-31e2ef80b9 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-31e2ef80b9

Comment 26 Fedora Update System 2023-07-02 16:41:58 UTC
FEDORA-2023-eeff822011 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2023-eeff822011

Comment 27 Fedora Update System 2023-07-03 01:37:26 UTC
FEDORA-2023-eeff822011 has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-eeff822011`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-eeff822011

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 28 Fedora Update System 2023-07-03 02:06:21 UTC
FEDORA-2023-31e2ef80b9 has been pushed to the Fedora 38 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-31e2ef80b9`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-31e2ef80b9

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 29 Fedora Update System 2023-07-05 01:19:15 UTC
FEDORA-2023-31e2ef80b9 has been pushed to the Fedora 38 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-31e2ef80b9`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-31e2ef80b9

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 30 Fedora Update System 2023-07-05 02:47:38 UTC
FEDORA-2023-eeff822011 has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-eeff822011`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-eeff822011

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 31 Fedora Update System 2023-07-06 02:19:31 UTC
FEDORA-2023-31e2ef80b9 has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 32 Fedora Update System 2023-07-07 01:54:06 UTC
FEDORA-2023-eeff822011 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.