Bug 1701096 - Fedora 29 - Kernel does not load signed modules
Summary: Fedora 29 - Kernel does not load signed modules
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 29
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1702844 1703599 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-18 04:16 UTC by Robert Holmes
Modified: 2019-05-07 04:50 UTC (History)
28 users (show)

Fixed In Version: kernel-5.0.9-301.fc30 kernel-5.0.10-200.fc29
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-03 03:42:12 UTC
Type: Bug


Attachments (Terms of Use)

Description Robert Holmes 2019-04-18 04:16:24 UTC
In recent versions of the kernel, namely 5.0.x, the kernel is not accepting valid signatures that use UEFI:db or UEFI:MokListRT keys. This appears to be due to the following patches:

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-5.0.y&id=9dc92c45177ab70e20ae94baa2f2e558da63a9c7
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-5.0.y&id=60740accf78494e166ec76bdc39b7d75fc2fe1c7

So these UEFI keys are not put in the .platform keyring, but this keyring is apparently not used to verify module signatures. This commit appears to solve this:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=219a3e8676f3132d27b530c7d2d6bcab89536b57

but it has not yet made it to linux-stable.

Comment 1 Robert Holmes 2019-04-18 04:26:56 UTC
Additional note: modules were still loaded up until 5.0.8, thought with a warning. But this patch made modules not load at all:

https://src.fedoraproject.org/rpms/kernel/c/24416b46798ee9a53c81faad67021a5df90dff30?branch=f29

Comment 2 Robert Holmes 2019-04-20 08:13:37 UTC
Ok, after some experimentation, it appears that commit 219a3e8676f3132d27b530c7d2d6bcab89536b57 is not sufficient to get signed modules to be accepted. The issue seems to be here:

https://elixir.bootlin.com/linux/v5.1-rc5/source/kernel/module_signing.c#L86

where there is seemingly no way to ever use VERIFY_USE_PLATFORM_KEYRING to verify the signatures.


As it currently stands, the instructions in https://docs.fedoraproject.org/en-US/fedora/f29/system-administrators-guide/kernel-module-driver-configuration/Working_with_Kernel_Modules/#sect-signing-kernel-modules-for-secure-boot do not work.

Comment 3 Robert Holmes 2019-04-20 15:58:48 UTC
Cherry-picking 219a3e8676f3132d27b530c7d2d6bcab89536b57 and 278311e417be60f7caef6fcb12bda4da2711ceff plus the following patch solve the issue:

diff --git a/kernel/module_signing.c b/kernel/module_signing.c
index 6b9a926fd86b..19ddd63a086b 100644
--- a/kernel/module_signing.c
+++ b/kernel/module_signing.c
@@ -49,6 +49,7 @@ int mod_verify_sig(const void *mod, struct load_info *info)
 {
        struct module_signature ms;
        size_t sig_len, modlen = info->len;
+       int ret;
 
        pr_devel("==>%s(,%zu)\n", __func__, modlen);
 
@@ -82,8 +83,15 @@ int mod_verify_sig(const void *mod, struct load_info *info)
                return -EBADMSG;
        }
 
-       return verify_pkcs7_signature(mod, modlen, mod + modlen, sig_len,
+       ret = verify_pkcs7_signature(mod, modlen, mod + modlen, sig_len,
                                      VERIFY_USE_SECONDARY_KEYRING,
                                      VERIFYING_MODULE_SIGNATURE,
                                      NULL, NULL);
+       if (ret == -ENOKEY && IS_ENABLED(CONFIG_INTEGRITY_PLATFORM_KEYRING)) {
+               ret = verify_pkcs7_signature(mod, modlen, mod + modlen, sig_len,
+                                     VERIFY_USE_PLATFORM_KEYRING,
+                                     VERIFYING_MODULE_SIGNATURE,
+                                     NULL, NULL);
+       }
+       return ret;
 }

Comment 4 Jeremy Cline 2019-04-22 17:14:42 UTC
Hi Robert,

Thanks for the thorough investigation and patch. Do you feel comfortable submitting this patch upstream? It seems like something upstream would want.

Comment 5 Robert Holmes 2019-04-23 07:46:56 UTC
Sent a patch here: https://lkml.org/lkml/2019/4/23/89

These patches should probably also be included in -stable, but I am not familiar with the process of nudging whoever's responsible to choose what gets there.

Comment 6 Jeremy Cline 2019-04-23 17:42:53 UTC
Great, thanks. The process for -stable is documented at https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html. If you end up needing to re-submit the patch you can add that, or ask the maintainer to consider adding the tag, or wait for it to get merged and email the stable list.

In the mean time, I've picked it up for today's Rawhide build and it should be in the next stable builds (5.0.10). The update system should comment on this bug when it's available for testing.

Comment 7 Kelvin J. Hill 2019-04-24 14:50:39 UTC
How on earth was this never picked up in testing? 

It's been a nightmare to workaround since running dnf this morning, removing lockdown and having to then manually load the missing 9 self-signed modules in my case.

Comment 8 Adam Williamson 2019-04-24 19:05:30 UTC
There is apparently some stuff related to module signing and loading that is not upstream, so this doesn't get caught by upstream testing and only shows up when we do downstream builds with our patches :/

Kernel updates in Bodhi tend to get an immediate flood of positive feedback, since everyone has the kernel installed and it'll work for lots of people - anyone on a BIOS install or non-SB UEFI install won't notice this problem, for instance, so they'll give the kernel positive feedback. That can lead to an update getting pushed stable very rapidly before a problem that only affects a specific configuration is noticed.

In this case, AIUI, *this* particular bug has not made it out of updates-testing: the 5.0.9 build for F29 was not yet submitted for stable - https://bodhi.fedoraproject.org/updates/FEDORA-2019-1e8a4c6958 . The current stable build is 5.0.8, which AIUI would suffer from 1696671 but not from this bug.

Comment 9 Adam Williamson 2019-04-24 19:08:48 UTC
BTW, for the record, the most recent F30 build, kernel-5.0.9-301.fc30 - https://koji.fedoraproject.org/koji/buildinfo?buildID=1253909 - should have the fix for this, so it should not suffer from #1696671 *or* this bug. That build is in the F30 release candidate compose that is currently spinning. If anyone notices that it still has bugs related to signed modules, please let us know. Thanks!

Comment 10 Robert Holmes 2019-04-24 19:12:13 UTC
(In reply to Adam Williamson from comment #8)
> In this case, AIUI, *this* particular bug has not made it out of
> updates-testing: the 5.0.9 build for F29 was not yet submitted for stable -
> https://bodhi.fedoraproject.org/updates/FEDORA-2019-1e8a4c6958 . The current
> stable build is 5.0.8, which AIUI would suffer from 1696671 but not from
> this bug.

5.0.8 does suffer from this bug, cf. https://bugzilla.redhat.com/show_bug.cgi?id=1701096#c1

Comment 11 Kelvin J. Hill 2019-04-24 19:13:04 UTC
(In reply to Adam Williamson from comment #8)
> There is apparently some stuff related to module signing and loading that is
> not upstream, so this doesn't get caught by upstream testing and only shows
> up when we do downstream builds with our patches :/
> 
> Kernel updates in Bodhi tend to get an immediate flood of positive feedback,
> since everyone has the kernel installed and it'll work for lots of people -
> anyone on a BIOS install or non-SB UEFI install won't notice this problem,
> for instance, so they'll give the kernel positive feedback. That can lead to
> an update getting pushed stable very rapidly before a problem that only
> affects a specific configuration is noticed.
> 
> In this case, AIUI, *this* particular bug has not made it out of
> updates-testing: the 5.0.9 build for F29 was not yet submitted for stable -
> https://bodhi.fedoraproject.org/updates/FEDORA-2019-1e8a4c6958 . The current
> stable build is 5.0.8, which AIUI would suffer from 1696671 but not from
> this bug.

I can confirm that 5.0.8-200, as updated by dnf today, does indeed suffer from the problem.
Non of the modules that would load, albeit noisily, on 5.0.7 fail on 5.0.8.

Comment 12 Adam Williamson 2019-04-24 19:14:01 UTC
Ah, OK. I was reading "up until 5.0.8" as meaning it worked *up to and including* 5.0.8, sorry.

Comment 13 Kelvin J. Hill 2019-04-24 19:16:13 UTC
(In reply to Kelvin J. Hill from comment #11) 
> I can confirm that 5.0.8-200, as updated by dnf today, does indeed suffer
> from the problem.
> Non of the modules that would load, albeit noisily, on 5.0.7 fail on 5.0.8.
Sorry, that should read "ALL of the modules that would load, albeit noisily, on 5.0.7 fail on 5.0.8."

Comment 14 Jeremy Cline 2019-04-25 01:39:21 UTC
*** Bug 1702844 has been marked as a duplicate of this bug. ***

Comment 15 Robert Holmes 2019-04-25 11:59:28 UTC
It appears upstream wants it that way: https://lkml.org/lkml/2019/4/25/409

Comment 16 Kelvin J. Hill 2019-04-25 12:50:52 UTC
Confirming that 5.0.9 shows the same problem.

From the upstream stance, how are self-signed modules supposed to be implemented then on a UEFI based installation?

Comment 17 Rick 2019-04-25 14:27:53 UTC
Besides disabling secure boot, is there another workaround?  Is there a way to add/link our MOK key to the .secondary_trusted_keys keyring?

Comment 18 Adam Williamson 2019-04-25 20:43:54 UTC
5.0.9-301 should *not* have this bug, please test that.

Comment 19 Kelvin J. Hill 2019-04-25 20:49:38 UTC
(In reply to Adam Williamson from comment #18)
> 5.0.9-301 should *not* have this bug, please test that.

This bug entry is against Fedora 29, which had 5.0.9-200.fc29 delivered by dnf today.
THAT kernel version still shows the problem.

Comment 20 Jan Hugo Prins 2019-04-25 20:54:55 UTC
Where is this 5.0.9-301 update available? I'm able to reproduce it very easily.

Comment 21 Kelvin J. Hill 2019-04-25 21:02:32 UTC
(In reply to Rick from comment #17)
> Besides disabling secure boot, is there another workaround?  Is there a way
> to add/link our MOK key to the .secondary_trusted_keys keyring?

On some motherboards, especially those from Asus, disabling secure boot can only be done by deleting keys. It is non-trivial...

Comment 22 Adam Williamson 2019-04-25 21:09:29 UTC
Kelvin: yes, that's true indeed, sorry - it'd still be very helpful to know if the -301.fc30 kernel is fixed, though. Installing it on F29 should be possible I believe.

Comment 23 Adam Williamson 2019-04-25 21:09:50 UTC
Jan: https://koji.fedoraproject.org/koji/buildinfo?buildID=1253909

Comment 24 Robert Holmes 2019-04-25 21:10:48 UTC
I can confirm -301.fc30 works fine; I've been running it on F29 since yesterday.

Comment 25 Kelvin J. Hill 2019-04-25 21:31:44 UTC
I can also confirm that 5.0.9-301.fc30 resolves the issue.

Comment 26 Adam Williamson 2019-04-25 21:33:37 UTC
Great. The fix should show up on F29 with the next build, I am not sure when kernel team plans on doing another build.

Comment 27 Jan Hugo Prins 2019-04-25 21:44:32 UTC
I can also confirm that this update fixes the problem.

Comment 28 Rick 2019-04-26 12:35:12 UTC
To avoid creating an even greater problem, what is the recommended way to start using the -301.fc30 kernel on my F29 system?  Are there any inherent dangers using an F30 kernel on a F29 system?  When a usable F29 kernel appears in the repo, is it a simple matter to return to normal F29 updates, etc.?

Comment 29 Jan Hugo Prins 2019-04-26 12:44:48 UTC
On my Fedora29 laptop I have done:

# dnf update --enablerepo=updates-testing --releasever=30 kernel kernel-devel kernel-modules-extra

The result was that the 5.0.9-301 build was installed, and it is working fine:

[root@capetown ~]# uname -r
5.0.9-301.fc30.x86_64

[root@capetown ~]# cat /etc/fedora-release 
Fedora release 29 (Twenty Nine)

Comment 30 Jeremy Cline 2019-04-26 14:19:13 UTC
(In reply to Rick from comment #28)
> To avoid creating an even greater problem, what is the recommended way to
> start using the -301.fc30 kernel on my F29 system?  Are there any inherent
> dangers using an F30 kernel on a F29 system?  When a usable F29 kernel
> appears in the repo, is it a simple matter to return to normal F29 updates,
> etc.?

Jan's command should get it installed and you shouldn't have any problems running a kernel from a newer version of Fedora. When 5.0.10-200.fc29 shows up, it should sort as newer than 5.0.9-301.fc30 and DNF will install it.

Comment 31 Adam Williamson 2019-04-26 15:48:09 UTC
Yeah. Mixing repos is not usually recommended, but trying a kernel like this is actually one of the safest things you can do with it, because Fedora always keeps three kernels installed at a time, so if it doesn't work you can just boot back to an F29 kernel. Kernels also usually rarely have many other dependencies because they're, well, the kernel, so they won't pull in a bunch of other F30 packages.

Comment 32 Jeremy Cline 2019-04-26 22:17:13 UTC
*** Bug 1703599 has been marked as a duplicate of this bug. ***

Comment 33 Mic 2019-04-29 08:20:56 UTC
Hi,

Just to let people know: When enabling "Windows-UEFI Secure Boot" on my ASUS motherboard, nvidia-drm module could not be loaded anymore since F29 5.0.9 (perhaps also 5.0.8), preventing NVIDIA drivers from installing. When enabling Secure boot but without Windows-UEFI mode, it started working again.

Windows-UEFI mode worked <= 5.0.7.

Don't know if it's a wanted behaviour, just posting here to let you guys know and help people stumbling on the issue. I myself am no wizard in all those questions, so I'll let you figure out if it's an issue or not!

Best

Comment 34 Kelvin J. Hill 2019-04-29 14:02:59 UTC
(In reply to Mic from comment #33)
> Hi,
> 
> Just to let people know: When enabling "Windows-UEFI Secure Boot" on my ASUS
> motherboard, nvidia-drm module could not be loaded anymore since F29 5.0.9
> (perhaps also 5.0.8), preventing NVIDIA drivers from installing. When
> enabling Secure boot but without Windows-UEFI mode, it started working again.
> 
> Windows-UEFI mode worked <= 5.0.7.
> 
> Don't know if it's a wanted behaviour, just posting here to let you guys
> know and help people stumbling on the issue. I myself am no wizard in all
> those questions, so I'll let you figure out if it's an issue or not!
> 
> Best

I tried that workaround on my ASUS Z-170A m/b but it still refused to load the self-signed modules. It may be m/b or BIOS version specific.

Only the fixed 5.0.9-301 works for me.

Comment 35 Fedora Update System 2019-04-29 21:06:37 UTC
kernel-5.0.10-200.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-8219efa9f6

Comment 36 Fedora Update System 2019-04-29 21:08:33 UTC
kernel-5.0.10-100.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2019-86e0db6dbb

Comment 37 Fedora Update System 2019-04-30 03:40:26 UTC
kernel-5.0.10-200.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-8219efa9f6

Comment 38 Fedora Update System 2019-04-30 15:01:23 UTC
kernel-5.0.10-100.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-86e0db6dbb

Comment 39 Robert Holmes 2019-04-30 15:04:24 UTC
kernel-5.0.10-200.fc29 works perfectly on my side, so as far as I'm concerned this can be closed.

Comment 40 Jeremy Cline 2019-04-30 15:10:56 UTC
Great, thanks for testing. The update system automatically closes bugs when the update is actually pushed to the stable repositories.

Comment 41 Benjamin 2019-05-01 01:11:05 UTC
Can confirm that this is fixed in kernel-5.0.9-301.fc30

Comment 42 Peter Robinson 2019-05-01 07:48:01 UTC
Closing as this landed with 5.0.8 and kernel-5.0.9-301.fc30 shipped in F-30 GA

Comment 43 Adam Williamson 2019-05-01 18:08:16 UTC
The bug is filed against 29, not 30. It even specifically says "Fedora 29" in the summary.

Comment 44 Fedora Update System 2019-05-02 20:37:26 UTC
kernel-5.0.11-100.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2019-a6cd583a8d

Comment 45 Fedora Update System 2019-05-03 02:02:11 UTC
kernel-5.0.11-100.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-a6cd583a8d

Comment 46 Fedora Update System 2019-05-03 03:42:12 UTC
kernel-5.0.10-200.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

Comment 47 Fedora Update System 2019-05-07 04:50:04 UTC
kernel-5.0.11-100.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.