Bug 1701096

Additional note: modules were still loaded up until 5.0.8, thought with a warning. But this patch made modules not load at all:

https://src.fedoraproject.org/rpms/kernel/c/24416b46798ee9a53c81faad67021a5df90dff30?branch=f29

Ok, after some experimentation, it appears that commit 219a3e8676f3132d27b530c7d2d6bcab89536b57 is not sufficient to get signed modules to be accepted. The issue seems to be here:

https://elixir.bootlin.com/linux/v5.1-rc5/source/kernel/module_signing.c#L86

where there is seemingly no way to ever use VERIFY_USE_PLATFORM_KEYRING to verify the signatures.


As it currently stands, the instructions in https://docs.fedoraproject.org/en-US/fedora/f29/system-administrators-guide/kernel-module-driver-configuration/Working_with_Kernel_Modules/#sect-signing-kernel-modules-for-secure-boot do not work.

Cherry-picking 219a3e8676f3132d27b530c7d2d6bcab89536b57 and 278311e417be60f7caef6fcb12bda4da2711ceff plus the following patch solve the issue:

diff --git a/kernel/module_signing.c b/kernel/module_signing.c
index 6b9a926fd86b..19ddd63a086b 100644
--- a/kernel/module_signing.c
+++ b/kernel/module_signing.c
@@ -49,6 +49,7 @@ int mod_verify_sig(const void *mod, struct load_info *info)
 {
        struct module_signature ms;
        size_t sig_len, modlen = info->len;
+       int ret;
 
        pr_devel("==>%s(,%zu)\n", __func__, modlen);
 
@@ -82,8 +83,15 @@ int mod_verify_sig(const void *mod, struct load_info *info)
                return -EBADMSG;
        }
 
-       return verify_pkcs7_signature(mod, modlen, mod + modlen, sig_len,
+       ret = verify_pkcs7_signature(mod, modlen, mod + modlen, sig_len,
                                      VERIFY_USE_SECONDARY_KEYRING,
                                      VERIFYING_MODULE_SIGNATURE,
                                      NULL, NULL);
+       if (ret == -ENOKEY && IS_ENABLED(CONFIG_INTEGRITY_PLATFORM_KEYRING)) {
+               ret = verify_pkcs7_signature(mod, modlen, mod + modlen, sig_len,
+                                     VERIFY_USE_PLATFORM_KEYRING,
+                                     VERIFYING_MODULE_SIGNATURE,
+                                     NULL, NULL);
+       }
+       return ret;
 }

Hi Robert,

Thanks for the thorough investigation and patch. Do you feel comfortable submitting this patch upstream? It seems like something upstream would want.

Sent a patch here: https://lkml.org/lkml/2019/4/23/89

These patches should probably also be included in -stable, but I am not familiar with the process of nudging whoever's responsible to choose what gets there.

Great, thanks. The process for -stable is documented at https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html. If you end up needing to re-submit the patch you can add that, or ask the maintainer to consider adding the tag, or wait for it to get merged and email the stable list.

In the mean time, I've picked it up for today's Rawhide build and it should be in the next stable builds (5.0.10). The update system should comment on this bug when it's available for testing.

How on earth was this never picked up in testing? 

It's been a nightmare to workaround since running dnf this morning, removing lockdown and having to then manually load the missing 9 self-signed modules in my case.

There is apparently some stuff related to module signing and loading that is not upstream, so this doesn't get caught by upstream testing and only shows up when we do downstream builds with our patches :/

Kernel updates in Bodhi tend to get an immediate flood of positive feedback, since everyone has the kernel installed and it'll work for lots of people - anyone on a BIOS install or non-SB UEFI install won't notice this problem, for instance, so they'll give the kernel positive feedback. That can lead to an update getting pushed stable very rapidly before a problem that only affects a specific configuration is noticed.

In this case, AIUI, *this* particular bug has not made it out of updates-testing: the 5.0.9 build for F29 was not yet submitted for stable - https://bodhi.fedoraproject.org/updates/FEDORA-2019-1e8a4c6958 . The current stable build is 5.0.8, which AIUI would suffer from 1696671 but not from this bug.

BTW, for the record, the most recent F30 build, kernel-5.0.9-301.fc30 - https://koji.fedoraproject.org/koji/buildinfo?buildID=1253909 - should have the fix for this, so it should not suffer from #1696671 *or* this bug. That build is in the F30 release candidate compose that is currently spinning. If anyone notices that it still has bugs related to signed modules, please let us know. Thanks!

(In reply to Adam Williamson from comment #8)
> In this case, AIUI, *this* particular bug has not made it out of
> updates-testing: the 5.0.9 build for F29 was not yet submitted for stable -
> https://bodhi.fedoraproject.org/updates/FEDORA-2019-1e8a4c6958 . The current
> stable build is 5.0.8, which AIUI would suffer from 1696671 but not from
> this bug.

5.0.8 does suffer from this bug, cf. https://bugzilla.redhat.com/show_bug.cgi?id=1701096#c1

(In reply to Adam Williamson from comment #8)
> There is apparently some stuff related to module signing and loading that is
> not upstream, so this doesn't get caught by upstream testing and only shows
> up when we do downstream builds with our patches :/
> 
> Kernel updates in Bodhi tend to get an immediate flood of positive feedback,
> since everyone has the kernel installed and it'll work for lots of people -
> anyone on a BIOS install or non-SB UEFI install won't notice this problem,
> for instance, so they'll give the kernel positive feedback. That can lead to
> an update getting pushed stable very rapidly before a problem that only
> affects a specific configuration is noticed.
> 
> In this case, AIUI, *this* particular bug has not made it out of
> updates-testing: the 5.0.9 build for F29 was not yet submitted for stable -
> https://bodhi.fedoraproject.org/updates/FEDORA-2019-1e8a4c6958 . The current
> stable build is 5.0.8, which AIUI would suffer from 1696671 but not from
> this bug.

I can confirm that 5.0.8-200, as updated by dnf today, does indeed suffer from the problem.
Non of the modules that would load, albeit noisily, on 5.0.7 fail on 5.0.8.

Ah, OK. I was reading "up until 5.0.8" as meaning it worked *up to and including* 5.0.8, sorry.

(In reply to Kelvin J. Hill from comment #11) 
> I can confirm that 5.0.8-200, as updated by dnf today, does indeed suffer
> from the problem.
> Non of the modules that would load, albeit noisily, on 5.0.7 fail on 5.0.8.
Sorry, that should read "ALL of the modules that would load, albeit noisily, on 5.0.7 fail on 5.0.8."

*** Bug 1702844 has been marked as a duplicate of this bug. ***

It appears upstream wants it that way: https://lkml.org/lkml/2019/4/25/409

Confirming that 5.0.9 shows the same problem.

From the upstream stance, how are self-signed modules supposed to be implemented then on a UEFI based installation?

Besides disabling secure boot, is there another workaround?  Is there a way to add/link our MOK key to the .secondary_trusted_keys keyring?

5.0.9-301 should *not* have this bug, please test that.

(In reply to Adam Williamson from comment #18)
> 5.0.9-301 should *not* have this bug, please test that.

This bug entry is against Fedora 29, which had 5.0.9-200.fc29 delivered by dnf today.
THAT kernel version still shows the problem.

Where is this 5.0.9-301 update available? I'm able to reproduce it very easily.

(In reply to Rick from comment #17)
> Besides disabling secure boot, is there another workaround?  Is there a way
> to add/link our MOK key to the .secondary_trusted_keys keyring?

On some motherboards, especially those from Asus, disabling secure boot can only be done by deleting keys. It is non-trivial...

Kelvin: yes, that's true indeed, sorry - it'd still be very helpful to know if the -301.fc30 kernel is fixed, though. Installing it on F29 should be possible I believe.

Jan: https://koji.fedoraproject.org/koji/buildinfo?buildID=1253909

I can confirm -301.fc30 works fine; I've been running it on F29 since yesterday.

I can also confirm that 5.0.9-301.fc30 resolves the issue.

Great. The fix should show up on F29 with the next build, I am not sure when kernel team plans on doing another build.

I can also confirm that this update fixes the problem.

To avoid creating an even greater problem, what is the recommended way to start using the -301.fc30 kernel on my F29 system?  Are there any inherent dangers using an F30 kernel on a F29 system?  When a usable F29 kernel appears in the repo, is it a simple matter to return to normal F29 updates, etc.?

On my Fedora29 laptop I have done:

# dnf update --enablerepo=updates-testing --releasever=30 kernel kernel-devel kernel-modules-extra

The result was that the 5.0.9-301 build was installed, and it is working fine:

[root@capetown ~]# uname -r
5.0.9-301.fc30.x86_64

[root@capetown ~]# cat /etc/fedora-release 
Fedora release 29 (Twenty Nine)

(In reply to Rick from comment #28)
> To avoid creating an even greater problem, what is the recommended way to
> start using the -301.fc30 kernel on my F29 system?  Are there any inherent
> dangers using an F30 kernel on a F29 system?  When a usable F29 kernel
> appears in the repo, is it a simple matter to return to normal F29 updates,
> etc.?

Jan's command should get it installed and you shouldn't have any problems running a kernel from a newer version of Fedora. When 5.0.10-200.fc29 shows up, it should sort as newer than 5.0.9-301.fc30 and DNF will install it.

Yeah. Mixing repos is not usually recommended, but trying a kernel like this is actually one of the safest things you can do with it, because Fedora always keeps three kernels installed at a time, so if it doesn't work you can just boot back to an F29 kernel. Kernels also usually rarely have many other dependencies because they're, well, the kernel, so they won't pull in a bunch of other F30 packages.

*** Bug 1703599 has been marked as a duplicate of this bug. ***

Hi,

Just to let people know: When enabling "Windows-UEFI Secure Boot" on my ASUS motherboard, nvidia-drm module could not be loaded anymore since F29 5.0.9 (perhaps also 5.0.8), preventing NVIDIA drivers from installing. When enabling Secure boot but without Windows-UEFI mode, it started working again.

Windows-UEFI mode worked <= 5.0.7.

Don't know if it's a wanted behaviour, just posting here to let you guys know and help people stumbling on the issue. I myself am no wizard in all those questions, so I'll let you figure out if it's an issue or not!

Best

(In reply to Mic from comment #33)
> Hi,
> 
> Just to let people know: When enabling "Windows-UEFI Secure Boot" on my ASUS
> motherboard, nvidia-drm module could not be loaded anymore since F29 5.0.9
> (perhaps also 5.0.8), preventing NVIDIA drivers from installing. When
> enabling Secure boot but without Windows-UEFI mode, it started working again.
> 
> Windows-UEFI mode worked <= 5.0.7.
> 
> Don't know if it's a wanted behaviour, just posting here to let you guys
> know and help people stumbling on the issue. I myself am no wizard in all
> those questions, so I'll let you figure out if it's an issue or not!
> 
> Best

I tried that workaround on my ASUS Z-170A m/b but it still refused to load the self-signed modules. It may be m/b or BIOS version specific.

Only the fixed 5.0.9-301 works for me.

kernel-5.0.10-200.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-8219efa9f6

kernel-5.0.10-100.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2019-86e0db6dbb

kernel-5.0.10-200.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-8219efa9f6

kernel-5.0.10-100.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-86e0db6dbb

kernel-5.0.10-200.fc29 works perfectly on my side, so as far as I'm concerned this can be closed.

Great, thanks for testing. The update system automatically closes bugs when the update is actually pushed to the stable repositories.

Can confirm that this is fixed in kernel-5.0.9-301.fc30

Closing as this landed with 5.0.8 and kernel-5.0.9-301.fc30 shipped in F-30 GA

The bug is filed against 29, not 30. It even specifically says "Fedora 29" in the summary.

kernel-5.0.11-100.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2019-a6cd583a8d

kernel-5.0.11-100.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-a6cd583a8d

kernel-5.0.10-200.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

kernel-5.0.11-100.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.