Bug 873796

Summary: dracut is stripping off kernel module signatures
Product: [Fedora] Fedora Reporter: Josh Boyer <jwboyer>
Component: dracutAssignee: dracut-maint
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 18CC: agrover, awilliam, dhowells, dracut-maint, harald, hdegoede, johannbg, jonathan, jreznik, mads, mikhail.v.gavrilov, pjones, redhat, robatino
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: AcceptedNTH RejectedBlocker
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-11-23 07:58:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 752664    
Attachments:
Description Flags
Don't strip kernel modules in the initramfs
none
dmesg output none

Description Josh Boyer 2012-11-06 18:11:04 UTC
Created attachment 639483 [details]
Don't strip kernel modules in the initramfs

Description of problem:

In the F18 and rawhide kernels, we have enabled signed modules using the newly included upstream modsign infrastructure.  This adds a signature to the end of all kernel modules, which the kernel will then verify against a certificate embedded in the kernel.

At the moment, dracut runs 'strip -g' on all of the modules.  This leaves the ELF sections intact but it will strip off the signature of the modules included in the initramfs.  This causes the module to fail signature verification.  That will either cause the module to be tainted as FORCED, or fail to load depending on the settings of the kernel verification mechanism.  An example can be seen here:

[jwboyer@vader ~]$ cat /proc/modules | grep F
broadcom 13170 0 - Live 0xffffffffa022a000 (F)
crc32c_intel 12901 0 - Live 0xffffffffa0222000 (F)
ghash_clmulni_intel 13180 0 - Live 0xffffffffa005f000 (F)
radeon 895780 3 - Live 0xffffffffa00e7000 (F)
i2c_algo_bit 13257 1 radeon, Live 0xffffffffa0040000 (F)
drm_kms_helper 44701 1 radeon, Live 0xffffffffa0048000 (F)
ttm 79760 1 radeon, Live 0xffffffffa00d2000 (F)
tg3 148915 0 - Live 0xffffffffa00a5000 (F)
drm 255010 5 radeon,drm_kms_helper,ttm, Live 0xffffffffa0065000 (F)
i2c_core 38314 5 i2c_i801,radeon,i2c_algo_bit,drm_kms_helper,drm, Live 0xffffffffa0054000 (F)
sunrpc 255586 17 nfsd,auth_rpcgss,nfs_acl,lockd, Live 0xffffffffa0000000 (F)
[jwboyer@vader ~]$

This is seen with dracut-024-5.git20121019.fc18.x86_64.

We actually don't need to strip the modules at all in Fedora, as the kernel build already runs the equivalent of strip -g when it does the kernel debuginfo package creation.  All dracut is doing in this case is stripping off the signature.

The attached patch adds a new --kmod-strip option and defaults it to no.  This seems to have fixed the issue on my local test machine.

Comment 1 Adam Williamson 2012-11-06 18:56:04 UTC
AIUI, this will result in Secure Boot-enabled systems failing to boot, yes? Impact on non-SB case:

<jwb> the kernel being tainted unecessarily, lockdep being disabled because of the taint (more of a kernel-debug thing, or rawhide), and possibly ABRT not filing backtraces
<jmoskovc> and yes, abrt refuses to report oopses in that case :-/

I'm at least +1 NTH given the above impacts. dracut folks, in preparing an update to fix this, can you please build one which *only* fixes this, no other changes? we're in Beta freeze and we'd like to avoid taking unnecessary changes to critical components at this point. Thanks! The build currently in stable is dracut-024-5.git20121019.fc18 , so a build which is just that plus this fix would be ideal.

Comment 2 Kevin Fenzi 2012-11-06 20:08:05 UTC
+1 NTH.

Comment 3 Jaroslav Reznik 2012-11-06 20:13:56 UTC
+1 NTH as we'd like to have SB working in Beta, seems like the patch is against the latest SPEC - just rebuild is needed when applied/no other changes should be pulled in.

Comment 4 Josh Boyer 2012-11-06 20:15:18 UTC
(In reply to comment #1)
> AIUI, this will result in Secure Boot-enabled systems failing to boot, yes?

It is possible it would cause SB systems to fail to boot, yes.  That depends on the system needing a module from the initramfs to load to access the root filesystem, or other similar scenarios.  It would certainly cause interesting issues with graphics working in the initramfs environment, given none of those modules are built-in.

Comment 5 Jóhann B. Guðmundsson 2012-11-06 20:19:01 UTC
Harald How risky is a fix for this?

Comment 6 Jóhann B. Guðmundsson 2012-11-06 20:26:54 UTC
From the looks of daz code it it just needs to be generated with --nostrip or NO by default +1 nth

Comment 7 Josh Boyer 2012-11-06 21:22:35 UTC
(In reply to comment #6)
> From the looks of daz code it it just needs to be generated with --nostrip
> or NO by default +1 nth

That's possible to do, but it treats everything the same.  The attached patch just does it for kernel modules.  I went for a solution that would be more acceptable from a generic upstream perspective.

Comment 8 Adam Williamson 2012-11-07 00:06:48 UTC
we seem to be solidly NTH on this one, setting acceptednth rejectedblocker. Again, please ensure any fix for this is as minimal as possible, we don't want to take any unnecessary changes to sensitive components at this point.

Comment 9 Josh Boyer 2012-11-13 17:06:45 UTC
Harald, Kay, <anyone else>?

Comment 10 Mads Kiilerich 2012-11-13 21:32:21 UTC
FWIW, Bug 867829 was closely related and could have been resolved by disabling strip completely ... but for some reason strip was kept.

One reason it was kept might be that it do make a difference.
In KVM, initramfs-3.6.6-3.fc18.x86_64.img
with strip:    7053874 bytes
without strip: 8009793 bytes

Comment 11 Harald Hoyer 2012-11-19 17:25:10 UTC
Well, customers run into problems with weak-modules and unstripped modules, resulting in initramfs sizes around 150MB, which do not boot anymore.

Can we teach "strip" to keep the kernel signatures somehow?

Comment 12 Josh Boyer 2012-11-19 18:22:53 UTC
(In reply to comment #11)
> Well, customers run into problems with weak-modules and unstripped modules,
> resulting in initramfs sizes around 150MB, which do not boot anymore.
> 
> Can we teach "strip" to keep the kernel signatures somehow?

Not that I'm aware of.  Both strip and eu-strip (binutils and elf-utils) will strip off the signature at the end because it isn't in an ELF section.  David Howells spent quite some time looking at this, so I'm putting him on CC.  Upstream chose the "append signatures" option, so here we are.

Fedora doesn't provide or support weak-modules, so that isn't particularly a concern here.  However, if that is a use case dracut has to support elsewhere, I'm not sure how it will work with upstream signed modules.

Also, the modules Fedora has actually _are_ stripped even without dracut doing it because of debuginfo, so the size of the initramfs is not really increased significantly when passing the new option in the patch.  Basically just the size of the module signature per kernel module.  I am unaware of how weak-modules are generated, but if they don't have associated debuginfo packages then I could see why leaving the modules unstripped would be an issue.

Comment 13 Josh Boyer 2012-11-19 19:22:21 UTC
*** Bug 878092 has been marked as a duplicate of this bug. ***

Comment 19 Fedora Update System 2012-11-21 16:40:24 UTC
dracut-024-10.git20121121.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/dracut-024-10.git20121121.fc18

Comment 20 Fedora Update System 2012-11-21 20:52:41 UTC
Package dracut-024-10.git20121121.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dracut-024-10.git20121121.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-18701/dracut-024-10.git20121121.fc18
then log in and leave karma (feedback).

Comment 21 Mikhail 2012-11-22 03:22:33 UTC
Seems now kernel not tainted.

Comment 22 Mikhail 2012-11-22 03:23:02 UTC
Created attachment 649483 [details]
dmesg output

Comment 23 Fedora Update System 2012-11-23 07:59:02 UTC
dracut-024-10.git20121121.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.