Bug 1856501
| Summary: | unable to boot RHCOS 4.5 with SecureBoot enabled | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Micah Abbott <miabbott> | |
| Component: | RHCOS | Assignee: | Colin Walters <walters> | |
| Status: | CLOSED DUPLICATE | QA Contact: | Michael Nguyen <mnguyen> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | urgent | |||
| Version: | 4.6 | CC: | bbreard, bgilbert, dcain, dornelas, imcleod, jjung, jligon, kholtz, lmohanty, mfuruta, nstielau, pjones, pmatouse, vlaad, william.caban, wking | |
| Target Milestone: | --- | Keywords: | UpcomingSprint | |
| Target Release: | 4.6.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1856820 (view as bug list) | Environment: | ||
| Last Closed: | 2020-07-17 14:21:11 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1856820, 1856821, 1856822 | |||
|
Description
Micah Abbott
2020-07-13 19:16:19 UTC
It looks to me like the kernel we chose might signed only with a "test" key; if you look at: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1214959 Note: `Task build (rhel-8.3.0-test-pesign, /rpms/kernel:62c14a5bb3a93e2ed34de201a92ae94702de249e)` In contrast say this kernel for 8.2.z: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1245471 `Task build (rhel-8.2.0-z-candidate, /rpms/kernel:6f03bc339daa3d3cbaa53f1e643b32d6ac824e9e)` To test with qemu/virt-install: `$ virt-install --features smm=on --machine q35 -n rhcos443-uefisb -r 4096 --os-variant=rhel8.0 --import --network network=default --disk=/var/lib/libvirt/images/rhcos-4.4.3-x86_64-qemu.x86_64.vm0713a.qcow2,format=qcow2,bus=virtio --qemu-commandline="-fw_cfg name=opt/com.coreos/config,file=/var/lib/libvirt/images/ignition.json" --boot loader=/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd,loader.readonly=yes,loader.type=pflash,loader_secure=yes` Once in the node, copy your pull secret to /var/lib/kubelet/config.json and use pivot to update to 4.5.0: `$ sudo pivot quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:784456823004f41b2718e45796423bd8bf6689ed6372b66f2e2f4db8e7d6bcb9` `cosa run --qemu-firmware=uefi-secure` also reproduces the failure FWIW. Upgrading the 4.4.3 VM to an *older* version of RHCOS 4.5 which has `kernel-4.18.0-193.11.1.el8_2.x86_64` signed by 199e2f91fd431d51 (Red Hat release key, I believe) works successfully. The RPM signature may be a red-herring; upon further inspection the kernel in the affected RHCOS 4.5 build (`kernel-4.18.0-193.12.1.el8_2.x86_64`) is signed with the same key as the unaffected kernel (`kernel-4.18.0-193.11.1.el8_2.x86_64`) We've determined that the affected kernel (`kernel-4.18.0-193.12.1.el8_2.x86_64`) was built with the incorrectly with regards to how it is signed for Secure Boot. (Note: this is different from how the RPM file itself is signed) How this happened is currently being investigated by the RHEL kernel team, who will also provide a new kernel package that was built properly. When the new kernel is provided, RHCOS will be able to consume it as part the normal build process. FYI, for future reference you can tell which key it was signed with either by reading the logs and trying to decode which invocation was used (that's the hard way) or the much easier way: random:~/tmp$ rpm2cpio ../Downloads/kernel-core-4.18.0-211.el8.x86_64.rpm | cpio -diu 58019 blocks random:~/tmp$ pesign -i ./lib/modules/4.18.0-211.el8.x86_64/vmlinuz -l --------------------------------------------- certificate address is 0x7fb263c9b978 Content was not encrypted. Content is detached; signature cannot be verified. The signer's common name is Red Hat Secure Boot Signing 3 (beta) No signer email address. Signing time: Thu Jun 04, 2020 There were certs or crls included. --------------------------------------------- We shipped the fix for this but this BZ didn't get attached to the errata. Ah, confusing. For 4.6 then it's a duplicate of 1857238 since we switched kernels back to 8.3. *** This bug has been marked as a duplicate of bug 1857238 *** Removing UpgradeBlocker from this older bug, to remove it from the suspect queue described in [1]. If you feel like this bug still needs to be a suspect, please add keyword again. [1]: https://github.com/openshift/enhancements/pull/475 |