When booting the 4.5 ISO with SecureBoot enabled, users get the following error on the console: error: ../../g rub-core/loader/i386/efi/linux.c:208:(hd0,gpt1)/ostree/rhcos-12763030471528c3bb 42044e109205afafb3693660ffb2a56d16fb5a2beaeacd/vmlinuz-4.18.0-193.12.1.el8_2.x8 6_64 has invalid signature. error: ../../grub-core/loader/i386/efi/linux.c:208:(hd0,gpt1)/ostree/rhcos-12763030471 528c3bb42044e109205afafb3693660ffb2a56d16fb5a2beaeacd/vmlinuz-4.18.0-193.12.1.e l8_2.x86_64 has invalid signature. error: ../../grub-core/loader/i386/efi/linux.c:93:you need to load the kernel first. error: ../../grub-core/loader/i386/efi/linux.c:93:you need to load the kernel first. Press any key to continue...Press any key to continue... Disabling SecureBoot will allow the host to boot successfully. Likewise, upgrading from a previous version (i.e. 4.4.3) to 4.5.0 via the `machine-os-content` causes the same error to occur when booting into the new deployment.
It looks to me like the kernel we chose might signed only with a "test" key; if you look at: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1214959 Note: `Task build (rhel-8.3.0-test-pesign, /rpms/kernel:62c14a5bb3a93e2ed34de201a92ae94702de249e)` In contrast say this kernel for 8.2.z: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1245471 `Task build (rhel-8.2.0-z-candidate, /rpms/kernel:6f03bc339daa3d3cbaa53f1e643b32d6ac824e9e)`
To test with qemu/virt-install: `$ virt-install --features smm=on --machine q35 -n rhcos443-uefisb -r 4096 --os-variant=rhel8.0 --import --network network=default --disk=/var/lib/libvirt/images/rhcos-4.4.3-x86_64-qemu.x86_64.vm0713a.qcow2,format=qcow2,bus=virtio --qemu-commandline="-fw_cfg name=opt/com.coreos/config,file=/var/lib/libvirt/images/ignition.json" --boot loader=/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd,loader.readonly=yes,loader.type=pflash,loader_secure=yes` Once in the node, copy your pull secret to /var/lib/kubelet/config.json and use pivot to update to 4.5.0: `$ sudo pivot quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:784456823004f41b2718e45796423bd8bf6689ed6372b66f2e2f4db8e7d6bcb9`
`cosa run --qemu-firmware=uefi-secure` also reproduces the failure FWIW.
Upgrading the 4.4.3 VM to an *older* version of RHCOS 4.5 which has `kernel-4.18.0-193.11.1.el8_2.x86_64` signed by 199e2f91fd431d51 (Red Hat release key, I believe) works successfully.
The RPM signature may be a red-herring; upon further inspection the kernel in the affected RHCOS 4.5 build (`kernel-4.18.0-193.12.1.el8_2.x86_64`) is signed with the same key as the unaffected kernel (`kernel-4.18.0-193.11.1.el8_2.x86_64`)
We've determined that the affected kernel (`kernel-4.18.0-193.12.1.el8_2.x86_64`) was built with the incorrectly with regards to how it is signed for Secure Boot. (Note: this is different from how the RPM file itself is signed) How this happened is currently being investigated by the RHEL kernel team, who will also provide a new kernel package that was built properly. When the new kernel is provided, RHCOS will be able to consume it as part the normal build process.
FYI, for future reference you can tell which key it was signed with either by reading the logs and trying to decode which invocation was used (that's the hard way) or the much easier way: random:~/tmp$ rpm2cpio ../Downloads/kernel-core-4.18.0-211.el8.x86_64.rpm | cpio -diu 58019 blocks random:~/tmp$ pesign -i ./lib/modules/4.18.0-211.el8.x86_64/vmlinuz -l --------------------------------------------- certificate address is 0x7fb263c9b978 Content was not encrypted. Content is detached; signature cannot be verified. The signer's common name is Red Hat Secure Boot Signing 3 (beta) No signer email address. Signing time: Thu Jun 04, 2020 There were certs or crls included. ---------------------------------------------
We shipped the fix for this but this BZ didn't get attached to the errata.
Ah, confusing. For 4.6 then it's a duplicate of 1857238 since we switched kernels back to 8.3. *** This bug has been marked as a duplicate of bug 1857238 ***
Removing UpgradeBlocker from this older bug, to remove it from the suspect queue described in [1]. If you feel like this bug still needs to be a suspect, please add keyword again. [1]: https://github.com/openshift/enhancements/pull/475