Bug 1856820 - [4.5.z] unable to boot RHCOS 4.5 with SecureBoot enabled
Summary: [4.5.z] unable to boot RHCOS 4.5 with SecureBoot enabled
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: RHCOS
Version: 4.5
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.5.z
Assignee: Micah Abbott
QA Contact: Michael Nguyen
URL:
Whiteboard:
Depends On: 1856501 1857238
Blocks: 1856821 1856822
TreeView+ depends on / blocked
 
Reported: 2020-07-14 14:09 UTC by Micah Abbott
Modified: 2021-04-05 17:46 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Booting RHCOS with Secure Boot enabled Consequence: RHCOS is unable to boot Fix: Including a rebuilt 8.2 kernel with proper signatures for Secure Boot Result: RHCOS is able to boot successfully with Secure Boot enabled
Clone Of: 1856501
: 1856821 (view as bug list)
Environment:
Last Closed: 2020-07-16 16:12:24 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:2909 0 None None None 2020-07-16 16:12:45 UTC

Comment 4 Micah Abbott 2020-07-15 03:28:42 UTC
Verified with 4.5.0-0.nightly-2020-07-14-213353

```
$ oc image info --output json $(oc adm release info -a ~/openshift-cluster-installs/all-the-pull-secrets.json --image-for=machine-os-content registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-07-14-213353) | jq .config.config.Labels.version
"45.82.202007141718-0"

$ curl -L -O https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.5/45.82.202007141718-0/x86_64/rhcos-45.82.202007141718-0-qemu.x86_64.qcow2.gz

$ gunzip rhcos-45.82.202007141718-0-qemu.x86_64.qcow2.gz

$ coreos-assembler run --qemu-firmware uefi-secure --qemu-image ./rhcos-45.82.202007141718-0-qemu.x86_64.qcow2
+ chrt --idle 0 podman run --rm -ti --security-opt label=disable --privileged --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap 1001:1001:64536 -v /var/home/miabbott/redhat-coreos:/srv/ --device /dev/kvm --device /dev2
virtio journal connected - sshd started
Warning: Permanently added '[127.0.0.1]:41655' (ECDSA) to the list of known hosts.
Red Hat Enterprise Linux CoreOS 45.82.202007141718-0
  Part of OpenShift 4.5, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.5/architecture/architecture-rhcos.html

---
Last login: Wed Jul 15 03:21:04 2020
[core@cosa-devsh ~]$ rpm-ostree status -b
State: idle
BootedDeployment:
* ostree://67315b4b010341ffd396fe699287defe530830b17879d695fec0243b87e97c82
                   Version: 45.82.202007141718-0 (2020-07-14T17:21:59Z)
[core@cosa-devsh ~]$ rpm -q kernel
kernel-4.18.0-193.13.2.el8_2.x86_64
[core@cosa-devsh ~]$ dmesg | grep -i secure
[    0.000000] secureboot: Secure boot enabled
[    0.000000] Kernel is locked down from EFI secure boot; see man kernel_lockdown.7
[    2.903033] integrity: Loaded X.509 cert 'Red Hat Secure Boot (CA key 1): 4016841644ce3a810408050766e8f8a29c65f85c'
```

Upgrading from 4.4.3

```
$ oc image info --output json $(oc adm release info -a ~/openshift-cluster-installs/all-the-pull-secrets.json --image-for=machine-os-content registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-07-14-213353) | jq .name
"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4915dc7f35a77a07fd4a1ae1c41463de17e03ebbaa5e9296dbc0acefa40f714d"

$ curl -LO https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.4/latest/rhcos-4.4.3-x86_64-qemu.x86_64.qcow2.gz

$ gunzip rhcos-4.4.3-x86_64-qemu.x86_64.qcow2.gz

$ coreos-assembler run --qemu-firmware uefi-secure --qemu-image ./rhcos-4.4.3-x86_64-qemu.x86_64.qcow2
+ chrt --idle 0 podman run --rm -ti --security-opt label=disable --privileged --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap 1001:1001:64536 -v /var/home/miabbott/redhat-coreos:/srv/ --device /dev/kvm --device /dev/fuse --tmpfs /tmp -v /var/tmp:/var/tmp --name coreos-assembler quay.io/coreos-assembler/coreos-assembler:latest run --qemu-firmware uefi-secure --qemu-image ./rhcos-4.4.3-x86_64-qemu.x86_64.qcow2
virtio journal connected - sshd started
Warning: Permanently added '[127.0.0.1]:37407' (ECDSA) to the list of known hosts.
Red Hat Enterprise Linux CoreOS 44.81.202004260825-0
  Part of OpenShift 4.4, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.4/architecture/architecture-rhcos.html

---
Last login: Wed Jul 15 03:24:08 2020
[core@cosa-devsh ~]$ rpm-ostree status -b
State: idle
AutomaticUpdates: disabled
BootedDeployment:
* ostree://2062bce64e4932160feb58ce4976a885172d3f1017dc01f09177504bd55e035b
                   Version: 44.81.202004260825-0 (2020-04-26T08:30:26Z)
[core@cosa-devsh ~]$ rpm -q kernel
kernel-4.18.0-147.8.1.el8_1.x86_64
[core@cosa-devsh ~]$ dmesg | grep -i secure
[    0.000000] secureboot: Secure boot enabled
[    0.000000] Kernel is locked down from EFI secure boot; see man kernel_lockdown.7
[    4.386216] integrity: Loaded X.509 cert 'Red Hat Secure Boot (CA key 1): 4016841644ce3a810408050766e8f8a29c65f85c'
[core@cosa-devsh ~]$ sudo mkdir -p /var/lib/kubelet
[core@cosa-devsh ~]$ sudo vi /var/lib/kubelet/config.json
[core@cosa-devsh ~]$ sudo pivot quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4915dc7f35a77a07fd4a1ae1c41463de17e03ebbaa5e9296dbc0acefa40f714d
I0715 03:25:12.316737    1844 rpm-ostree.go:366] Running captured: rpm-ostree status --json
I0715 03:25:12.357904    1844 rpm-ostree.go:159] Current origin is not custom
I0715 03:25:12.358300    1844 run.go:16] Running: podman pull -q --authfile /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4915dc7f35a77a07fd4a1ae1c41463de17e03ebbaa5e9296dbc0acefa40f714d
aa9557bde2f3e1699a119eea4fe53bfef7232628a8c03597816b58a77cd47297
I0715 03:26:50.674455    1844 rpm-ostree.go:366] Running captured: podman inspect --type=image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4915dc7f35a77a07fd4a1ae1c41463de17e03ebbaa5e9296dbc0acefa40f714d
I0715 03:26:50.806449    1844 rpm-ostree.go:366] Running captured: podman create --net=none --annotation=org.openshift.machineconfigoperator.pivot=true --name ostree-container-pivot-8b4e3f7c-e206-4c55-bf20-090008761f0a quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4915dc7f35a77a07fd4a1ae1c41463de17e03ebbaa5e9296dbc0acefa40f714d
I0715 03:26:50.916467    1844 rpm-ostree.go:366] Running captured: podman mount 42018988f5c19692a71e23eda9247ade30114e840e86195d7df529c759054b67
I0715 03:26:51.001808    1844 rpm-ostree.go:246] Pivoting to: 45.82.202007141718-0 (67315b4b010341ffd396fe699287defe530830b17879d695fec0243b87e97c82)
[core@cosa-devsh ~]$ rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
  pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4915dc7f35a77a07fd4a1ae1c41463de17e03ebbaa5e9296dbc0acefa40f714d
              CustomOrigin: Managed by machine-config-operator
                   Version: 45.82.202007141718-0 (2020-07-14T17:21:59Z)
                      Diff: 225 upgraded, 4 downgraded, 3 removed, 29 added

* ostree://2062bce64e4932160feb58ce4976a885172d3f1017dc01f09177504bd55e035b
                   Version: 44.81.202004260825-0 (2020-04-26T08:30:26Z)
[core@cosa-devsh ~]$ sudo systemctl reboot
Connection to 127.0.0.1 closed by remote host.
Connection to 127.0.0.1 closed.
Disconnected, attempting to reconnect (Ctrl-C to exit)
kex_exchange_identification: read: Connection reset by peer
Connection reset by 127.0.0.1 port 37407
Disconnected, attempting to reconnect (Ctrl-C to exit)
kex_exchange_identification: read: Connection reset by peer
Connection reset by 127.0.0.1 port 37407
Disconnected, attempting to reconnect (Ctrl-C to exit)
Warning: Permanently added '[127.0.0.1]:37407' (ECDSA) to the list of known hosts.
Red Hat Enterprise Linux CoreOS 45.82.202007141718-0
  Part of OpenShift 4.5, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.5/architecture/architecture-rhcos.html

---
Last login: Wed Jul 15 03:28:06 2020
[core@cosa-devsh ~]$ rpm-ostree status -b
State: idle
BootedDeployment:
* pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4915dc7f35a77a07fd4a1ae1c41463de17e03ebbaa5e9296dbc0acefa40f714d
              CustomOrigin: Managed by machine-config-operator
                   Version: 45.82.202007141718-0 (2020-07-14T17:21:59Z)
[core@cosa-devsh ~]$ rpm -q kernel
kernel-4.18.0-193.13.2.el8_2.x86_64
[core@cosa-devsh ~]$ dmesg | grep -i secure
[    0.000000] secureboot: Secure boot enabled
[    0.000000] Kernel is locked down from EFI secure boot; see man kernel_lockdown.7
[    2.780632] integrity: Loaded X.509 cert 'Red Hat Secure Boot (CA key 1): 4016841644ce3a810408050766e8f8a29c65f85c'
```

Comment 6 errata-xmlrpc 2020-07-16 16:12:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2909

Comment 7 W. Trevor King 2021-04-05 17:46:23 UTC
Removing UpgradeBlocker from this older bug, to remove it from the suspect queue described in [1].  If you feel like this bug still needs to be a suspect, please add keyword again.

[1]: https://github.com/openshift/enhancements/pull/475


Note You need to log in before you can comment on or make changes to this bug.