Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1856820

Summary: [4.5.z] unable to boot RHCOS 4.5 with SecureBoot enabled
Product: OpenShift Container Platform Reporter: Micah Abbott <miabbott>
Component: RHCOSAssignee: Micah Abbott <miabbott>
Status: CLOSED ERRATA QA Contact: Michael Nguyen <mnguyen>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 4.5CC: bbreard, bgilbert, dornelas, imcleod, jjung, jligon, lmohanty, mnguyen, nstielau, rheinzma, rsandu, smilner, vlaad, walters, wking
Target Milestone: ---Keywords: UpcomingSprint
Target Release: 4.5.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Booting RHCOS with Secure Boot enabled Consequence: RHCOS is unable to boot Fix: Including a rebuilt 8.2 kernel with proper signatures for Secure Boot Result: RHCOS is able to boot successfully with Secure Boot enabled
 Story Points: ---
Clone Of: 1856501
: 1856821 (view as bug list) Environment:
Last Closed: 2020-07-16 16:12:24 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1856501, 1857238    
Bug Blocks: 1856821, 1856822    

Comment 4 Micah Abbott 2020-07-15 03:28:42 UTC 
Verified with 4.5.0-0.nightly-2020-07-14-213353

```
$ oc image info --output json $(oc adm release info -a ~/openshift-cluster-installs/all-the-pull-secrets.json --image-for=machine-os-content registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-07-14-213353) | jq .config.config.Labels.version
"45.82.202007141718-0"

$ curl -L -O https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.5/45.82.202007141718-0/x86_64/rhcos-45.82.202007141718-0-qemu.x86_64.qcow2.gz

$ gunzip rhcos-45.82.202007141718-0-qemu.x86_64.qcow2.gz

$ coreos-assembler run --qemu-firmware uefi-secure --qemu-image ./rhcos-45.82.202007141718-0-qemu.x86_64.qcow2
+ chrt --idle 0 podman run --rm -ti --security-opt label=disable --privileged --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap 1001:1001:64536 -v /var/home/miabbott/redhat-coreos:/srv/ --device /dev/kvm --device /dev2
virtio journal connected - sshd started
Warning: Permanently added '[127.0.0.1]:41655' (ECDSA) to the list of known hosts.
Red Hat Enterprise Linux CoreOS 45.82.202007141718-0
  Part of OpenShift 4.5, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.5/architecture/architecture-rhcos.html

---
Last login: Wed Jul 15 03:21:04 2020
[core@cosa-devsh ~]$ rpm-ostree status -b
State: idle
BootedDeployment:
* ostree://67315b4b010341ffd396fe699287defe530830b17879d695fec0243b87e97c82
                   Version: 45.82.202007141718-0 (2020-07-14T17:21:59Z)
[core@cosa-devsh ~]$ rpm -q kernel
kernel-4.18.0-193.13.2.el8_2.x86_64
[core@cosa-devsh ~]$ dmesg | grep -i secure
[    0.000000] secureboot: Secure boot enabled
[    0.000000] Kernel is locked down from EFI secure boot; see man kernel_lockdown.7
[    2.903033] integrity: Loaded X.509 cert 'Red Hat Secure Boot (CA key 1): 4016841644ce3a810408050766e8f8a29c65f85c'
```

Upgrading from 4.4.3

```
$ oc image info --output json $(oc adm release info -a ~/openshift-cluster-installs/all-the-pull-secrets.json --image-for=machine-os-content registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-07-14-213353) | jq .name
"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4915dc7f35a77a07fd4a1ae1c41463de17e03ebbaa5e9296dbc0acefa40f714d"

$ curl -LO https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.4/latest/rhcos-4.4.3-x86_64-qemu.x86_64.qcow2.gz

$ gunzip rhcos-4.4.3-x86_64-qemu.x86_64.qcow2.gz

$ coreos-assembler run --qemu-firmware uefi-secure --qemu-image ./rhcos-4.4.3-x86_64-qemu.x86_64.qcow2
+ chrt --idle 0 podman run --rm -ti --security-opt label=disable --privileged --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap 1001:1001:64536 -v /var/home/miabbott/redhat-coreos:/srv/ --device /dev/kvm --device /dev/fuse --tmpfs /tmp -v /var/tmp:/var/tmp --name coreos-assembler quay.io/coreos-assembler/coreos-assembler:latest run --qemu-firmware uefi-secure --qemu-image ./rhcos-4.4.3-x86_64-qemu.x86_64.qcow2
virtio journal connected - sshd started
Warning: Permanently added '[127.0.0.1]:37407' (ECDSA) to the list of known hosts.
Red Hat Enterprise Linux CoreOS 44.81.202004260825-0
  Part of OpenShift 4.4, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.4/architecture/architecture-rhcos.html

---
Last login: Wed Jul 15 03:24:08 2020
[core@cosa-devsh ~]$ rpm-ostree status -b
State: idle
AutomaticUpdates: disabled
BootedDeployment:
* ostree://2062bce64e4932160feb58ce4976a885172d3f1017dc01f09177504bd55e035b
                   Version: 44.81.202004260825-0 (2020-04-26T08:30:26Z)
[core@cosa-devsh ~]$ rpm -q kernel
kernel-4.18.0-147.8.1.el8_1.x86_64
[core@cosa-devsh ~]$ dmesg | grep -i secure
[    0.000000] secureboot: Secure boot enabled
[    0.000000] Kernel is locked down from EFI secure boot; see man kernel_lockdown.7
[    4.386216] integrity: Loaded X.509 cert 'Red Hat Secure Boot (CA key 1): 4016841644ce3a810408050766e8f8a29c65f85c'
[core@cosa-devsh ~]$ sudo mkdir -p /var/lib/kubelet
[core@cosa-devsh ~]$ sudo vi /var/lib/kubelet/config.json
[core@cosa-devsh ~]$ sudo pivot quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4915dc7f35a77a07fd4a1ae1c41463de17e03ebbaa5e9296dbc0acefa40f714d
I0715 03:25:12.316737    1844 rpm-ostree.go:366] Running captured: rpm-ostree status --json
I0715 03:25:12.357904    1844 rpm-ostree.go:159] Current origin is not custom
I0715 03:25:12.358300    1844 run.go:16] Running: podman pull -q --authfile /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4915dc7f35a77a07fd4a1ae1c41463de17e03ebbaa5e9296dbc0acefa40f714d
aa9557bde2f3e1699a119eea4fe53bfef7232628a8c03597816b58a77cd47297
I0715 03:26:50.674455    1844 rpm-ostree.go:366] Running captured: podman inspect --type=image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4915dc7f35a77a07fd4a1ae1c41463de17e03ebbaa5e9296dbc0acefa40f714d
I0715 03:26:50.806449    1844 rpm-ostree.go:366] Running captured: podman create --net=none --annotation=org.openshift.machineconfigoperator.pivot=true --name ostree-container-pivot-8b4e3f7c-e206-4c55-bf20-090008761f0a quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4915dc7f35a77a07fd4a1ae1c41463de17e03ebbaa5e9296dbc0acefa40f714d
I0715 03:26:50.916467    1844 rpm-ostree.go:366] Running captured: podman mount 42018988f5c19692a71e23eda9247ade30114e840e86195d7df529c759054b67
I0715 03:26:51.001808    1844 rpm-ostree.go:246] Pivoting to: 45.82.202007141718-0 (67315b4b010341ffd396fe699287defe530830b17879d695fec0243b87e97c82)
[core@cosa-devsh ~]$ rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
  pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4915dc7f35a77a07fd4a1ae1c41463de17e03ebbaa5e9296dbc0acefa40f714d
              CustomOrigin: Managed by machine-config-operator
                   Version: 45.82.202007141718-0 (2020-07-14T17:21:59Z)
                      Diff: 225 upgraded, 4 downgraded, 3 removed, 29 added

* ostree://2062bce64e4932160feb58ce4976a885172d3f1017dc01f09177504bd55e035b
                   Version: 44.81.202004260825-0 (2020-04-26T08:30:26Z)
[core@cosa-devsh ~]$ sudo systemctl reboot
Connection to 127.0.0.1 closed by remote host.
Connection to 127.0.0.1 closed.
Disconnected, attempting to reconnect (Ctrl-C to exit)
kex_exchange_identification: read: Connection reset by peer
Connection reset by 127.0.0.1 port 37407
Disconnected, attempting to reconnect (Ctrl-C to exit)
kex_exchange_identification: read: Connection reset by peer
Connection reset by 127.0.0.1 port 37407
Disconnected, attempting to reconnect (Ctrl-C to exit)
Warning: Permanently added '[127.0.0.1]:37407' (ECDSA) to the list of known hosts.
Red Hat Enterprise Linux CoreOS 45.82.202007141718-0
  Part of OpenShift 4.5, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.5/architecture/architecture-rhcos.html

---
Last login: Wed Jul 15 03:28:06 2020
[core@cosa-devsh ~]$ rpm-ostree status -b
State: idle
BootedDeployment:
* pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4915dc7f35a77a07fd4a1ae1c41463de17e03ebbaa5e9296dbc0acefa40f714d
              CustomOrigin: Managed by machine-config-operator
                   Version: 45.82.202007141718-0 (2020-07-14T17:21:59Z)
[core@cosa-devsh ~]$ rpm -q kernel
kernel-4.18.0-193.13.2.el8_2.x86_64
[core@cosa-devsh ~]$ dmesg | grep -i secure
[    0.000000] secureboot: Secure boot enabled
[    0.000000] Kernel is locked down from EFI secure boot; see man kernel_lockdown.7
[    2.780632] integrity: Loaded X.509 cert 'Red Hat Secure Boot (CA key 1): 4016841644ce3a810408050766e8f8a29c65f85c'
```
Comment 6 errata-xmlrpc 2020-07-16 16:12:24 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2909
Comment 7 W. Trevor King 2021-04-05 17:46:23 UTC 
Removing UpgradeBlocker from this older bug, to remove it from the suspect queue described in [1].  If you feel like this bug still needs to be a suspect, please add keyword again.

[1]: https://github.com/openshift/enhancements/pull/475