MCO uses rpm-ostree install when installing shipped extensions. However, the image may contain enabled repos, so external repositories may be used to install RPMs during firstboot. This mostly affects OKD, which has FCOS with standard Fedora repos enabled by default. As a result installed RPMs being pulled from ostree repo instead of embedded RPM repository
This doesn't seem like a blocker. I'm going to move out it. I trust Vadim will backport it if necessary for OKD.
Verified with 4.7.0-0.nightly-2020-11-18-085225 Booted a cluster in AWS and confirmed the `machine-config-daemon-firstboot.service` has the change included ``` $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.0-0.nightly-2020-11-18-085225 True False 12m Cluster version is 4.7.0-0.nightly-2020-11-18-085225 $ oc get nodes NAME STATUS ROLES AGE VERSION ip-10-0-132-182.us-west-1.compute.internal Ready master 16m v1.19.2+99c8f3a ip-10-0-156-55.us-west-1.compute.internal Ready worker 12m v1.19.2+99c8f3a ip-10-0-159-3.us-west-1.compute.internal Ready master 17m v1.19.2+99c8f3a ip-10-0-170-78.us-west-1.compute.internal Ready worker 11m v1.19.2+99c8f3a ip-10-0-217-171.us-west-1.compute.internal Ready worker 11m v1.19.2+99c8f3a ip-10-0-251-93.us-west-1.compute.internal Ready master 17m v1.19.2+99c8f3a $ oc debug node/ip-10-0-170-78.us-west-1.compute.internal Starting pod/ip-10-0-170-78us-west-1computeinternal-debug ... To use host binaries, run `chroot /host` Pod IP: 10.0.170.78 If you don't see a command prompt, try pressing enter. sh-4.4# chroot /host sh-4.4# systemctl cat machine-config-daemon-firstboot.service # /etc/systemd/system/machine-config-daemon-firstboot.service [Unit] Description=Machine Config Daemon Firstboot # Make sure it runs only on OSTree booted system ConditionPathExists=/run/ostree-booted # Removal of this file signals firstboot completion ConditionPathExists=/etc/ignition-machine-config-encapsulated.json After=machine-config-daemon-pull.service Before=crio.service crio-wipe.service Before=kubelet.service [Service] Type=oneshot RemainAfterExit=yes # Disable existing repos (if any) so that OS extensions would use embedded RPMs only ExecStartPre=-/usr/bin/sh -c "sed -i 's/enabled=1/enabled=0/' /etc/yum.repos.d/*.repo" ExecStart=/run/bin/machine-config-daemon firstboot-complete-machineconfig [Install] WantedBy=multi-user.target RequiredBy=crio.service kubelet.service sh-4.4# journalctl -u machine-config-daemon-firstboot -- Logs begin at Wed 2020-11-18 14:29:06 UTC, end at Wed 2020-11-18 14:45:29 UTC. -- Nov 18 14:29:55 ip-10-0-170-78 systemd[1]: Starting Machine Config Daemon Firstboot... Nov 18 14:29:55 ip-10-0-170-78 sh[1937]: sed: can't read /etc/yum.repos.d/*.repo: No such file or directory Nov 18 14:29:55 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:29:55.441795 1940 rpm-ostree.go:261] Running captured: rpm-ostree status --json Nov 18 14:29:56 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:29:56.470075 1940 daemon.go:226] Booted osImageURL: (47.82.202010211043-0) Nov 18 14:29:57 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:29:57.520983 1940 daemon.go:233] Installed Ignition binary version: 2.7.0 Nov 18 14:29:57 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:29:57.522396 1940 update.go:417] Checking Reconcilable for config mco-empty-mc to rendered-worker-222fc194a5e52c6b3b8c1aebd0de2deb Nov 18 14:29:57 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:29:57.523001 1940 update.go:1607] Starting update from mco-empty-mc to rendered-worker-222fc194a5e52c6b3b8c1aebd0de2deb: &{osUpdate:true kargs:false fips:false passwd:false files:false units:false kernelType:false extensions:false} Nov 18 14:29:57 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:29:57.526751 1940 update.go:1027] Updating files Nov 18 14:29:57 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:29:57.527069 1940 update.go:1064] Deleting stale data Nov 18 14:29:57 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:29:57.528827 1940 run.go:18] Running: nice -- ionice -c 3 oc image extract --path /:/run/mco-machine-os-content/os-content-993375167 --registry-config /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ebab03e4ee0 > Nov 18 14:31:20 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:31:20.686609 1940 update.go:1485] Updating OS to quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ebab03e4ee05f6c0844632fabb2ae31dad29b68da0f53f0d11d3e20e80f5590 Nov 18 14:31:20 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:31:20.686793 1940 rpm-ostree.go:261] Running captured: rpm-ostree status --json Nov 18 14:31:20 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:31:20.748936 1940 rpm-ostree.go:184] Current origin is not custom Nov 18 14:31:22 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:31:22.250546 1940 rpm-ostree.go:211] Pivoting to: 47.82.202011171242-0 (28197f2e2acd42dc8f72265a0774789ec1acb9f3b5c8d71cdd057e3831b716ab) Nov 18 14:31:22 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:31:22.250572 1940 rpm-ostree.go:243] Executing rebase from repo path /run/mco-machine-os-content/os-content-993375167/srv/repo with customImageURL pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ebab03e4ee05f6c0844632fabb2ae31dad29 > Nov 18 14:31:22 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:31:22.250584 1940 rpm-ostree.go:261] Running captured: rpm-ostree rebase --experimental /run/mco-machine-os-content/os-content-993375167/srv/repo:28197f2e2acd42dc8f72265a0774789ec1acb9f3b5c8d71cdd057e3831b716ab --custom-origin-url pivot://quay.io > Nov 18 14:31:34 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:31:34.282507 1940 update.go:1607] initiating reboot: Completing firstboot provisioning to rendered-worker-222fc194a5e52c6b3b8c1aebd0de2deb Nov 18 14:31:34 ip-10-0-170-78 systemd[1]: machine-config-daemon-firstboot.service: Main process exited, code=killed, status=15/TERM Nov 18 14:31:34 ip-10-0-170-78 systemd[1]: machine-config-daemon-firstboot.service: Failed with result 'signal'. Nov 18 14:31:34 ip-10-0-170-78 systemd[1]: Stopped Machine Config Daemon Firstboot. Nov 18 14:31:34 ip-10-0-170-78 systemd[1]: machine-config-daemon-firstboot.service: Consumed 17.942s CPU time ```
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633