Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1884165

Summary: Repos should be disabled in -firstboot.service before OS extensions are applied
Product: OpenShift Container Platform Reporter: Vadim Rutkovsky <vrutkovs>
Component: Machine Config OperatorAssignee: Vadim Rutkovsky <vrutkovs>
Status: CLOSED ERRATA QA Contact: Michael Nguyen <mnguyen>
Severity: low Docs Contact:
Priority: unspecified    
Version: 4.6CC: bleanhar, jerzhang, miabbott, mkrejci
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-24 15:22:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1888467    

Description Vadim Rutkovsky 2020-10-01 08:34:30 UTC 
MCO uses rpm-ostree install when installing shipped extensions. However, the image may contain enabled repos, so external repositories may be used to install RPMs during firstboot.

This mostly affects OKD, which has FCOS with standard Fedora repos enabled by default. As a result installed RPMs being pulled from ostree repo instead of embedded RPM repository
Comment 1 Brenton Leanhardt 2020-10-02 19:24:40 UTC 
This doesn't seem like a blocker.  I'm going to move out it.  I trust Vadim will backport it if necessary for OKD.
Comment 3 Micah Abbott 2020-11-18 15:02:23 UTC 
Verified with 4.7.0-0.nightly-2020-11-18-085225

Booted a cluster in AWS and confirmed the `machine-config-daemon-firstboot.service` has the change included

```
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2020-11-18-085225   True        False         12m     Cluster version is 4.7.0-0.nightly-2020-11-18-085225

$ oc get nodes
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-0-132-182.us-west-1.compute.internal   Ready    master   16m   v1.19.2+99c8f3a
ip-10-0-156-55.us-west-1.compute.internal    Ready    worker   12m   v1.19.2+99c8f3a                                  
ip-10-0-159-3.us-west-1.compute.internal     Ready    master   17m   v1.19.2+99c8f3a  
ip-10-0-170-78.us-west-1.compute.internal    Ready    worker   11m   v1.19.2+99c8f3a                                                
ip-10-0-217-171.us-west-1.compute.internal   Ready    worker   11m   v1.19.2+99c8f3a            
ip-10-0-251-93.us-west-1.compute.internal    Ready    master   17m   v1.19.2+99c8f3a

$ oc debug node/ip-10-0-170-78.us-west-1.compute.internal
Starting pod/ip-10-0-170-78us-west-1computeinternal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.170.78
If you don't see a command prompt, try pressing enter.
sh-4.4# chroot /host
sh-4.4# systemctl cat machine-config-daemon-firstboot.service
# /etc/systemd/system/machine-config-daemon-firstboot.service
[Unit]
Description=Machine Config Daemon Firstboot
# Make sure it runs only on OSTree booted system
ConditionPathExists=/run/ostree-booted
# Removal of this file signals firstboot completion
ConditionPathExists=/etc/ignition-machine-config-encapsulated.json
After=machine-config-daemon-pull.service
Before=crio.service crio-wipe.service
Before=kubelet.service

[Service]
Type=oneshot
RemainAfterExit=yes
# Disable existing repos (if any) so that OS extensions would use embedded RPMs only
ExecStartPre=-/usr/bin/sh -c "sed -i 's/enabled=1/enabled=0/' /etc/yum.repos.d/*.repo"
ExecStart=/run/bin/machine-config-daemon firstboot-complete-machineconfig

[Install]
WantedBy=multi-user.target
RequiredBy=crio.service kubelet.service
sh-4.4# journalctl -u machine-config-daemon-firstboot
-- Logs begin at Wed 2020-11-18 14:29:06 UTC, end at Wed 2020-11-18 14:45:29 UTC. --
Nov 18 14:29:55 ip-10-0-170-78 systemd[1]: Starting Machine Config Daemon Firstboot...
Nov 18 14:29:55 ip-10-0-170-78 sh[1937]: sed: can't read /etc/yum.repos.d/*.repo: No such file or directory
Nov 18 14:29:55 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:29:55.441795    1940 rpm-ostree.go:261] Running captured: rpm-ostree status --json
Nov 18 14:29:56 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:29:56.470075    1940 daemon.go:226] Booted osImageURL:  (47.82.202010211043-0)
Nov 18 14:29:57 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:29:57.520983    1940 daemon.go:233] Installed Ignition binary version: 2.7.0
Nov 18 14:29:57 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:29:57.522396    1940 update.go:417] Checking Reconcilable for config mco-empty-mc to rendered-worker-222fc194a5e52c6b3b8c1aebd0de2deb
Nov 18 14:29:57 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:29:57.523001    1940 update.go:1607] Starting update from mco-empty-mc to rendered-worker-222fc194a5e52c6b3b8c1aebd0de2deb: &{osUpdate:true kargs:false fips:false passwd:false files:false units:false kernelType:false extensions:false}
Nov 18 14:29:57 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:29:57.526751    1940 update.go:1027] Updating files
Nov 18 14:29:57 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:29:57.527069    1940 update.go:1064] Deleting stale data
Nov 18 14:29:57 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:29:57.528827    1940 run.go:18] Running: nice -- ionice -c 3 oc image extract --path /:/run/mco-machine-os-content/os-content-993375167 --registry-config /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ebab03e4ee0
>
Nov 18 14:31:20 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:31:20.686609    1940 update.go:1485] Updating OS to quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ebab03e4ee05f6c0844632fabb2ae31dad29b68da0f53f0d11d3e20e80f5590
Nov 18 14:31:20 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:31:20.686793    1940 rpm-ostree.go:261] Running captured: rpm-ostree status --json
Nov 18 14:31:20 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:31:20.748936    1940 rpm-ostree.go:184] Current origin is not custom
Nov 18 14:31:22 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:31:22.250546    1940 rpm-ostree.go:211] Pivoting to: 47.82.202011171242-0 (28197f2e2acd42dc8f72265a0774789ec1acb9f3b5c8d71cdd057e3831b716ab)
Nov 18 14:31:22 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:31:22.250572    1940 rpm-ostree.go:243] Executing rebase from repo path /run/mco-machine-os-content/os-content-993375167/srv/repo with customImageURL pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ebab03e4ee05f6c0844632fabb2ae31dad29
>
Nov 18 14:31:22 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:31:22.250584    1940 rpm-ostree.go:261] Running captured: rpm-ostree rebase --experimental /run/mco-machine-os-content/os-content-993375167/srv/repo:28197f2e2acd42dc8f72265a0774789ec1acb9f3b5c8d71cdd057e3831b716ab --custom-origin-url pivot://quay.io
>
Nov 18 14:31:34 ip-10-0-170-78 machine-config-daemon[1940]: I1118 14:31:34.282507    1940 update.go:1607] initiating reboot: Completing firstboot provisioning to rendered-worker-222fc194a5e52c6b3b8c1aebd0de2deb
Nov 18 14:31:34 ip-10-0-170-78 systemd[1]: machine-config-daemon-firstboot.service: Main process exited, code=killed, status=15/TERM
Nov 18 14:31:34 ip-10-0-170-78 systemd[1]: machine-config-daemon-firstboot.service: Failed with result 'signal'.
Nov 18 14:31:34 ip-10-0-170-78 systemd[1]: Stopped Machine Config Daemon Firstboot.
Nov 18 14:31:34 ip-10-0-170-78 systemd[1]: machine-config-daemon-firstboot.service: Consumed 17.942s CPU time
```
Comment 6 errata-xmlrpc 2021-02-24 15:22:14 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633