Description of problem: using rpm-ostree for the following custom file insertion failed to work. Initramfs: -I /etc/systemd/system.conf.d/setAffinity.conf there was no complain from the rpm-ostree and after reboot the status shows: pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a41afef678a6d50c0d1459380531bd90eb7c3d4b33feddeabf7ab77916f33f77 CustomOrigin: Managed by machine-config-operator Version: 44.81.202002070530-0 (2020-02-07T05:36:00Z) RemovedBasePackages: kernel-core kernel-modules kernel kernel-modules-extra 4.18.0-147.5.1.el8_1 LocalPackages: kernel-rt-modules-4.18.0-147.5.1.rt24.98.el8_1.x86_64 kernel-rt-core-4.18.0-147.5.1.rt24.98.el8_1.x86_64 kernel-rt-modules-extra-4.18.0-147.5.1.rt24.98.el8_1.x86_64 Initramfs: -I /etc/systemd/system.conf.d/setAffinity.conf lsinitrd shows the setAffinity.conf is not in the initramfs. lsinitrd /boot/ostree/rhcos-089075598d1903a31ac6733b4d25475bccf24d2fab16f9d750195c65bf31c95c/initramfs-4.18.0-147.5.1.rt24.98.el8_1.x86_64.img | grep setAffinity.conf Arguments: --reproducible --gzip -v --add 'ostree' --tmpdir '/tmp/dracut' -f -I '/etc/systemd/system.conf /etc/systemd/system.conf.d/setAffinity.conf' --kver '4.18.0-147.5.1.rt24.98.el8_1.x86_64' [root@perf150 tmp]# if directly using dracut, this file shows up in a temporary initramfs image: dracut -I /etc/systemd/system.conf.d/setAffinity.conf test.img [root@perf150 tmp]# lsinitrd test.img | grep setAffinity.conf Arguments: -I '/etc/systemd/system.conf.d/setAffinity.conf' -rwx------ 1 root root 27 Jan 1 1970 etc/systemd/system.conf.d/setAffinity.conf Version-Release number of selected component (if applicable): # rpm-ostree status State: idle AutomaticUpdates: disabled Deployments: ● pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a41afef678a6d50c0d1459380531bd90eb7c3d4b33feddeabf7ab77916f33f77 CustomOrigin: Managed by machine-config-operator Version: 44.81.202002070530-0 (2020-02-07T05:36:00Z) RemovedBasePackages: kernel-core kernel-modules kernel kernel-modules-extra 4.18.0-147.5.1.el8_1 LocalPackages: kernel-rt-modules-4.18.0-147.5.1.rt24.98.el8_1.x86_64 kernel-rt-core-4.18.0-147.5.1.rt24.98.el8_1.x86_64 kernel-rt-modules-extra-4.18.0-147.5.1.rt24.98.el8_1.x86_64 Initramfs: -I /etc/systemd/system.conf /etc/systemd/system.conf.d/setAffinity.conf pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a41afef678a6d50c0d1459380531bd90eb7c3d4b33feddeabf7ab77916f33f77 CustomOrigin: Managed by machine-config-operator Version: 44.81.202002070530-0 (2020-02-07T05:36:00Z) RemovedBasePackages: kernel-core kernel-modules kernel kernel-modules-extra 4.18.0-147.5.1.el8_1 LocalPackages: kernel-rt-modules-4.18.0-147.5.1.rt24.98.el8_1.x86_64 kernel-rt-core-4.18.0-147.5.1.rt24.98.el8_1.x86_64 kernel-rt-modules-extra-4.18.0-147.5.1.rt24.98.el8_1.x86_64 Initramfs: -I /etc/systemd/system.conf.d/setAffinity.conf How reproducible: the insertion may work right after the fresh coreos install (not sure but saw it worked once); after other rpm-ostree operation this issue starts to show up and then always happen.
I'm testing patches for this and should have something out soon. Though note in the meantime, if this is blocking other tests from proceeding, you can temporarily hack around it by creating an overlay CPIO containing the right files. E.g. mkdir rootfs && cd rootfs mkdir -p etc/systemd/system.conf.d echo 'foo' > etc/systemd/system.conf echo 'bar' > etc/systemd/system.conf.d/affinity.conf find . | cpio -co > /boot/foobar.img And then modify the BLS config file in /boot/loader/entries to have the second initrd: initrd /foobar.img
(In reply to Jonathan Lebon from comment #3) > I'm testing patches for this and should have something out soon. Though note > in the meantime, if this is blocking other tests from proceeding, you can > temporarily hack around it by creating an overlay CPIO containing the right > files. E.g. > > mkdir rootfs && cd rootfs > mkdir -p etc/systemd/system.conf.d > echo 'foo' > etc/systemd/system.conf > echo 'bar' > etc/systemd/system.conf.d/affinity.conf > find . | cpio -co > /boot/foobar.img > > And then modify the BLS config file in /boot/loader/entries to have the > second initrd: > initrd /foobar.img This would mean we will have a persistent entry in /boot/loader/entries which will need to disappear somehow in upgrades
*** Bug 1806954 has been marked as a duplicate of this bug. ***
I've verified that the upstream patches linked here fix this exact issue (using the exact same kernel-rt NEVRA and initramfs injected files).
This is still assigned but I've verified the upstream BZ[0] using RHCOS from OCP 4.4.0-0.nightly-2020-03-05-175156. We can move this to verified once its ON_QA. [0] https://bugzilla.redhat.com/show_bug.cgi?id=1808459 $ oc image info -a ../all-the-pull-secrets.json $(oc adm release -a ../all-the-pull-secrets.json info --image-for=machine-os-content registry.svc.ci.openshift.org/ocp/release:4.4.0-0.nightly-2020-03-05-175156) Name: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8d15e78a81ee0f2a725d88c08f086b3178a5e898d74a5bf89d91f3eec73f1345 Media Type: application/vnd.docker.distribution.manifest.v2+json Created: 8h ago Image Size: 804.2MB OS: linux Arch: amd64 Entrypoint: /noentry Labels: com.coreos.ostree-commit=a33130bb69d2edeb5a255b9059d2bfc975219ecd735fb12fa6ec0c7838bf02bd io.buildah.version=1.14.0 version=44.81.202003051130-0 [root@localhost ~]# rpm -q rpm-ostree rpm-ostree-2019.6-8.el8.x86_64 [root@localhost ~]# rpm-ostree override remove kernel-core kernel-modules kernel kernel-modules-extra --install kernel-rt-core-4.18.0-147.5.1.rt24.98.el8_1.x86_64.rpm --install kernel-rt-modules-4.18.0-147.5.1.rt24.98.el8_1.x86_64.rpm --install kernel-rt-modules-extra-4.18.0-147.5.1.rt24.98.el8_1.x86_64.rpm Checking out tree a33130b... done Enabled rpm-md repositories: Importing rpm-md... done Resolving dependencies... done Applying 4 overrides and 3 overlays Processing packages... done Running pre scripts... done Running post scripts... done Running posttrans scripts... done Writing rpmdb... done Generating initramfs... done Writing OSTree commit... done Staging deployment... done Removed: kernel-4.18.0-147.5.1.el8_1.x86_64 kernel-core-4.18.0-147.5.1.el8_1.x86_64 kernel-modules-4.18.0-147.5.1.el8_1.x86_64 kernel-modules-extra-4.18.0-147.5.1.el8_1.x86_64 Added: kernel-rt-core-4.18.0-147.5.1.rt24.98.el8_1.x86_64 kernel-rt-modules-4.18.0-147.5.1.rt24.98.el8_1.x86_64 kernel-rt-modules-extra-4.18.0-147.5.1.rt24.98.el8_1.x86_64 Run "systemctl reboot" to start a reboot root@localhost ~]# rpm-ostree status State: idle AutomaticUpdates: disabled Deployments: ostree://a33130bb69d2edeb5a255b9059d2bfc975219ecd735fb12fa6ec0c7838bf02bd Version: 44.81.202003051130-0 (2020-03-05T11:35:49Z) Diff: 4 removed, 3 added RemovedBasePackages: kernel-core kernel-modules kernel kernel-modules-extra 4.18.0-147.5.1.el8_1 LocalPackages: kernel-rt-modules-4.18.0-147.5.1.rt24.98.el8_1.x86_64 kernel-rt-core-4.18.0-147.5.1.rt24.98.el8_1.x86_64 kernel-rt-modules-extra-4.18.0-147.5.1.rt24.98.el8_1.x86_64 ● ostree://a33130bb69d2edeb5a255b9059d2bfc975219ecd735fb12fa6ec0c7838bf02bd Version: 44.81.202003051130-0 (2020-03-05T11:35:49Z) [root@localhost ~]# echo '#' > /etc/systemd/system.conf.d/setAffinity.conf [root@localhost ~]# cat /etc/systemd/system.conf.d/setAffinity.conf # [root@localhost ~]# cat /etc/systemd/system.conf # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the GNU Lesser General Public License as published by # the Free Software Foundation; either version 2.1 of the License, or # (at your option) any later version. # # Entries in this file show the compile time defaults. # You can change settings by editing this file. # Defaults can be restored by simply deleting this file. # # See systemd-system.conf(5) for details. [Manager] #LogLevel=info #LogTarget=journal-or-kmsg #LogColor=yes #LogLocation=no #DumpCore=yes #ShowStatus=yes #CrashChangeVT=no #CrashShell=no #CrashReboot=no #CtrlAltDelBurstAction=reboot-force #CPUAffinity=1 2 #JoinControllers=cpu,cpuacct net_cls,net_prio #RuntimeWatchdogSec=0 #ShutdownWatchdogSec=10min #CapabilityBoundingSet= #NoNewPrivileges=no #SystemCallArchitectures= #TimerSlackNSec= #DefaultTimerAccuracySec=1min #DefaultStandardOutput=journal #DefaultStandardError=inherit #DefaultTimeoutStartSec=90s #DefaultTimeoutStopSec=90s #DefaultRestartSec=100ms #DefaultStartLimitIntervalSec=10s #DefaultStartLimitBurst=5 #DefaultEnvironment= #DefaultCPUAccounting=no #DefaultIOAccounting=no #DefaultIPAccounting=no #DefaultBlockIOAccounting=no #DefaultMemoryAccounting=yes #DefaultTasksAccounting=yes #DefaultTasksMax=80% #DefaultLimitCPU= #DefaultLimitFSIZE= #DefaultLimitDATA= #DefaultLimitSTACK= #DefaultLimitCORE= #DefaultLimitRSS= #DefaultLimitNOFILE= #DefaultLimitAS= #DefaultLimitNPROC= #DefaultLimitMEMLOCK= #DefaultLimitLOCKS= #DefaultLimitSIGPENDING= #DefaultLimitMSGQUEUE= #DefaultLimitNICE= #DefaultLimitRTPRIO= #DefaultLimitRTTIME= #IPAddressAllow= #IPAddressDeny= [root@localhost ~]# rpm-ostree initramfs --enable --arg="-I" --arg="/etc/systemd/system.conf /etc/systemd/system.conf.d/setAffinity.conf" Checking out tree a33130b... done Enabled rpm-md repositories: Importing rpm-md... done Resolving dependencies... done Applying 4 overrides and 3 overlays Processing packages... done Running pre scripts... done Running post scripts... done Running posttrans scripts... done Writing rpmdb... done Generating initramfs... done Writing OSTree commit... done Staging deployment... done Freed: 109.4 MB (pkgcache branches: 0) Initramfs regeneration is now: enabled [root@localhost ~]# sudo rpm-ostree status State: idle AutomaticUpdates: disabled Deployments: ostree://a33130bb69d2edeb5a255b9059d2bfc975219ecd735fb12fa6ec0c7838bf02bd Version: 44.81.202003051130-0 (2020-03-05T11:35:49Z) Diff: 4 removed, 3 added RemovedBasePackages: kernel-core kernel-modules kernel kernel-modules-extra 4.18.0-147.5.1.el8_1 LocalPackages: kernel-rt-modules-4.18.0-147.5.1.rt24.98.el8_1.x86_64 kernel-rt-core-4.18.0-147.5.1.rt24.98.el8_1.x86_64 kernel-rt-modules-extra-4.18.0-147.5.1.rt24.98.el8_1.x86_64 Initramfs: -I '/etc/systemd/system.conf /etc/systemd/system.conf.d/setAffinity.conf' ● ostree://a33130bb69d2edeb5a255b9059d2bfc975219ecd735fb12fa6ec0c7838bf02bd Version: 44.81.202003051130-0 (2020-03-05T11:35:49Z) [root@localhost ~]# systemctl reboot [root@localhost ~]# Connection to 192.168.122.249 closed by remote host. == RECONNECT == [root@localhost ~]# cd /boot/ostree/ [root@localhost ostree]# lsinitrd /boot/ostree/rhcos-261af2cc637b4270c88dfa8e3611b226b77528a50ada141d281a943c09c7ddda/initramfs-4.18.0-147.5.1.rt24.98.el8_1.x86_64.img | grep setAffinity.conf Arguments: --reproducible --gzip -v --add 'ostree' --tmpdir '/tmp/dracut' -f -I '/etc/systemd/system.conf /etc/systemd/system.conf.d/setAffinity.conf' --kver '4.18.0-147.5.1.rt24.98.el8_1.x86_64' -rw-r--r-- 1 root root 2 Jan 1 1970 etc/systemd/system.conf.d/setAffinity.conf == IN INITRAMFS == Entering emergency mode. Exit the shell to continue. Type "journalctl" to view system logs. You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot after mounting them and attach it to a bug report. switch_root:/# ls bin dracut-state.sh etc kernel lib64 root sbin sys tmp var dev early_cpio init lib proc run shutdown sysroot usr switch_root:/# cd etc/systemd/ switch_root:/etc/systemd# ls journald.conf journald.conf.d system system.conf system.conf.d switch_root:/etc/systemd# cat system.conf # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the GNU Lesser General Public License as published by # the Free Software Foundation; either version 2.1 of the License, or # (at your option) any later version. # # Entries in this file show the compile time defaults. # You can change settings by editing this file. # Defaults can be restored by simply deleting this file. # # See systemd-system.conf(5) for details. [Manager] #LogLevel=info #LogTarget=journal-or-kmsg #LogColor=yes #LogLocation=no #DumpCore=yes #ShowStatus=yes #CrashChangeVT=no #CrashShell=no #CrashReboot=no #CtrlAltDelBurstAction=reboot-force #CPUAffinity=1 2 #JoinControllers=cpu,cpuacct net_cls,net_prio #RuntimeWatchdogSec=0 #ShutdownWatchdogSec=10min #CapabilityBoundingSet= #NoNewPrivileges=no #SystemCallArchitectures= #TimerSlackNSec= #DefaultTimerAccuracySec=1min #DefaultStandardOutput=journal #DefaultStandardError=inherit #DefaultTimeoutStartSec=90s #DefaultTimeoutStopSec=90s #DefaultRestartSec=100ms #DefaultStartLimitIntervalSec=10s #DefaultStartLimitBurst=5 #DefaultEnvironment= #DefaultCPUAccounting=no #DefaultIOAccounting=no #DefaultIPAccounting=no #DefaultBlockIOAccounting=no #DefaultMemoryAccounting=yes #DefaultTasksAccounting=yes #DefaultTasksMax=80% #DefaultLimitCPU= #DefaultLimitFSIZE= #DefaultLimitDATA= #DefaultLimitSTACK= #DefaultLimitCORE= #DefaultLimitRSS= #DefaultLimitNOFILE= #DefaultLimitAS= #DefaultLimitNPROC= #DefaultLimitMEMLOCK= #DefaultLimitLOCKS= #DefaultLimitSIGPENDING= #DefaultLimitMSGQUEUE= #DefaultLimitNICE= #DefaultLimitRTPRIO= #DefaultLimitRTTIME= #IPAddressAllow= #IPAddressDeny= switch_root:/etc/systemd# cat system system/ system.conf system.conf.d/ switch_root:/etc/systemd# cat system.conf.d/setAffinity.conf # switch_root:/etc/systemd#
Recent systemd now includes a kernel cmdline option for this: https://github.com/systemd/systemd/pull/14154 We should at some point get that backported to RHEL and teach the performance operator to use it.
Colin, that is great news. Any estimate on when this will land into RHCOS?
The fixed package landed in RHCOS 44.81.202003062006-0; all subsequent builds should have the fix.
@Martin please follow https://bugzilla.redhat.com/show_bug.cgi?id=1812894 Remember RHCOS *is* RHEL - so systemd patches go there.
Moving to VERIFIED based on comment #7
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581