Bug 1806588 - rpm-ostree insert file into initramfs may not work
Summary: rpm-ostree insert file into initramfs may not work
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: RHCOS
Version: 4.4
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ---
: 4.4.0
Assignee: Jonathan Lebon
QA Contact: Michael Nguyen
URL:
Whiteboard:
: 1806954 (view as bug list)
Depends On: 1808459
Blocks: 1771572
TreeView+ depends on / blocked
 
Reported: 2020-02-24 15:10 UTC by jianzzha
Modified: 2020-05-04 11:39 UTC (History)
12 users (show)

Fixed In Version: rpm-ostree-2019.6-8.el8
Doc Type: Bug Fix
Doc Text:
Previously, using `rpm-ostree initramfs` to pass `-I /etc/foobar.conf` to dracut while having a replaced kernel would not work: the file would not appear in the regenerated initramfs. Now, rpm-ostree correctly handles this scenario.
Clone Of:
Environment:
Last Closed: 2020-05-04 11:39:23 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github coreos rpm-ostree pull 1997 None closed compose: Include base dracut args in commitmeta 2020-09-15 20:24:23 UTC
Github coreos rpm-ostree pull 1998 None closed initramfs: Fix using local /etc when also replacing kernel 2020-09-15 20:24:21 UTC

Description jianzzha 2020-02-24 15:10:15 UTC
Description of problem:
using rpm-ostree for the following custom file insertion failed to work. 
Initramfs: -I /etc/systemd/system.conf.d/setAffinity.conf 

there was no complain from the rpm-ostree and after reboot the status shows:
  pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a41afef678a6d50c0d1459380531bd90eb7c3d4b33feddeabf7ab77916f33f77
              CustomOrigin: Managed by machine-config-operator
                   Version: 44.81.202002070530-0 (2020-02-07T05:36:00Z)
       RemovedBasePackages: kernel-core kernel-modules kernel kernel-modules-extra 4.18.0-147.5.1.el8_1
             LocalPackages: kernel-rt-modules-4.18.0-147.5.1.rt24.98.el8_1.x86_64 kernel-rt-core-4.18.0-147.5.1.rt24.98.el8_1.x86_64
                            kernel-rt-modules-extra-4.18.0-147.5.1.rt24.98.el8_1.x86_64
                 Initramfs: -I /etc/systemd/system.conf.d/setAffinity.conf 

lsinitrd shows the setAffinity.conf is not in the initramfs.
lsinitrd /boot/ostree/rhcos-089075598d1903a31ac6733b4d25475bccf24d2fab16f9d750195c65bf31c95c/initramfs-4.18.0-147.5.1.rt24.98.el8_1.x86_64.img | grep setAffinity.conf
Arguments: --reproducible --gzip -v --add 'ostree' --tmpdir '/tmp/dracut' -f -I '/etc/systemd/system.conf /etc/systemd/system.conf.d/setAffinity.conf' --kver '4.18.0-147.5.1.rt24.98.el8_1.x86_64'
[root@perf150 tmp]#

if directly using dracut, this file shows up in a temporary initramfs image:
dracut -I /etc/systemd/system.conf.d/setAffinity.conf test.img
[root@perf150 tmp]# lsinitrd test.img | grep setAffinity.conf
Arguments: -I '/etc/systemd/system.conf.d/setAffinity.conf'
-rwx------   1 root     root           27 Jan  1  1970 etc/systemd/system.conf.d/setAffinity.conf



Version-Release number of selected component (if applicable):
# rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
● pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a41afef678a6d50c0d1459380531bd90eb7c3d4b33feddeabf7ab77916f33f77
              CustomOrigin: Managed by machine-config-operator
                   Version: 44.81.202002070530-0 (2020-02-07T05:36:00Z)
       RemovedBasePackages: kernel-core kernel-modules kernel kernel-modules-extra 4.18.0-147.5.1.el8_1
             LocalPackages: kernel-rt-modules-4.18.0-147.5.1.rt24.98.el8_1.x86_64 kernel-rt-core-4.18.0-147.5.1.rt24.98.el8_1.x86_64
                            kernel-rt-modules-extra-4.18.0-147.5.1.rt24.98.el8_1.x86_64
                 Initramfs: -I /etc/systemd/system.conf /etc/systemd/system.conf.d/setAffinity.conf 

  pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a41afef678a6d50c0d1459380531bd90eb7c3d4b33feddeabf7ab77916f33f77
              CustomOrigin: Managed by machine-config-operator
                   Version: 44.81.202002070530-0 (2020-02-07T05:36:00Z)
       RemovedBasePackages: kernel-core kernel-modules kernel kernel-modules-extra 4.18.0-147.5.1.el8_1
             LocalPackages: kernel-rt-modules-4.18.0-147.5.1.rt24.98.el8_1.x86_64 kernel-rt-core-4.18.0-147.5.1.rt24.98.el8_1.x86_64
                            kernel-rt-modules-extra-4.18.0-147.5.1.rt24.98.el8_1.x86_64
                 Initramfs: -I /etc/systemd/system.conf.d/setAffinity.conf 


How reproducible:
the insertion may work right after the fresh coreos install (not sure but saw it worked once); after other rpm-ostree operation this issue starts to show up and then always happen.

Comment 3 Jonathan Lebon 2020-02-26 15:06:29 UTC
I'm testing patches for this and should have something out soon. Though note in the meantime, if this is blocking other tests from proceeding, you can temporarily hack around it by creating an overlay CPIO containing the right files. E.g.

mkdir rootfs && cd rootfs
mkdir -p etc/systemd/system.conf.d
echo 'foo' > etc/systemd/system.conf
echo 'bar' > etc/systemd/system.conf.d/affinity.conf
find . | cpio -co > /boot/foobar.img

And then modify the BLS config file in /boot/loader/entries to have the second initrd:
initrd /foobar.img

Comment 4 Yanir Quinn 2020-02-27 08:30:56 UTC
(In reply to Jonathan Lebon from comment #3)
> I'm testing patches for this and should have something out soon. Though note
> in the meantime, if this is blocking other tests from proceeding, you can
> temporarily hack around it by creating an overlay CPIO containing the right
> files. E.g.
> 
> mkdir rootfs && cd rootfs
> mkdir -p etc/systemd/system.conf.d
> echo 'foo' > etc/systemd/system.conf
> echo 'bar' > etc/systemd/system.conf.d/affinity.conf
> find . | cpio -co > /boot/foobar.img
> 
> And then modify the BLS config file in /boot/loader/entries to have the
> second initrd:
> initrd /foobar.img

This would mean we will have a persistent entry in /boot/loader/entries which will need to disappear somehow in upgrades

Comment 5 Colin Walters 2020-02-27 15:46:52 UTC
*** Bug 1806954 has been marked as a duplicate of this bug. ***

Comment 6 Jonathan Lebon 2020-03-02 17:12:12 UTC
I've verified that the upstream patches linked here fix this exact issue (using the exact same kernel-rt NEVRA and initramfs injected files).

Comment 7 Michael Nguyen 2020-03-05 22:52:02 UTC
This is still assigned but I've verified the upstream BZ[0] using RHCOS from OCP 4.4.0-0.nightly-2020-03-05-175156.  We can move this to verified once its ON_QA.


[0] https://bugzilla.redhat.com/show_bug.cgi?id=1808459

$ oc image info -a ../all-the-pull-secrets.json $(oc adm release -a ../all-the-pull-secrets.json info --image-for=machine-os-content registry.svc.ci.openshift.org/ocp/release:4.4.0-0.nightly-2020-03-05-175156)
Name:       quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8d15e78a81ee0f2a725d88c08f086b3178a5e898d74a5bf89d91f3eec73f1345
Media Type: application/vnd.docker.distribution.manifest.v2+json
Created:    8h ago
Image Size: 804.2MB
OS:         linux
Arch:       amd64
Entrypoint: /noentry
Labels:     com.coreos.ostree-commit=a33130bb69d2edeb5a255b9059d2bfc975219ecd735fb12fa6ec0c7838bf02bd
            io.buildah.version=1.14.0
            version=44.81.202003051130-0


[root@localhost ~]# rpm -q rpm-ostree
rpm-ostree-2019.6-8.el8.x86_64
[root@localhost ~]# rpm-ostree override remove kernel-core kernel-modules kernel kernel-modules-extra --install kernel-rt-core-4.18.0-147.5.1.rt24.98.el8_1.x86_64.rpm  --install kernel-rt-modules-4.18.0-147.5.1.rt24.98.el8_1.x86_64.rpm --install kernel-rt-modules-extra-4.18.0-147.5.1.rt24.98.el8_1.x86_64.rpm
Checking out tree a33130b... done
Enabled rpm-md repositories:
Importing rpm-md... done
Resolving dependencies... done
Applying 4 overrides and 3 overlays
Processing packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Generating initramfs... done
Writing OSTree commit... done
Staging deployment... done
Removed:
  kernel-4.18.0-147.5.1.el8_1.x86_64
  kernel-core-4.18.0-147.5.1.el8_1.x86_64
  kernel-modules-4.18.0-147.5.1.el8_1.x86_64
  kernel-modules-extra-4.18.0-147.5.1.el8_1.x86_64
Added:
  kernel-rt-core-4.18.0-147.5.1.rt24.98.el8_1.x86_64
  kernel-rt-modules-4.18.0-147.5.1.rt24.98.el8_1.x86_64
  kernel-rt-modules-extra-4.18.0-147.5.1.rt24.98.el8_1.x86_64
Run "systemctl reboot" to start a reboot
root@localhost ~]# rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
  ostree://a33130bb69d2edeb5a255b9059d2bfc975219ecd735fb12fa6ec0c7838bf02bd
                   Version: 44.81.202003051130-0 (2020-03-05T11:35:49Z)
                      Diff: 4 removed, 3 added
       RemovedBasePackages: kernel-core kernel-modules kernel kernel-modules-extra 4.18.0-147.5.1.el8_1
             LocalPackages: kernel-rt-modules-4.18.0-147.5.1.rt24.98.el8_1.x86_64
                            kernel-rt-core-4.18.0-147.5.1.rt24.98.el8_1.x86_64
                            kernel-rt-modules-extra-4.18.0-147.5.1.rt24.98.el8_1.x86_64

● ostree://a33130bb69d2edeb5a255b9059d2bfc975219ecd735fb12fa6ec0c7838bf02bd
                   Version: 44.81.202003051130-0 (2020-03-05T11:35:49Z)

[root@localhost ~]# echo '#' > /etc/systemd/system.conf.d/setAffinity.conf
[root@localhost ~]# cat /etc/systemd/system.conf.d/setAffinity.conf
#
[root@localhost ~]# cat /etc/systemd/system.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See systemd-system.conf(5) for details.

[Manager]
#LogLevel=info
#LogTarget=journal-or-kmsg
#LogColor=yes
#LogLocation=no
#DumpCore=yes
#ShowStatus=yes
#CrashChangeVT=no
#CrashShell=no
#CrashReboot=no
#CtrlAltDelBurstAction=reboot-force
#CPUAffinity=1 2
#JoinControllers=cpu,cpuacct net_cls,net_prio
#RuntimeWatchdogSec=0
#ShutdownWatchdogSec=10min
#CapabilityBoundingSet=
#NoNewPrivileges=no
#SystemCallArchitectures=
#TimerSlackNSec=
#DefaultTimerAccuracySec=1min
#DefaultStandardOutput=journal
#DefaultStandardError=inherit
#DefaultTimeoutStartSec=90s
#DefaultTimeoutStopSec=90s
#DefaultRestartSec=100ms
#DefaultStartLimitIntervalSec=10s
#DefaultStartLimitBurst=5
#DefaultEnvironment=
#DefaultCPUAccounting=no
#DefaultIOAccounting=no
#DefaultIPAccounting=no
#DefaultBlockIOAccounting=no
#DefaultMemoryAccounting=yes
#DefaultTasksAccounting=yes
#DefaultTasksMax=80%
#DefaultLimitCPU=
#DefaultLimitFSIZE=
#DefaultLimitDATA=
#DefaultLimitSTACK=
#DefaultLimitCORE=
#DefaultLimitRSS=
#DefaultLimitNOFILE=
#DefaultLimitAS=
#DefaultLimitNPROC=
#DefaultLimitMEMLOCK=
#DefaultLimitLOCKS=
#DefaultLimitSIGPENDING=
#DefaultLimitMSGQUEUE=
#DefaultLimitNICE=
#DefaultLimitRTPRIO=
#DefaultLimitRTTIME=
#IPAddressAllow=
#IPAddressDeny=


[root@localhost ~]# rpm-ostree initramfs --enable --arg="-I" --arg="/etc/systemd/system.conf /etc/systemd/system.conf.d/setAffinity.conf"
Checking out tree a33130b... done
Enabled rpm-md repositories:
Importing rpm-md... done
Resolving dependencies... done
Applying 4 overrides and 3 overlays
Processing packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Generating initramfs... done
Writing OSTree commit... done
Staging deployment... done
Freed: 109.4 MB (pkgcache branches: 0)
Initramfs regeneration is now: enabled
[root@localhost ~]# sudo rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
  ostree://a33130bb69d2edeb5a255b9059d2bfc975219ecd735fb12fa6ec0c7838bf02bd
                   Version: 44.81.202003051130-0 (2020-03-05T11:35:49Z)
                      Diff: 4 removed, 3 added
       RemovedBasePackages: kernel-core kernel-modules kernel kernel-modules-extra 4.18.0-147.5.1.el8_1
             LocalPackages: kernel-rt-modules-4.18.0-147.5.1.rt24.98.el8_1.x86_64 kernel-rt-core-4.18.0-147.5.1.rt24.98.el8_1.x86_64
                            kernel-rt-modules-extra-4.18.0-147.5.1.rt24.98.el8_1.x86_64
                 Initramfs: -I '/etc/systemd/system.conf /etc/systemd/system.conf.d/setAffinity.conf' 

● ostree://a33130bb69d2edeb5a255b9059d2bfc975219ecd735fb12fa6ec0c7838bf02bd
                   Version: 44.81.202003051130-0 (2020-03-05T11:35:49Z)
[root@localhost ~]# systemctl reboot
[root@localhost ~]# Connection to 192.168.122.249 closed by remote host.

== RECONNECT ==
[root@localhost ~]# cd /boot/ostree/
[root@localhost ostree]# lsinitrd /boot/ostree/rhcos-261af2cc637b4270c88dfa8e3611b226b77528a50ada141d281a943c09c7ddda/initramfs-4.18.0-147.5.1.rt24.98.el8_1.x86_64.img | grep setAffinity.conf
Arguments: --reproducible --gzip -v --add 'ostree' --tmpdir '/tmp/dracut' -f -I '/etc/systemd/system.conf /etc/systemd/system.conf.d/setAffinity.conf' --kver '4.18.0-147.5.1.rt24.98.el8_1.x86_64'
-rw-r--r--   1 root     root            2 Jan  1  1970 etc/systemd/system.conf.d/setAffinity.conf



== IN INITRAMFS ==
Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.


switch_root:/# ls
bin  dracut-state.sh  etc   kernel  lib64  root  sbin	   sys	    tmp  var
dev  early_cpio       init  lib     proc   run	 shutdown  sysroot  usr
switch_root:/# cd etc/systemd/      
switch_root:/etc/systemd# ls
journald.conf  journald.conf.d	system	system.conf  system.conf.d
switch_root:/etc/systemd# cat system.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See systemd-system.conf(5) for details.

[Manager]
#LogLevel=info
#LogTarget=journal-or-kmsg
#LogColor=yes
#LogLocation=no
#DumpCore=yes
#ShowStatus=yes
#CrashChangeVT=no
#CrashShell=no
#CrashReboot=no
#CtrlAltDelBurstAction=reboot-force
#CPUAffinity=1 2
#JoinControllers=cpu,cpuacct net_cls,net_prio
#RuntimeWatchdogSec=0
#ShutdownWatchdogSec=10min
#CapabilityBoundingSet=
#NoNewPrivileges=no
#SystemCallArchitectures=
#TimerSlackNSec=
#DefaultTimerAccuracySec=1min
#DefaultStandardOutput=journal
#DefaultStandardError=inherit
#DefaultTimeoutStartSec=90s
#DefaultTimeoutStopSec=90s
#DefaultRestartSec=100ms
#DefaultStartLimitIntervalSec=10s
#DefaultStartLimitBurst=5
#DefaultEnvironment=
#DefaultCPUAccounting=no
#DefaultIOAccounting=no
#DefaultIPAccounting=no
#DefaultBlockIOAccounting=no
#DefaultMemoryAccounting=yes
#DefaultTasksAccounting=yes
#DefaultTasksMax=80%
#DefaultLimitCPU=
#DefaultLimitFSIZE=
#DefaultLimitDATA=
#DefaultLimitSTACK=
#DefaultLimitCORE=
#DefaultLimitRSS=
#DefaultLimitNOFILE=
#DefaultLimitAS=
#DefaultLimitNPROC=
#DefaultLimitMEMLOCK=
#DefaultLimitLOCKS=
#DefaultLimitSIGPENDING=
#DefaultLimitMSGQUEUE=
#DefaultLimitNICE=
#DefaultLimitRTPRIO=
#DefaultLimitRTTIME=
#IPAddressAllow=
#IPAddressDeny=
switch_root:/etc/systemd# cat system
system/        system.conf    system.conf.d/ 
switch_root:/etc/systemd# cat system.conf.d/setAffinity.conf 
#
switch_root:/etc/systemd#

Comment 9 Colin Walters 2020-03-09 19:16:50 UTC
Recent systemd now includes a kernel cmdline option for this: https://github.com/systemd/systemd/pull/14154

We should at some point get that backported to RHEL and teach the performance operator to use it.

Comment 10 Martin Sivák 2020-03-10 10:06:09 UTC
Colin, that is great news. Any estimate on when this will land into RHCOS?

Comment 12 Micah Abbott 2020-03-12 12:58:21 UTC
The fixed package landed in RHCOS 44.81.202003062006-0; all subsequent builds should have the fix.

Comment 13 Colin Walters 2020-03-12 13:09:45 UTC
@Martin please follow https://bugzilla.redhat.com/show_bug.cgi?id=1812894

Remember RHCOS *is* RHEL - so systemd patches go there.

Comment 16 Micah Abbott 2020-03-12 15:06:57 UTC
Moving to VERIFIED based on comment #7

Comment 18 errata-xmlrpc 2020-05-04 11:39:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.