Bug 2161740

I tested Robbie's scratch build with the PR, and it works as intended.

https://beaker.engineering.redhat.com/recipes/13248677

from beaker xml:
      <ks_appends>
        <ks_append><![CDATA[
        %post
curl -o /etc/yum.repos.d/s390utils.repo http://brew-task-repos.usersys.redhat.com/repos/scratch/rharwood/s390utils/2.25.0/999.el9/s390utils-2.25.0-999.el9-scratch.repo
dnf install -y s390utils

curl -o /etc/yum.repos.d/kernel.repo http://brew-task-repos.usersys.redhat.com/repos/official/kernel/5.14.0/235.el9/kernel-5.14.0-235.el9.repo
dnf update -y kernel
        %end
      ]]></ks_append>
      </ks_appends>

[root@rtt4 ~]# grubby --info ALL
index=0
kernel="/boot/vmlinuz-5.14.0-235.el9.s390x"
args="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0"
root="/dev/mapper/rhel_rtt4-root"
initrd="/boot/initramfs-5.14.0-235.el9.s390x.img"
title="(5.14.0-235.el9.s390x) "
id="3276f4c6bea34a489ceee2954dde2036-5.14.0-235.el9.s390x"
index=1
kernel="/boot/vmlinuz-5.14.0-230.el9.s390x"
args="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0"
root="/dev/mapper/rhel_rtt4-root"
initrd="/boot/initramfs-5.14.0-230.el9.s390x.img"
title="(5.14.0-230.el9.s390x) "
id="3276f4c6bea34a489ceee2954dde2036-5.14.0-230.el9.s390x"
index=2
kernel="/boot/vmlinuz-0-rescue-3276f4c6bea34a489ceee2954dde2036"
args="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0"
root="/dev/mapper/rhel_rtt4-root"
initrd="/boot/initramfs-0-rescue-3276f4c6bea34a489ceee2954dde2036.img"
title="(0-rescue-3276f4c6bea34a489ceee2954dde2036) "
id="3276f4c6bea34a489ceee2954dde2036-0-rescue"

[root@rtt4 ~]# uname -r
5.14.0-235.el9.s390x

[root@rtt4 ~]# cat /etc/kernel/cmdline 
root=/dev/mapper/rhel_rtt4-root crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0

Yes, you need to use s390utils-core-2.22.0-2.el9.s390x to reproduce this. 

This bug is contingent on doing the kernel update during the installation. Kernel installation scripts read their arguments from /proc/cmdline, which-- only during installation-- contains arguments passed to the installer: https://bugzilla.redhat.com/show_bug.cgi?id=1940288#c21

I am wondering how the change in the kernel install script will work with the scenario described in bug #1899759 ... The change has been pushed to Fedora, so hopefully tomorrow there will be an ISO to try.

If I inst.addrepo (to s390utils) on the kernel cmdline, there is a failure in kernel post trans scriptlet: https://beaker.engineering.redhat.com/jobs/7451863

from console log of that job:
[  110.173248] anaconda[2552]: Configuring kernel-core.s390x                     
[  112.202201] anaconda[2552]: The installation was stopped due to an error whic 
h occurred while running in non-interactive cmdline mode. Since there cannot be  
any questions in cmdline mode, edit your kickstart file and retry installation.  
                                                                                 
[  112.202250] anaconda[2552]: The exact error message is:                       
[  112.202270] anaconda[2552]: Non interactive installation failed: DNF error: E 
rror in POSTTRANS scriptlet in rpm package kernel-core.                          
[  112.202288] anaconda[2552]: The installer will now terminate.  


But if I add the repo in the post installation script, the machine boots: https://beaker.engineering.redhat.com/jobs/7452156

[root@rtt4 ~]# grubby --info ALL
index=0
kernel="/boot/vmlinuz-5.14.0-235.el9.s390x"
args="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0"
root="/dev/mapper/rhel_rtt4-root"
initrd="/boot/initramfs-5.14.0-235.el9.s390x.img"
title="(5.14.0-235.el9.s390x) "
id="61dc6f783d574d01b7fda64c66ee4fe6-5.14.0-235.el9.s390x"
index=1
kernel="/boot/vmlinuz-5.14.0-230.el9.s390x"
args="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0"
root="/dev/mapper/rhel_rtt4-root"
initrd="/boot/initramfs-5.14.0-230.el9.s390x.img"
title="(5.14.0-230.el9.s390x) "
id="61dc6f783d574d01b7fda64c66ee4fe6-5.14.0-230.el9.s390x"
index=2
kernel="/boot/vmlinuz-0-rescue-61dc6f783d574d01b7fda64c66ee4fe6"
args="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0"
root="/dev/mapper/rhel_rtt4-root"
initrd="/boot/initramfs-0-rescue-61dc6f783d574d01b7fda64c66ee4fe6.img"
title="(0-rescue-61dc6f783d574d01b7fda64c66ee4fe6) "
id="61dc6f783d574d01b7fda64c66ee4fe6-0-rescue"
[root@rtt4 ~]# uname -r
5.14.0-235.el9.s390x

[root@rtt4 ~]# rpm -qa s390utils\*
s390utils-core-2.25.0-2.el9.s390x
s390utils-iucvterm-2.25.0-2.el9.s390x
s390utils-mon_statd-2.25.0-2.el9.s390x
s390utils-cpuplugd-2.25.0-2.el9.s390x
s390utils-osasnmpd-2.25.0-2.el9.s390x
s390utils-base-2.25.0-2.el9.s390x
s390utils-ziomon-2.25.0-2.el9.s390x
s390utils-2.25.0-2.el9.s390x

[root@rtt4 ~]# cat /etc/kernel/cmdline 
root=/dev/mapper/rhel_rtt4-root crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0

I wonder if the patched script works for installing the first kernel when doing the installation ...

so yes, the 20-zipl-kernel.install script is broken for the initial kernel installation :-(

Jan 23 08:06:29 fedora anaconda[1831]: anaconda: modules.payloads.payload.dnf.transaction_progress: Configuring (running scriptlet for): kernel-core-6.2.0-0.rc4.20230120git>
Jan 23 08:06:29 fedora anaconda[2519]: dnf: awk: fatal: cannot open file `/boot/loader/entries/*.conf' for reading: No such file or directory
                                       Could not determine the kernel command line parameters.
                                       Please specify the kernel command line in /etc/kernel/cmdline!
                                       warning: %posttrans(kernel-core-6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.s390x) scriptlet failed, exit status 1
Jan 23 08:06:29 fedora anaconda[2519]: dnf: Error in POSTTRANS scriptlet in rpm package kernel-core

I think we should move the creation of /etc/kernel/cmdline (if it doesn't exist yet) to
https://src.fedoraproject.org/rpms/s390utils/blob/rawhide/f/20-zipl-kernel.install#_111
after the BLS entry will be created. And keep the previous logic of determining the BOOT_OPTIONS.

Robbie, what do you think?

and it won't work as well ...

using the following patch instead

diff --git a/20-zipl-kernel.install b/20-zipl-kernel.install
index 4fadd1d..365330b 100755
--- a/20-zipl-kernel.install
+++ b/20-zipl-kernel.install
@@ -101,6 +101,17 @@ case "$COMMAND" in
                     "${BOOT_OPTIONS[*]}" >"${BLS_TARGET}"
             fi
 
+            if [[ ! -f /etc/kernel/cmdline ]]; then
+                last=''
+                for entry in "${BLS_DIR}"/*.conf; do
+                    last="$entry"
+                done
+                if [[ ! -z "$last" ]]; then
+                    args="$(awk '/options/{$1="";$0=$0;$1=$1;print}' $last)"
+                    echo "$args" > /etc/kernel/cmdline
+                fi
+            fi
+
             if [[ "$KERNEL_VERSION" == *\+* ]] && [ "x$DEFAULTDEBUG" != "xyes" ]; then
                 UPDATEDEFAULT="no"
             fi


I am getting the following after a successful instalation

[anaconda root@fedora /]# cat /etc/kernel/cmdline 
inst.ks=file:/vm.ks inst.sshd inst.addrepo=s390utils,http://junkyard.khw1.lab.eng.bos.redhat.com/f38/s390x/repo/ inst.repo=https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230122.n.0/compose/Everything/s390x/os/

[anaconda root@fedora entries]# cat 975fff50b3df4c7b87f8b949bd7dac61-6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.s390x.conf
title  (6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.s390x) 
version 6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.s390x
linux /boot/vmlinuz-6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.s390x
initrd /boot/initramfs-6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.s390x.img
options root=UUID=04808e15-2c7d-4c1c-829f-356f3b3a248f
id -20230123123011-6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.s390x
grub_users $grub_users
grub_arg --unrestricted
grub_class kernel

which means the initial BLS snippet contain the "garbage" from /proc/cmdline, but it is fixed later.

I believe writing of the initial (install time) /etc/kernel/cmdline must happen at the same time (or later) when the "options" in the BLS snipped get converted to the final/working state. And it might be somewhere in anaconda.

I think
https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/modules/storage/bootloader/zipl.py#L110
is the place where /etc/kernel/cmdline should be created. But only from the options for the production kernel IMO.

The question is how it is handled by other platforms? And another question is how "/etc/kernel/cmdline" is defined, what component is responsible for providing it?

I think creating it in anaconda is a good way to go.

On other platforms, grub spits it out during mkconfig, and there's some logic to run that when needed from kernel-install.  Because mkconfig isn't run after install, grubby can also produce it when needed.  Since existing systems would need an upgrade path anyway, I didn't want to involve anaconda... but if there's no other way to get the initial correct arguments on s390x, then I guess we have to.

addendum: grubby will also update this file (and can create it if it doesn't exist) but only when args are added/removed.

I think writing /etc/kernel/cmdline initially by anaconda should be done on all platforms, because it sets the initial cmdline. In the grub2 case, there is write_defaults() in https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/modules/storage/bootloader/grub2.py#L253 and it could create /etc/kernel/cmdline. Then instead of having GRUB_CMDLINE_LINUX="arg1 arg2 ..." it could do GRUB_CMDLINE_LINUX=$(cat /etc/kernel/cmdline) like it does for GRUB_DISTRIBUTOR.

Then we can say
- /etc/kernel/cmdline is the default place for setting kernel boot parameters
- it is written initially by the installer
- (and probably more, but I can't recall now :-))


Marta, could you bring it to the installer team? I can draft an initial PR (for Rawhide, it will need some time in Fedora) for the change, but it shouldn't be difficult.

Of course. Thank you, Dan.

I have brought this to the attention of anaconda folks and am reassigning as per comment#19