I tested Robbie's scratch build with the PR, and it works as intended.

https://beaker.engineering.redhat.com/recipes/13248677

from beaker xml:
      <ks_appends>
        <ks_append><![CDATA[
        %post
curl -o /etc/yum.repos.d/s390utils.repo http://brew-task-repos.usersys.redhat.com/repos/scratch/rharwood/s390utils/2.25.0/999.el9/s390utils-2.25.0-999.el9-scratch.repo
dnf install -y s390utils

curl -o /etc/yum.repos.d/kernel.repo http://brew-task-repos.usersys.redhat.com/repos/official/kernel/5.14.0/235.el9/kernel-5.14.0-235.el9.repo
dnf update -y kernel
        %end
      ]]></ks_append>
      </ks_appends>

[root@rtt4 ~]# grubby --info ALL
index=0
kernel="/boot/vmlinuz-5.14.0-235.el9.s390x"
args="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0"
root="/dev/mapper/rhel_rtt4-root"
initrd="/boot/initramfs-5.14.0-235.el9.s390x.img"
title="(5.14.0-235.el9.s390x) "
id="3276f4c6bea34a489ceee2954dde2036-5.14.0-235.el9.s390x"
index=1
kernel="/boot/vmlinuz-5.14.0-230.el9.s390x"
args="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0"
root="/dev/mapper/rhel_rtt4-root"
initrd="/boot/initramfs-5.14.0-230.el9.s390x.img"
title="(5.14.0-230.el9.s390x) "
id="3276f4c6bea34a489ceee2954dde2036-5.14.0-230.el9.s390x"
index=2
kernel="/boot/vmlinuz-0-rescue-3276f4c6bea34a489ceee2954dde2036"
args="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0"
root="/dev/mapper/rhel_rtt4-root"
initrd="/boot/initramfs-0-rescue-3276f4c6bea34a489ceee2954dde2036.img"
title="(0-rescue-3276f4c6bea34a489ceee2954dde2036) "
id="3276f4c6bea34a489ceee2954dde2036-0-rescue"

[root@rtt4 ~]# uname -r
5.14.0-235.el9.s390x

[root@rtt4 ~]# cat /etc/kernel/cmdline 
root=/dev/mapper/rhel_rtt4-root crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0

Yes, you need to use s390utils-core-2.22.0-2.el9.s390x to reproduce this. 

This bug is contingent on doing the kernel update during the installation. Kernel installation scripts read their arguments from /proc/cmdline, which-- only during installation-- contains arguments passed to the installer: https://bugzilla.redhat.com/show_bug.cgi?id=1940288#c21

I am wondering how the change in the kernel install script will work with the scenario described in bug #1899759 ... The change has been pushed to Fedora, so hopefully tomorrow there will be an ISO to try.

If I inst.addrepo (to s390utils) on the kernel cmdline, there is a failure in kernel post trans scriptlet: https://beaker.engineering.redhat.com/jobs/7451863

from console log of that job:
[  110.173248] anaconda[2552]: Configuring kernel-core.s390x                     
[  112.202201] anaconda[2552]: The installation was stopped due to an error whic 
h occurred while running in non-interactive cmdline mode. Since there cannot be  
any questions in cmdline mode, edit your kickstart file and retry installation.  
                                                                                 
[  112.202250] anaconda[2552]: The exact error message is:                       
[  112.202270] anaconda[2552]: Non interactive installation failed: DNF error: E 
rror in POSTTRANS scriptlet in rpm package kernel-core.                          
[  112.202288] anaconda[2552]: The installer will now terminate.  


But if I add the repo in the post installation script, the machine boots: https://beaker.engineering.redhat.com/jobs/7452156

[root@rtt4 ~]# grubby --info ALL
index=0
kernel="/boot/vmlinuz-5.14.0-235.el9.s390x"
args="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0"
root="/dev/mapper/rhel_rtt4-root"
initrd="/boot/initramfs-5.14.0-235.el9.s390x.img"
title="(5.14.0-235.el9.s390x) "
id="61dc6f783d574d01b7fda64c66ee4fe6-5.14.0-235.el9.s390x"
index=1
kernel="/boot/vmlinuz-5.14.0-230.el9.s390x"
args="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0"
root="/dev/mapper/rhel_rtt4-root"
initrd="/boot/initramfs-5.14.0-230.el9.s390x.img"
title="(5.14.0-230.el9.s390x) "
id="61dc6f783d574d01b7fda64c66ee4fe6-5.14.0-230.el9.s390x"
index=2
kernel="/boot/vmlinuz-0-rescue-61dc6f783d574d01b7fda64c66ee4fe6"
args="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0"
root="/dev/mapper/rhel_rtt4-root"
initrd="/boot/initramfs-0-rescue-61dc6f783d574d01b7fda64c66ee4fe6.img"
title="(0-rescue-61dc6f783d574d01b7fda64c66ee4fe6) "
id="61dc6f783d574d01b7fda64c66ee4fe6-0-rescue"
[root@rtt4 ~]# uname -r
5.14.0-235.el9.s390x

[root@rtt4 ~]# rpm -qa s390utils\*
s390utils-core-2.25.0-2.el9.s390x
s390utils-iucvterm-2.25.0-2.el9.s390x
s390utils-mon_statd-2.25.0-2.el9.s390x
s390utils-cpuplugd-2.25.0-2.el9.s390x
s390utils-osasnmpd-2.25.0-2.el9.s390x
s390utils-base-2.25.0-2.el9.s390x
s390utils-ziomon-2.25.0-2.el9.s390x
s390utils-2.25.0-2.el9.s390x

[root@rtt4 ~]# cat /etc/kernel/cmdline 
root=/dev/mapper/rhel_rtt4-root crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0

I wonder if the patched script works for installing the first kernel when doing the installation ...

so yes, the 20-zipl-kernel.install script is broken for the initial kernel installation :-(

Jan 23 08:06:29 fedora anaconda[1831]: anaconda: modules.payloads.payload.dnf.transaction_progress: Configuring (running scriptlet for): kernel-core-6.2.0-0.rc4.20230120git>
Jan 23 08:06:29 fedora anaconda[2519]: dnf: awk: fatal: cannot open file `/boot/loader/entries/*.conf' for reading: No such file or directory
                                       Could not determine the kernel command line parameters.
                                       Please specify the kernel command line in /etc/kernel/cmdline!
                                       warning: %posttrans(kernel-core-6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.s390x) scriptlet failed, exit status 1
Jan 23 08:06:29 fedora anaconda[2519]: dnf: Error in POSTTRANS scriptlet in rpm package kernel-core

I think we should move the creation of /etc/kernel/cmdline (if it doesn't exist yet) to
https://src.fedoraproject.org/rpms/s390utils/blob/rawhide/f/20-zipl-kernel.install#_111
after the BLS entry will be created. And keep the previous logic of determining the BOOT_OPTIONS.

Robbie, what do you think?

and it won't work as well ...

using the following patch instead

diff --git a/20-zipl-kernel.install b/20-zipl-kernel.install
index 4fadd1d..365330b 100755
--- a/20-zipl-kernel.install
+++ b/20-zipl-kernel.install
@@ -101,6 +101,17 @@ case "$COMMAND" in
                     "${BOOT_OPTIONS[*]}" >"${BLS_TARGET}"
             fi
 
+            if [[ ! -f /etc/kernel/cmdline ]]; then
+                last=''
+                for entry in "${BLS_DIR}"/*.conf; do
+                    last="$entry"
+                done
+                if [[ ! -z "$last" ]]; then
+                    args="$(awk '/options/{$1="";$0=$0;$1=$1;print}' $last)"
+                    echo "$args" > /etc/kernel/cmdline
+                fi
+            fi
+
             if [[ "$KERNEL_VERSION" == *\+* ]] && [ "x$DEFAULTDEBUG" != "xyes" ]; then
                 UPDATEDEFAULT="no"
             fi


I am getting the following after a successful instalation

[anaconda root@fedora /]# cat /etc/kernel/cmdline 
inst.ks=file:/vm.ks inst.sshd inst.addrepo=s390utils,http://junkyard.khw1.lab.eng.bos.redhat.com/f38/s390x/repo/ inst.repo=https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230122.n.0/compose/Everything/s390x/os/

[anaconda root@fedora entries]# cat 975fff50b3df4c7b87f8b949bd7dac61-6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.s390x.conf
title  (6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.s390x) 
version 6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.s390x
linux /boot/vmlinuz-6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.s390x
initrd /boot/initramfs-6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.s390x.img
options root=UUID=04808e15-2c7d-4c1c-829f-356f3b3a248f
id -20230123123011-6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.s390x
grub_users $grub_users
grub_arg --unrestricted
grub_class kernel

which means the initial BLS snippet contain the "garbage" from /proc/cmdline, but it is fixed later.

I believe writing of the initial (install time) /etc/kernel/cmdline must happen at the same time (or later) when the "options" in the BLS snipped get converted to the final/working state. And it might be somewhere in anaconda.

I think
https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/modules/storage/bootloader/zipl.py#L110
is the place where /etc/kernel/cmdline should be created. But only from the options for the production kernel IMO.

The question is how it is handled by other platforms? And another question is how "/etc/kernel/cmdline" is defined, what component is responsible for providing it?

I think creating it in anaconda is a good way to go.

On other platforms, grub spits it out during mkconfig, and there's some logic to run that when needed from kernel-install.  Because mkconfig isn't run after install, grubby can also produce it when needed.  Since existing systems would need an upgrade path anyway, I didn't want to involve anaconda... but if there's no other way to get the initial correct arguments on s390x, then I guess we have to.

addendum: grubby will also update this file (and can create it if it doesn't exist) but only when args are added/removed.

I think writing /etc/kernel/cmdline initially by anaconda should be done on all platforms, because it sets the initial cmdline. In the grub2 case, there is write_defaults() in https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/modules/storage/bootloader/grub2.py#L253 and it could create /etc/kernel/cmdline. Then instead of having GRUB_CMDLINE_LINUX="arg1 arg2 ..." it could do GRUB_CMDLINE_LINUX=$(cat /etc/kernel/cmdline) like it does for GRUB_DISTRIBUTOR.

Then we can say
- /etc/kernel/cmdline is the default place for setting kernel boot parameters
- it is written initially by the installer
- (and probably more, but I can't recall now :-))


Marta, could you bring it to the installer team? I can draft an initial PR (for Rawhide, it will need some time in Fedora) for the change, but it shouldn't be difficult.

Of course. Thank you, Dan.

I have brought this to the attention of anaconda folks and am reassigning as per comment#19

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.