Description of problem: Installing / upgrading kernel in kickstart post installation script creates a boot entry without root set and with the wrong kernel cmdline arguments such that the entry boots into the emergency shell Version-Release number of selected component (if applicable): s390utils-core-2.22.0-2.el9.s390x How reproducible: always Steps to Reproduce: 1. Add another kernel / installation repo in %post 2. dnf install -y kernel in %post Actual results: root= is not set, kernel cmdline arguments passed to new kernel come from installation (/proc/cmdline) Machine reboots to emergency shell Expected results: root and cmdline arguments are set correctly and machine boots new kernel Additional info: This was fixed incidentally on all others arches that use grub2 as a bootloader: https://bugzilla.redhat.com/show_bug.cgi?id=1940288 rharwood has already opened a PR which should fix this issue: https://src.fedoraproject.org/rpms/s390utils/pull-request/5 This is a high priority bug which customers are waiting for and will need to be quickly backported to RHEL 9.0
I tested Robbie's scratch build with the PR, and it works as intended. https://beaker.engineering.redhat.com/recipes/13248677 from beaker xml: <ks_appends> <ks_append><![CDATA[ %post curl -o /etc/yum.repos.d/s390utils.repo http://brew-task-repos.usersys.redhat.com/repos/scratch/rharwood/s390utils/2.25.0/999.el9/s390utils-2.25.0-999.el9-scratch.repo dnf install -y s390utils curl -o /etc/yum.repos.d/kernel.repo http://brew-task-repos.usersys.redhat.com/repos/official/kernel/5.14.0/235.el9/kernel-5.14.0-235.el9.repo dnf update -y kernel %end ]]></ks_append> </ks_appends> [root@rtt4 ~]# grubby --info ALL index=0 kernel="/boot/vmlinuz-5.14.0-235.el9.s390x" args="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0" root="/dev/mapper/rhel_rtt4-root" initrd="/boot/initramfs-5.14.0-235.el9.s390x.img" title="(5.14.0-235.el9.s390x) " id="3276f4c6bea34a489ceee2954dde2036-5.14.0-235.el9.s390x" index=1 kernel="/boot/vmlinuz-5.14.0-230.el9.s390x" args="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0" root="/dev/mapper/rhel_rtt4-root" initrd="/boot/initramfs-5.14.0-230.el9.s390x.img" title="(5.14.0-230.el9.s390x) " id="3276f4c6bea34a489ceee2954dde2036-5.14.0-230.el9.s390x" index=2 kernel="/boot/vmlinuz-0-rescue-3276f4c6bea34a489ceee2954dde2036" args="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0" root="/dev/mapper/rhel_rtt4-root" initrd="/boot/initramfs-0-rescue-3276f4c6bea34a489ceee2954dde2036.img" title="(0-rescue-3276f4c6bea34a489ceee2954dde2036) " id="3276f4c6bea34a489ceee2954dde2036-0-rescue" [root@rtt4 ~]# uname -r 5.14.0-235.el9.s390x [root@rtt4 ~]# cat /etc/kernel/cmdline root=/dev/mapper/rhel_rtt4-root crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0
Yes, you need to use s390utils-core-2.22.0-2.el9.s390x to reproduce this. This bug is contingent on doing the kernel update during the installation. Kernel installation scripts read their arguments from /proc/cmdline, which-- only during installation-- contains arguments passed to the installer: https://bugzilla.redhat.com/show_bug.cgi?id=1940288#c21
I am wondering how the change in the kernel install script will work with the scenario described in bug #1899759 ... The change has been pushed to Fedora, so hopefully tomorrow there will be an ISO to try.
If I inst.addrepo (to s390utils) on the kernel cmdline, there is a failure in kernel post trans scriptlet: https://beaker.engineering.redhat.com/jobs/7451863 from console log of that job: [ 110.173248] anaconda[2552]: Configuring kernel-core.s390x [ 112.202201] anaconda[2552]: The installation was stopped due to an error whic h occurred while running in non-interactive cmdline mode. Since there cannot be any questions in cmdline mode, edit your kickstart file and retry installation. [ 112.202250] anaconda[2552]: The exact error message is: [ 112.202270] anaconda[2552]: Non interactive installation failed: DNF error: E rror in POSTTRANS scriptlet in rpm package kernel-core. [ 112.202288] anaconda[2552]: The installer will now terminate. But if I add the repo in the post installation script, the machine boots: https://beaker.engineering.redhat.com/jobs/7452156 [root@rtt4 ~]# grubby --info ALL index=0 kernel="/boot/vmlinuz-5.14.0-235.el9.s390x" args="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0" root="/dev/mapper/rhel_rtt4-root" initrd="/boot/initramfs-5.14.0-235.el9.s390x.img" title="(5.14.0-235.el9.s390x) " id="61dc6f783d574d01b7fda64c66ee4fe6-5.14.0-235.el9.s390x" index=1 kernel="/boot/vmlinuz-5.14.0-230.el9.s390x" args="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0" root="/dev/mapper/rhel_rtt4-root" initrd="/boot/initramfs-5.14.0-230.el9.s390x.img" title="(5.14.0-230.el9.s390x) " id="61dc6f783d574d01b7fda64c66ee4fe6-5.14.0-230.el9.s390x" index=2 kernel="/boot/vmlinuz-0-rescue-61dc6f783d574d01b7fda64c66ee4fe6" args="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0" root="/dev/mapper/rhel_rtt4-root" initrd="/boot/initramfs-0-rescue-61dc6f783d574d01b7fda64c66ee4fe6.img" title="(0-rescue-61dc6f783d574d01b7fda64c66ee4fe6) " id="61dc6f783d574d01b7fda64c66ee4fe6-0-rescue" [root@rtt4 ~]# uname -r 5.14.0-235.el9.s390x [root@rtt4 ~]# rpm -qa s390utils\* s390utils-core-2.25.0-2.el9.s390x s390utils-iucvterm-2.25.0-2.el9.s390x s390utils-mon_statd-2.25.0-2.el9.s390x s390utils-cpuplugd-2.25.0-2.el9.s390x s390utils-osasnmpd-2.25.0-2.el9.s390x s390utils-base-2.25.0-2.el9.s390x s390utils-ziomon-2.25.0-2.el9.s390x s390utils-2.25.0-2.el9.s390x [root@rtt4 ~]# cat /etc/kernel/cmdline root=/dev/mapper/rhel_rtt4-root crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.4018 rd.dasd=0.0.4118 rd.dasd=0.0.4218 rd.dasd=0.0.4318 rd.dasd=0.0.4019 rd.dasd=0.0.4119 rd.dasd=0.0.4219 rd.dasd=0.0.4319 rd.lvm.lv=rhel_rtt4/root rd.lvm.lv=rhel_rtt4/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0
I wonder if the patched script works for installing the first kernel when doing the installation ...
so yes, the 20-zipl-kernel.install script is broken for the initial kernel installation :-( Jan 23 08:06:29 fedora anaconda[1831]: anaconda: modules.payloads.payload.dnf.transaction_progress: Configuring (running scriptlet for): kernel-core-6.2.0-0.rc4.20230120git> Jan 23 08:06:29 fedora anaconda[2519]: dnf: awk: fatal: cannot open file `/boot/loader/entries/*.conf' for reading: No such file or directory Could not determine the kernel command line parameters. Please specify the kernel command line in /etc/kernel/cmdline! warning: %posttrans(kernel-core-6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.s390x) scriptlet failed, exit status 1 Jan 23 08:06:29 fedora anaconda[2519]: dnf: Error in POSTTRANS scriptlet in rpm package kernel-core
I think we should move the creation of /etc/kernel/cmdline (if it doesn't exist yet) to https://src.fedoraproject.org/rpms/s390utils/blob/rawhide/f/20-zipl-kernel.install#_111 after the BLS entry will be created. And keep the previous logic of determining the BOOT_OPTIONS. Robbie, what do you think?
and it won't work as well ... using the following patch instead diff --git a/20-zipl-kernel.install b/20-zipl-kernel.install index 4fadd1d..365330b 100755 --- a/20-zipl-kernel.install +++ b/20-zipl-kernel.install @@ -101,6 +101,17 @@ case "$COMMAND" in "${BOOT_OPTIONS[*]}" >"${BLS_TARGET}" fi + if [[ ! -f /etc/kernel/cmdline ]]; then + last='' + for entry in "${BLS_DIR}"/*.conf; do + last="$entry" + done + if [[ ! -z "$last" ]]; then + args="$(awk '/options/{$1="";$0=$0;$1=$1;print}' $last)" + echo "$args" > /etc/kernel/cmdline + fi + fi + if [[ "$KERNEL_VERSION" == *\+* ]] && [ "x$DEFAULTDEBUG" != "xyes" ]; then UPDATEDEFAULT="no" fi I am getting the following after a successful instalation [anaconda root@fedora /]# cat /etc/kernel/cmdline inst.ks=file:/vm.ks inst.sshd inst.addrepo=s390utils,http://junkyard.khw1.lab.eng.bos.redhat.com/f38/s390x/repo/ inst.repo=https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230122.n.0/compose/Everything/s390x/os/ [anaconda root@fedora entries]# cat 975fff50b3df4c7b87f8b949bd7dac61-6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.s390x.conf title (6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.s390x) version 6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.s390x linux /boot/vmlinuz-6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.s390x initrd /boot/initramfs-6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.s390x.img options root=UUID=04808e15-2c7d-4c1c-829f-356f3b3a248f id -20230123123011-6.2.0-0.rc4.20230120gitd368967cb103.35.fc38.s390x grub_users $grub_users grub_arg --unrestricted grub_class kernel which means the initial BLS snippet contain the "garbage" from /proc/cmdline, but it is fixed later. I believe writing of the initial (install time) /etc/kernel/cmdline must happen at the same time (or later) when the "options" in the BLS snipped get converted to the final/working state. And it might be somewhere in anaconda.
I think https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/modules/storage/bootloader/zipl.py#L110 is the place where /etc/kernel/cmdline should be created. But only from the options for the production kernel IMO. The question is how it is handled by other platforms? And another question is how "/etc/kernel/cmdline" is defined, what component is responsible for providing it?
I think creating it in anaconda is a good way to go. On other platforms, grub spits it out during mkconfig, and there's some logic to run that when needed from kernel-install. Because mkconfig isn't run after install, grubby can also produce it when needed. Since existing systems would need an upgrade path anyway, I didn't want to involve anaconda... but if there's no other way to get the initial correct arguments on s390x, then I guess we have to.
addendum: grubby will also update this file (and can create it if it doesn't exist) but only when args are added/removed.
I think writing /etc/kernel/cmdline initially by anaconda should be done on all platforms, because it sets the initial cmdline. In the grub2 case, there is write_defaults() in https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/modules/storage/bootloader/grub2.py#L253 and it could create /etc/kernel/cmdline. Then instead of having GRUB_CMDLINE_LINUX="arg1 arg2 ..." it could do GRUB_CMDLINE_LINUX=$(cat /etc/kernel/cmdline) like it does for GRUB_DISTRIBUTOR. Then we can say - /etc/kernel/cmdline is the default place for setting kernel boot parameters - it is written initially by the installer - (and probably more, but I can't recall now :-)) Marta, could you bring it to the installer team? I can draft an initial PR (for Rawhide, it will need some time in Fedora) for the change, but it shouldn't be difficult.
Of course. Thank you, Dan.
I have brought this to the attention of anaconda folks and am reassigning as per comment#19