Description of problem:

Based on https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9-beta/html/managing_monitoring_and_updating_the_kernel/configuring-kernel-command-line-parameters_managing-monitoring-and-updating-the-kernel, the correct way of updating the kernel command line parameters for all kernels is using

   grubby --update-kernel=ALL --args="<NEW_PARAMETER>"

There is an implied assumption that this will affect not just the currently installed kernels but will set the system so that upgraded kernels get the new set of kernel command line parameters as well. That's how things work on RHEL 8 x86_64, even if it is currently broken on RHEL 9: regression bug 2021193.

However, on s390x which uses zipl, grubby does not seem to do that even on RHEL 8. Therefore, on s390x there is currently no documented way for modifying the kernel command line parameters.

Version-Release number of selected component (if applicable):

grubby-8.40-54.el9.s390x

How reproducible:

Deterministic.

Steps to Reproduce:
1. grep -r jezek= /boot /etc || :
2. grubby --update-kernel=ALL --args=jezek=8
3. zipl
4. grep -r jezek= /boot /etc
5. dnf reinstall -y kernel-core
6. grep -r jezek= /boot /etc
7. reboot
8. grep jezek=8 /proc/cmdline

Actual results:

1. no output

2. no output

3.
Using config file '/etc/zipl.conf'
Using BLS config file '/boot/loader/entries/52b83a83975040039e2c15d5a7add464-5.14.0-11.el9.s390x.conf'
Using BLS config file '/boot/loader/entries/52b83a83975040039e2c15d5a7add464-0-rescue.conf'
Building bootmap in '/boot'
Building menu 'zipl-automatic-menu'
Adding #1: IPL section '(5.14.0-11.el9.s390x)' (default)
Adding #2: IPL section '(0-rescue-52b83a83975040039e2c15d5a7add464)'
Preparing boot device: dasda (0120).
Done.

4.
/boot/loader/entries/52b83a83975040039e2c15d5a7add464-5.14.0-11.el9.s390x.conf:options root=/dev/mapper/rhel_ibm--z--502-root crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.0120 rd.lvm.lv=rhel_ibm-z-502/root rd.lvm.lv=rhel_ibm-z-502/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0600,0.0.0601,0.0.0602,layer2=1,portno=0 jezek=8
/boot/loader/entries/52b83a83975040039e2c15d5a7add464-0-rescue.conf:options root=/dev/mapper/rhel_ibm--z--502-root crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.0120 rd.lvm.lv=rhel_ibm-z-502/root rd.lvm.lv=rhel_ibm-z-502/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0600,0.0.0601,0.0.0602,layer2=1,portno=0 jezek=8
grep: /boot/bootmap: binary file matches

5.
[...]
Running transaction
  Preparing        :                                                        1/1 
  Reinstalling     : kernel-core-5.14.0-11.el9.s390x                        1/2 
  Running scriptlet: kernel-core-5.14.0-11.el9.s390x                        1/2 
  Running scriptlet: kernel-core-5.14.0-11.el9.s390x                        2/2 
  Cleanup          : kernel-core-5.14.0-11.el9.s390x                        2/2 
  Running scriptlet: kernel-core-5.14.0-11.el9.s390x                        2/2 
dracut-install: ERROR: installing 'vmcp'
dracut: FAILED: /usr/lib/dracut/dracut-install -D /var/tmp/dracut.VQSzDo/initramfs -a chzdev lszdev vmcp

  Verifying        : kernel-core-5.14.0-11.el9.s390x                        1/2 
  Verifying        : kernel-core-5.14.0-11.el9.s390x                        2/2 
Installed products updated.

Reinstalled:
  kernel-core-5.14.0-11.el9.s390x                                               

Complete!

6.
/boot/loader/entries/52b83a83975040039e2c15d5a7add464-0-rescue.conf:options root=/dev/mapper/rhel_ibm--z--502-root crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.0120 rd.lvm.lv=rhel_ibm-z-502/root rd.lvm.lv=rhel_ibm-z-502/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0600,0.0.0601,0.0.0602,layer2=1,portno=0 jezek=8
grep: /boot/bootmap: binary file matches

8. no output

Expected results:

Output 6 the same as output 4, and also likely listing /etc/kernel/cmdline.

Additional info:

On x390x with zipl, the mechanism equivalent to grub's GRUB_CMDLINE_LINUX seems to be to set /etc/kernel/cmdline. However, it is not documented anywhere, so it's hard to say how stable that mechanism is.

I did not use --zipl option to grubby above because I assume that is the default on s390x. But even with that option added, the result is the same.