(In reply to Marius Vollmer from comment #0)
> Description of problem:
> 
> It used to be so that new kernel command line arguments added with grubby
> were picked up by new boot loader entries added by kernel-install. This no
> longer happens.
> 
> This might be a bug in grubby, or in kernel-install, or in how we use them,
> of course.
> 
> The bigger picture is the feature of Cockpit that allows the user to
> activate certain hardware specific mitigations.  We let people add "nosmt"
> to the kernel command line, and we expect "nosmt" to be in place also after
> upgrading the kernel.
> 
> Version-Release number of selected component (if applicable):
> rhel-guest-image-9.0-1582.x86_64.qcow2 and updated from RHEL-NIGHTLY as of
> 2021-06-07.
> 
> How reproducible:
> Always
> 
> Steps to Reproduce:
> 1. grubby --args="nosmt" --update-kernel=ALL
> 2. shutdown -r now
> 3. touch /boot/vmlinuz-42.0.0; mkdir -p /lib/modules/42.0.0/
> 4. kernel-install add 42.0.0 /boot/vmlinuz-42.0.0
> 
> Actual results:
> The new BLS entry in /boot/load/entries does not contain "nosmt".
> 
> Expected results:
> The new BLS entry in /boot/load/entries does contain "nosmt".  This used to
> be the case in older nightlies.
> 
> 
> Additional info:
> 
> This change in behavior is caused by the presence of /etc/kernel/cmdline.  
> 
> Earlier, /etc/kernel/cmdline did not exist and kernel-install would copy
> /proc/cmdline into the new BLS entry.  At the time our tests run
> kernel-install, it does contain "nosmt".
> 
> Now, /etc/kernel/cmdline does exist but it si not affected by grubby or the
> reboot.
> 
> Cockpit could add "nosmt" also to /etc/kernel/cmdline in addition to running
> grubby. Would that be the right thing to do?

I think the correct thing would be for grubby to learn about /etc/kernel/cmdline
and update that file if is present. Since this has precedence over /proc/cmdline.

I was quite surprised about /etc/kernel/cmdline -- after /etc/default/grub, /boot/grub2/grubenv (grub2-editenv), BLS, and rpm-ostree kargs this is now the fifth place to check for/change kernel arguments. Is that documented somewhere?

cockpit's (and administrators') life/script [1] is complicated enough as it is, so fixing that in grubby would be much appreciated (rather than manually editing it). Thank you!

[1] https://github.com/cockpit-project/cockpit/blob/master/pkg/systemd/kernelopt.sh

(In reply to Javier Martinez Canillas from comment #1)

> I think the correct thing would be for grubby to learn about
> /etc/kernel/cmdline
> and update that file if is present. Since this has precedence over
> /proc/cmdline.

That would be appreciated.

One can also argue that this whole scenario only worked by accident earlier.  If there would be no reboot between running grubby and running kernel-install, the "nosmt" option wouldn't be in the new BLS entry ever.  If grubby writes options also into /etc/kernel/cmdline, then the reboot is optional.

*** Bug 2007297 has been marked as a duplicate of this bug. ***

*** Bug 2021193 has been marked as a duplicate of this bug. ***

*** Bug 2078397 has been marked as a duplicate of this bug. ***

I reported bug 2078397 yesterday, but wasn't aware of this duplicate. This is really not a feature, but a regression. It already got fixed in RHEL 8 in bug 1690765, works fine in Fedora, and somehow broke again in RHEL 9.

Note that grubby-bls is the same on Fedora and RHEL 9 right now, so either that's incorrect or something else is in play here.

grubby has never been intended to apply arguments to future installed kernels.  Just because it happened to work does not mean it was supposed to work, or even expected.  Nowhere in our documentation (man pages, etc.) are future kernels mentioned.  Right now it looks like it works because it's picking up the current kernel's arguments for new installed kernels - arguably a bug on its own - so if one reboots between the grubby invocation and the new kernel install, it'll pick them up.

(Doesn't mean that's how it *should* work, of course - still thinking about that.)

Then I still wonder what *is* the official method now, so that we can add it to the growing pile of hacks and workarounds [1]. /etc/default/grub has been the place for many years and in many distros, then it got broken [2]. Then we got told "use grubby", even in official security docs [3], then broke that [4], fixed it again after half a year, and now broke it again (this bug). There's also grub2-editenv with $kernelopts, but that does not really work either, as (1) our BLS scripts don't consistently use that, and it is also fragile [5].

I'm sorry, but this is simply unacceptable. There just must be *some* supported and regularly tested solution to set a kernel parameter that doesn't break every few months. Silently dropping "nosmt" or other important arguments is hard to notice if you are not looking, and can have dire consequences.

Please point me to some documentation how kernel parameters should be set in Fedora and RHEL , and I'll be happy to implement it and point users to it who ask. The Fedora CoreOS docs [6] are a good example for that.

Thanks!

[1] https://github.com/cockpit-project/cockpit/blob/rhel-9.0/pkg/systemd/kernelopt.sh
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1654322 and various duplicates
[3] https://access.redhat.com/solutions/rhel-smt
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1690765
[5] https://bugzilla.redhat.com/show_bug.cgi?id=1785460 https://bugzilla.redhat.com/show_bug.cgi?id=2036520 https://bugzilla.redhat.com/show_bug.cgi?id=1669252
[6] https://docs.fedoraproject.org/en-US/fedora-coreos/kernel-args/

(In reply to Robbie Harwood from comment #11)
> 
> grubby has never been intended to apply arguments to future installed kernels.

Robbie, I agree with Martin here.

It does not really matter what the original grubby intent was. Once it started to be recommended by the bootloader team as *the* way of managing the kernel parameters (bug 1926715 comment 3), it implicitly inherited the functional assumption of the previous mechanisms. It was also explicitly stated in that bugzilla: "and make the new installed kernel use the same kernel parameter too".

And frankly -- setting the kernel command-line parameters just for the current kernel with no guarantees for persistence of those parameters across kernel updates brings very little value so it it weird that you'd suggest that's the whole scope of functionality that bootloader team's recommended tooling guarantees.

I agree with Martin and Jan. This affects also in-place upgrades between RHEL systems. If I am counting right, 3+ SSTs (kernel sst included) are now affected by that and waiting for the resolution as nobody know how to correctly handle this problem as everyone expected (expect!) this should be the way and then everyone is surprised it's not.

Btw, if any different way will become to be expected solution, then it should be pretty well communicated & announced, as a new way will be literally surprise for most people. But I really hope the solution will be the fix in grubby and this will be well tested in future for to prevent regressions.

I literally said that I'm still thinking about what the right way to solve the problem is.  You all need to calm down, a *lot*.

If you have your comment 12 in mind, I read it as a fix/clarification to your previous comment

  Just because it happened to work does not mean it was supposed to work.

which suggested that even the current limited functionality is somehow not desired.

So as long as we are exchanging recommendations, please try to be clearer in your comments.

> which suggested that even the current limited functionality is somehow not desired.

No, that message says exactly what I meant to convey: it's not a regression because it was never an intended use case.  Doesn't mean we don't want to support the use case, just that we didn't plan on it initially.

(In reply to Robbie Harwood from comment #17)
> I literally said that I'm still thinking about what the right way to solve
> the problem is.  You all need to calm down, a *lot*.

What do you think of comment 1?

I have built grubby-8.40-60.fc37 in rawhide.  Once installed, future applications of grubby with --update-kernel=ALL will write out /etc/kernel/cmdline.  Please let me know if there are problems with this build or whether it works for your needs.

Robbie, could you do a RHEL 9 (scratch) build as well?

Reproduced using grubby-8.40-55.el9 from released RHEL-9

1. grubby --args="nosmt" --update-kernel=ALL
2. install new kernel
3. shutdown -r now
4. restart
5. check arguments in kernel cmdline and/or boot into new kernel and check /proc/cmdline for "nosmt"

nosmt is not there!

Preverified using grubby-8.40-56.el9
"nosmt" is present for all kernels

setting verified: tested

(In reply to Marta Lewandowska from comment #23)
> 
> Preverified using grubby-8.40-56.el9
> "nosmt" is present for all kernels
> 
> setting verified: tested

Hello Marta, which architecture was this on? On s390x there's zipl instead of grub2, so it might make sense to test that separately.

(In reply to Jan Pazdziora from comment #27)
> (In reply to Marta Lewandowska from comment #23)
> > 
> > Preverified using grubby-8.40-56.el9
> > "nosmt" is present for all kernels
> > 
> > setting verified: tested
> 
> Hello Marta, which architecture was this on? On s390x there's zipl instead
> of grub2, so it might make sense to test that separately.

This testing was on x86 VM, but it works the same way on s390.
The only difference is that after running grubby, zipl also has to be run.

# rpm -qa grubby\*
grubby-8.40-56.el9.s390x

# cat /proc/cmdline 
root=/dev/mapper/rhel_rtt5-root crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.dasd=0.0.3025 rd.dasd=0.0.3125 rd.dasd=0.0.3225 rd.dasd=0.0.3325 rd.dasd=0.0.3425 rd.dasd=0.0.3525 rd.lvm.lv=rhel_rtt5/root rd.lvm.lv=rhel_rtt5/swap cio_ignore=all,!condev rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0 nosmt BOOT_IMAGE=0

I came across this issue after debugging why our newly-installed CentOS Stream 9 machines with -56 wouldn't boot after installing a new kernel. Turns out the reason is that /etc/kernel/cmdline was getting populated from $GRUB_CMDLINE_LINUX, and it doesn't include the root UUID.

# cat /etc/default/grub
GRUB_TIMEOUT=1
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="console=ttyS0,115200 console=tty0 rd.auto net.ifnames=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

# cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt1)/boot/vmlinuz-5.14.0-115.el9.x86_64 root=UUID=91c75e1d-2376-4c44-bd53-2c46b4d7860b ro console=ttyS0,115200 console=tty0 rd.auto net.ifnames=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M

-57 has reverted this change, which is good, but if I understand comment 21 correctly, the intention is to write to /etc/kernel/cmdline in the future. If so, my question is why doesn't GRUB_CMDLINE_LINUX already include the root UUID parameter? What is responsible for setting that?

We haven't seen this bug anymore in our CI in for 24 days.  Thanks!

Thanks for the message, Marius. I'm happy this is working now too! :)

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (grubby bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8339