Bug 1978226

*** Bug 2008131 has been marked as a duplicate of this bug. ***

Please note that this should, as I perceive it, apply also to kernel downgrades.

> 1. disable SMT: run "grubby --args=nosmt --update-kernel=DEFAULT"

DEFAULT here causes it to be applied to the entry which is the default one, not all entries.  I believe this would work if one said --update-kernel=ALL instead.

However, if the machine is not rebooted (your step 2), it may not.  I'd like to use this bug as the RHEL-8 equivalent of #1969362 - to fix the behavior for new kernel installations after setting an ALL.

fwiw, I tested both --update-kernel=DEFAULT and --update-kernel=ALL, and you're right, Robbie, that the latter works even with the older version of grubby on all arches except s390x, even without the reboot. The new version grubby-8.40-43.el8, unfortunately, acts in the same way: works for all arches except s390. But, it also does not appear to break anything...

https://beaker.engineering.redhat.com/jobs/6725758

Testing using grubby-8.40-45.el8 passes for all architectures: https://beaker.engineering.redhat.com/jobs/6913883

Setting Verifed: Tested

grubby-8.40-45.el8.x86_64 is present in nightly compose RHEL-8.7.0-20220826.0

Since this bug was documented, I believe documentation will need to be updated to reflect this [new] behavior..?

Hi, we found several problems with this fix. Checking patches, I guess that it should be:

~~~
diff --git a/grubby-bls b/grubby-bls
index 6357e38..2d18c91 100755
--- a/grubby-bls
+++ b/grubby-bls
@@ -526,8 +526,8 @@ update_bls_fragment() {
 
     if [[ $bootloader = grub2 ]] && [[ ! -f /etc/kernel/cmdline ]]; then
        opts=$(grub2-editenv "${env}" list | grep kernelopts | sed 's/kernelopts=//')
-       if [[ -n $opts ]]; then
-           echo "opts" > /etc/kernel/cmdline
+       if [[ -n "$opts" ]]; then
+           echo "$opts" > /etc/kernel/cmdline
        fi
     fi
~~~

This leads to to one of errors we are hitting when
    # cat /etc/kernel/cmdline
    opts
which produces invalid grub entries with options "opts", so the kernel is unbootable.



There are additional bugs present, e.g. missing 'root=...' on s390x:
~~~~
    [root@localhost yum.repos.d]# cat /etc/kernel/cmdline/
    inst.sshd inst.repo=cdrom inst.ks=cdrom:/ks.cfg nicdelay=60 net.ifnames=0 console=ttysclp0,38400


    [root@localhost yum.repos.d]# grubby --info ALL
    index=0
    kernel="/boot/vmlinuz-4.18.0-425.3.1.el8.s390x"
    args="console=tty0 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap net.ifnames=0"
    root="/dev/mapper/rhel-root"
    initrd="/boot/initramfs-4.18.0-425.3.1.el8.s390x.img"
    title="Red Hat Enterprise Linux (4.18.0-425.3.1.el8.s390x) 8.7 (Ootpa)"
    id="7df90c6e34204f03aab8f70bdd5a64f1-4.18.0-425.3.1.el8.s390x"
    index=1
    kernel="/boot/vmlinuz-0-rescue-7df90c6e34204f03aab8f70bdd5a64f1"
    args="console=tty0 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap net.ifnames=0"
    root="/dev/mapper/rhel-root"
    initrd="/boot/initramfs-0-rescue-7df90c6e34204f03aab8f70bdd5a64f1.img"
    title="Red Hat Enterprise Linux (0-rescue-7df90c6e34204f03aab8f70bdd5a64f1) 8.7 (Ootpa)"
    id="7df90c6e34204f03aab8f70bdd5a64f1-0-rescue"


    [root@localhost yum.repos.d]# dnf update
    ...[snip]....
    Installing:
     kernel         s390x 4.18.0-429.el8 brew-task-repo-kernel-4.18.0-429.el8 8.8 M
    Upgrading:
     bpftool        s390x 4.18.0-429.el8 brew-task-repo-kernel-4.18.0-429.el8  12 M
     kernel-headers s390x 4.18.0-429.el8 brew-task-repo-kernel-4.18.0-429.el8  10 M
     kernel-tools   s390x 4.18.0-429.el8 brew-task-repo-kernel-4.18.0-429.el8 8.9 M
     python3-perf   s390x 4.18.0-429.el8 brew-task-repo-kernel-4.18.0-429.el8 9.0 M
    Installing dependencies:
     kernel-core    s390x 4.18.0-429.el8 brew-task-repo-kernel-4.18.0-429.el8  30 M
     kernel-modules s390x 4.18.0-429.el8 brew-task-repo-kernel-4.18.0-429.el8  10 M
    Installing weak dependencies:
     kernel-devel   s390x 4.18.0-429.el8 brew-task-repo-kernel-4.18.0-429.el8  21 M
    ...[snip]....


    [root@localhost yum.repos.d]# grubby --info ALL
    index=0
    kernel="/boot/vmlinuz-4.18.0-429.el8.s390x"
    args="inst.sshd inst.repo=cdrom inst.ks=cdrom:/ks.cfg nicdelay=60 net.ifnames=0 console=ttysclp0,38400"
    initrd="/boot/initramfs-4.18.0-429.el8.s390x.img"
    title="Red Hat Enterprise Linux (4.18.0-429.el8.s390x) 8.8 (Ootpa)"
    id="7df90c6e34204f03aab8f70bdd5a64f1-4.18.0-429.el8.s390x"
    index=1
    kernel="/boot/vmlinuz-4.18.0-425.3.1.el8.s390x"
    args="console=tty0 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap net.ifnames=0"
    root="/dev/mapper/rhel-root"
    initrd="/boot/initramfs-4.18.0-425.3.1.el8.s390x.img"
    title="Red Hat Enterprise Linux (4.18.0-425.3.1.el8.s390x) 8.7 (Ootpa)"
    id="7df90c6e34204f03aab8f70bdd5a64f1-4.18.0-425.3.1.el8.s390x"
    index=2
    kernel="/boot/vmlinuz-0-rescue-7df90c6e34204f03aab8f70bdd5a64f1"
    args="console=tty0 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap net.ifnames=0"
    root="/dev/mapper/rhel-root"
    initrd="/boot/initramfs-0-rescue-7df90c6e34204f03aab8f70bdd5a64f1.img"
    title="Red Hat Enterprise Linux (0-rescue-7df90c6e34204f03aab8f70bdd5a64f1) 8.7 (Ootpa)"
    id="7df90c6e34204f03aab8f70bdd5a64f1-0-rescue"
~~~~

See the newly created entry does not contain the "root=" option. So after the reboot system is unbootable again.

This also introduces regression for of in-place upgrades RHEL 8.7 -> RHEL 9.0 (see bug 2129740), where the grub entry for rhel9 could look e.g. like this (see the "opts" string):
~~~
    [root@localhost ~]# grubby --info ALL
    index=0
    kernel="/boot/vmlinuz-5.14.0-70.13.1.el9_0.x86_64"
    args="opts $tuned_params net.ifnames=0"
    initrd="/boot/initramfs-5.14.0-70.13.1.el9_0.x86_64.img $tuned_initrd"
    title="Red Hat Enterprise Linux (5.14.0-70.13.1.el9_0.x86_64) 9.0 (Plow)"
    id="e34284b9133641fca763c5dc2d60499d-5.14.0-70.13.1.el9_0.x86_64"
~~~

Regarding this output, is "$tuned_params" expected value in the args?
Also see bug 1900829 which could be related as well.
The fix for RHEL 9.1 seems to be very different from from this one (bug 1969362) - I haven't tested RHEL 9.1...

This is potential release blocker for RHEL 8.7.

In the future, please keep comments in the other bug.  This simplifies problem investigations.

My mistake, originally I thought the problem is going to be resolved in this bug as I thought it's going to be marked as failedqa regarding the original fix is from my pov invalid.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (grubby bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:7799