Bug 2113883

I am seeing this on F36 with grub2-common-2.06-47.fc36.noarch. I routinely run "grub2-install /dev/sda" and "grub2-mkconfig -o /boot/grub2/grub.conf" (on my BIOS machines) after every grub2 update so -47 is apparently the first affected version.

Just to check, are you sure you're using the right location for the .cfg file? I thought it was /boot/grub2/grub.cfg or /etc/grub2.cfg (for a BIOS system). I always use autocomplete so am quite sure I got the name right and started seeing this on F36 with -47. (I mistyped it in comment 1 but wouldn't have done that when running the command.)

After installing the latest kernel in my branched (37) guest, I see TWO copies of "rhgb quiet" for that kernel in the grub menu. The original kernel doesn't have it, as expected. (I edited /etc/default/grub shortly after install to remove "rhgb quiet", then ran "grub2-mkconfig -o /boot/grub2/grub.cfg".) The branched guest has grub2-common-2.06-47.fc37.noarch.

The preferred way to update the cmdline is with grubby.

But let me see if I can think of a way to make this work.

I can't reproduce #c0:

[root@f36-efi ~]# rpm -qv grub2-tools grubby
grub2-tools-2.06-47.fc36.x86_64
grubby-8.40-64.fc36.x86_64
[root@f36-efi ~]# grep GRUB_CMDLINE_LINUX /etc/default/grub
GRUB_CMDLINE_LINUX="rhgb quiet aabb"
[root@f36-efi ~]# grub2-mkconfig -o /etc/grub2.cfg 
Generating grub configuration file ...
Adding boot menu entry for UEFI Firmware Settings ...
done
[root@f36-efi ~]# grubby --info=ALL | grep args=
args="ro rhgb quiet aabb"
args="ro rhgb quiet aabb"
args="ro rhgb quiet aabb"
[root@f36-efi ~]# 

Any more information you can give me?  Anything look different (other than the output file for grub2-mkconfig)?  I checked both Fedora 36 and rawhide.

In F36, I found that if I run "grub2-mkconfig -o /boot/grub2/grub.cfg" again AFTER the new kernel was installed, it removes the unwanted "rhgb quiet" for that kernel, which must have been added by grubby. So for now I can get rid of it by manually running grub2-mkconfig after installing each new kernel. The same works in F37 - both unwanted copies of "rhgb quiet" are removed from the kernel installed after previously running grub2-mkconfig. So, for me at least, it looks like grub2-mkconfig works okay and the problem is with grubby.

Andre, if you edit the file /etc/kernel/cmdline to remove "rhgb quiet", I think your problem will go away and you'll no longer need to run mkconfig.

Alternately, you could run `grubby --update-kernel=ALL --remove-args='rhgb quiet'` or so and it should resolve itself.

The problem has to do with the initial population of /etc/kernel/cmdline - its contents don't *quite* match what's in grub configuration, and I'm unfortunately stuck applying a heuristic.  It shouldn't affect any other arguments - just those two.

I suspect this is not related to the bug in #c0, merely exposed at the same time.

Thanks. Running "grubby --update-kernel=ALL --remove-args='rhgb quiet' " does indeed fix the problem. Interestingly, before running it, the arguments "ro rootflags=subvol=root" were contained in both the output of "grubby --info=ALL | grep args=" and in /etc/kernel/cmdline. After running it, they were removed from the latter (in addition to removing "rhgb quiet") but were still in the former, and if I reboot, those arguments are still present.

Changing the Version back to 35 since that's what it was before I changed it.

(In reply to Andre Robatino from comment #1)
> I am seeing this on F36 with grub2-common-2.06-47.fc36.noarch. I routinely
> run "grub2-install /dev/sda" and "grub2-mkconfig -o /boot/grub2/grub.conf"
> (on my BIOS machines) after every grub2 update so -47 is apparently the
> first affected version.

I only see this on F35 when creating the bug report. 

(In reply to Andre Robatino from comment #2)
> Just to check, are you sure you're using the right location for the .cfg
> file? I thought it was /boot/grub2/grub.cfg or /etc/grub2.cfg (for a BIOS
> system). I always use autocomplete so am quite sure I got the name right and
> started seeing this on F36 with -47. (I mistyped it in comment 1 but
> wouldn't have done that when running the command.)

Yes, /boot/grub2/grub.cfg should be the right location although "grub2-mkconfig -o /boot/grub2.cfg" also seems to work.

(In reply to Robbie Harwood from comment #5)
> I can't reproduce #c0:
> 
> [root@f36-efi ~]# rpm -qv grub2-tools grubby
> grub2-tools-2.06-47.fc36.x86_64
> grubby-8.40-64.fc36.x86_64
> [root@f36-efi ~]# grep GRUB_CMDLINE_LINUX /etc/default/grub
> GRUB_CMDLINE_LINUX="rhgb quiet aabb"
> [root@f36-efi ~]# grub2-mkconfig -o /etc/grub2.cfg 
> Generating grub configuration file ...
> Adding boot menu entry for UEFI Firmware Settings ...
> done
> [root@f36-efi ~]# grubby --info=ALL | grep args=
> args="ro rhgb quiet aabb"
> args="ro rhgb quiet aabb"
> args="ro rhgb quiet aabb"
> [root@f36-efi ~]# 
> 
> Any more information you can give me?  Anything look different (other than
> the output file for grub2-mkconfig)?  I checked both Fedora 36 and rawhide.

I've updated the bug report to include the version info. Note I don't see this issue on 36 and rawhide. It only happens to current F35. I can reproduce it using 1minutetip machine.

So what is /etc/kernel/cmdline ? I mean traditionally, the canonical source of command line was /etc/default/grub.. I assume /etc/kernel/cmdline is "new" ... what uses it ? ie. which scripts ?

I wonder if we should ensure that GRUB_CMDLINE_LINUX no longer exist, along with a comment in /etc/default/grub to point users at /etc/kernel/cmdline ? (and the necessary transition hacks in the spec file)

> So what is /etc/kernel/cmdline ? I mean traditionally, the canonical source of command line was /etc/default/grub.. I assume /etc/kernel/cmdline is "new" ... what uses it ? ie. which scripts ?

kernel-install.  kernel-install isn't using /etc/default/grub - it tries /etc/kernel/cmdline, /usr/lib/kernel/cmdline, and finally falls back to /proc/cmdline.  So the fact that the current setup appears to work is something of an accident: you lose the race condition if you reboot at the wrong time wrt updating arguments.

> I've updated the bug report to include the version info. Note I don't see this issue on 36 and rawhide. It only happens to current F35. I can reproduce it using 1minutetip machine.

Thanks.  So I think this is the old, broken behavior I mentioned above.

I would like to use this bug *only* for the problem in #c0: that is, behavior on Fedora 35.  Anything else needs a different bug.

FEDORA-2022-a3480ad0d3 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-a3480ad0d3

Hrm... or rather the scripts in /usr/lib/kernel/install.d each of which does ... different things. Not even talking about BLS vs !BLS (and thus different grubbys)

This is a complete mess if you ask me :-)

It's still rather problematic that grub-install and kernel install will end up picking up command lines from different parts depending on how many flies happen to be in the room at that time. Not counting the cases where grubby will try to re-use whatever command line the current kernel is using or the fallback cases in 20-grub.install where it looks at /proc/cmdline.

On Amazon Linux, I've solved it for now by not having /etc/kernel/cmdline and modifying our 20-grub.install as follow:

@@ -82,8 +82,28 @@ case "$COMMAND" in
                 read -r -d '' -a BOOT_OPTIONS < /etc/kernel/cmdline
             elif [[ -f /usr/lib/kernel/cmdline ]]; then
                 read -r -d '' -a BOOT_OPTIONS < /usr/lib/kernel/cmdline
-	    elif [ ! -z "$GRUB_CMDLINE_LINUX_DEFAULT" ]; then
-		BOOT_OPTIONS=("$GRUB_CMDLINE_LINUX_DEFAULT")
+	    elif [ ! -z "$GRUB_CMDLINE_LINUX" ] || [ ! -z "$GRUB_CMDLINE_LINUX_DEFAULT" ]; then
+                declare -a BOOT_OPTIONS
+
+		# Grab the root device and "ro" option from the current command line
+		# because doing otherwise is too hard
+                read -r -d '' -a line < /proc/cmdline
+                for i in "${line[@]}"; do
+                    if [[ "${i#root=*}" != "$i" ]] ; then
+			BOOT_OPTIONS+=("$i")
+		    fi
+		    if [ "$i" == "ro" ]; then
+			BOOT_OPTIONS+=("$i")
+		    fi
+                done
+
+		# Grab everything else from /etc/default/grub
+		if  [ ! -z "$GRUB_CMDLINE_LINUX" ]; then
+		    BOOT_OPTIONS+=("$GRUB_CMDLINE_LINUX")
+		fi
+		if  [ ! -z "$GRUB_CMDLINE_LINUX_DEFAULT" ]; then
+		    BOOT_OPTIONS+=("$GRUB_CMDLINE_LINUX_DEFAULT")
+		fi
 	    else
                 declare -a BOOT_OPTIONS

because I noticed things were all over the place without this. I was meaning to get back to Fedora folks to discuss it and forgot ... that was before your involvement :-)

The problems is users are trained to use grub2-mkconfig, it's all over the place and we even have a "script" that wraps it and grub2-install (and redo the stub config in the EFI partition) all at once. In all those cases, things will use /etc/default/grub, and I'm not super keen on having a "new" canonical location at that time. If you want to move Fedora in that direction however, I think the key is to ensure grub always sources it from there as well, ie, take out the entry in /etc/default/grub, maybe add some warning if present that it's deprecated, and have the scripts always use /etc/kernel/ ... and ensure everything else in /usr/lib/kernel/install.d does so too.

Right now my gut feeling is that it's not clear what comes from where and when...

Test on ARM64 (AMI ami-0689a64e9dedab538 --> Fedora-Cloud-Base-36-20220817.0.aarch64-hvm-ap-southeast-2-gp2-0). All updates from updates-testing + grub2/grubby from the latest builds (https://bodhi.fedoraproject.org/updates/FEDORA-2022-a3480ad0d3).

New kernel install:
-----
# dnf -y update https://kojipkgs.fedoraproject.org//packages/kernel/5.18.18/200.fc36/aarch64/kernel-core-5.18.18-200.fc36.aarch64.rpm
Last metadata expiration check: 0:04:01 ago on Wed 17 Aug 2022 23:35:02.
kernel-core-5.18.18-200.fc36.aarch64.rpm        5.4 MB/s |  56 MB     00:10    
Dependencies resolved.
================================================================================
 Package          Architecture Version                  Repository         Size
================================================================================
Installing:
 kernel-core      aarch64      5.18.18-200.fc36         @commandline       56 M

Transaction Summary
================================================================================
Install  1 Package

Total size: 56 M
Installed size: 141 M
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Installing       : kernel-core-5.18.18-200.fc36.aarch64                   1/1 
  Running scriptlet: kernel-core-5.18.18-200.fc36.aarch64                   1/1 
Generating grub configuration file ...
/usr/share/os-prober/common.sh: line 328: which: command not found
Adding boot menu entry for UEFI Firmware Settings ...
done

  Verifying        : kernel-core-5.18.18-200.fc36.aarch64                   1/1 

Installed:
  kernel-core-5.18.18-200.fc36.aarch64                                          

Complete!
-----

Probably want to use command instead of which there or something like that, right?

Here is the contents of loader conf file (i.e. rootflags is there):
-----
# cat /boot/loader/entries/599428aa056541aa951e2e2d515758f8-5.18.18-200.fc36.aarch64.conf 
title Fedora Linux (5.18.18-200.fc36.aarch64) 36 (Cloud Edition)
version 5.18.18-200.fc36.aarch64
linux /vmlinuz-5.18.18-200.fc36.aarch64
initrd /initramfs-5.18.18-200.fc36.aarch64.img
options root=UUID=29a82369-d013-4c09-815d-ec13c467d2ff ro rootflags=subvol=root no_timer_check net.ifnames=0 console=tty1 console=ttyS0,115200n8 console=tty0
grub_users $grub_users
grub_arg --unrestricted
grub_class fedora
-----

Another test, reinstall kernel after reboot:
-----
# dnf -y reinstall kernel-core
Last metadata expiration check: 0:10:46 ago on Wed 17 Aug 2022 23:35:02.
Dependencies resolved.
================================================================================
 Package            Architecture   Version                 Repository      Size
================================================================================
Reinstalling:
 kernel-core        aarch64        5.18.17-200.fc36        updates         55 M

Transaction Summary
================================================================================

Total download size: 55 M
Installed size: 141 M
Downloading Packages:
kernel-core-5.18.17-200.fc36.aarch64.rpm         52 MB/s |  55 MB     00:01    
--------------------------------------------------------------------------------
Total                                            28 MB/s |  55 MB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Reinstalling     : kernel-core-5.18.17-200.fc36.aarch64                   1/2 
  Running scriptlet: kernel-core-5.18.17-200.fc36.aarch64                   1/2 
  Running scriptlet: kernel-core-5.18.17-200.fc36.aarch64                   2/2 
  Cleanup          : kernel-core-5.18.17-200.fc36.aarch64                   2/2 
  Running scriptlet: kernel-core-5.18.17-200.fc36.aarch64                   2/2 
  Verifying        : kernel-core-5.18.17-200.fc36.aarch64                   1/2 
  Verifying        : kernel-core-5.18.17-200.fc36.aarch64                   2/2 

Reinstalled:
  kernel-core-5.18.17-200.fc36.aarch64                                          

Complete!
-----

Loader conf still looks OK:
-----
# cat /boot/loader/entries/599428aa056541aa951e2e2d515758f8-5.18.17-200.fc36.aarch64.conf 
title Fedora Linux (5.18.17-200.fc36.aarch64) 36 (Cloud Edition)
version 5.18.17-200.fc36.aarch64
linux /vmlinuz-5.18.17-200.fc36.aarch64
initrd /initramfs-5.18.17-200.fc36.aarch64.img
options root=UUID=29a82369-d013-4c09-815d-ec13c467d2ff ro rootflags=subvol=root no_timer_check net.ifnames=0 console=tty1 console=ttyS0,115200n8 console=tty0
grub_users $grub_users
grub_arg --unrestricted
grub_class fedora
-----

Machine still alive post reboot, so apart from that minor which issue, it appears this did fix it.

FEDORA-2022-a3480ad0d3 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-a3480ad0d3`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-a3480ad0d3

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

(In reply to Benjamin Herrenschmidt from comment #15)
I agree it's a mess, and I'm... well, not happy, but definitely willing to discuss it further :)

Fundamentally, what we're trying to solve is having a place to store default arguments for future installed kernels. kernel-install picked three places this could be, and without the first two being written, it pulls /proc/cmdline.  So we can either try to get it to pull somewhere else - hence writing /etc/kernel/cmdline - or we can renegotiate the entire interface boundary.

FEDORA-2022-a3480ad0d3 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-a3480ad0d3`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-a3480ad0d3

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Right, I agree.

It boils down to:

Users have (too) long be trained to treat /etc/default/grub as the main source. Well, those who aren't hacking directly with grubby but I'll ignore these here, so when grub is in use, this should remain the "canonical" source of thruth and thus it makes sense to, in turn, update /etc/kernel/cmdline accordingly. It also makes sense that everything else relies on /etc/kernel/cmdline, so that the !grub case needs only to make sure this gets populated properly and everything will derive from it.

This will work ... until something starts whacking /etc/kernel/cmdline directly on a grub system :-) (some other admin tool ?) and we are back to being out of sync ... oh well

I only have a one, probably minor, comment at this point:

The 20-grub.install patch that does this makes me nervous:

         if [[ "x${GRUB_ENABLE_BLSCFG}" = "xtrue" ]] || [[ ! -f /sbin/new-kernel-pkg ]]; then
             if [[ -f /etc/kernel/cmdline ]]; then
+                if [[ /etc/kernel/cmdline -ot /etc/default/grub ]]; then
+                    # user modified /etc/default/grub manually; sync
+                    grub2-mkconfig -o /etc/grub2.cfg
+                fi
                 read -r -d '' -a BOOT_OPTIONS < /etc/kernel/cmdline
             elif [[ -f /usr/lib/kernel/cmdline ]]; then

Under what circumstances do we think this is needed ?

It was never expected that changing /etc/default/grub without a following grub2-mkconfig would do anything anyways, any specific reason we are trying address this here ? Ie, if grub2-mkconfig now updates /etc/kernel/cmdline, we should be fine, shouldn't we ?

Otherwise I think the rest of your changes should allow me to also remove my hacks to 20-grub.install so all good :-)

> This will work ... until something starts whacking /etc/kernel/cmdline directly on a grub system :-) (some other admin tool ?) and we are back to being out of sync ... oh well

I think at that point the desync is intentional - either the user is managing it directly and bypassing grub's, or something else is responsible for configuration.  I'm not aware of anything doing this, and wouldn't be thrilled about tools doing it either (just call grubby please), but I'm not about to fight with admins over their own systems.

> Under what circumstances do we think this is needed ?

I suspect it's not needed at all, but couldn't easily prove that.  I chose to steer to the side of caution in order to minimize testing cycles.

Now that said, if there's something you're worried about going wrong with that snippet, that's something I'd want to address too :)

FEDORA-2022-a3480ad0d3 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-a3480ad0d3`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-a3480ad0d3

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-a3480ad0d3 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.