Bug 1625124

Did you edit the grubenv manually at some point? The grub2-editenv command is quite fragile and is common for it to fail writing to the environment block if the grubenv file was externally edited.

Also, since you are not using the grubenv (you mention that the default saved_entry doesn't even exist in your grub.cfg file) could you try creating a new grubenv file:

$ grub2-editenv create

And then attempt again to install your kernel.

at some point in the past, i had have to edit that file manually, yes. But i have no idea, if that entry was made manually. Also i doubt it, as 4.16 was instable too, and i would have set it back to  4.15, not 4.16. 

i will try it out... tried it..

interessting handling.. as "create" clears the file "/boot/grub2/grubenv",
but "set saved_entry=..." does not work without the exact hint to the file you wanne edit.

Sorry say that, but that's s....inconsistent :) As if two different people coded those functions and did never talk to each other ;)

BTW: the installation and boot of that kernel worked, besides that specific entry. 

BTW²: it does not matter whats inside that grubenv, as it gets ignored. The first entry in the grub.conf is loaded.

the grub.conf says:

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub2-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
set pager=1

if [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="${saved_entry}"
fi

.....

Hint: when/where does "$prefix" gets defined and to what ?

no "prefix" in /etc/default/grub:

[root@s121 ~]# cat /etc/default/grub 
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rhgb quiet audit=0"
GRUB_DISABLE_RECOVERY="true"

nor in the templates nor anywhere under/etc . 

I pretty sure, that that information got lost when we upgraded from grub1 to grub2 years ago, but never was a problem as the newest kernel got sorted on top of the bootentry list. 

Let me know, if you need some more details.

This message is a reminder that Fedora 27 is nearing its end of life.
On 2018-Nov-30  Fedora will stop maintaining and issuing updates for
Fedora 27. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora  'version' of '27'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 27 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 27 changed to end-of-life (EOL) status on 2018-11-30. Fedora 27 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

this happened on FC29 and grubenv was never user modified

'grub2env create' solved the issue

fwiw, grubenv was a symlink in /boot/grub2 to /boot/efi/EFI/fedora/grubenv

@Mai Ling :

to clarify: you mean, that the grubenv was filled up with "###" and you never touched that file manually. Correct?

Ran into the same issue but I did edit mine manually because it continued to default to an older kernel even when two new ones had been installed.

got this repeatedly with a fresh f29 install from november 2019. last saw this today w/ 4.19.15-300.fc29.x86_64.

(In reply to Mai Ling from comment #5)
> 'grub2env create' solved the issue

that should be 'grub2-editenv create', right?

(In reply to customercare from comment #6)
> @Mai Ling :
> 
> to clarify: you mean, that the grubenv was filled up with "###" and you
> never touched that file manually. Correct?

correct

(In reply to jan p. springer from comment #9)
> (In reply to Mai Ling from comment #5)
> > 'grub2env create' solved the issue
> 
> that should be 'grub2-editenv create', right?

oops typo my bad, yes, grub2-editenv

(In reply to Mai Ling from comment #10)
> (In reply to customercare from comment #6)
> > @Mai Ling :
> > 
> > to clarify: you mean, that the grubenv was filled up with "###" and you
> > never touched that file manually. Correct?
> 
> correct

What was the exact command that caused grub2-editenv to complain about the grubenv file size? I'm trying to get a reproducer of the issue.

afaict, this only turns up during kernel updates. however, with 4.20.x i did not (yet) see it.

(In reply to jan p. springer from comment #13)
> afaict, this only turns up during kernel updates. however, with 4.20.x i did
> not (yet) see it.

Yes, on kernel installs the grubenv is modified to updated the saved_entry variable (if UPDATEDEFAULT=yes is set in /etc/sysconfig/kernel).

So it would be good if this is still present or not. It's expected to fail if someone modified the grubenv without grub2-editenv, but some users reported that it failed even when the grubenv file was only modified with grub2-editenv.

it seems to be random. i did not manual changes nor anything else with grubenv, when all of a sudden a kernel update warned about grubenv size. after searching the net i used grub2-editenv to correct grubenv. two or three kernel updates later the same thing happened again. again i used grub2-editenv to correct grubenv. other than those two times i did not touch the grubenv file.

Since i opened the bugreport, i upgraded to f28 and it happens with f28 kernels as well.

My F29 laptop with 4.20.x does it too sometimes.

The big question is : 

What exactly is too small for which information ? It's a textfile , it has unlimited space. I would rule this out.

is it a an internal variable length that gets exhausted?

(In reply to customercare from comment #16)
> Since i opened the bugreport, i upgraded to f28 and it happens with f28
> kernels as well.
> 
> My F29 laptop with 4.20.x does it too sometimes.
> 

I was not able to reproduce this. Could you please provide your grubenv file (/boot/grub2/grubenv) the next time that happens to you.

> The big question is : 
> 
> What exactly is too small for which information ? It's a textfile , it has
> unlimited space. I would rule this out.
> 
> is it a an internal variable length that gets exhausted?

The grubenv file has a fixed size of 1024 bytes. It contains a signature (# GRUB Environment Block) followed by a string of # characters up until 1024 bytes.

When a new variable is added with grub2-editenv, the key=value pair is stored in grubenv and the file is then filled again with # characters up to 1024 bytes. The number of # characters are the "free space" that the grubenv file has for new variables.

That's why editing manually the file breaks this assumption, because the # characters are not replaced by the variable added to the file.

So what is important to know when the "grub2-editenv: Error: Environment-Block is too small" error happens is:

a) What's the size of the grubenv file (/boot/efi/EFI/fedora/grubenv for EFI or /boot/grub2/grubenv for BIOS)

b) The content of the file to see why grub2-editenv complains.

Ok, in my case I manually edited mine because I got stuck booting on a previous kernel even though there had been several new kernels installed and this whole thing isn't exactly intuitive, but the real question is...

Why is grub2-editenv so fragile?

(In reply to Richard Shaw from comment #18)
> Ok, in my case I manually edited mine because I got stuck booting on a
> previous kernel even though there had been several new kernels installed and
> this whole thing isn't exactly intuitive, but the real question is...
> 
> Why is grub2-editenv so fragile?

Yes, I do wonder the same.

*** Bug 1673909 has been marked as a duplicate of this bug. ***

Filed an RFE upstream.

i just got this :

[root@surface ~]# cat /boot/grub2/grubenv
saved_entry="Fedora (4.19.23-surface-pro4.fc29.x86_64+) 29 (Workstation Edition)"
[root@surface ~]# grub2-editenv create
[root@surface ~]# cat /boot/grub2/grubenv
# GRUB Environment Block
#######################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################[root@surface ~]# 


...  broken beyond repair ...

One way to make this happen is when a text editor has added an <eol> at the end of the last line. Even when the file is exactly 1024 chars, "grub2-editenv /boot/efi/EFI/fedora/grubenv set saved-entry=..." will fail with "error: environment block too small.". 

Apparently, grub2-editenv ignores the last <eol> when reading this file. Interestingly however, "grub2-editenv /boot/efi/EFI/fedora/grubenv set boot_indeterminate=0" works without error in this case.

For the googlers, you can repair the grubenv file with vim by entering binary mode with ":set binary noeol". You will have to add one extra # to the padding to compensate for the <eol> that will be removed from the last line.

Same thing happens on F30. I manually edited the file because I'm puzzled by the thought that hiding Grub boot menu and being able to choose various kernels is a good idea. So I manually put menu_auto_hide=0 and removed the 'rhgb' flag in the kernel line, because I want to see my boot process and I want to catch services that possibly die during bootstrap. So now my file looks like this:

# GRUB Environment Block
saved_entry=98d7696763cd423cb298caa133e0129a-5.2.8-200.fc30.x86_64
menu_auto_hide=0
boot_success=1
boot_indeterminate=0
kernelopts=root=UUID=e0797368-596a-4b28-9c9b-f6906a310b88 ro resume=UUID=1f169d9f-f92f-4a15-a334-df8bfe81dc96
############################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################

root@fedora:~# cat /boot/efi/EFI/fedora/grubenv  | wc -m
1020

The vim trick worked, although I added a bunch of extra # since I removed 4 or 5 characters from the file so apparently it needs to be 1024 chars or more plus the vim trick:

root@fedora:~# cat /boot/efi/EFI/fedora/grubenv  | wc -m
1033

Recommendation: regenerate grubenv based on the context from /etc/default/grub, as that file is there for a reason.

This message is a reminder that Fedora 29 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '29'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 29 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

*** Bug 1714533 has been marked as a duplicate of this bug. ***

This bug appears to have been reported against 'rawhide' during the Fedora 32 development cycle.
Changing version to 32.

Just saw this whilst installing kernels:-

kernel-core-5.5.11-200.fc31.x86_64

and

kernel-core-5.5.10-200.fc31.x86_64

I further observerve that this time-

a) Editing /boot/efi/EFI/fedora/grubenv to make it >= 1024 bytes long or longer then reinstalling 5.5.11 did not work; still get the error message.
b) The grubenv file had somehow got a newline character right at the end and had also somehow become shorter than 1024 bytes, so I was surprised that a) did not work.
c) Removing the grubenv file, recreating it, and then reinstall the kernel *did* work:-

# cp -p /boot/efi/EFI/fedora/grubenv /boot/efi/EFI/fedora/grubenv.old
# rm /boot/efi/EFI/fedora/grubenv
rm: remove regular file '/boot/efi/EFI/fedora/grubenv'? y
# grub2-editenv create
# cat /boot/efi/EFI/fedora/grubenv
# GRUB Environment Block
#######################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################ll /boot/efi/EFI/fedora/grubenv
-rwx------ 1 root root 1024 Mar 30 13:15 /boot/efi/EFI/fedora/grubenv
# dnf reinstall kernel*-5.5.11-200.fc31.x86_64                       

...

Reinstalled:
  kernel-5.5.11-200.fc31.x86_64          kernel-core-5.5.11-200.fc31.x86_64          kernel-devel-5.5.11-200.fc31.x86_64          kernel-modules-5.5.11-200.fc31.x86_64         

Complete!
# ll /boot/efi/EFI/fedora/grubenv
-rwx------ 1 root root 1024 Mar 30 13:16 /boot/efi/EFI/fedora/grubenv
# cat /boot/efi/EFI/fedora/grubenv
# GRUB Environment Block
saved_entry=d17c54aa0897427887fb56d7f7e4dc7c-5.5.11-200.fc31.x86_64
####################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################

d) I don't really see why this file is needed.  The grub boot menu should be adequate on its own.

I opened the github issue a year ago now. I don't think complaining on BZ is going to help unfortunately. Maybe nagging on the linked github issue would be more effective?

@Comment 29:

> d) I don't really see why this file is needed.  The grub boot menu should be adequate on its own. 

Because it's holding the information, which kernel should be booted on default. 

If you enable BLS config on your system, it also contains the information about all the kernel args used. Which is even more important, than knowing the default kernel.

(In reply to customercare from comment #31)
> @Comment 29:
> 
> > d) I don't really see why this file is needed.  The grub boot menu should be adequate on its own. 
> 
> Because it's holding the information, which kernel should be booted on
> default. 
> 
> If you enable BLS config on your system, it also contains the information
> about all the kernel args used. Which is even more important, than knowing
> the default kernel.

We will change that from F33 onwards, because the grubenv is very fragile and having the kernel command line parameters there caused more harm than good.

Instead the cmdline will be stored in the BLS snippets itself, as it's already the case for BLS snippets used by other bootloaders like zipl and sd-boot.

@RS: can you supply a like, i would like to give some boot to the head ;)

It's at the top under "Links" but here it is:

https://github.com/rhboot/grub2/issues/50

Hello Richard,

(In reply to Richard Shaw from comment #30)
> I opened the github issue a year ago now. I don't think complaining on BZ is
> going to help unfortunately. Maybe nagging on the linked github issue would
> be more effective?

Sorry about that. The https://github.com/rhboot/grub2 repository is just to maintain the downstream patches that we are carrying in the grub2 Fedora package, the GRUB mainline repository is http://git.savannah.gnu.org/cgit/grub.git.

So I don't think that anyone pays attention to the issues created in that repository since the correct place to file bugs for Fedora is in Bugzilla. I've disabled the issues feature now in the rhboot grub2 repository to avoid misunderstanding.

Now, the grub2-editenv tool being fragile is an issue not specific to Fedora, so you could try to file an issue in upstream GRUB bug tracker (https://savannah.gnu.org/bugs/?group=grub) or report it to the grub-devel mailing list (https://lists.gnu.org/mailman/listinfo/grub-devel).

I have in my TODO to take a look to this but I don't know when I'll have the time to do that.

I can see with over 300 open bugs this would not be a high priority but it also makes a poor user experience...

I'm not really a programmer but the psudeau loging would be something like:

1. check end if of the file, if it's a newline, remove it
2. check again, is it a "#"?
3a. If so pad or unpad as needed to fix the file size
3b. Throw a real error that the user mucked it up.

pseudo logic... I usually don't bother to fix spelling errors in BZ but I really messed that one up.

per request from Javier in IRC,

on F32, booted EFI,

	uname -rm
		5.7.6-201.fc32.x86_64 x86_64

with

	rpm -qa | grep grub2 | sort
		grub2-common-2.04-21.fc32.noarch
		grub2-efi-x64-2.04-21.fc32.x86_64
		grub2-tools-2.04-21.fc32.x86_64
		grub2-tools-efi-2.04-21.fc32.x86_64
		grub2-tools-extra-2.04-21.fc32.x86_64
		grub2-tools-minimal-2.04-21.fc32.x86_64
		terminus-fonts-grub2-4.48-7.fc32.noarch



on exec of 

	grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg


i still see error


	Generating grub configuration file ...
		/usr/bin/grub2-editenv: error: environment block too small.

checking

	ls -al /boot/efi/EFI/fedora/grubenv /boot/grub2/grubenv
		-rwx------  1 root root 1.0K Jul  2 12:58 /boot/efi/EFI/fedora/grubenv*
		lrwxrwxrwx. 1 root root   25 Jun 19 09:24 /boot/grub2/grubenv -> ../efi/EFI/fedora/grubenv*

	grub2-editenv list
		saved_entry=6863dfea801840279b5e8c09adeed241-5.7.6-201.fc32.x86_64
		boot_success=1

	cat /boot/loader/entries/6863dfea801840279b5e8c09adeed241-5.7.6-201.fc32.x86_64.conf
		title Fedora (5.7.6-201.fc32.x86_64) 32 (Thirty Two)
		version 5.7.6-201.fc32.x86_64
		linux /vmlinuz-5.7.6-201.fc32.x86_64
		initrd /initramfs-5.7.6-201.fc32.x86_64.img
		options $kernelopts
		grub_users $grub_users
		grub_arg --unrestricted
		grub_class kernel

	cat /boot/grub2/grubenv
		# GRUB Environment Block
		saved_entry=6863dfea801840279b5e8c09adeed241-5.7.6-201.fc32.x86_64
		boot_success=1
		#####################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################

Created attachment 1699737 [details]
grubenv.tar.gz, request from javier

(In reply to pgnet.dev from comment #39)
> Created attachment 1699737 [details]
> grubenv.tar.gz, request from javier

I'm not able to reproduce the issue with your attached grubenv file.

$ tar -xvf grubenv.tar.gz 
grubenv

$ cat grubenv
# GRUB Environment Block
saved_entry=6863dfea801840279b5e8c09adeed241-5.7.6-201.fc32.x86_64
boot_success=1
#####################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################

$ grub2-editenv grubenv set foo=bar

$ cat grubenv
# GRUB Environment Block
saved_entry=6863dfea801840279b5e8c09adeed241-5.7.6-201.fc32.x86_64
boot_success=1
foo=bar
#############################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################

This is with grub2-tools-minimal-2.04-21.fc32.x86_64 that is the same version that you have.

Can you please share your GRUB_CMDLINE_LINUX from /etc/default/grub ?

as it seems, your grubenv is to small, it needs to be 1024 bytes long .. exact! 

just delete the grubenv and recreate it : grubby --set-default-index=1  ( or which ever kernel you want ) . It will have the correct size afterwards.

> just delete the grubenv and recreate it

As I'd discussed in #irc, but did not mention here ...

that's

-- certainly doable
-- been done depeatedly
-- insufficient, as it's not 'persistent'.  the problem returns.

e.g.

grubby --default-title  && \
grubby --default-kernel && \
grubby --default-index

	Fedora (5.7.6-201.fc32.x86_64) 32 (Thirty Two)
	/boot/vmlinuz-5.7.6-201.fc32.x86_64
	0

rm -f /boot/efi/EFI/fedora/grubenv

grubby --set-default-index=0

grub2-editenv create
ls -al /boot/efi/EFI/fedora/grubenv
	-rwx------ 1 root root 1.0K Jul  3 08:59 /boot/efi/EFI/fedora/grubenv*

grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
	Generating grub configuration file ...
	/usr/bin/grub2-editenv: error: environment block too small.

(In reply to pgnet.dev from comment #43)

[snip]

> 
> grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
> 	Generating grub configuration file ...
> 	/usr/bin/grub2-editenv: error: environment block too small.


Can you please share your GRUB_CMDLINE_LINUX as mentioned in Comment 40? That way I could attempt to reproduce your issue. I tried several times and wasn't able to do it.

sorry, missed the request :-/

for this instance I'm working on,

GRUB_CMDLINE_LINUX="vconsole.keymap=us vconsole.font=eurlatgr vconsole.font_map=trivial domdadm dolvm lvmwait=/dev/mapper/VG_MX1DEV_VB-LV_MX1DEV_ROOT mitigations=auto transparent_hugepage=never max_loop=128 kdbus=0 apparmor=0 enforcing=0 selinux=0 rd.plymouth=0 plymouth.enable=0 audit=0 fsck.mode=auto fsck.repair=preen net.ifnames=1 scsi_mod.use_blk_mq=1 acpi_osi=Linux enable_mtrr_cleanup mtrr_spare_reg_nr=1 mtrr_gran_size=32M mtrr_chunk_size=128M mce=bootlog usbcore.autosuspend=-1 nmi_watchdog=0 debug showopts noquiet print_fatal_signals=1 loglevel=3 rd.systemd.show_status=1 root=/dev/mapper/VG_MX1DEV_VB-LV_MX1DEV_ROOT rootfstype=ext4 rootflags=journal_checksum rd.lvm.lv=VG_MX1DEV_VB/LV_MX1DEV_ROOT rd.lvm.lv=VG_MX1DEV_VB/LV_MX1DEV_SWAP rd.debug=0 rd.shell=1 rd.auto=1 rd.udev.log_priority=info systemd.log_target=kmsg log_buf_len=1M printk.devkmsg=on systemd.log_level=info console=tty0 earlycon earlyprintk=vga,keep"

cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.7.6-201.fc32.x86_64 root=/dev/mapper/VG_MX1DEV_VB-LV_MX1DEV_ROOT ro vconsole.keymap=us vconsole.font=eurlatgr vconsole.font_map=trivial domdadm dolvm lvmwait=/dev/mapper/VG_MX1DEV_VB-LV_MX1DEV_ROOT mitigations=auto transparent_hugepage=never max_loop=128 kdbus=0 apparmor=0 enforcing=0 selinux=0 rd.plymouth=0 plymouth.enable=0 audit=0 fsck.mode=auto fsck.repair=preen net.ifnames=1 scsi_mod.use_blk_mq=1 acpi_osi=Linux enable_mtrr_cleanup mtrr_spare_reg_nr=1 mtrr_gran_size=32M mtrr_chunk_size=128M mce=bootlog usbcore.autosuspend=-1 nmi_watchdog=0 debug showopts noquiet print_fatal_signals=1 loglevel=3 rd.systemd.show_status=1 root=/dev/mapper/VG_MX1DEV_VB-LV_MX1DEV_ROOT rootfstype=ext4 rootflags=journal_checksum rd.lvm.lv=VG_MX1DEV_VB/LV_MX1DEV_ROOT rd.lvm.lv=VG_MX1DEV_VB/LV_MX1DEV_SWAP rd.debug=0 rd.shell=1 rd.auto=1 rd.udev.log_priority=info systemd.log_target=kmsg log_buf_len=1M printk.devkmsg=on systemd.log_level=info console=tty0 earlycon earlyprintk=vga,keep

I just checked the sources .. that messages is mostly wrong all day long. 

     if (! grub_envblk_set (envblk, argv[0], p))
        grub_util_error ("%s", _("environment block too small"));

for any reason the function fails, it gives out the same error.

(In reply to pgnet.dev from comment #45)
> sorry, missed the request :-/
> 
> for this instance I'm working on,
> 
> GRUB_CMDLINE_LINUX="vconsole.keymap=us vconsole.font=eurlatgr
> vconsole.font_map=trivial domdadm dolvm
> lvmwait=/dev/mapper/VG_MX1DEV_VB-LV_MX1DEV_ROOT mitigations=auto
> transparent_hugepage=never max_loop=128 kdbus=0 apparmor=0 enforcing=0
> selinux=0 rd.plymouth=0 plymouth.enable=0 audit=0 fsck.mode=auto
> fsck.repair=preen net.ifnames=1 scsi_mod.use_blk_mq=1 acpi_osi=Linux
> enable_mtrr_cleanup mtrr_spare_reg_nr=1 mtrr_gran_size=32M
> mtrr_chunk_size=128M mce=bootlog usbcore.autosuspend=-1 nmi_watchdog=0 debug
> showopts noquiet print_fatal_signals=1 loglevel=3 rd.systemd.show_status=1
> root=/dev/mapper/VG_MX1DEV_VB-LV_MX1DEV_ROOT rootfstype=ext4
> rootflags=journal_checksum rd.lvm.lv=VG_MX1DEV_VB/LV_MX1DEV_ROOT
> rd.lvm.lv=VG_MX1DEV_VB/LV_MX1DEV_SWAP rd.debug=0 rd.shell=1 rd.auto=1
> rd.udev.log_priority=info systemd.log_target=kmsg log_buf_len=1M
> printk.devkmsg=on systemd.log_level=info console=tty0 earlycon
> earlyprintk=vga,keep"
> 

Wow, that's a huge kernel cmdline. That explains then why you are getting out of space in the grubenv file, since only what's set in the GRUB_CMDLINE_LINUX variable is 908 characters long.

If you add the root parameter that's added by GRUB and the kernelopts variable itself, only the cmdline is almost the size of the whole grubenv file (which is only 1024 bytes).

For example, using your cmdline in a VM I get the following:

$ grub2-editenv list | grep kernelopts | wc -c
952

The default boot entry is also set in the grubenv when a new kernel is installed (if UPDATEDEFAULT=yes is set in /etc/sysconfig/kernel) and is stored in the saved_entry variable. On my F32 laptop the length of this with the latest kernel is 67 bytes:

$ grub2-editenv list | grep saved_entry | wc -c
67

But with your cmdline, I only get 48 chars left in the grubenv so then setting the default entry won't be possible and grub2-editenv will fail:

$ grub2-editenv create && grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
Generating grub configuration file ...
Adding boot menu entry for UEFI Firmware Settings ...
done

$ cat /boot/efi/EFI/fedora/grubenv | grep "^#*$" | wc -c
48

$ grub2-editenv - set save_default=5c131dfee3249cbb9891c2641d8e350-5.7.7-200.fc32.x86_64
grub2-editenv: error: environment block too small.

Conversely, if after creating an empty grubenv, you set something there, then it will not have enough space anymore for your cmdline:

$ grub2-editenv create

$ grub2-editenv - set saved_entry=6863dfea801840279b5e8c09adeed241-5.7.6-201.fc32.x86_64

$ cat /boot/efi/EFI/fedora/grubenv | grep "^#*$" | wc -c
933

So there's only 933 characters left, but as mentioned I would need 952 to store your cmdline which will lead to grub2-mkconfig failing due not having enough space to store:

$ grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
Generating grub configuration file ...
/usr/bin/grub2-editenv: error: environment block too small.

I understand this is not ideal and that shows yet another reason why storing the cmdline as a variable in the grubenv is fragile. This won't be an issue anymore from F33 onwards, since as mentioned the cmdline will be stored in the BLS snippets.

But is also true that your use case is a corner case, most users won't need such a long kernel cmdline. One solution for this could be to backport to F32 the changes for F33 that drops the kernelopts variable, but I think that is a drastic change that shouldn't be made after a Fedora version has been released.

Maybe you could revisit some of the options that you are passing to the kernel? To make enough room in the grubenv for the other variables that are stored there, i.e: saved_entry, menu_auto_hide, and boot_{success,indeterminate}.

> most users won't need such a long kernel cmdline

Perhaps, but that's an assumption.  It may be correct, but I don't see how it's particularly relevant to the point.

The command line is perfectly valid, unless GRUB &/or the kernel have a limit on cmd length?
If not, the 'assumption' can't be considered a reasons to rationalize the breakage.

The commands are there for operational reasons that vary to some extent by hardware.

> Maybe you could revisit some of the options that you are passing to the kernel? To make enough room in the grubenv for the other variables that are stored there, i.e: saved_entry, menu_auto_hide, and boot_{success,indeterminate}.

Sure, one can try that.  But the point is that that's a workround to the fact of the problem; which itself remains unfixed; Simply, it's broken for the moment.

And such a workaround is, here, not a solution I can entertain, or maintain, for 1K+ machines; not that they're not all Fedora32 ... today.

A far simpler 'workaround' -- seems to be to DISABLE BLSCFG, i.e. set "GRUB_ENABLE_BLSCFG=false".  still not ideal, but simpler.  With that, the greubenv error, at least, ceases.

>  but I think that is a drastic change that shouldn't be made after a Fedora version has been released.

I hear you, and understand the argument.  And know that the decision 'for F32' will be made on its own grounds.

Here, the inclusion of BLSCFG & this breakage are more 'drastic' ... it already causes this not insignificant effort/workload.  Monkeying with each/every machine's grub cmd line to 'make it fit' this workaround is not a viable solution. Here.

(In reply to pgnet.dev from comment #48)
> > most users won't need such a long kernel cmdline
> 
> Perhaps, but that's an assumption.  It may be correct, but I don't see how
> it's particularly relevant to the point.
>

Right, it's an assumption indeed (which may even be wrong). The reason why I brought it was to weight between fixing this issue in F32, which as mentioned I think is to stop using the grubenv to store the cmdline.
 
> The command line is perfectly valid, unless GRUB &/or the kernel have a
> limit on cmd length?

Right, it's an artificial restriction of the current Fedora implementation. The Linux kernel documentation (https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html) says:

"The number of kernel parameters is not limited, but the length of the complete command line (parameters including spaces etc.) is limited to a fixed number of characters. This limit depends on the architecture and is between 256 and 4096 characters. It is defined in the file ./include/asm/setup.h as COMMAND_LINE_SIZE"

I've to check what that is for x86.

> If not, the 'assumption' can't be considered a reasons to rationalize the
> breakage.
> 
> The commands are there for operational reasons that vary to some extent by
> hardware.
>

Ok.
 
> > Maybe you could revisit some of the options that you are passing to the kernel? To make enough room in the grubenv for the other variables that are stored there, i.e: saved_entry, menu_auto_hide, and boot_{success,indeterminate}.
> 
> Sure, one can try that.  But the point is that that's a workround to the
> fact of the problem; which itself remains unfixed; Simply, it's broken for
> the moment.
>

Yes, I didn't mean to claim otherwise.

> And such a workaround is, here, not a solution I can entertain, or maintain,
> for 1K+ machines; not that they're not all Fedora32 ... today.
> 
> A far simpler 'workaround' -- seems to be to DISABLE BLSCFG, i.e. set
> "GRUB_ENABLE_BLSCFG=false".  still not ideal, but simpler.  With that, the
> greubenv error, at least, ceases.
> 
> >  but I think that is a drastic change that shouldn't be made after a Fedora version has been released.
> 
> I hear you, and understand the argument.  And know that the decision 'for
> F32' will be made on its own grounds.
>

My worry is what I mentioned, that this change even if being the proper fix for your use case could introduce regressions for other users that have a smaller cmdline and not hitting this bug. But making the change in Rawhide/F33 we make sure to have enough time and testing to catch and fix those issues before pushing the change to a released Fedora version.
 
> Here, the inclusion of BLSCFG & this breakage are more 'drastic' ... it
> already causes this not insignificant effort/workload.  Monkeying with
> each/every machine's grub cmd line to 'make it fit' this workaround is not a
> viable solution. Here.

Sorry again about this issue, and disabling BLS support is also another workaround as you said. Just keep in mind that if you choose to do that, you will need to install the grubby-deprecated package that contains the old grubby tool used to modify the non-BLS grub configuration.

> My worry is what I mentioned, that this change even if being the proper fix for your use case could introduce regressions

sure.

tho, the current situation is (IIUC?):

-- valid-but-long cmd line used to work just fine on Fedora b4 BLS
-- BLS was introduced, & switched on as default, triggering this^ problem

'worked b4, but doesn't now' _is_ a regression itself.  at least that's my $0.02.

and tbh, I don't yet know what Fedora policy _is_ on regressions: are they fixed in existing/stable release? or pushed to next release? etc ...

> you will need to install the grubby-deprecated package

wasn't even *aware* of 'grubby tool' until you mentioned it.

sigh.  more wrapper 'cruft' ;-)

sounds like best _available_ option is then to disable BLS + downgrade to grubby-deprecated.

i need to poke around and see if that's a migration deal-breaker for me, or not; i don't _think_ it should be ....

one question re: the grubby-deprecated recommenation; related, but admittedly off-topic ...

a 'grubby-deprecated' INSTALL appear to leave 'grubby' in place.

can/should 'grubby' be REMOVED?

an attempt to do so rm's BOTH:


Removing:
 grubby                         x86_64                8.40-40.fc32 @anaconda                 62 k
Removing dependent packages:
 fips-mode-setup                noarch                20200619-1.git781bbd4.fc32                  @updates                 6.7 k

not sure what problems rm'ing either does cause; deleting 'fips*' _seems_ a bad idea ...

(In reply to pgnet.dev from comment #50)
> > My worry is what I mentioned, that this change even if being the proper fix for your use case could introduce regressions
> 
> sure.
> 
> tho, the current situation is (IIUC?):
> 
> -- valid-but-long cmd line used to work just fine on Fedora b4 BLS
> -- BLS was introduced, & switched on as default, triggering this^ problem
> 
> 'worked b4, but doesn't now' _is_ a regression itself.  at least that's my
> $0.02.
>

Yes, I agree with you that's a regression and that's why we are changing in F33 (among other reasons).
 
> and tbh, I don't yet know what Fedora policy _is_ on regressions: are they
> fixed in existing/stable release? or pushed to next release? etc ...
>

Regressions should be fixed in my opinion, the problem is when the fix is to change how the configuration works where the risk to introduce other regressions may be high. Or maybe I'm over cautious here since the change dropping the kenrelopts variable has been in Rawhide for two months now and there weren't issues reported. So yes, it might make sense to backport this for F32.
 
> > you will need to install the grubby-deprecated package
> 
> wasn't even *aware* of 'grubby tool' until you mentioned it.
> 
> sigh.  more wrapper 'cruft' ;-)
> 
> sounds like best _available_ option is then to disable BLS + downgrade to
> grubby-deprecated.
> 
> i need to poke around and see if that's a migration deal-breaker for me, or
> not; i don't _think_ it should be ....

Yeah, getting rid of the grubby cruft was the main motivation for the migration to BLS, so we could have a consistent configuration across all the bootloaders and arches supported by Fedora.

I now realized that there is a better workaround for you. And that is to not set the kernelopts in the grubenv at all. Since the grub.cfg already sets a default kernelopts as a fallback if there isn't one present in the grubenv file, you don't really need one.

The convenience of having that variable in the grubenv was to allow changing the kernel cmdline without the need to modify the grub.cfg, but that came with more disadvantages than advantages.

So you could do something like the following:

$ sudo grub2-editenv create

$ sudo grub2-mkconfig --no-grubenv-update -o /boot/efi/EFI/fedora/grub.cfg

$ sudo cat /boot/efi/EFI/fedora/grubenv | grep "^#*$" | wc -c
1000

The drawback is that if you change a kernel command line parameter the grub.cfg will need to be re-generated again. But if I understood correctly that is what you where doing anyways.

(In reply to pgnet.dev from comment #51)
> one question re: the grubby-deprecated recommenation; related, but
> admittedly off-topic ...
> 
> a 'grubby-deprecated' INSTALL appear to leave 'grubby' in place.
> 
> can/should 'grubby' be REMOVED?
> 
> an attempt to do so rm's BOTH:
> 
> 
> Removing:
>  grubby                         x86_64                8.40-40.fc32 @anaconda
> 62 k
> Removing dependent packages:
>  fips-mode-setup                noarch               
> 20200619-1.git781bbd4.fc32                  @updates                 6.7 k
> 
> not sure what problems rm'ing either does cause; deleting 'fips*' _seems_ a
> bad idea ..

Yes, don't install grubby-deprecated and instead use the workaround I mentioned in Comment 52. I think that's the best approach and will also play nice with F33 when you upgrade (or if the change gets pushed to F32) since when that happens the kernelopts variable won't be used anymore anyways (because the cmdline will be in the BLS files instead of using GRUB env vars).

edit

-	GRUB_ENABLE_BLSCFG=false
+	GRUB_ENABLE_BLSCFG=true

delete 'grubby deprecated'

exec

	grub2-switch-to-blscfg
	grub2-editenv create
	grub2-mkconfig --no-grubenv-update -o /boot/efi/EFI/fedora/grub.cfg

		Generating grub configuration file ...
		Adding boot menu entry for EFI firmware configuration
		done

no more error.

sanity check

	cat /boot/efi/EFI/fedora/grubenv | grep "^#*$" | wc -c
		1000
	shutdown -r now

after reboot, all's good. so that seems to work as a workaround.

or, at least, there's no immediately visible problem either at the -mkconfig exec, or in the nicely-verbose boot-log/dmesg output my "wow that's long" cmd line provides! ;-)

one thing I _do_ notice is, immediately after that^ reboot,

	cat /boot/efi/EFI/fedora/grubenv | grep "^#*$" | wc -c
		985

not sure what/why the grubenv content length _changes_.
iiuc, simply rm'ing the grubenv mod from the loop _should_ have no negative effect ...

> But if I understood correctly that is what you where doing anyways.

yep.  that do-it-manually scripting is a carryover behavior from flaky dracut releases, borked mobo EFI, etc etc.

(In reply to pgnet.dev from comment #54)

[snip]

> 
> one thing I _do_ notice is, immediately after that^ reboot,
> 
> 	cat /boot/efi/EFI/fedora/grubenv | grep "^#*$" | wc -c
> 		985
> 
> not sure what/why the grubenv content length _changes_.

I think that changes because the grub-boot-success.service setting the boot_success variable or GRUB setting the boot_indeterminate variable.

> iiuc, simply rm'ing the grubenv mod from the loop _should_ have no negative
> effect ...
> 

Yes, which is the reason why the fallback kernelopts is set in the grub.cfg file, to prevent the boot entries to not have a cmdline if the grubenv is removed / corrupted. This again won't be an issue anymore once the cmdline is in the BLS snippets.

The only impact when the grubenv is removed would be on the default entry to boot (saved_entry variable) since that is stored there. In that case GRUB would just choose to boot the first one from the list which should be the latest installed kernel.

The 'workaround' appears to be fragile, or at least tempermental ...

On a new F32 VM, attempting to do the same as above, after F32 kernel update to 5.7.7, I _see_ the 5.7.7 menu entry in the grub menu, but it's not selected by default.  The prior 5.7.6 entry is.

Even though,

grubby --info=ALL
index=0
kernel="/boot/vmlinuz-5.7.7-200.fc32.x86_64"
args="$kernelopts"
initrd="/boot/initramfs-5.7.7-200.fc32.x86_64.img"
title="Fedora (5.7.7-200.fc32.x86_64) 32 (Thirty Two)"
id="b860b55ce0e44708acc76eb5e1f395b4-5.7.7-200.fc32.x86_64"
index=1
kernel="/boot/vmlinuz-5.7.6-201.fc32.x86_64"
args="$kernelopts"
initrd="/boot/initramfs-5.7.6-201.fc32.x86_64.img"
title="Fedora (5.7.6-201.fc32.x86_64) 32 (Thirty Two)"
id="b860b55ce0e44708acc76eb5e1f395b4-5.7.6-201.fc32.x86_64"
index=2
kernel="/boot/vmlinuz-5.6.19-300.fc32.x86_64"
args="$kernelopts"
initrd="/boot/initramfs-5.6.19-300.fc32.x86_64.img"
title="Fedora (5.6.19-300.fc32.x86_64) 32 (Thirty Two)"
id="b860b55ce0e44708acc76eb5e1f395b4-5.6.19-300.fc32.x86_64"
index=3
kernel="/boot/vmlinuz-0-rescue-b860b55ce0e44708acc76eb5e1f395b4"
args="$kernelopts"
initrd="/boot/initramfs-0-rescue-b860b55ce0e44708acc76eb5e1f395b4.img"
title="Fedora (0-rescue-b860b55ce0e44708acc76eb5e1f395b4) 32 (Thirty Two)"
id="b860b55ce0e44708acc76eb5e1f395b4-0-rescue"

grubby --default-kernel
  /boot/vmlinuz-5.7.7-200.fc32.x86_64

I *can* manually select 7 boot to 5.7.7,

  uname -rm
    5.7.7-200.fc32.x86_64 x86_64

but, it's not picking up _my_ cmd line, rather just the fallback,

cat /proc/cmdline
  BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.7.7-200.fc32.x86_64 root=/dev/mapper/VG_02-VG_02_ROOT ro resume=/dev/mapper/VG_02-VG_02_SWAP rd.lvm.lv=VG_02/VG_02_ROOT rd.lvm.lv=VG_02/VG_02_SWAP rhgb quiet

working on trying to figure out what's different/broken with _this_ current setup :-/

Ok, so there are two issues then:

1) the latest kernel not being picked

Do you have a saved_entry set in your grubenv?

2) the cmdline not being what you expected?

Do you have the correct cmdline in GRUB_CMDLINE_LINUX of your /etc/default/grub? What's the value of kernelopts in your grub.cfg ?

This message is a reminder that Fedora 32 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 32 on 2021-05-25.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '32'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 32 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

This package has changed maintainer in Fedora. Reassigning to the new maintainer of this component.

Fedora 32 changed to end-of-life (EOL) status on 2021-05-25. Fedora 32 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.