Bug 1693515 - boot failure after F29 to F30 dnf system upgrade, arg.c:362:missing mandatory option for `users`.
Summary: boot failure after F29 to F30 dnf system upgrade, arg.c:362:missing mandatory...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: grub2
Version: 30
Hardware: ppc64le
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Peter Jones
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-28 03:43 UTC by Chris Murphy
Modified: 2019-09-20 14:44 UTC (History)
10 users (show)

Fixed In Version: grub2-2.02-75.fc30
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1755815 (view as bug list)
Environment:
Last Closed: 2019-04-01 00:01:23 UTC
Type: Bug


Attachments (Terms of Use)
photo1 (62.31 KB, image/jpeg)
2019-03-28 03:45 UTC, Chris Murphy
no flags Details
photo2 (317.27 KB, image/jpeg)
2019-03-28 03:52 UTC, Chris Murphy
no flags Details
photo4 (325.77 KB, image/jpeg)
2019-03-28 03:58 UTC, Chris Murphy
no flags Details
photo3 (132.93 KB, image/jpeg)
2019-03-28 04:03 UTC, Chris Murphy
no flags Details
f30 grub.cfg (5.77 KB, text/plain)
2019-03-28 04:07 UTC, Chris Murphy
no flags Details

Description Chris Murphy 2019-03-28 03:43:38 UTC
Description of problem:

Following an upgrade from Fedora 29 to Fedora 30 using `dnf system-upgrade`, boot fails.

Version-Release number of selected component (if applicable):
grub2-ppc64le-2.02-72.fc30.ppc64le

How reproducible:
Always


Steps to Reproduce:
1. Fedora 29 minimal install, qemu-ppc/kvm
2. use `dnf system-upgrade` per https://fedoraproject.org/wiki/DNF_system_upgrade
3. system reboots, and upgrades as expected, reboots again and...

Actual results:

Minimal BASH-like line editing is support...

grub> _


(i.e. grub prompt, no boot menu)


Expected results:

Either a grub menu (it is first boot after an upgrade), or it should boot successfully if hidden grub menu feature applies here.

Additional info:


Very briefly, between the prepboot and GRUB menu appears are three lines (photo1) which I'm able to replicate by trying to manually load the grub.cfg using configfile (photo2).

lsmod (photo3)
set (photo4)

It looks like GRUB does find the grub.cfg but then fails to read or parse it for some reason.

Comment 1 Chris Murphy 2019-03-28 03:45:06 UTC
Created attachment 1548812 [details]
photo1

Comment 2 Chris Murphy 2019-03-28 03:52:24 UTC
Created attachment 1548813 [details]
photo2

Enable debug, then use configfile to try to manually load the grub.cfg.

Comment 3 Chris Murphy 2019-03-28 03:58:21 UTC
Created attachment 1548814 [details]
photo4

grub environment variables

Comment 4 Chris Murphy 2019-03-28 04:03:03 UTC
Created attachment 1548815 [details]
photo3

lsmod

Comment 5 Chris Murphy 2019-03-28 04:07:38 UTC
Created attachment 1548816 [details]
f30 grub.cfg

This is the Fedora 30 /boot/grub2/grub.cfg

Comment 6 Chris Murphy 2019-03-28 04:31:31 UTC
- manually booting with `linux` `initrd` and `boot` commands does work, so at least that much is working and seems isolated to the grub.cfg
- running `grub2-mkconfig -o /boot/grub2/grub.cfg` doesn't fix the problem
- running `grub2-install --no-nvram --no-floppy /dev/vda1` does fix the problem, but I don't know why: lsmod shows the same modules loaded, and set shows the same environment variables (although timeout=5 is missing now for some reason, not sure if it's the grub reinstall or recreating the grub.cfg, anyway, unrelated)

Comment 7 Javier Martinez Canillas 2019-03-28 13:38:19 UTC
Hello Chris,

Thanks a lot for the report.

(In reply to Chris Murphy from comment #6)
> - manually booting with `linux` `initrd` and `boot` commands does work, so
> at least that much is working and seems isolated to the grub.cfg

You are correct, configfile /grub2/grub.cfg.rpmsave also works

> - running `grub2-mkconfig -o /boot/grub2/grub.cfg` doesn't fix the problem
> - running `grub2-install --no-nvram --no-floppy /dev/vda1` does fix the
> problem, but I don't know why: lsmod shows the same modules loaded, and set
> shows the same environment variables (although timeout=5 is missing now for
> some reason, not sure if it's the grub reinstall or recreating the grub.cfg,
> anyway, unrelated)

The problem is the --users $grub_users option in the menu entries. The --users option is used to restrict the access to specific menu entries only to a set of users. But the option requires an argument to either be a constant or a variable that has been set. So for example the following:
    
      menuentry "May be run by superusers or users in $users" --users $users {
                linux /vmlinuz
      }

Would fail if $users is not defined and GRUB would discard the menu entry (which happens in this bug and that's why you see the error messages and there are no entries in the menu). Instead GRUB should allow the --users option to have an optional argument and ignore the option if the argument was not set.

That was fixed in GRUB and that's why it works on a fresh F30 install, but it's not fixed in the F29 GRUB. And since we don't update the GRUB core on a package install, it fails on a system upgrade.

But looking at the F29 grub2.cfg, the menu entries don't have the --users option anyways and just the --unrestricted one. So we will just do the same in F30 to prevent this issue and allow it to work with older GRUB installations that don't have the fix for the --users option.


NOTE: You might be wondering why we have the menu entries in the GRUB config file and are not using the blscfg module for ppc64le? The answer is that we only have GRUB support for ppc64le on a VM (PowerVM) which uses the Open Firmware interface. On ppc64le bare metal (PowerNV) the OPAL firmware interface is used, and the bootloader ins't GRUB in that case but a Linux kernel with a built-in initramfs that contains a user-space bootloader (Petitboot).

We added BLS support to Petitboot, but that's fairly recent and found that many machines are still running old versions of the firmware that don't have the BLS support. Since the OPAL firmware is not controlled by us and updates are provided by the hardware vendor, for now what we do is to generate a grub2.cfg section by using the BLS snippets in /boot/loader/entries (similar to what ostree still does). That way it works with any OPAL firmware version.

Comment 8 Javier Martinez Canillas 2019-03-28 16:33:08 UTC
Please test installing the following grub2-2.02-75.fc30 build [0] and then re-generate your grub2.cfg with `grub2-mkconfig -o /boot/grub2/grub.cfg`.

[0]: build https://koji.fedoraproject.org/koji/taskinfo?taskID=33804676

Comment 9 Chris Murphy 2019-03-28 17:03:40 UTC
(In reply to Javier Martinez Canillas from comment #8)
> Please test installing the following grub2-2.02-75.fc30 build [0] and then
> re-generate your grub2.cfg with `grub2-mkconfig -o /boot/grub2/grub.cfg`.

After doing this, the VM boots successfully.

Comment 10 Fedora Update System 2019-03-29 16:45:10 UTC
grub2-2.02-75.fc30 systemd-241-4.gitcbf14c9.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-616045ca76

Comment 11 Fedora Update System 2019-03-29 20:32:37 UTC
grub2-2.02-75.fc30, systemd-241-4.gitcbf14c9.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-616045ca76

Comment 12 Fedora Update System 2019-04-01 00:01:23 UTC
grub2-2.02-75.fc30, systemd-241-4.gitcbf14c9.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 13 az.simple.az.that 2019-05-02 22:26:10 UTC
I can confirm the boot fail exactly as described above. Just finished my ugrade to FC30...

Can some friendly soul out there give me a heads-up on how to fix it. (Step by step please). I am not very savy with the grub2 boot loader menu and its options. 

Thanks in advance.

Comment 14 az.simple.az.that 2019-05-02 23:14:16 UTC
I was able to detect what grub2 has problems with:

After the bios messages it says:

GRUB loading.
Welcome to GRUB!

error: file '/grub2/locale/en.mo.gz' not found
_


The above it present for a short split second and then the machine reboots.

I am not even able to get to the grub2 prompt.

Comment 15 az.simple.az.that 2019-05-03 06:00:49 UTC
It looks like a legacy bug ... This one: https://bugzilla.redhat.com/show_bug.cgi?id=817187

Comment 16 az.simple.az.that 2019-05-03 08:54:13 UTC
My initial situation was boot failure after F29 to F30 dnf system upgrade.

The system this concerns has been upgraded since at least Fedora 17. It had reached Fedora 29 without problems. It is the first time I have encountered a boot problem with it after an upgrade. Hence my posts here.

It can be fixed, and there is a relation to this one: https://bugzilla.redhat.com/show_bug.cgi?id=817187.  

The fix for this boot problem after having upgraded my Fedora 29 to Fedora 30 was to boot the system off an USB CentOS 7 install image and go to "rescue a CentOS system" (that's the rescue boot I had available)

Then a chroot /mnt/sysimage (then you are at your Fedora 30 installation)

I then did:

0) cp /boot/grub2/locale/en@quot.mo /boot/grub2/locale/en.mo as adviced in bug 817187 comment 28 since that was the likely fix for the initial symptom BUT: Boot still failed hence ...
1) Edit /etc/default/grub.cfg and added the line GRUB_TERMINAL_OUTPUT=console as adviced in bug 817187 comment 16. (also removed the "rhgb quite" from the kernel boot options)
2) grub2-mkconfig -o /boot/grub2/grub.cfg
3) lsblk to check the device (mine was /dev/sda)
4) grub2-install  .. reinstalled grub2 on my system. 

The above is like a vanilla rescue for grub2 and it worked. 

After having reinstalled grub2 as described above in 0) to 4) I was able to boot the system successfully into Fedora 30 (which then relabelled and rebooted once more which took a long time on this older host)

Comment 17 Kåre Løvgren 2019-05-05 13:57:12 UTC
I ran into the same problem when updating F29 to F30 yesterday. After lots of anxiety and searching the net (on another PC), I found the solution here:
  https://fedoraproject.org/wiki/Common_F30_bugs#GRUB_boot_menu_is_not_populated_after_an_upgrade
and it saved my day :-)

It would have been nice to know _before_ updating that I should have updated GRUB...
(this PC started its life on F20, and all other updates have caused no problems)

Comment 18 zingale 2019-05-06 18:26:28 UTC
Same problem here.  I'll try the solution above, but dnf system update should check for this.

Comment 19 zingale 2019-05-06 18:33:56 UTC
That solution from the Common_F30_bugs works for me too.  Thanks for sharing.

Comment 20 Amit 2019-05-07 16:04:44 UTC
I am facing the same problem.....I am unable to boot after upgrade to fedora 30........ Can any body specifies steps to recover this error.....

I do not have any bootable disk.... Moreover my system is dual boot

Any help is highly appreciated

Comment 21 Chris Murphy 2019-05-07 17:11:31 UTC
Amit, bugs aren't for troubleshooting or support. This bug has been fixed. See comment 17 for the link to Common Bugs which has the specific steps you seek. If you need more help, check out either the user's list or askfedora.org

https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org/

Comment 22 Dominique Brazziel 2019-05-09 20:41:57 UTC
Actually, the bug has not been fixed. I just did a system upgrade from 29 to 30 today and got his with this bug. The Common F30 problems page gives a workaround for the problem, but doesn't reference a bug number or this bug in particular. As the the poster in Comment 17 said "It would have been nice to know _before_ updating that I should have updated GRUB...", and the post in Comment 18 said "dnf system update should check for this.".

Comment 23 Dominique Brazziel 2019-05-09 20:47:00 UTC
Bug says "Fixed in Version grub2-2.02-75.fc30" 

rpm -qi grub2-common
Name        : grub2-common
Epoch       : 1
Version     : 2.02
Release     : 78.fc30
Architecture: noarch
Install Date: Thu 09 May 2019 11:17:38 AM EDT
Group       : Unspecified
Size        : 5041036
License     : GPLv3+
Signature   : RSA/SHA256, Thu 18 Apr 2019 01:41:51 PM EDT, Key ID ef3c111fcfc659b9
Source RPM  : grub2-2.02-78.fc30.src.rpm
Build Date  : Thu 18 Apr 2019 01:39:47 PM EDT
Build Host  : buildvm-armv7-08.arm.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : http://www.gnu.org/software/grub/
Bug URL     : https://bugz.fedoraproject.org/grub2
Summary     : grub2 common layout
Description :
This package provides some directories which are required by various grub2
subpackages.

Comment 24 Chris Murphy 2019-05-09 21:02:08 UTC
(In reply to Dominique Brazziel from comment #22)
> Actually, the bug has not been fixed. I just did a system upgrade from 29 to
> 30 today and got his with this bug. The Common F30 problems page gives a
> workaround for the problem, but doesn't reference a bug number or this bug
> in particular. 

Common F30 bugs page does list the proper bug right below the title:
🔗 GRUB boot menu is not populated after an upgrade
link to this item - Bugzilla: #1652806 

The bug you're running into is that bug, not this one.

>As the the poster in Comment 17 said "It would have been nice
> to know _before_ updating that I should have updated GRUB...", and the post
> in Comment 18 said "dnf system update should check for this.".

There's no practical way for dnf system-update to check for this, all we could do is change the present policy and always overwrite BIOS GRUB on major upgrades. This is being discussed on the devel@ list.

The way to know to update GRUB in advance is to read the release notes including Common Bugs in advance. That's always been recommended on Fedora since I can remember.

Comment 25 Dominique Brazziel 2019-05-09 21:32:55 UTC
The 'Common Bugs' page is not referenced in the Fedora 30 Release Notes:

https://docs.fedoraproject.org/en-US/fedora/f30/release-notes/index.html

nor in the manual page for 'dnf.plugin.system-upgrade'.  It used to be mentioned in the documentation for 'fedup', and I used to review the bugs religiously. A link to a 'Common F(releasver) Bugs' should be included in the 'dnf.plugin.system-upgrade' man page.

Comment 26 Chris Murphy 2019-05-09 22:02:48 UTC
It's buried in the Feedback section and I agree it needs to be made more prominent.
https://docs.fedoraproject.org/en-US/fedora/f30/release-notes/welcome/Feedback/
https://pagure.io/fedora-docs/release-notes/issue/337

Comment 27 Amit 2019-05-10 08:40:47 UTC
I am unable to boot my fedora 30 machine right now. One solution reported here is in Comment 17. But, for that I need to be in grub prompt (and it comes by booting the fedora 30 in rescue mode, I THINK....). I tried creating fedora 30 live USB and booted from it. But, in troubleshooting menu, there are no option regarding "Rescue Mode". I have downloaded and created fedora 30 live USB from the address https://download.fedoraproject.org/pub/fedora/linux/releases/30/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-30-1.2.iso (It does not show rescue mode option).

I have some urgent data created on fedora 29 (before upgrade). I have not taken any backup and clueless right now. Kindly help me making my system in working condition.

Thanks in advance.

Comment 28 Javier Martinez Canillas 2019-05-10 09:26:33 UTC
(In reply to Amit from comment #27)
> I am unable to boot my fedora 30 machine right now. One solution reported
> here is in Comment 17. But, for that I need to be in grub prompt (and it
> comes by booting the fedora 30 in rescue mode, I THINK....). I tried
> creating fedora 30 live USB and booted from it. But, in troubleshooting
> menu, there are no option regarding "Rescue Mode". I have downloaded and
> created fedora 30 live USB from the address
> https://download.fedoraproject.org/pub/fedora/linux/releases/30/Workstation/
> x86_64/iso/Fedora-Workstation-Live-x86_64-30-1.2.iso (It does not show
> rescue mode option).
> 
> I have some urgent data created on fedora 29 (before upgrade). I have not
> taken any backup and clueless right now. Kindly help me making my system in
> working condition.
> 
> Thanks in advance.

This bugzilla is for a bug that was only relevant to the ppc64le architecture. You likely are hitting Bug #1652806 instead.

If you have a GRUB prompt, then you can follow the instructions in https://fedoraproject.org/wiki/Common_F30_bugs#GRUB_boot_menu_is_not_populated_after_an_upgrade.

I understand that you don't even have a GRUB prompt? In that case, then you could start a Live Image (i.e: https://download.fedoraproject.org/pub/fedora/linux/releases/30/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-30-1.2.iso) and use a terminal to cp your grub.cfg.rpmsave to grub.cfg. Something like the following:

$ mkdir /mnt/boot && mount /dev/sda1 /mnt/boot
$ cp /mnt/boot/grub2/grub.cfg.rpmsave /mnt/boot/grub2/grub.cfg
$ umount /mnt/boot && reboot

Comment 29 Chris Murphy 2019-05-10 16:53:42 UTC
(In reply to Amit from comment #27)
> I have some urgent data created on fedora 29 (before upgrade). I have not
> taken any backup and clueless right now. Kindly help me making my system in
> working condition.

Amit, like I said in comment 21, bug reports aren't for support, they're for progressing the bug. This bug is fixed. Only dvd and netinstall images have "Rescue a Fedora system" option, not lives. If you need more help you need to check out forums intended for help.

ask.fedoraproject.org
IRC freenode.net #fedora
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org/

Comment 30 omashkoozinini 2019-09-20 14:44:23 UTC
I had boot issues with FC 30 too.

First got caught by the legacy problem of the missing en.mo file as in 817187 comment 4.

Booted to FC 30 with DVD, mounted my boot partition (which for me is /dev/sda1) and ran

  'cp /usr/share/locale/en@quot/LC_MESSAGES/grub.mo /boot/grub2/locale/en.mo'

to get past the missing file issue.

Still wouldn't boot so I followed up with comment 28 above as follows:

$ mkdir /mnt/boot && mount /dev/sda1 /mnt/boot
$ cp /mnt/boot/grub2/grub.cfg.rpmsave /mnt/boot/grub2/grub.cfg
$ umount /mnt/boot && reboot

and voila - it lives!

Just copying a couple of files around using a bootable DVD saved my bacon. Thanks everyone...


Note You need to log in before you can comment on or make changes to this bug.