Proposed as a Blocker for 29-final by Fedora user mtasaka using the blocker tracking app because:

 All release-blocking images must boot in their supported configurations.

That's a Windows screen, by the looks of it. Is this a dual-boot install? How did you initially install the OSes on it?

Adam, this is looks like a dup of my bug 1626862 (I've experienced the same outcome after upgrading to Fedora 29).

Apparently, some UEFIs try (but fail) to boot something else if default option does not work (which is the case if user had GRUB as default and it became broken by upgrading to fc29 grub packages).

mtasaka, does rolling back to FC28 versions fix the issue for you? Does booting Windows directly from UEFI Boot Menu work?

Well, initially this was Windows 10 machine, but (I thought) I (almost?) deleted it by install Fedora:

# LC_ALL=C fdisk -l
Disk /dev/sda: 238.5 GiB, 256060514304 bytes, 500118192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: B16244E8-A000-4051-859C-1E1EF731A5C6

Device        Start       End   Sectors   Size Type
/dev/sda1      2048    534527    532480   260M EFI System
/dev/sda2    534528   2631679   2097152     1G EFI System
/dev/sda3   2631680  13117439  10485760     5G Linux filesystem
/dev/sda4  13117440 500117503 487000064 232.2G Linux LVM

And /dev/sda2 is mounted as /boot/efi . /dev/sda1 is perhaps some garbage (some Windows file left? maybe some backup files are left on /dev/sda1)


> mtasaka, does rolling back to FC28 versions fix the issue for you? 
Yes, as I wrote in comment 0 .

> Does booting Windows directly from UEFI Boot Menu work?
Well, as I said (I think) I deleted Windows 10, however some backup files may be left on /dev/sda1 .

Is it possible for you to attach program.log from the installed image?  My thinking here is that somehow efibootmgr failed (or the firmware didn't honor it), and the firmware is running the windows boot recovery from the "first" EFI System Partition.

Mamoru: that is, try and mount the installed system, and attach the file /var/log/anaconda/program.log from it.

However...pjones, Mamoru says this broke "After upgrading from Fedora 28 to Fedora 29". That implies it was working fine after it was first installed. So I don't think program.log is going to be relevant? It seems like somehow F29 grub2 dies on this 'multiple ESP' situation, but F28 grub2 is OK with it.

I can try reproducing this in a VM today...

Well, I kinda tried to reproduce this by installing F27 and using custom partitioning to create a 260MB /dev/vda1 EFI system partition mounted as '/otherefi', a 1GB /dev/vda2 EFI system partition mounted as /boot/efi , and the rest of the disk as a plain ext4 fs mounted as / . This booted fine with F27. I updated grub packages to the latest F28 stable grub2 - 2.02-38 - and the system continued to boot fine. I then updated grub packages plus dependencies (efi-rpm-macros and grubby) to the latest F29 stable - 2.02-59...and the system *still* boots fine.

So there must be something somehow different to that going on, whether it's the initial deployment or the firmware or what, I'm not sure. I was testing in a qemu VM with firmware from edk2-ovmf-20180815gitcb5f4f45ce-1.fc29.noarch .

Discussed during the 2018-10-01 blocker review meeting: [1]

The decision to delay the classification of this as a blocker bug was made as we are homing in on what's going on in this case (likely to do with multiple ESPs), but still need a bit more info to determine how significant it's likely to be. We will try to reproduce this independently and investigate further.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2018-10-01/f29-blocker-review.2018-10-01-16.00.txt

Well, as Adam said, I upgraded from F-28 to F-29 and program.log (by anaconda, okay?) is much older, is that okay?

Anyway once showing (perhaps) related part from (rather old) /var/log/anaconda/program.log:

21:18:54,432 INFO program: Running... new-kernel-pkg --rpmposttrans 4.8.6-300.fc25.x86_64
21:19:26,800 INFO program: Initializing machine ID from random generator.
21:19:26,800 DEBUG program: Return code: 0
21:19:26,985 INFO program: Running... efibootmgr
21:19:27,007 INFO program: BootCurrent: 0006
21:19:27,008 INFO program: Timeout: 0 seconds
21:19:27,008 INFO program: BootOrder: 0004,0005,0006
21:19:27,009 INFO program: Boot0004* Windows Boot Manager
21:19:27,009 INFO program: Boot0005* Solid State Disk
21:19:27,009 INFO program: Boot0006* USB Drive (UEFI) - TEAC DV-W28S-W L.0A
21:19:27,010 DEBUG program: Return code: 0
21:19:27,011 INFO program: Running... efibootmgr -c -w -L Fedora -d /dev/sda -p 2 -l \EFI\fedora\shim.efi
21:19:27,039 INFO program: BootCurrent: 0006
21:19:27,039 INFO program: Timeout: 0 seconds
21:19:27,040 INFO program: BootOrder: 0000,0004,0005,0006
21:19:27,040 INFO program: Boot0004* Windows Boot Manager
21:19:27,040 INFO program: Boot0005* Solid State Disk
21:19:27,040 INFO program: Boot0006* USB Drive (UEFI) - TEAC DV-W28S-W L.0A
21:19:27,040 INFO program: Boot0000* Fedora
21:19:27,040 DEBUG program: Return code: 0
21:19:27,042 INFO program: Running... grub2-set-default 0
21:19:27,070 DEBUG program: Return code: 0
21:19:27,071 INFO program: Running... grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
21:19:29,488 INFO program: Generating grub configuration file ...
21:19:29,489 INFO program: Found linux image: /boot/vmlinuz-4.8.6-300.fc25.x86_64
21:19:29,489 INFO program: Found initrd image: /boot/initramfs-4.8.6-300.fc25.x86_64.img
21:19:29,489 INFO program: Found linux image: /boot/vmlinuz-0-rescue-4444d860031845eb9bea12a137c4c220
21:19:29,489 INFO program: Found initrd image: /boot/initramfs-0-rescue-4444d860031845eb9bea12a137c4c220.img
21:19:29,489 INFO program: Found Windows Boot Manager on /dev/sda1@/EFI/Microsoft/Boot/bootmgfw.efi
21:19:29,489 INFO program: done

And currently working efibootmgr-16-2.fc28.x86_64 reports:
# efibootmgr -v
BootCurrent: 0000
Timeout: 0 seconds
BootOrder: 0000,0004,0005,0007,0008
Boot0000* Fedora	HD(2,GPT,3e92f9ad-b65a-4efd-b290-500cae67462e,0x82800,0x200000)/File(\EFI\fedora\shim.efi)
Boot0004* Windows Boot Manager	HD(1,GPT,25f1da69-d7d0-4dd5-9952-9bb55c7f9705,0x800,0x82000)/File(\EFI\Microsoft\Boot\bootmgfw.efi)WINDOWS.........x...B.C.D.O.B.J.E.C.T.=.{.9.d.e.a.8.6.2.c.-.5.c.d.d.-.4.e.7.0.-.a.c.c.1.-.f.3.2.b.3.4.4.d.4.7.9.5.}...9................
Boot0005* Solid State Disk	PciRoot(0x0)/Pci(0x17,0x0)/Sata(1,65535,0)/HD(1,GPT,25f1da69-d7d0-4dd5-9952-9bb55c7f9705,0x800,0x82000)..BO
Boot0007* Solid State Disk	PciRoot(0x0)/Pci(0x17,0x0)/Sata(1,65535,0)/HD(2,GPT,3e92f9ad-b65a-4efd-b290-500cae67462e,0x82800,0x200000)..BO
Boot0008* USB Drive (UEFI) - TEAC DV-W28S-W L.0A	PciRoot(0x0)/Pci(0x14,0x0)/USB(2,0)/CDROM(1,0x712d,0x4664)..BO

By the way when I use linux-host VirtualBox-5.2.18-1.fc29.x86_64 and use Fedora-Workstation-Live-x86_64-29-20180929.n.0.iso as guest, it boots fine (on VM). So maybe this is real-machine specific??

Oh, after upgrading to shim-x64-15-5.x86_64, (and this also upgrades efibootmgr to 16-4.fc29)

# rpm -ql shim-x64-15-5.x86_64
/boot/efi/EFI/BOOT/BOOTX64.EFI
/boot/efi/EFI/BOOT/fbx64.efi
/boot/efi/EFI/fedora/BOOTX64.CSV
/boot/efi/EFI/fedora/mmx64.efi
/boot/efi/EFI/fedora/shimx64-fedora.efi
/boot/efi/EFI/fedora/shimx64.efi

So there is no /boot/efi/EFI/fedora/shim.efi . but efibootmgr to 16-4.fc29 still says:
# efibootmgr -v
BootCurrent: 0000
Timeout: 0 seconds
BootOrder: 0000,0004,0005,0007,0008
Boot0000* Fedora	HD(2,GPT,3e92f9ad-b65a-4efd-b290-500cae67462e,0x82800,0x200000)/File(\EFI\fedora\shim.efi) <=======================
Boot0004* Windows Boot Manager	HD(1,GPT,25f1da69-d7d0-4dd5-9952-9bb55c7f9705,0x800,0x82000)/File(\EFI\Microsoft\Boot\bootmgfw.efi)WINDOWS.........x...B.C.D.O.B.J.E.C.T.=.{.9.d.e.a.8.6.2.c.-.5.c.d.d.-.4.e.7.0.-.a.c.c.1.-.f.3.2.b.3.4.4.d.4.7.9.5.}...9................
Boot0005* Solid State Disk	PciRoot(0x0)/Pci(0x17,0x0)/Sata(1,65535,0)/HD(1,GPT,25f1da69-d7d0-4dd5-9952-9bb55c7f9705,0x800,0x82000)..BO
Boot0007* Solid State Disk	PciRoot(0x0)/Pci(0x17,0x0)/Sata(1,65535,0)/HD(2,GPT,3e92f9ad-b65a-4efd-b290-500cae67462e,0x82800,0x200000)..BO
Boot0008* USB Drive (UEFI) - TEAC DV-W28S-W L.0A	PciRoot(0x0)/Pci(0x14,0x0)/USB(2,0)/CDROM(1,0x712d,0x4664)..BO

So maybe this is the reason??

For reference:
# rpm -qlp shim-x64-15-2.x86_64.rpm
/boot/efi
/boot/efi/EFI
/boot/efi/EFI/BOOT
/boot/efi/EFI/BOOT/BOOTX64.EFI
/boot/efi/EFI/BOOT/fallback.efi
/boot/efi/EFI/BOOT/fbx64.efi
/boot/efi/EFI/fedora
/boot/efi/EFI/fedora/BOOT.CSV
/boot/efi/EFI/fedora/BOOTX64.CSV
/boot/efi/EFI/fedora/MokManager.efi
/boot/efi/EFI/fedora/mmx64.efi
/boot/efi/EFI/fedora/shim.efi
/boot/efi/EFI/fedora/shimx64-fedora.efi
/boot/efi/EFI/fedora/shimx64.efi

So now I additionally installed shim-ia32-15-5.x86_64, and now my machine boots fine!!!

And it seems that other people are reporting the same issue:

https://bugzilla.redhat.com/show_bug.cgi?id=1631989

I see the same issue: After a system upgrade from F28 to F29, my system does not boot any more. I have not reported the issue because I had no time nor data to dig deep into what caused the issue.

(In reply to Mamoru TASAKA from comment #12)
> So now I additionally installed shim-ia32-15-5.x86_64, and now my machine
> boots fine!!!

Same workaround here. The system needs shim-ia32 to reach grub2 boot menu. This was different on Fedora 28 where I did not have shim-ia32 installed.

OK, yeah. Let's make this a dupe of that. Looks like the issue's actually quite straightforward...

*** This bug has been marked as a duplicate of bug 1631989 ***