Bug 1260799

Created attachment 1071131 [details]
File: anaconda-tb

Created attachment 1071132 [details]
File: anaconda.log

Created attachment 1071133 [details]
File: environ

Created attachment 1071134 [details]
File: journalctl

Created attachment 1071135 [details]
File: lsblk_output

Created attachment 1071136 [details]
File: nmcli_dev_list

Created attachment 1071137 [details]
File: os_info

Created attachment 1071138 [details]
File: program.log

Created attachment 1071139 [details]
File: storage.log

Created attachment 1071140 [details]
File: ifcfg.log

Created attachment 1071141 [details]
File: packaging.log

Okay, so in the kernel log we see:
Sep 07 16:32:20 localhost kernel: Initializing cgroup subsys cpuset
Sep 07 16:32:20 localhost kernel: Initializing cgroup subsys cpu
Sep 07 16:32:20 localhost kernel: Initializing cgroup subsys cpuacct
Sep 07 16:32:20 localhost kernel: Linux version 4.2.0-1.fc23.x86_64 (mockbuild.fedoraproject.org) (gcc version 5.1.1 20150618 (Red Hat 5.1.1-4) (GCC) ) #1 SMP Mon Aug 31 15:57:27 UTC 2015
Sep 07 16:32:20 localhost kernel: Command line: BOOT_IMAGE=/isolinux/vmlinuz0 root=live:LABEL=Fedora-Live-WS-x86_64-23-2015090 ro rd.live.image quiet rhgb
Sep 07 16:32:20 localhost kernel: tseg: 00bff00000
Sep 07 16:32:20 localhost kernel: x86/fpu: Legacy x87 FPU detected.
Sep 07 16:32:20 localhost kernel: x86/fpu: Using 'lazy' FPU context switches.
Sep 07 16:32:20 localhost kernel: e820: BIOS-provided physical RAM map:
Sep 07 16:32:20 localhost kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
Sep 07 16:32:20 localhost kernel: BIOS-e820: [mem 0x0000000000100000-0x00000000bf68efff] usable
Sep 07 16:32:20 localhost kernel: BIOS-e820: [mem 0x00000000bf68f000-0x00000000bf907fff] reserved
Sep 07 16:32:20 localhost kernel: BIOS-e820: [mem 0x00000000bf908000-0x00000000bfb0afff] ACPI NVS
Sep 07 16:32:20 localhost kernel: BIOS-e820: [mem 0x00000000bfb0b000-0x00000000bfd0efff] usable
Sep 07 16:32:20 localhost kernel: BIOS-e820: [mem 0x00000000bfd0f000-0x00000000bfd5cfff] ACPI NVS
Sep 07 16:32:20 localhost kernel: BIOS-e820: [mem 0x00000000bfd5d000-0x00000000bfd65fff] ACPI data
Sep 07 16:32:20 localhost kernel: BIOS-e820: [mem 0x00000000bfd66000-0x00000000bfd68fff] ACPI NVS
Sep 07 16:32:20 localhost kernel: BIOS-e820: [mem 0x00000000bfd69000-0x00000000bfd69fff] usable
...
Sep 07 16:32:20 localhost kernel: efi: EFI v2.00 by American Megatrends
Sep 07 16:32:20 localhost kernel: efi:  SMBIOS=0xbf906a18  ACPI=0xbfd5d000  ACPI 2.0=0xbfd5d000  MPS=0xfcaf0
...
Sep 07 16:32:20 localhost kernel: efifb: probing for efifb
Sep 07 16:32:20 localhost kernel: efifb: framebuffer at 0xc0000000, mapped to 0xffffc90001000000, using 1876k, total 1875k
Sep 07 16:32:20 localhost kernel: efifb: mode is 800x600x32, linelength=3200, pages=1
Sep 07 16:32:20 localhost kernel: efifb: scrolling: redraw
Sep 07 16:32:20 localhost kernel: efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
Sep 07 16:32:20 localhost kernel: Console: switching to colour frame buffer device 100x37
Sep 07 16:32:20 localhost kernel: fb0: EFI VGA frame buffer device
...
Sep 07 16:32:20 localhost kernel: fb: switching to radeondrmfb from EFI VGA
...

What's happening here is this: the kernel is being called from the EFI entry point, and it's finding the EFI system table, so the EFI stub can do things like set the video mode parameters and such, but for whatever reason, it never actually finds the EFI memory map.  So it creates /sys/firmware/efi , and rightly so, but efi_enabled(EFI_RUNTIME_SERVICES) inside the kernel is never set to true, so /sys/firmware/efi/efivars is never mounted.

So in this case anaconda has done the right thing - this is an EFI machine, but also efibootmgr is correct - the machine, as exposed to userland, does not support EFI variables.

I think this is either a kernel or firmware bug; either the memory map isn't being exposed correctly, or the kernel is messing it up somehow.

Can you boot with "efi=debug" on the kernel command line and attach a copy of /var/log/messages that results from it?

This and https://bugzilla.redhat.com/show_bug.cgi?id=1260999 look like dupes, and as it seems like everyone who tries a UEFI install of Beta TC4 is hitting this (on systems which previously installed fine), it's not a firmware issue.

Also note this from that bug:

"However, Lukas forgot to mention that if you switch to VT2 and run "efibootmgr", it prints current boot options just fine. So there seems to be some difference between running it in a VT and anaconda running it inside its process (in a chroot?)."

*** Bug 1260999 has been marked as a duplicate of this bug. ***

Created attachment 1071450 [details]
journal from boot with efi=debug on affected VM

Here's the journal contents after booting with 'efi=debug' on an affected VM.

(In reply to awilliam from comment #16)
> Created attachment 1071450 [details]
> journal from boot with efi=debug on affected VM
> 
> Here's the journal contents after booting with 'efi=debug' on an affected VM.

This doesn't look like the same issue at all; the original has:

Sep 07 16:32:20 localhost kernel: efi: EFI v2.00 by American Megatrends
Sep 07 16:32:20 localhost kernel: efi:  SMBIOS=0xbf906a18  ACPI=0xbfd5d000  ACPI 2.0=0xbfd5d000  MPS=0xfcaf0
Sep 07 16:32:20 localhost kernel: SMBIOS 2.6 present.
Sep 07 16:32:20 localhost kernel: DMI: System manufacturer System Product Name/F1A75-V PRO, BIOS 2302 09/26/2012

(i.e. no EFI memory map), and your log has:

Sep 08 18:16:33 localhost kernel: reserve setup_data: [mem 0x000000007fff0000-0x000000007fffffff] usable
Sep 08 18:16:33 localhost kernel: efi: EFI v2.50 by EDK II
Sep 08 18:16:33 localhost kernel: efi:  SMBIOS=0x7fec3000  ACPI=0x7fef2000  ACPI 2.0=0x7fef2014 
Sep 08 18:16:33 localhost kernel: efi: mem00: [Boot Code          |   |  |  |  |   |WB|WT|WC|UC] range=[0x0000000000000000-0x0000000000001000) (0MB)
...

(Hrm, might be the same after all - jwb tells me the dumping of the efi memory map has moved to efi=debug only, which surprised me because it's much more useful than the e820 map...)

pjones: in the dupe bug you can find the output *without* efi=debug from my system, for comparison, if it helps - https://bugzilla.redhat.com/attachment.cgi?id=1071442

Not particularly helpful collapsing these two bugs into one yet, but I guess we'll work through it.

Can everyone see if efivarfs is mounted on their afflicted machine?

'mount | grep efivarfs' in a vt will do (or terminal).  It should be mounted on /sys/firmware/efi/efivars/

Josh: it really seems pretty certain that we're all hitting the same thing. This isn't some system-specific firmware issue that just came up - it's something that changed circa Beta TC4 which somehow makes efibootmgr when called by the installer not think efivar support is available. Every report I've seen so far has the "

Did you note the thing from #c14? efibootmgr *does* work apparently OK from tty2. It's only when anaconda calls it that it's not working right.

Yes, efivarfs is mounted on /sys/firmware/efi/efivars .

Looking at the code, anaconda does apparently run efibootmgr from within a chroot to the installed system.

Yeah, so if I do this (after install has run):

chroot /mnt/sysimage
efibootmgr

I get the same error. As mentioned before, running 'efibootmgr' outside the chroot runs fine.

Inside the chroot, /sys/firmware/efi/efivars is not populated. How new is this efivarfs thing?

I guess I'd better double-check exactly when this broke, too. I'll check Beta TC1 and Alpha.

(In reply to awilliam from comment #23)
> Yeah, so if I do this (after install has run):
> 
> chroot /mnt/sysimage
> efibootmgr
> 
> I get the same error. As mentioned before, running 'efibootmgr' outside the
> chroot runs fine.
> 
> Inside the chroot, /sys/firmware/efi/efivars is not populated. How new is
> this efivarfs thing?

It isn't new at all.  If it isn't available in the chroot, perhaps it is because something did not bind mount it into the chroot before chroot was done.

> I guess I'd better double-check exactly when this broke, too. I'll check
> Beta TC1 and Alpha.

My best guess is the following:

The kernel used to be built with both EFI_VARS (old interface) and EFIVARFS (new).  A couple weeks ago, we disabled EFI_VARS because 1) it's broken, and 2) it's extra broken when both of them are available.

If the theory about the efivarfs mount not being in the chroot is accurate, then perhaps what is happening is that efibootmgr tries to use efivarfs and it fails.  Then it falls back to trying the old efivars interface, and that now also fails because we disabled it in the kernel.

(Which is somewhat frightening, since that means the old interface has been used for a long time when the efivarfs interface was intended to be used for quite a while.)

That sounds very plausible, because I just tested Beta TC1 - and indeed install works, and running 'efibootmgr' manually in the sysimage chroot works. But /sys/firmware/efi/efivars is definitely not present in the chroot, and neither efibootmgr nor efivar-libs has changed at all between TC1 and TC4, the versions are identical.

So that sounds like it's almost certainly what's going on, yes.

"If it isn't available in the chroot, perhaps it is because something did not bind mount it into the chroot before chroot was done."

I don't think anything even *tries* to do that, unless I'm missing it. If you dig through all the anaconda utility functions you wind up here:

https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/iutil.py#L126

here is where it sets up a chroot, if needed:

https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/iutil.py#L164

it just runs chroot, there's no bind mounting or anything else clever done. I poked around other bits of anaconda but can't find anything else that does any bind mounts into /mnt/sysimage except some stuff in the exotic ostree payload (which we're not using here).

Created attachment 1071514 [details]
Possible fix

https://github.com/rhinstaller/blivet/pull/219

Give this a try and see if it works.

Can affected folks please test https://www.happyassassin.net/updates/1260799.1.img ? It's a slightly tweaked version of bcl's patch (see discussion on the github issue), seems to work for me.

(In reply to awilliam from comment #28)
> Can affected folks please test
> https://www.happyassassin.net/updates/1260799.1.img ? It's a slightly
> tweaked version of bcl's patch (see discussion on the github issue), seems
> to work for me.

The update fixed the problem for me.

*** Bug 1262071 has been marked as a duplicate of this bug. ***

Discussed at 2015-09-10 blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2015-09-10/f23-blocker-review.2015-09-10-16.00.log.txt . Accepted as a blocker: this violates "A system installed with a release-blocking desktop must boot to a log in screen where it is possible to log in to a working desktop using a user account created during installation or a 'first boot' utility." in the case of a UEFI install where the fallback mechanism doesn't work (we know of several scenarios where that would be the case), which we consider a broad enough impact to make this a Beta blocker.

python-blivet-1.12.3-1.fc23 anaconda-23.19.3-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-15562

anaconda-23.19.3-1.fc23, python-blivet-1.12.3-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.\nIf you want to test the update, you can install it with \n su -c 'yum --enablerepo=updates-testing update anaconda python-blivet'. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-15562

python-blivet-1.12.4-1.fc23 anaconda-23.19.4-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-15846

When would install images be available with this update?

Beta RC1 will include it. You should still be able to use the updates.img linked above for now, as well.

anaconda-23.19.4-1.fc23, python-blivet-1.12.4-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.\nIf you want to test the update, you can install it with \n su -c 'yum --enablerepo=updates-testing update anaconda python-blivet'. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-15846

Verified in openQA testing.

anaconda-23.19.4-1.fc23, python-blivet-1.12.4-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.