Bug 1488204

P.S. As mentioned in the bug title the reformatting appears to happen even though check box is not ticked

Needs logs from /tmp attached, from the 2nd installation. And also reproduce steps are missing - there's like 8000 ways to do an installation. What's confusing is you say the kernel and initramfs are erased/overwritten (two different things, which is it?) which are located in /boot, but the title is that /boot/efi is reformatted. So this bug report is just confusing. On UEFI /boot and /boot/efi are complete different file system volumes.

Sorry for the confusion: /boot/efi with grub is shared but two distinct /boot with kernels etc. exist. The following steps may reproduce the problem:

1. install the first instance of Fedora 26 in on a clean disk under custom mode with btrfs partitioning

2. install the second instance of Fedora 26 on the same disk under custom mode with btrfs partitioning - note the shared /boot/efi partition and do not tick the reformat option.

3. after reboot, the entries of the first instance under /boot/efi have vanished.

I can send logs, no problem, if needed and unless clearer from this description.

I think I understand.

The entire contents of /boot/efi/EFI/fedora is replaced for each installation. Your EFI system partition isn't being reformatted, you can look at the UUID entry for /boot/efi in the /etc/fstab for the two intallations to confirm, I bet they are the same. But the contents of the fedora directory are blown away at the 2nd installation, including the previous grub.cfg, in favor of the new installation.

In theory, grub2-mkconfig should find the previous btrfs installation, and create (suboptimal) boot entries, but this is just a fakaked design. There is no practical way for two OS's to share a grub.cfg, either upstream GRUB's design or Fedora's implemenation of it (which then uses grubby to update grub.cfg, not grub2-mkconfig). So it's just double or triple fakaked design wise for your use case.

Last time I checked, upstream GRUB expects the distro to create a generic grub.cfg, on the EFI System partition, to search by fsuuid and configfile to load the distro specific grub.cfg found at /boot/grub. Fedora breaks upstream's design by putting the distro specific grub.cfg on the EFI system partition, making it impossible to share with two Fedora's on the same system.

To unbreak this lazy shit:

a. 
For each distro version:
grub2-mkconfig /boot/grub2/grub.cfg

Yes, you're reverting to the upstream way, which on Fedora happens to be the command for BIOS systems, rather than for UEFI. This is obviously easy to do for Fedora 26 2nd instance, but how to get a new grub.cfg in Fedora 26 1st instance?

One way, 'grep mount /var/log/anaconda/program.log' for hint how to assemble a Fedora system, and do that for Fedora 26a, chroot, then run grub2-mkconfig. Done. Easy.

Another is have rescue mode on DVD or Netinstall image on a USB stick do the assembly for you. Maybe it'll work, but I'm not sure how it distinguishes between two Fedoras of the same version, probably you see two options to assemble with identical names. *shrug*

Another is you reboot normally and at the grub menu edit the first menu entry with 'e', go to linuxefi line, remove root=UUID=longuuidstring and replace that with root=/dev/sdXY where XY is designation for Fedora 26a's rootfs.

At the end of this, you should have Fedora 26a grub.cfg located on Fedora 26a's /boot, and a Fedora 26b grub.cfg on Fedora 26b's /boot. Next a way for GRUB to find them, which right now it won't.


b.
Hand create a new grub.cfg on the EFI System partition at /boot/efi/EFI/fedora/grub.cfg that's just super simple.

menuentry 'Fedora 26a' {
	load_video
	set gfxpayload=keep
	insmod gzio
	insmod part_gpt
	insmod ext2
	search --no-floppy --fs-uuid --set=root  <fsuuid for /boot for this distro, will match uuid in its /etc/fstab>
        configfile /grub2/grub.cfg
}
menuentry 'Fedora 26b' {
	load_video
	set gfxpayload=keep
	insmod gzio
	insmod part_gpt
	insmod ext2
        search --no-floppy --fs-uuid --set=root  <fsuuid for /boot for this distro, will match uuid in its /etc/fstab>
        configfile /grub2/grub.cfg
}


This works because you have two boot partitions for F26a and F26b, they each have different fsuuids. So this will find those unique fsuuids, set as root, and then finds the distro specific grub.cfg on that unique partition.

But grubby won't know WTF to do because it wants to edit the ESP grub.cfg, and that brings us to:

c.

In each Fedora distro version, fix this crap in /etc/
grub2.cfg -> ../boot/grub2/grub.cfg
grub2-efi.cfg -> ../boot/efi/EFI/fedora/grub.cfg

remake the symlink for grub2-efi.cfg to point to /boot/grub2/grub.cfg; that way on UEFI grubby will try to edit only its own /boot/grub2/grub.cfg and not the other distro's grub.cfg, and not your forwarding grub.cfg on the ESP.

I understand all is understood and mastered:-)

I followed the above instruction but upon installation of Fedora 26a, I get this error:

ln -s ../boot/grub2/grub.cfg grub2-efi.cfg

[root@localhost-live etc]# grub2-mkconfig -o /boot/grub2/grub.cfg
/usr/sbin/grub2-probe: error: failed to get canonical path of ‘/dev/nvme1n1p1’.

I have to admit I was being adventurous and placed the /root n /home partitions on another disk, see below. I always use btrfs for these.

Is the error related to btrfs or something else?

Cheers, Thomas


[liveuser@localhost-live ~]$ sudo fdisk -l
Disk /dev/nvme0n1: 477 GiB, 512110190592 bytes, 1000215216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 06CD7AFE-9476-4A67-8704-DF27C875BD6F

Device       	Start 	End Sectors  Size Type
/dev/nvme0n1p1	2048 1026047 1024000  500M EFI System
/dev/nvme0n1p2 1026048 3123199 2097152	1G Linux filesystem


Disk /dev/nvme1n1: 238.5 GiB, 256060514304 bytes, 500118192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 053C3D3A-2FB3-4EAF-B7EC-5FD21212D18E

Device     	Start   	End   Sectors  Size Type
/dev/nvme1n1p1  2048 203048959 203046912 96.8G Linux filesystem

The error is related to the way I access:

sudo mount -t btrfs -o subvol=root /dev/nvme1n1p1 /mnt
sudo mount /dev/nvme0n1p2 /mnt/boot
sudo mount /dev/nvme0n1p1 /mnt/boot/efi 
sudo systemd-nspawn -b -D /mnt

It does not happen when I am normally logged on.

I did solve the ultimate problem of two side-by-side installations this way:

1. did a backup of the 1st "fedora" installation /boot, /boot/efi

2. installed "fedora2" which overwrites /boot/efi/EFI/fedora

3. renamed /fedora to /fedora2 and restored /fedora (using nspawn from above)

4. F11 boot options key on an MSI AM4 board now displays "fedora"

5. in "fedora" I decided to stay with EFI (not following Chris' advice, though) and run

grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg

which recognized "fedora2". Done.

All of this is maybe not intended and elegant but should help more savy people to find a better way how to implement it in the future.

I may have had all this trouble due to btrfs (see above) but on fast SSDs this is what you want.

correction ad 1): omitt /boot (just did a backup of /boot/efi)

Chris,

Would it be possible to change /boot/efi/EFI/fedora/ for the second installation? For example to /boot/efi/EFI/fedora2/? Would that be possible or is the path hard coded into grub2-efi installation?

If it is not hard coded, one could probably create /boot/efi/EFI/fedora2/ and symlink /boot/efi/EFI/fedora/ to /boot/efi/EFI/fedora2/?

Gentlemen, we may have to close this "bug" since it is not apparent or obvious except that some experts may spot a problem in there which actually is a bug e.g. overwriting /boot/efi/EFI/fedora.

Here some personal notes I jotted down helping me get back into the matter after a while. I eventually got to grips with Chris' useful hints. I went beyond my previous approach. Beware: I may have misconceived one or the other fact. I am running happily these four OSs side-by-side. I am not a computer scientist.

Please advise, if I  should close the bug or how to proceed.

Appreciate, Thomas

Notes--------------------------------------------------------------

multiboot
  disk 1
    CentOS 7: efi and os partition on disk 1
    Windows 10: efi on disk 2, os on disk 1
  disk 2
    Fedora rawhide: both on disk 2
    Fedora release: both on disk 2

0. concept
	nvram (non-volatile-ram) = "BIOS"
		F11 (AM4 board) to Boot Menu
		contains bootstrapper accessible via efibootmgr
		knows about various efi/EFI partitions
		installers register their OS here
	grub = bootloader
		redirected to from BOOT (here, secure boot comes in)
		reads grub.cfg possibly master grub

1. todo
gparted creates a fat32 partition for esp, efi manually
	no matter on which of several disks (I have it on disk 2)
	modern OS including multiple Fedora installations grab it > master grub!
	centOS 7 creates its own - no problem (access via F11 boot menu)
	Fedora always creates a "fedora" subdirectory
anaconda
        allocate a spedific "volume" e.g. release,  rawhide, cuda for /root and /home (hit update settings frequently) if using btrfs
        always overrides "fedora" i.e. best save disk uuid directly after install for later

2. master grub which calls specific grub via the line " configfile /grub2/grub.cfg"
	/boot/efi/EFI/fedora/grub.cfg

3. check in /etc/fstab or Disks UUID for /boot  that changes  on every new install  (grub option possible to avoid this) and past into grub.cfg

c269b3b9-ec6f-4dfa-a4a4-b62cc5b7cbd0 (e.g. rawhide)
0e259bde-6299-4795-8510-8796a30887cb (e.g release)

4. change symbolic links for /etc/grub* from /boot/efi to /boot/grub2 (important)

5. grub2-mkconfig -o /boot/grub2/grub.cfg 

possilbly disable in /etc/default/grub
GRUB_DISABLE_OS_PROBER="true"
https://help.ubuntu.com/community/Grub2/Setup#Configuring_GRUB_2 (just because it is a good documentation not because I am thinkin of using ubuntu)

6. summa summarum:

I boot into each OS via F11 Boot Menu. The two Fedora intallations are selected by their master grub.

Thanks for all your info. These steps could be pretty valuable for others.

# example master boot configuration file
# /boot/efi/EFI/fedora/grub.cfg

### BEGIN /etc/grub.d/00_header ###
set pager=1

if [ -f ${config_directory}/grubenv ]; then
  load_env -f ${config_directory}/grubenv
elif [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="${saved_entry}"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

terminal_output console
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=5
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/01_users ###
if [ -f ${prefix}/user.cfg ]; then
  source ${prefix}/user.cfg
  if [ -n "${GRUB2_PASSWORD}" ]; then
    set superusers="root"
    export superusers
    password_pbkdf2 root ${GRUB2_PASSWORD}
  fi
fi
### END /etc/grub.d/01_users ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'release' {
	load_video
	set gfxpayload=keep
	insmod gzio
	insmod part_gpt
	insmod ext2
	      search --no-floppy --fs-uuid --set=root 0e29bde-6399-4745-8510-8746a30887cb
        configfile /grub2/grub.cfg
}
menuentry 'rawhide' {
	load_video
	set gfxpayload=keep
	insmod gzio
	insmod part_gpt
	insmod ext2
        search --no-floppy --fs-uuid --set=root c262b2b9-ec6f-4dfa-a4a4-b82cc4b7cbd0
        configfile /grub2/grub.cfg
}
if [ "x$default" = 'Fedora (4.14.0-0.rc2.git3.1.fc28.x86_64) 28 (Workstation Edition)' ]; then default='Advanced options for Fedora>Fedora (4.14.0-0.rc2.git3.1.fc28.x86_64) 28 (Workstation Edition)'; fi;
### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###

### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/20_ppc_terminfo ###
### END /etc/grub.d/20_ppc_terminfo ###

### BEGIN /etc/grub.d/30_os-prober ###
menuentry 'Windows Boot Manager (on /dev/nvme1n1p1)' --class windows --class os $menuentry_id_option 'osprober-efi-9783-1C46' {
	insmod part_gpt
	insmod fat
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root  9783-1C46
	else
	  search --no-floppy --fs-uuid --set=root 9783-1C46
	fi
	chainloader /EFI/Microsoft/Boot/bootmgfw.efi
}

### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###

(In reply to Daniel Laczi from comment #9)
> Would it be possible to change /boot/efi/EFI/fedora/ for the second
> installation? For example to /boot/efi/EFI/fedora2/? Would that be possible
> or is the path hard coded into grub2-efi installation?

It is possible, but this results in separate bootloaders (grub2 efi binary, and grub2 configuration) in each path. There is fairly poor intelligence by GRUB when it comes to discovering other grub configurations, or other Linux installations. So you can have seperate paths for each fedora, separate bootloaders, separate bootloader configs, and use the UEFI boot manager to choose between them. But if you load the 'fedora' bootloader, it's not going to see the 'fedora2' grub.cfg - they're all completely separate.

The effort required here exceeds the interest and expertise of pretty much everyone. Fedora carries a lot of patches for GRUB that make it behave in non-standard ways. And upstream GRUB is not terribly interested in making GRUB smarter, including dynamically finding new kernels rather than having to completely obliterate the grub.cfg and create a new one from scratch. They also don't care about drop in configuration snippets like the bootloaderspec proposal, of which variations are supported by project atomic and the bls.mod module for GRUB, again a Fedora only patch.


> If it is not hard coded, one could probably create /boot/efi/EFI/fedora2/
> and symlink /boot/efi/EFI/fedora/ to /boot/efi/EFI/fedora2/?

I don't think FAT supports symlinks. Anyway, I'm not a fan of anything constantly modifying the EFI system partition for just bootloader changes. That should be a rare thing, and whatever is doing that modification should mount the ESP volume, modify it, and then umount it. I think it's pretty sloppy that it's always persistently mounted at /boot/efi but the reason we do this is because it's easy, no one responsible for kernel updating wants to create the scripts that determine whether this is on UEFI and then how to discover the proper ESP, mount it, modify it, and then umount it. But that's what other OS's do.

Addendum (just for completeness of the my above record, I acknowledge Chris' comments):

Upon start with a clean disk, gparted enlarges an existing EFI fat16 partition 
(which I had to get from a preceding fake fedora installation!) to 500 MB and most importantly converts it to fat32 so that other OS can grab it.

Addendum 2: as non-IT specialist I once again installed multi-boot from scratch and noticed that I forgot to mention one point: as soon as Windows gets involved (WHQL seems to be a synonym for UEFI and is listed on my msi AM4 motherboard under Windows) one needs to - under Windows Power Shell (not Command Prompt because of the quotes):

bcdedit /enum
bcdedit /set '{bootmgr}' path \EFI\fedora\shim.efi

Just to complete the record above.

*** Bug 1516475 has been marked as a duplicate of this bug. ***

In the above, 
    configfile /grub2/grub.cfg
should be
    configfile /boot/grub2/grub.cfg

The menuentry items that use that feature, could be added to
/etc/grub.d/40_custom

Question:  I noticed /boot/grub2 had: grubenv -> /boot/efi/EFI/fedora/grubenv
Should that link be replaced with the file?
Since the goal is to have each system have its own grub 
(independent of other partitions and of the /boot/efi ESP),
I did replace the link with the file.  It is working as I expect,
and I can boot into Windows 10, Fedora 25, 26, and 27.

As far as I can tell, yes. If upstream Fedora projects implemented a working process in anaconda for multiple installations, they would have an edge over other OSs. It is a kind of low hanging fruit!

For a 2 ESP and 2 /boot possible solution see https://bugzilla.redhat.com/show_bug.cgi?id=1598857

A small comment for people reaching this issue form a search engine.

The aforementioned procedure looks quite complex and generates a real risk of breaking the things. I was somehow anxious with a vision of installing the second Fedora instance on the same drive to occasionally play with Rawhide or some other (potentially) less stable things.

However, I have found that guide https://www.redips.net/linux/fedora-triple-boot/ which takes an another approach. In short:
1. Use two EFI partitions (on the same drive).
2. Install the completely independent second Fedora instance pointing the second EFI partition.
3. Use efibootmgr to select which Fedora should be booted by default.
4. Switch the EFI partition to boot at UEFI/BIOS level (on demand to boot the second Fedora).

Thanks to that the installation of the second Fedora looks the same (simple) as the first one (in Fedora 32 I haven't encountered the issues the article had with Grub configuration/customization - it just works out of the box).

Some of this bug is obsoleted by BLS work [1], but still complicated due to the grubenv containing boot parameters. The root=uuid= hint is in there. It's maybe possible to put two sets of kernel args in there, the file is only 1024 bytes so it's a tight fit.

The idea of two ESPs in c20 is safer in that everything is completely separate: two ESPs, two boots, two roots (you could share a single VG if using LVM, or Btrfs, and custom partitioning). But switching between them is still a bit of a hurdle: either use the firmware's built-in boot manager; or manually add both ESP as search locations to each grub.cfg, so both sets of BLS snippets are read, and thus appear in either GRUB menu. Since the grub.cfg is static in a BLS world, it's OK to do this - but will be overwritten if you call grub2-mkconfig, which is now largely not needed in most cases, since the GRUB menu is populated by the BLS snippets in /boot/loader/entries.

Future change will likely de-emphasize grubenv, putting the boot parameters into the BLS snippets, making it more like the upstream BLS implementation.

[1]
https://fedoraproject.org/wiki/Changes/BootLoaderSpecByDefault