Bug 1405295 - mactel-boot needs script to handle generating boot.efi for HFS+ /boot/efi to allow boot selector to work
Summary: mactel-boot needs script to handle generating boot.efi for HFS+ /boot/efi to ...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: mactel-boot
Version: rawhide
Hardware: x86_64
OS: Mac OS
unspecified
high
Target Milestone: ---
Assignee: Matthew Garrett
QA Contact: Fedora Extras Quality Assurance
URL: https://bugs.launchpad.net/ubuntu/+so...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-12-16 06:13 UTC by Jack Howarth
Modified: 2016-12-17 18:15 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-12-17 17:46:51 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
script to automate building and installing boot.efi on Macs for a boot selector friendly /boot/efi partition (1.37 KB, application/x-shellscript)
2016-12-16 06:13 UTC, Jack Howarth
no flags Details
additional program to disable journal on HFS+ for write access (1.17 KB, text/plain)
2016-12-16 06:19 UTC, Jack Howarth
no flags Details
final copy of the update-efi-booter with bug fixes (1.48 KB, text/plain)
2016-12-17 04:44 UTC, Jack Howarth
no flags Details

Description Jack Howarth 2016-12-16 06:13:23 UTC
Created attachment 1232440 [details]
script to automate building and installing boot.efi on Macs for a boot selector friendly /boot/efi partition

Description of problem: 

Currently, while Fedora boots on Macs, it doesn't appear as a bootable OS in the boot selector which forces people to resort to rEFind.  The attached script assumes a separate partition for /boot/efi, as used by Debian/Ubuntu, which has been formatted as HFS+. The script generates the approrpiate architecture of boot.efi depending on whether EFI-32 or EFI-64 firmware is detected. The resulting boot.efi, generated in /tmp, is installed in /boot/efi/System/Library/CoreServices along side the SystemVersions.plist from the mactel-boot package files and the required /boot/efi/mach_kernel file. The final step is blessing the newly installed boot.efi file with hfs-bless from the mactel-boot package. 

Note that the HFS+ /boot/efi partition is only temporarily remounted as read/write during the file installations and blessing steps to avoid tickling the fragility of the HFS+ file system support in Linux.

Comment 1 Jack Howarth 2016-12-16 06:18:10 UTC
Note that if program similar to posted to pastebin

http://pastebin.com/7rvxR38d

for disabling journaling of HFS+ volumes is added to the mactel-boot package as well, its use could also be added to the automated script.

Comment 2 Jack Howarth 2016-12-16 06:19:14 UTC
Created attachment 1232444 [details]
additional program to disable journal on HFS+ for write access

Comment 3 Jack Howarth 2016-12-16 06:21:10 UTC
Comment on attachment 1232440 [details]
script to automate building and installing boot.efi on Macs for a boot selector friendly /boot/efi partition

#!/bin/bash -ex
# update-efi-booter script
pushd /
grub-mkconfig -o /boot/grub/grub.cfg
efi_arch="x86_64"
coreservices_dir="/boot/efi/System/Library/CoreServices/"
boot_efi_dir="/boot/efi/"
mactel_boot_dir="/usr/share/mactel-boot/"
mactel_boot_logo_dir="/usr/share/mactel-boot-logo/"
grep -q "32" /sys/firmware/efi/fw_platform_size && efi_arch="i386"
echo 'Creating boot.efi for '$efi_arch
case "$efi_arch" in
x86_64)
  grub-mkstandalone -o /tmp/boot.efi -d usr/lib/grub/x86_64-efi -O x86_64-efi --compress=xz boot/grub/grub.cfg
  ;;
i386)
  grub-mkstandalone -o /tmp/boot.efi -d usr/lib/grub/i386-efi -O i386-efi --compress=xz boot/grub/grub.cfg
  ;;
esac
echo 'Installing boot.efi for '$efi_arch 
mount -t hfsplus -o force,remount,rw $boot_efi_dir && sleep 1
test ! -d $coreservices_dir && install -d $coreservices_dir
install -m 644 /tmp/boot.efi $coreservices_dir
test ! -f $boot_efi_dir/.VolumeIcon.icns && -f $mactel_boot_logo_dir/ubuntu.icons && install -m 644 $mactel_boot_logo_dir/ubuntu.icns $boo
t_efi_dir/.VolumeIcon.icns
test ! -f $boot_efi_dir/mach_kernel && echo "This file is required for booting" > $boot_efi_dir/mach_kernel
test ! -f $coreservices_dir/SystemVersion.plist && -f $mactel_boot_dir/SysytemVersion.plist && install -m 644 $mactel_boot_dir/SystemVersi
on.plist $coreservices_dir

Comment 4 Jack Howarth 2016-12-16 06:27:36 UTC
Note that this has been tested on Ubuntu 16.40 x86_64 Linux installed on a MacBook Pro 2,1 with EFI-32 firmware and the file locations haven't been tweaked for Fedora yet. Also anaconda will have to adjusted to swap the use of a /boot partition for a /boot/efi like Debian/Ubuntu (which retains /boot as part of /). This allows us to limit the effected files stored on the, normally, read-only HFS+ partition to just those residing in /boot/efi.

Comment 5 Matthew Garrett 2016-12-16 07:32:28 UTC
I'm very confused. mactel-boot and Anaconda should already be installing things appropriately. This won't work on 32-bit firmware Mac systems for a number of reasons, but should be fine on 64-bit. boot.efi should be a link to grub.efi. Anaconda should be generating an HFS+ partition for /boot/efi already. If any of this is going wrong then we should be fixing that rather than adding additional scripts.

Comment 6 Jack Howarth 2016-12-16 07:46:37 UTC
It works perfectly fine here on a Early 2006 MacBook Pro 2,1 with EFI firmware. Note that while the Fedora 24/25 DVDs will install on the same machine, no other distribution currently can boot an installer from a USB device on GPT except for the Debian Jessie multi-arch installer from...

http://cdimage.debian.org/mirror/debian-cd/current/multi-arch/iso-cd/debian-8.6.0-amd64-i386-netinst.iso

which leverages the new supprt for mixed modes that is described in https://wiki.debian.org/UEFI as...

Support for mixed-mode systems: 64-bit system with 32-bit UEFI
Some systems have been released containing 64-bit Intel Atom CPUs (such as the Bay Trail), but unfortunately use 32-bit UEFI firmware with no BIOS compatibility mode. Using the 32-bit UEFI x86 support, an i386 installation should be possible on these machines but it won't make the most of the 64-bit hardware.

Debian Jessie (8.0) was the first Linux distribution to include full support for mixed-mode UEFI installation on these machines. The multi-arch installation media (available in netinst and DVD form) include the UEFI boot loaders necessary for both i386 and amd64 boot. By selecting "64-bit install" from the initial boot menu, debian-installer will install a 64-bit (amd64) version of Debian. The system will automatically detect that the underlying UEFI firmware is 32-bit and will install the appropriate version of grub-efi to work with it.

Note that the boot.efi being used is completely distinct from the kernel runnig on top of it. On my MacBook Pro 2,1 with EFI-32 I have...

$ uname -a
Linux howarth-macbook-pro 4.8.0-30-generic #32-Ubuntu SMP Fri Dec 2 03:43:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

and

$ file /boot/efi/System/Library/CoreServices/boot.efi
/boot/efi/System/Library/CoreServices/boot.efi: PE32 executable (EFI application) Intel 80386 (stripped to external PDB), for MS Windows

which not only boots without resorting to rEFind but shows up in the option-key boot selector with the Ubuntu icon.

Comment 7 Matthew Garrett 2016-12-16 07:53:02 UTC
Fedora doesn't currently support mixed mode UEFI installs in general - that needs to be fixed before it works for Apple hardware. mactel-boot should then work fine without modification.

Comment 8 Jack Howarth 2016-12-16 08:02:51 UTC
I should clarify my exact installation. I wanted to use x86_64 Ubuntu 16.10 on my MacBook Pro 2,1 release but had no spare optical media laying around to burn a DVD so I became focused on the USB booting problem. After testing every other distribution, I stumbled upon the Debian Jessie multi-arch iso purely by chance. That installs as well on a MacBook Pro 2,1 as the Fedora 24/25 except that Debian hasn't scripted handling of the System/Library/CoreServices files in /boot/efi. Note that their bootable installation of a x86_64 Linux kernel on EFI-32 firmware installs in /boot/efi/EFI/debian the files..

grub.efi:     PE32 executable (EFI application) Intel 80386 (stripped to external PDB), for MS Windows
grubia32.efi: PE32 executable (EFI application) Intel 80386 (stripped to external PDB), for MS Windows

I transitioned to the desired Ubuntu 16.10 by copying over the /etc from a fresh Ubuntu 16.10 installation on a MacPro 3,1 with EFI-32 firmware while retaining the fstab, passwd and shadow files. Then I installed the Ubuntu 16.10 Live installer package set with...

apt-get -f update
apt-get clean
for pkg in `dpkg --get-selections | awk '{print $1}' | egrep -v '(dpkg|apt|mysql
|mythtv)'` ; do apt-get -y  -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confnew" --force-yes install --reinstall $pkg ; done

to purge any remnants of the Debian Jessie installation (which which only had the additional System Tools software package set selected in their installer).

Comment 9 Jack Howarth 2016-12-16 08:11:37 UTC
Also, if we want to discuss broken installers, I can find absolutely NO permutation for burning the Fedora 24/25 Live x86_64 Linux isos onto a usb stick which will boot on a EFI-64 MacPro 3,1. I have tried using Etcher, Unetbootin, Rufus, RMPrepUSB, XBoot, Startup Disk Creator and bunch more. Neither Debian Jessie nor Ubuntu 16.10 or their immediate previous releases have any problems in installing off of a GPT volume unlike Fedora which is a true horror show.

Comment 10 Jack Howarth 2016-12-16 08:22:01 UTC
Note that the update-efi-booter script can be made more robust with the addition of...

# make sure HFS+ Journaling is disabled
disable_journal `cat /proc/mounts | grep '/boot/efi' | cut -f1 -d' '`

provided that the mactel-boot package adds a disable_journal program based on  http://pastebin.com/7rvxR38d

Comment 11 Matthew Garrett 2016-12-16 16:57:26 UTC
Just dd the ISO onto a USB stick and it should work fine. It sounds like there isn't actually a bug here - you installed the system in an unsupported way, and some things don't work as expected as a result?

Comment 12 Jack Howarth 2016-12-17 04:44:40 UTC
Created attachment 1232819 [details]
final copy of  the  update-efi-booter with bug fixes

Tested on a fresh install of Ubuntu 16.10 x86_64 by...

gcc -o disable_journal disable_journal.c
chmod ugo+x update-efi-booter 
install disable_journal update-efi-booter /usr/local/bin

identify the device that the /boot/efi partition is mounted on, umount it and then use gdisk to delete that partition and create a new on with the code AF00 for HFS+. Write the changes out in gdisk and then execute mkfs.hfsplus on the device to format it. Now change the boot/efi entry in fstab to have 'hfsplus defaults'. Now the remount the boot/efi partition and just execute the 'update-efi-booter' script.

Now the /boot/efi subdirectory will contain...
# cd /boot/efi/      
# ls -lR .
.:
total 4
-rw-r--r-- 1 root root 34 Dec 16 23:04 mach_kernel
drwxr-xr-x 1 root root  3 Dec 16 23:04 System

./System:
total 0
drwxr-xr-x 1 root root 3 Dec 16 23:04 Library

./System/Library:
total 0
drwxr-xr-x 1 root root 4 Dec 16 23:04 CoreServices

./System/Library/CoreServices:
total 1968
-rw-r--r-- 1 root root 2009088 Dec 16 23:04 boot.efi
-rw-r--r-- 1 root root     380 Dec 16 23:04 SystemVersion.plist

and where the boot.efi is now blessed on a HFS+ filesystem. This allows the option key boot selector on Macs to display the Ubuntu operating system with a customized icon. Thus rEFind is no longer needed to be able to switch between other installed OS releases.

Comment 13 Matthew Garrett 2016-12-17 17:03:34 UTC
Issues with Ubuntu probably need to be filed on Launchpad. Is there an actual bug in Fedora here?

Comment 14 Jack Howarth 2016-12-17 17:39:32 UTC
(In reply to Matthew Garrett from comment #13)
> Issues with Ubuntu probably need to be filed on Launchpad. Is there an
> actual bug in Fedora here?

It does also exist on launchpad but the issue also exists on Fedora. Again, this issue is quite simple to grasp. The current Fedora installation fails to register with the Apple EFI boot selector as a valid bootable OS requiring users to resort to third-party utilities like the deprecated rEFit (http://refit.sourceforge.net/) or the currently supported rEFind (http://www.rodsbooks.com/refind/). However those solutions are sub-optimal compared to achieving generic support for the built-in EFI boot selector on the hardware itself as each time the user uses the Disk Startup panel to select a disk, it blows away the rEFit or rEFind configuration and they must be re-installed which is a real PITA.

Again, I don't understand why everyone is being so provincial about this issue. Fedora can be adapted to support the a Apple firmware EFI boot selector with just two minor changes. On installations on Apple hardware, the user must be offered by Anaconda the opportunity to install a /boot/efi partition which is HFS+ formatted. Due to the fragility of the Linux HFS+ support it is best to leave this partition as the default read-only and just switch it to rw when updating the efi booter files. The proposed update-efi-booter script does just that with the lines...

mount -t hfsplus -o force,remount,rw $boot_efi_dir && sleep 1
,,,
hfs-bless $coreservices_dir/boot.efi && sleep 1 && mount -t hfsplus -o remount $boot_efi_dir

Again note that following things must exist on the /boot/efi partition for the Apple EFI boot selector code to recognize it as containing a viable bootable OS..

/boot/efi/mach_kernel
/boot/efi/System/Library/CoreServices/System/SystemVersions.plist
/boot/efi/System/Library/CoreServices/System/boot.efi (which is blessed on the HFS+ filesystem with mactel-boot's hfs-bless program).

Fedora already has the first two items here. What it lacks is the presence of the blessed boot.efi file on a HFS+ partition to make the OS visible to the Apple EFI boot selector firmware.

Also note that if Fedora places its company logo in an icon and places it at /boot/efi/.VolumeIcon.icns, the user will be presented with this icon in the option key boot selector.

Lastly, the boot selector is highly desirable for folks using dual boots of Linux and OS X on their Macs as that is how they have to get to their OS X recovery partitions.

Comment 15 Matthew Garrett 2016-12-17 17:46:51 UTC
The current Fedora installer does register as long as you install Fedora in the supported manner. It sounds like you installed Fedora in some other way, on an unsupported hardware configuration, and as a result it doesn't work. If there's a bug here, it's that Fedora doesn't support mixed-mode UEFI installs. If you can reproduce this issue after installing Fedora in a supported configuration then please reopen this bug and we'll figure out when this broke, but the expected behaviour is certainly for Anaconda to handle this rather than for mactel-boot to ship additional scripts.

Comment 16 Jack Howarth 2016-12-17 17:55:45 UTC
Have you actually tried booting up with the option key pressed? Fedora most certainly does NOT appear in the option key boot selector. Again, you are only two small steps away from achieving that functionality. The only things missing are...

the presences of a boot.efi generated for the architecture matching the running EFI firmware (ie i386 for EFI-32 and x86_64 for EFI-64) on a HFS+ filesystem which would allow the OS to be seen as bootable by the built in EFI boot selector.

In case you are unfamiliar with that see...

https://support.apple.com/en-us/HT202796

under the section 'Use Startup Manager'.

Comment 17 Matthew Garrett 2016-12-17 18:08:35 UTC
I wrote this code. I wrote the Anaconda code that makes use of it. Yes, I have actually tried this. If you installed Fedora using a supported mechanism then you should already have an HFS+ /boot/efi. If you installed Fedora using some unsupported mechanism (such as doing a BIOS install and then trying to set up EFI support afterwards) then, well, like I said - the right fix is to get mixed-mode EFI support into Fedora. The current state of affairs is that Fedora doesn't support 32-bit EFI systems, and as a result things like this will be randomly broken.

Comment 18 Matthew Garrett 2016-12-17 18:15:09 UTC
For reference, I also wrote https://mjg59.dreamwidth.org/7468.html and https://mjg59.dreamwidth.org/12037.html - like I said, if you're having problems booting in a supported configuration (ie, a Mac with 64-bit EFI firmware that was installed via Anaconda) then that's a genuine problem and we should fix it. But the right place to fix it is still going to be in Anaconda, because that's where the setup code lives.


Note You need to log in before you can comment on or make changes to this bug.