Bug 873629

Summary: Malformed variable content during system installation
Product: [Fedora] Fedora Reporter: Johan Olby <johan>
Component: efibootmgrAssignee: Matt Domsch <matt_domsch>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 19CC: jordan_hargrave, mads, matt_domsch, mjg59, murray.alex, pjones, satellitgo, surya_prabhakar
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: efibootmgr-0.7.0-3.fc20 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-06-29 02:54:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 752665    
Attachments:
Description Flags
Zero out 31st bit of attributes when reading from sysfs to make sure we don't get any invalid looking stuff from Apple EFI firmware none

Description Johan Olby 2012-11-06 11:12:51 UTC 
Description of problem:
System instalation from live cd failes when running efibootmgr on mac mini rev 6 (late 2012) 

If there is an existing fedora boot entry the installer fails with the following log:
Running... efibootmgr -b 0000  -B
kernel: efivars: Malformed variable content

If the entry is removed prior to running the installer:
Running... efibootmgr -c -w -L Fedora -d /dev/sda -p 4 -L \EFI\fedora\shim.efi
kernel: efivars: Malformed variable content


Version-Release number of selected component (if applicable):
Fedora 18 Beta Test Compose 7

How reproducible:
Boot the tc7 live cd on a mac mini 6.2 and click install with default config. Post install fails when running efibootmgr

Steps to Reproduce:
1.Boot live cd on mac mini 6.2
2.Install to harddrive 
3.Error is displayed in a message box
  
Actual results:


Expected results:


Additional info:
Comment 1 Alex Murray 2012-11-27 03:29:10 UTC 
@Johan - I've hit what is probably this same bug in bugs #866028 and #880143 (ie. failing to set the old / new entry as you describe above) - have you managed to work around this and successfully install F18 Beta yet?
Comment 2 Alex Murray 2012-11-27 10:15:21 UTC 
Resetting the PRAM didn't help in my case... so either Apple's EFI Firmware does have invalid variables in it OR the latest kernel's are flagging valid variables as invalid...
Comment 3 Alex Murray 2012-11-27 12:59:01 UTC 
Ahh seems I misinterpreted what the validation code in efivars.c was doing - it validates newly created or changed variables, so it seems efibootmgr is sending invalid looking stuff to the kernel... hence the bug is therefore in efibootmgr OR in the validation code in the kernel. Will keep investigating...
Comment 4 Alex Murray 2012-11-27 13:00:33 UTC 
*** Bug 880143 has been marked as a duplicate of this bug. ***
Comment 5 Alex Murray 2012-11-27 19:37:03 UTC 
So strace'ing efibootmgr shows that for both cases it performs two operations - the first being the create new entry / delete old entry respectively which both succeed. The however it goes on to update the BootCurrent variable to ensure the newly created entry is added to it / the old deleted entry is removed from it - and it is this operation which fails.
Comment 6 Alex Murray 2012-11-27 23:38:03 UTC 
Whoops meant BootOrder not BootCurrent...

Looking more at the efibootmgr source - when it updates BootOrder it does so by reading the existing value and then modifying it (to add / remove the entry) and writing back the result - and this fails. However, you can also get efibootmgr to set BootOrder directly, in which case it does not first read it out but instead constructs the data manually from the values provided on the command-line - so I am thinking the existing value of the variable in the Apple EFI firmware contains something which makes it appear invalid when trying to write it back - and I can confirm this since:

sudo cat /sys/firmware/efi/vars/BootOrder-xxx/raw_var > /tmp/raw
cat /tmp/raw | sudo tee /sys/firmware/efi/vars/BootOrder-xxx/raw_var

Fails with the malformed variable content error.
Comment 7 Alex Murray 2012-11-28 09:54:08 UTC 
So I can confirm the problem as:

With the latest Apple EFI firmware, all variables have the 31st bit set in their attributes:

sudo hexdump -C /sys/firmware/efi/vars/BootOrder-8be4df61-93ca-11d2-aa0d-00e098032b8c/raw_var 
00000000  42 00 6f 00 6f 00 74 00  4f 00 72 00 64 00 65 00  |B.o.o.t.O.r.d.e.|
00000010  72 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |r...............|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400  61 df e4 8b ca 93 d2 11  aa 0d 00 e0 98 03 2b 8c  |a.............+.|
00000410  04 00 00 00 00 00 00 00  00 00 80 00 00 00 00 00  |................|
00000420  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000820  07 00 00 80                                       |....|
00000824


- ie the 80 at the very end - the problem is the more recent kernels reject setting any of the bits which it believes are invalid [1] and the way efibootmgr works is to generally read the existing value of a variable, modify the data and then write it back - so in this case it reads out the invalid attributes, and then writes it back with these same attributes still set which the kernel borks on.

Since the easiest way for me to fix this in the live environment was to patch efibootmgr, I've done that - and you can see the attached patch which basically just zeroes out the 31st bit of Attributes when reading it from sysfs so it should never get set back.

Long term I'm not sure what the best fix is - I'm CCing mjg since he wrote the original EFI variable validation code so perhaps a kernel side fix where when reading out an EFI var, we zero out any invalid attributes so they never get exposed could be a good fix - but hopefully he can provide some guidance.

[1] http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=drivers/firmware/efivars.c;h=d10c9873dd9a8684d7137e49245afc9b40d13d60;hb=HEAD#l522
Comment 8 Alex Murray 2012-11-28 09:56:34 UTC 
Created attachment 653399 [details]
Zero out 31st bit of attributes when reading from sysfs to make sure we don't get any invalid looking stuff from Apple EFI firmware
Comment 9 Fedora Update System 2012-11-28 22:46:27 UTC 
efibootmgr-0.5.4-14.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/efibootmgr-0.5.4-14.fc18
Comment 10 Matt Domsch 2012-11-29 21:34:15 UTC 
Alex, any idea what Apple is using that high bit to mean?  It's clearly not defined in the UEFI 2.3.1 spec.
Comment 11 Alex Murray 2012-11-29 21:56:17 UTC 
No I haven't got a clue unfortunately.
Comment 12 Fedora Update System 2012-11-30 06:40:35 UTC 
Package efibootmgr-0.5.4-14.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing efibootmgr-0.5.4-14.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-19382/efibootmgr-0.5.4-14.fc18
then log in and leave karma (feedback).
Comment 13 Fedora Update System 2012-12-07 04:28:08 UTC 
efibootmgr-0.5.4-14.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 14 satellitgo 2012-12-09 00:59:55 UTC 
(In reply to comment #13)
> efibootmgr-0.5.4-14.fc18 has been pushed to the Fedora 18 stable repository.
> If problems still persist, please make note of it in this bug report.

https://bugzilla.redhat.com/show_bug.cgi?id=879046
Comment 15 Matthew Garrett 2012-12-09 01:02:38 UTC 
*** Bug 879046 has been marked as a duplicate of this bug. ***
Comment 16 satellitgo 2012-12-09 01:03:32 UTC 
anaconda USB HD install not show the EFI USB icons after install
Comment 17 satellitgo 2012-12-09 02:28:36 UTC 
(In reply to comment #16)
> anaconda USB HD install not show the EFI USB icons after install

I made a dd of TC-1 DVD x86_64 then booted it on MacBookPro. from USB (EFI) right icon. Installed to an External USB HD with anaconda. install looked normal did reboot at end and removed install USB  reboot with option key did not show any USB (EFI) icons....
Comment 18 Alex Murray 2013-04-14 11:29:20 UTC 
So it seems I am getting this exact same bug occurring AGAIN but in F19 Alpha TC6 - so probably some extra validation has again been added to the kernel which is rejecting the EXISTING format of efi variables in the Apple firmware:

[liveuser@Device-D47978 ~]$ uname -a
Linux Device-D47978 3.9.0-0.rc6.git2.1.fc19.x86_64 #1 SMP Thu Apr 11 17:11:04 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
[liveuser@Device-D47978 ~]$ sudo cat /sys/firmware/efi/vars/BootOrder-8be4df61-93ca-11d2-aa0d-00e098032b8c/raw_var > /tmp/raw
[liveuser@Device-D47978 ~]$ cat /tmp/raw | sudo tee /sys/firmware/efi/vars/BootOrder-8be4df61-93ca-11d2-aa0d-00e098032b8c/raw_var 
��+���tee: /sys/firmware/efi/vars/BootOrder-8be4df61-93ca-11d2-aa0d-00e098032b8c/raw_var: Invalid argument
[liveuser@Device-D47978 ~]$ sudo hexdump -C /sys/firmware/efi/vars/BootOrder-8be4df61-93ca-11d2-aa0d-00e098032b8c/raw_var 
00000000  42 00 6f 00 6f 00 74 00  4f 00 72 00 64 00 65 00  |B.o.o.t.O.r.d.e.|
00000010  72 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |r...............|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400  61 df e4 8b ca 93 d2 11  aa 0d 00 e0 98 03 2b 8c  |a.............+.|
00000410  02 00 00 00 00 00 00 00  80 00 00 00 00 00 00 00  |................|
00000420  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000820  07 00 00 80                                       |....|
00000824

@mjg - Any ideas on what may have changed in the efivars validation code to cause this??
Comment 19 Alex Murray 2013-04-14 12:05:19 UTC 
Actually it seems this is a packaging error - efibootmgr in F19 doesn't actually include the patch which was added to F18 above - ie it's missing this entire entry from the changelog and the associated patches:

* Wed Nov 28 2012 Matthew Garrett <mjg59.org> - 0.5.4-14
- efibootmgr-0.5.4-Work-around-broken-Apple-firmware.patch
  Resolves: #873629
- efibootmgr-0.5.4-Remove-device-path-padding-on-non-Itanium.patch - improve
  spec conformance
- efibootmgr-0.5.4-fix-minor-memory-leak.patch - from upstream
- efibootmgr-0.5.4-fix-disk-minor-number-discovery.patch - from upstream
- efibootmgr-0.5.4-make_boot_var-does-not-check-for-failed-status-with-.patch -
  from upstream

mjg - can you rebuild it with this same lot of changes (or better yet update the package to the newer 0.6.0 - https://bugzilla.redhat.com/show_bug.cgi?id=922368)
Comment 20 Fedora Update System 2014-06-20 14:55:01 UTC 
efibootmgr-0.7.0-2.fc19,efivar-0.10-2.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/efibootmgr-0.7.0-2.fc19,efivar-0.10-2.fc19
Comment 21 Fedora Update System 2014-06-20 14:55:34 UTC 
efibootmgr-0.7.0-3.fc20,efivar-0.10-2.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/efibootmgr-0.7.0-3.fc20,efivar-0.10-2.fc20
Comment 22 Fedora Update System 2014-06-21 02:57:39 UTC 
Package efibootmgr-0.7.0-3.fc20, efivar-0.10-2.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing efibootmgr-0.7.0-3.fc20 efivar-0.10-2.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-7564/efibootmgr-0.7.0-3.fc20,efivar-0.10-2.fc20
then log in and leave karma (feedback).
Comment 23 Fedora Update System 2014-06-29 02:54:34 UTC 
efibootmgr-0.7.0-3.fc20, efivar-0.10-2.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 24 Fedora Update System 2014-07-30 21:58:33 UTC 
efibootmgr-0.7.0-2.fc19, efivar-0.10-2.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.