Bug 1728652

Summary: RFE: Ship the JSON firmware "descriptor files" as part of EDK2
Product: [Fedora] Fedora Reporter: Kashyap Chamarthy <kchamart>
Component: edk2Assignee: Kashyap Chamarthy <kchamart>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 30CC: berrange, crobinso, kraxel, lersek, pbonzini, philmd, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: edk2-20190501stable-2.fc30 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-15 18:08:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
proposed firmware descriptor files none

Description Kashyap Chamarthy 2019-07-10 11:16:12 UTC 
Description of the problem
--------------------------

From version 4.1 (due in August 2019) onwards, QEMU ships the so-called
firmware "descriptor files".  These are small JSON files that describe
details about UEFI firmware binaries — such as the fimware binary path,
its architecture, supported machine type, NVRAM template and so forth.

You can see examples of these files from the QEMU upstream Git:
https://git.qemu.org/?p=qemu.git;a=tree;f=pc-bios/descriptors

    $> tree descriptors/
    descriptors/
    ├── 50-edk2-i386-secure.json
    ├── 50-edk2-x86_64-secure.json
    ├── 60-edk2-aarch64.json
    ├── 60-edk2-arm.json
    ├── 60-edk2-i386.json
    └── 60-edk2-x86_64.json

QEMU 4.1 itself will ship the above files.  However, Fedora needs[*] to 
ship these file as part of its EDK2 package.

What problem does this solve?
-----------------------------

The said descritptor files allow high-level virtualization management
tools to systemtically query the features of a given firmware binary,
based on which the tool can make further decisions.  E.g. the libvirt
project has been wired up to take advantage of these descriptor files;
with that, libvirt knows the "characteristics" and features of a 
firmware binary.  This info, in turn, can be used by tools OpenStack
(which uses libvirt APIs) to make further decisions -- e.g. whether to
allow UEFI Secure Boot for guests, etc.


[*] Why should Fedora ship these when QEMU already bundles them?
----------------------------------------------------------------

Quoting (with minor formatting edits) Laszlo Ersek:

    Distributions providing their own EDK2 packages would not include
    the descriptors from upstream QEMU, even if they otherwise package
    QEMU.  That's beause the descriptor files in QEMU match the firmware
    bundled with QEMU -- but the firmware images in the distros' own
    EDK2 packages are different.  So, if a distro provides an EDK2
    package, then the same EDK2 package should offer matching
    descriptors.  QEMU offers descriptors (soon) because QEMU
    technically distributes edk2 firmware binaries (soon).  [Where
    "soon" == QEMU 4.1)

Also, as Dan Berrangé reminded me on IRC:

    In Fedora, we need to ship them [the "descriptor files"] as part of
    the EDK2 package, because Fedora throws away all the firmware files
    that QEMU bundles, because we're [Fedora] required to rebuild
    everything from pristine source.
Comment 1 Kashyap Chamarthy 2019-07-10 18:20:20 UTC 
WIP pull request:

    https://src.fedoraproject.org/rpms/edk2/pull-request/3
Comment 2 Laszlo Ersek 2019-07-11 15:14:17 UTC 
Created attachment 1589575 [details]
proposed firmware descriptor files

The directory structure within the tarball reflects the subpackages of the Fedora edk2 package:

fedora-edk2/
fedora-edk2/aarch64/
fedora-edk2/aarch64/70-edk2-aarch64-verbose.json
fedora-edk2/arm/
fedora-edk2/arm/70-edk2-arm-verbose.json
fedora-edk2/ovmf-ia32/
fedora-edk2/ovmf-ia32/40-edk2-ovmf-ia32-sb-enrolled.json
fedora-edk2/ovmf-ia32/50-edk2-ovmf-ia32-sb.json
fedora-edk2/ovmf-ia32/60-edk2-ovmf-ia32.json
fedora-edk2/ovmf/
fedora-edk2/ovmf/40-edk2-ovmf-sb-enrolled.json
fedora-edk2/ovmf/50-edk2-ovmf-sb.json
fedora-edk2/ovmf/60-edk2-ovmf.json

This is meant to show what JSON file(s) should be included in what subpackage.

With regard to installing the JSON files on the target system, all files should go flat under "/usr/share/qemu/firmware/" (the filenames are unique without the directory prefixes too):

40-edk2-ovmf-ia32-sb-enrolled.json
40-edk2-ovmf-sb-enrolled.json
50-edk2-ovmf-ia32-sb.json
50-edk2-ovmf-sb.json
60-edk2-ovmf-ia32.json
60-edk2-ovmf.json
70-edk2-aarch64-verbose.json
70-edk2-arm-verbose.json
Comment 3 Cole Robinson 2019-07-12 00:10:25 UTC 
As another data point, Gerd's nightly builds have a script for generating json files: https://git.kraxel.org/cgit/jenkins/edk2/commit/?id=d69296594f3cfdba817483b6c8aeea65a18987f4
Comment 4 Kashyap Chamarthy 2019-07-12 09:01:48 UTC 
(In reply to Cole Robinson from comment #3)
> As another data point, Gerd's nightly builds have a script for generating
> json files:
> https://git.kraxel.org/cgit/jenkins/edk2/commit/
> ?id=d69296594f3cfdba817483b6c8aeea65a18987f4

(In reply to Laszlo Ersek from comment #2)
> Created attachment 1589575 [details]
> proposed firmware descriptor files
> 
> The directory structure within the tarball reflects the subpackages of the
> Fedora edk2 package:
> 
> fedora-edk2/
> fedora-edk2/aarch64/
> fedora-edk2/aarch64/70-edk2-aarch64-verbose.json
> fedora-edk2/arm/
> fedora-edk2/arm/70-edk2-arm-verbose.json
> fedora-edk2/ovmf-ia32/
> fedora-edk2/ovmf-ia32/40-edk2-ovmf-ia32-sb-enrolled.json
> fedora-edk2/ovmf-ia32/50-edk2-ovmf-ia32-sb.json
> fedora-edk2/ovmf-ia32/60-edk2-ovmf-ia32.json
> fedora-edk2/ovmf/
> fedora-edk2/ovmf/40-edk2-ovmf-sb-enrolled.json
> fedora-edk2/ovmf/50-edk2-ovmf-sb.json
> fedora-edk2/ovmf/60-edk2-ovmf.json
>
> This is meant to show what JSON file(s) should be included in what
> subpackage.

Yeah, that's useful.
 
> With regard to installing the JSON files on the target system, all files
> should go flat under "/usr/share/qemu/firmware/" (the filenames are unique
> without the directory prefixes too):
> 
> 40-edk2-ovmf-ia32-sb-enrolled.json
> 40-edk2-ovmf-sb-enrolled.json
> 50-edk2-ovmf-ia32-sb.json
> 50-edk2-ovmf-sb.json
> 60-edk2-ovmf-ia32.json
> 60-edk2-ovmf.json
> 70-edk2-aarch64-verbose.json
> 70-edk2-arm-verbose.json

Noted; thanks for the descriptor files.  And the review.
Comment 5 Kashyap Chamarthy 2019-07-12 09:02:13 UTC 
(In reply to Cole Robinson from comment #3)
> As another data point, Gerd's nightly builds have a script for generating
> json files:
> https://git.kraxel.org/cgit/jenkins/edk2/commit/
> ?id=d69296594f3cfdba817483b6c8aeea65a18987f4

Ah, interesting.  Thanks for the pointer, Cole.
Comment 6 Cole Robinson 2019-07-12 20:11:47 UTC 
Okay a build is in progress, thanks for the help folks: https://koji.fedoraproject.org/koji/buildinfo?buildID=1312544

Some small changes I made:

* non-ia32 edk2-ovmf*.json files are renamed to edk2-ovmf-x64*.json to make it easier to glob by arch
* ovmf-ia32/ now has its own _VARS files so it doesn't need to implicitly depend on the ones in ovmf/

I tested with latest libvirt and virt-install --boot firmware=efi and it all seemed to work well, except aarch64 which I think is a libvirt problem: https://bugzilla.redhat.com/show_bug.cgi?id=1729604

Does anyone know what the plan in libvirt is to actually allow users to determine choices? Like there's 70-edk2-aarch64-verbose.json, but what if we start shipping a non-verbose version, is it intended that libvirt users can make that distinction via the XML?
Comment 7 Laszlo Ersek 2019-07-15 12:45:45 UTC 
- If we start shipping a non-verbose version, that should get priority 60 (at the system level); per <https://src.fedoraproject.org/rpms/edk2/pull-request/3#comment-27523>.

- Overriding priorities *per domain* doesn't seem useful; if it's about a specific domain, the user can already pick a specific firmware binary + varstore template.

- Setting preferences (defaults for new VMs) with different priorities is supported; please see the following snippet from "docs/interop/firmware.json" in the QEMU tree (I think it also may be documented somewhere in the libvirt docs, but I don't recall where):

# It is recommended to create firmware JSON files (each containing a
# single @Firmware root element) with a double-digit prefix, for example
# "50-ovmf.json", "50-seabios-256k.json", etc, so they can be sorted in
# predictable order. The firmware JSON files should be searched for in
# three directories:
#
#   - /usr/share/qemu/firmware -- populated by distro-provided firmware
#                                 packages (XDG_DATA_DIRS covers
#                                 /usr/share by default),
#
#   - /etc/qemu/firmware -- exclusively for sysadmins' local additions,
#
#   - $XDG_CONFIG_HOME/qemu/firmware -- exclusively for per-user local
#                                       additions (XDG_CONFIG_HOME
#                                       defaults to $HOME/.config).
#
# Top-down, the list of directories goes from general to specific.
#
# Management software should build a list of files from all three
# locations, then sort the list by filename (i.e., last pathname
# component). Management software should choose the first JSON file on
# the sorted list that matches the search criteria. If a more specific
# directory has a file with same name as a less specific directory, then
# the file in the more specific directory takes effect. If the more
# specific file is zero length, it hides the less specific one.
#
# For example, if a distro ships
#
#   - /usr/share/qemu/firmware/50-ovmf.json
#
#   - /usr/share/qemu/firmware/50-seabios-256k.json
#
# then the sysadmin can prevent the default OVMF being used at all with
#
#   $ touch /etc/qemu/firmware/50-ovmf.json
#
# The sysadmin can replace/alter the distro default OVMF with
#
#   $ vim /etc/qemu/firmware/50-ovmf.json
#
# or they can provide a parallel OVMF with higher priority
#
#   $ vim /etc/qemu/firmware/10-ovmf.json
#
# or they can provide a parallel OVMF with lower priority
#
#   $ vim /etc/qemu/firmware/99-ovmf.json
Comment 8 Kashyap Chamarthy 2019-07-15 15:13:34 UTC 
Merged:

    https://src.fedoraproject.org/rpms/edk2/c/674b3c8a27a8 (Ship the JSON firmware "descriptor files")

    https://src.fedoraproject.org/rpms/edk2/c/429a2758cc5c (Copy VARS from plain ovmf/ to ovmf-ia32/)

    https://src.fedoraproject.org/rpms/edk2/c/167eadb8ba32 (Give all json file categories a unique matchable prefix)
Comment 9 Fedora Update System 2019-07-15 19:58:18 UTC 
FEDORA-2019-d47a9d4b8b has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-d47a9d4b8b
Comment 10 Fedora Update System 2019-07-16 00:54:28 UTC 
edk2-20190501stable-2.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-d47a9d4b8b
Comment 11 Fedora Update System 2019-08-15 18:08:30 UTC 
edk2-20190501stable-2.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.