Robert,

What are versions of related components (libvirt, qemu and edk2-ovmf) before and after upgrade?

While I'm setting up environment to reproduce it locally,
can reporter test it with downgraded edk2-ovmf (8.2 version)?

looks like firmware or PXE server configuration issue, for now moving it to firmware component.

Please capture and provide debug output from OVMF, see
https://bugzilla.redhat.com/show_bug.cgi?id=1450345#c2

Also reading https://bugzilla.redhat.com/show_bug.cgi?id=1450345#c2 
might be useful for troubleshooting.

Test it on rhel8.4 host,it works well.

Host env:
kernel-4.18.0-259.el8.x86_64
qemu-kvm-5.2.0-1.module+el8.4.0+9091+650b220a.x86_64
libvirt-client-6.10.0-1.module+el8.4.0+8898+a84e86e1.x86_64
edk2-ovmf-20200602gitca407c7246bf-4.el8.noarch

1.Boot with:
#!/bin/sh
VMNAME=pxe
sudo rm -f  /home/pxe/$VMNAME.raw
sudo fallocate -l 50G /home/pxe/$VMNAME.raw
virt-install --name $VMNAME --boot uefi --pxe  --ram 32768 --vcpus 4 --cpu host --disk path="/home/pxe/$VMNAME.raw",target.bus=virtio --os-variant rhel8.3 --network bridge=private-switch,model.type=virtio,mac.address="52:54:00:aa:ad:04" --connect qemu:///system

Result:
Installation successful.

2.Boot with:
#!/bin/sh
VMNAME=cdrom
sudo rm -f  /home/pxe/$VMNAME.raw
sudo fallocate -l 50G /home/pxe/$VMNAME.raw
virt-install --name $VMNAME --boot uefi --cdrom /home/RHEL8.3.0-BaseOS-x86_64.iso  --ram 32768 --vcpus 4 --cpu host --disk path="/home/pxe/$VMNAME.raw",target.bus=virtio --os-variant rhel8.3 --network bridge=switch,model.type=virtio,mac.address="52:54:00:aa:ad:05" --connect qemu:///system

Result:
Installation successful.

Reproduced problem locally, with debug log enabled.
which yields:

[Bds] Expand PciRoot(0x0)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(52540046A78E,0x1)/IPv4(0.0.0.0,0x0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0) -> PciRoot(0x0)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(52540046A78E,0x1)/IPv4(0.0.0.0,0x0,DHCP,0.0.
0.0,0.0.0.0,0.0.0.0)
[Security] 3rd party image[0] can be loaded after EndOfDxe: PciRoot(0x0)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(52540046A78E,0x1)/IPv4(0.0.0.0,0x0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0).
DxeImageVerificationLib: Image is not signed and SHA256 hash of image is not found in DB/DBX.
...
The image doesn't pass verification: PciRoot(0x0)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(52540046A78E,0x1)/IPv4(0.0.0.0,0x0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0)

Which results in "Access denied" massage in provided screenshot.

As workaround,
ipxe loads fine if secureboot is disabled.
(btw: I hasn't been able to boot VM with ipxe and secure boot enabled yet)

It looks like in 8.2 creates VM with secureboot disabled by default (using /usr/share/OVMF/OVMF_VARS.fd) and hence loading and starting  unsigned ipxe.efi works.
However 8.3 creates VM with secureboot enabled by default which works with properly signed ISO images but fails loading unsigned ipxe.efi with "access denied".

If host that creates VMs dedicated to bootstrap VM exclusively from network (ipxe), then as workaround
one can temporarily replace /usr/share/OVMF/OVMF_VARS.secboot.fd with /usr/share/OVMF/OVMF_VARS.fd
which should revert behaviour to 8.2 and allow boot with ipxe succeed.

(alternatively in case of using virt-install one can use --boot loader option to explicitly specify which var store to use)

Test PXE boot with RHEL 8.2 and RHEL 8.3,still works well.

1.Test with RHEL8.2 host

env:
kernel-4.18.0-193.el8.x86_64
qemu-kvm-2.12.0-99.module+el8.2.0+5827+8c39933c.x86_64
libvirt-client-4.5.0-42.module+el8.2.0+6024+15a2423f.x86_64
edk2-ovmf-20190829git37eef91017ad-9.el8.noarch

Result:
Installation successful.

2.Test with RHEL8.3 host(upgrade from 8.2 to 8.3)

env:
kernel-core-4.18.0-240.el8.x86_64
qemu-kvm-4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64
libvirt-client-6.0.0-28.module+el8.3.0+7827+5e65edd7.x86_64
edk2-ovmf-20200602gitca407c7246bf-3.el8.noarch

Result:
Installation successful.

virt-install command:
#!/bin/sh
VMNAME=pxe
sudo rm -f  /home/pxe/$VMNAME.raw
sudo fallocate -l 50G /home/pxe/$VMNAME.raw
virt-install --name $VMNAME --boot uefi --pxe  --ram 32768 --vcpus 4 --cpu host --disk path="/home/pxe/$VMNAME.raw",target.bus=virtio --os-variant rhel8.2 --network bridge=private-switch,model.type=virtio,mac.address="52:54:00:aa:ad:04" --connect qemu:///system

Thank you everyone for contributing to the analysis thus far. I've just
returned from PTO.

Since RHEL-8.1 bug 1600230, we provide firmware descriptor metafiles in
the edk2-ovmf and edk2-aarch64 subpackages. Please see the message on
downstream commit 80c3c179a421 ("redhat: provide firmware descriptor
meta-files", 2019-04-09):

> Add two edk2-ovmf and two edk2-aarch64 metafiles, advertizing four
> firmware use cases in total to management software. For interpreting
> the contents and (installed) names of the JSON files, please consult
> "docs/interop/firmware.json" in the QEMU source directory.
>
> The priority prefixes are made up as follows:
>
> - 4x: Secure Boot feature present, keys enrolled    (edk2-ovmf-sb)
> - 5x: Secure Boot feature present, no keys enrolled (edk2-ovmf)
> - 6x: Secure Boot feature absent, silent build      (edk2-aarch64)
> - 7x: Secure Boot feature absent, verbose build     (edk2-aarch64-verbose)

By default, newly defined domains will instantiate their UEFI variable
store files, from "/usr/share/edk2/ovmf/OVMF_VARS.secboot.fd". This
enables the Secure Boot operational mode right off the bat, for the
domain, when it is started for the very first time.

Rejecting an unsigned ipxe.efi binary (loaded from local media or
remotely from the network) is proper behavior in this case.

In addition, signing the utilities on UefiShell.iso would be wrong -- we
offer two binaries on that ISO image, the UEFI shell, and the
EnrollDefaultKeys utility.

- Signing the former (the UEFI shell) would be self-defeating, as UEFI
  shell commands / UEFI shell scripts do not undergo SB verification,
  but they can be used to undermine SB. Avoiding this scenario is
  exactly why we *exclude* the UEFI shell from the OVMF firmware binary
  image itself, and only offer the shell on a separate ISO image.

- Regarding EnrollDefaultKeys.efi, you don't *need* to run that binary
  when SB is already enabled. EnrollDefaultKeys.efi is useful for two
  purposes: (i) as part of the package build in Brew, when we prepare
  "OVMF_VARS.secboot.fd" in the first place, and (ii) if you want to
  expose the Platform Key / First Key Exchange Key to the guest firmware
  via an SMBIOS Type 11 table from the host side, and enroll those
  certificates by invoking EnrollDefaultKeys.efi manually from the UEFI
  shell command line. Neither of these cases apply when the SB
  operational mode is already enabled.

If you want the previous behavior (that is, SB operational mode disabled
for a newly defined domain), then please refer to bullet (7) at:

  https://bugzilla.redhat.com/show_bug.cgi?id=1869625#c9

As described there, there are three options (alternatives):

(a) For defining just one domain with the SB operational mode disabled,
    invoke "virt-install" with the following option explicitly:

      --boot uefi,nvram_template=/usr/share/edk2/ovmf/OVMF_VARS.fd

(b) For defining just one domain with the SB operational mode disabled,
    invoke "virsh define" with the following domain XML snippet:

      <domain type='kvm'>
        <os>
          <loader
           readonly='yes'
           secure='yes'
           type='pflash'
          >/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader>
          <nvram template='/usr/share/edk2/ovmf/OVMF_VARS.fd'/>
        </os>
        <features>
          <smm state='on'/>
        </features>
      </domain>

(c) For defining all further (system-level) domains with the SB
    operational mode disabled, invoke the following commands as root:

  mkdir -p /etc/qemu/firmware
  touch /etc/qemu/firmware/40-edk2-ovmf-sb.json

This will "hide" the firmware descriptor
"/usr/share/qemu/firmware/40-edk2-ovmf-sb.json", and allow
"/usr/share/qemu/firmware/50-edk2-ovmf.json" to take priority.


IMPORTANT:

- Do not overwrite files under /usr/share.

- Do not install edk2 packages from other distributions, in a RHEL8
  production environment.

- Do not reference files under /usr/share in the *text contents* of the
  <nvram> element in the domain XML.

- Referencing files under /usr/share in the *template attribute* of the
  <nvram> element is fine.

All alternatives (a) through (c), listed above, honor these
requirements.

(In the longer term, we might want to turn this into a Knowledge Base
article or something similar, as the same topic had come up elsewhere,
for example <https://bugzilla.redhat.com/show_bug.cgi?id=1867449>.)

Thanks
Laszlo

Clearing needinfo requests per comment 23.

Also, to clarify, this is NOTABUG.

I'll let Robert close the BZ as such, after advising the customer according to comment 23.

Clarifying yet further, PXEv4 / PXEv6 boot with the SB operational mode
enabled is expected to work; it has been tested several times. You just
have to set up the UEFI PXE boot environment properly, exactly as
described in the RHEL8 installation manual.

In particular, the DHCPv4 / DHCPv6 server must advertize the "shim"
utility as the remote boot loader. "shim" can be PXE-booted by OVMF just
fine, with the SB operational mode enabled. Subsequently, "shim" will
PXE-load "grub" (and verify the signature), and "grub" will PXE-load the
kernel image and the initial ramdisk (and very the signatures on those).

If you attempt to PXE-boot an unsigned "ipxe.efi" binary as first stage
from OVMF, or you attempt to omit "shim" and directly PXE-load "grub",
it's expected that those will be rejected with SB enabled.

Some help on configuring UEFI netboot environments:

- PXEv4 setup:
  https://bugzilla.redhat.com/show_bug.cgi?id=1462351#c9

- PXEv6 setup (builds upon the PXEv4 setup):
  https://bugzilla.redhat.com/show_bug.cgi?id=1536627#c3

- HTTP(S)v4/v6 setup:
  https://bugzilla.redhat.com/show_bug.cgi?id=1536624#c58

(In reply to Laszlo Ersek from comment #25)
> Also, to clarify, this is NOTABUG.
> 
> I'll let Robert close the BZ as such, after advising the customer according
> to comment 23.

Even though the behavior of QEMU/edk2 is correct, the user-experience is not what one would expect. So let's not close this bug until we have a way to improve that, with something like a better error message, different defaults, or at the very least, some documentation.

It would not be difficult to invert the priorities between the firmware descriptor files

  /usr/share/qemu/firmware/40-edk2-ovmf-sb.json
  /usr/share/qemu/firmware/50-edk2-ovmf.json

in the edk2-ovmf package. Exchanging the "40" and "50" priority prefixes would result in new domains being defined with the SB operational mode disabled.

If that's the desired way forward, please reassign the BZ to the edk2 component. Thank you!

(In reply to Laszlo Ersek from comment #29)
> It would not be difficult to invert the priorities between the firmware
> descriptor files
> 
>   /usr/share/qemu/firmware/40-edk2-ovmf-sb.json
>   /usr/share/qemu/firmware/50-edk2-ovmf.json
> 
> in the edk2-ovmf package. Exchanging the "40" and "50" priority prefixes
> would result in new domains being defined with the SB operational mode
> disabled.
> 
> If that's the desired way forward, please reassign the BZ to the edk2
> component. Thank you!

Isn't this shifting the brokeness from one group of people (those who don't care about secure boot), onto a different group of people (those expecting secureboot to be operational). Doesn't feel like a net win in general.

We have a specific issue with use of virt-install, so I don't think we should be changing the host-wide defaults that apply to all applications.

It feels to me like libvirt isn't giving enough flexibility in configuration.

We have "secure=yes|no" to state whether secure boot should be present, but nothing to state whether we're requiring it to be operational or not.

eg virt-install should have a way to specify whether secure boot is operational or not, which it can set in libvirt - it shouldn't need users to specify full file paths.

Agreed with Dan, changing the default will not solve the problem for all users.

We definitely need to update documentation to explicitly state what the default is and provide steps to change it to the other configuration.

Now for the steps there is no easy way to set it in libvirt XML so currently the only option is providing full path to the varstore. To have it more user-friendly it would require new value for "secure" attribute or new attribute and wiring it up in virt-install as well.

(In reply to Daniel Berrangé from comment #32)
> (In reply to Laszlo Ersek from comment #29)
> > It would not be difficult to invert the priorities between the
> > firmware descriptor files
> >
> >   /usr/share/qemu/firmware/40-edk2-ovmf-sb.json
> >   /usr/share/qemu/firmware/50-edk2-ovmf.json
> >
> > in the edk2-ovmf package. Exchanging the "40" and "50" priority
> > prefixes would result in new domains being defined with the SB
> > operational mode disabled.
> >
> > If that's the desired way forward, please reassign the BZ to the
> > edk2 component. Thank you!
>
> Isn't this shifting the brokeness from one group of people (those who
> don't care about secure boot), onto a different group of people (those
> expecting secureboot to be operational).

Yes, that's correct.

Not dissimilar to buying a physical laptop with SB enabled out of the
box (with a particular set of certificates enrolled) vs. one that has SB
disabled.

In comment 23, I quoted the RHEL-8.1 commit that introduced the
priorities. In that commit message, I was careful to use the term "made
up". I had figured we should pick the "strongest defaults" in RHEL,
having done no actual "market research". If customers turn out to prefer
a different default, we can adapt. But, we can't satisfy two opposite
preferences with a single default.

> Doesn't feel like a net win in general.

I agree. It would be better to stick with the current default.

> We have a specific issue with use of virt-install, so I don't think we
> should be changing the host-wide defaults that apply to all
> applications.
>
> It feels to me like libvirt isn't giving enough flexibility in
> configuration.
>
> We have "secure=yes|no" to state whether secure boot should be
> present, but nothing to state whether we're requiring it to be
> operational or not.
>
> eg virt-install should have a way to specify whether secure boot is
> operational or not, which it can set in libvirt - it shouldn't need
> users to specify full file paths.

This should be doable (with libvirt daemon development); namely if you
diff the two descriptor files now, you get

> --- /usr/share/qemu/firmware/40-edk2-ovmf-sb.json     2020-11-10 11:11:07.000000000 +0100
> +++ /usr/share/qemu/firmware/50-edk2-ovmf.json        2020-11-10 11:11:07.000000000 +0100
> @@ -1,5 +1,5 @@
>  {
> -    "description": "OVMF with SB+SMM, SB enabled, MS certs enrolled",
> +    "description": "OVMF with SB+SMM, empty varstore",
>      "interface-types": [
>          "uefi"
>      ],
> @@ -10,7 +10,7 @@
>              "format": "raw"
>          },
>          "nvram-template": {
> -            "filename": "/usr/share/edk2/ovmf/OVMF_VARS.secboot.fd",
> +            "filename": "/usr/share/edk2/ovmf/OVMF_VARS.fd",
>              "format": "raw"
>          }
>      },
> @@ -25,7 +25,6 @@
>      "features": [
>          "acpi-s3",
>          "amd-sev",
> -        "enrolled-keys",
>          "requires-smm",
>          "secure-boot",
>          "verbose-dynamic"

So the feature to look for is "enrolled-keys".

The domain XML syntax for firmware auto-selection is:

  <os firmware='efi'>
    <loader secure='yes'/>
  </os>

So I guess (a) virt-install would have to generate such an XML -- which
I understand it doesn't do yet? --, and (b) libvirtd would have to
introduce new syntax (a new attribute?) and new logic to filter for the
"enrolled-keys" feature.

Related:
- https://bugzilla.redhat.com/show_bug.cgi?id=1564270
- https://bugzilla.redhat.com/show_bug.cgi?id=1776949

>   <os firmware='efi'>
>     <loader secure='yes'/>
>   </os>
> 
> So I guess (a) virt-install would have to generate such an XML -- which
> I understand it doesn't do yet? --, and (b) libvirtd would have to
> introduce new syntax (a new attribute?) and new logic to filter for the
> "enrolled-keys" feature.

Well, its not the only option we have.  The alternatives I see are:

  (1) ship signed ipxe roms
  (2) support virtio-net only (that should work fine without ipxe
      given that edk2 has a virtio-net driver, right?)

just my two cents.

(In reply to Gerd Hoffmann from comment #36)
> >   <os firmware='efi'>
> >     <loader secure='yes'/>
> >   </os>
> >
> > So I guess (a) virt-install would have to generate such an XML --
> > which I understand it doesn't do yet? --, and (b) libvirtd would
> > have to introduce new syntax (a new attribute?) and new logic to
> > filter for the "enrolled-keys" feature.
>
> Well, its not the only option we have.  The alternatives I see are:
>
>   (1) ship signed ipxe roms
>   (2) support virtio-net only (that should work fine without ipxe
>       given that edk2 has a virtio-net driver, right?)

There are two kinds of iPXE binaries to consider:

(i) The "CONFIG=qemu" builds that we include in ipxe-roms-qemu.

    These are stripped-down builds which only provide
    EFI_SIMPLE_NETWORK_PROTOCOL implementations for a number of emulated
    NICs; they enable the edk2 network stack to work with some QEMU NICs
    that differ from virtio-net.

(ii) The full-blown build (ipxe.efi) that we include in ipxe-bootimgs.

     This is a binary that includes a bunch of advanced iPXE features,
     such as scripting, iPXE's own IP/TCP/HTTP... stacks, and so on.

     This binary is possible to use in an OVMF guest as a *second stage*
     netboot program (i.e., once it has been downloaded by the edk2
     network stack).


The binaries in (i) need not be signed, because  driver binaries that
come from PCI option ROM BARs are exempt from SB verification in OVMF.

(Exception: when running in an SEV guest, even UEFI drivers of such
origin are subjected to SB verification. However, iPXE is not
SEV-capable in the first place, so if you want to netboot an SEV OVMF
guest, you have to use the built-in VirtioNetDxe driver anyway.)


The binary in (ii) could be signed, but:

- ProdSec would have to be involved, and we'd have to figure out what
  our signature means (e.g., only that ipxe.efi comes from us?).

- Signing anything would require quite a bit of infrastructure (Peter
  Jones could advise us on how the private key is handled that shim.efi
  is signed with), and then a further question is whether we use
  different keys for signing development builds vs. release builds (cf.
  test vs. release kernels are signed with different keys).

- Customers choose to PXE-boot a wide range of UEFI boot loaders, such
  as:

  - an ipxe.efi they downloaded from the iPXE website,

  - an ipxe.efi they built with a custom feature set,

  - various Windows netboot loaders (variants of WDSNBP.COM for
    example).

  Signing our own "ipxe.efi" in ipxe-bootimgs would not help with these
  use cases.


Restricting our set of supported NICs to virtio-net would also not help
with said custom (2nd stage) netboot programs, as customers use those
UEFI boot loaders for their "business logic" (such as iPXE scripting, or
custom Windows deployment). And that's orthogonal to the underlying NIC
model.

Upstream commits:

c91fa27306 qemu: implement support for firmware auto-selection feature filtering
cff524af6c conf: introduce support for firmware auto-selection feature filtering
6330be1ba3 conf: use switch in virDomainDefParseBootOptions
108cb29c1c conf: introduce virDomainDefParseBootAcpiOptions
b8dd70db4e conf: introduce virDomainDefParseBootLoaderOptions
bcf97abfc6 conf: introduce virDomainDefParseBootFirmwareOptions
bf9b3f8e57 conf: introduce virDomainDefParseBootKernelOptions
b07116438c conf: introduce virDomainDefParseBootInitOptions
f47d06260b docs: improve description of secure attribute for loader element

Hi, Pavel:

I have two questions when I test this bug, can you help to review them?

1) When start the guest with the following xml, it will hit an error.
# virsh dumpxml ovmf | grep /os -B7
  <os firmware='efi'>
    <type arch='x86_64' machine='pc-q35-6.0'>hvm</type>
    <firmware>
      <feature enabled='yes' name='enrolled-keys'/>
      <feature enabled='no' name='secure-boot'/>
    </firmware>
    <boot dev='hd'/>
  </os>
# virsh start ovmf
error: Failed to start domain 'ovmf'
error: operation failed: Unable to find any firmware to satisfy 'efi'
-----------
I think sometimes this error will confuse the user. It's better to have an clear message, for example, not support this combination. 
Or provide a related firmware descriptor metafiles.

2) From the patches we can know that it support "<firmware type='efi'/>". But actually it's not supported in guest xml.
# cat ovmf.xml | grep efi 
  <os firmware='efi'>
    <firmware type='efi'/>
# virsh define ovmf.xml 
Domain 'ovmf' defined from ovmf.xml
# virsh dumpxml --inactive ovmf | grep /os -B5
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os firmware='efi'>
    <type arch='x86_64' machine='pc-q35-6.0'>hvm</type>
    <boot dev='hd'/>
  </os>

Test Version:
# rpm -q libvirt qemu-kvm
libvirt-7.3.0-1.fc33.x86_64
qemu-kvm-6.0.0-0.1.rc2.fc35.x86_64


Thanks

(In reply to Meina Li from comment #40)
> Hi, Pavel:
> 
> I have two questions when I test this bug, can you help to review them?
> 
> 1) When start the guest with the following xml, it will hit an error.
> # virsh dumpxml ovmf | grep /os -B7
>   <os firmware='efi'>
>     <type arch='x86_64' machine='pc-q35-6.0'>hvm</type>
>     <firmware>
>       <feature enabled='yes' name='enrolled-keys'/>
>       <feature enabled='no' name='secure-boot'/>
>     </firmware>
>     <boot dev='hd'/>
>   </os>
> # virsh start ovmf
> error: Failed to start domain 'ovmf'
> error: operation failed: Unable to find any firmware to satisfy 'efi'
> -----------
> I think sometimes this error will confuse the user. It's better to have an
> clear message, for example, not support this combination. 
> Or provide a related firmware descriptor metafiles.

The firmware descriptor files are provided by QEMU and that combination doesn't make any sense because without secure-boot the enrolled-keys would not be used.
Having better error message would be nice but I don't think it should block this BZ. This would be ideal to track in upstream gitlab issues and it would eventually get into RHEL as well.

> 2) From the patches we can know that it support "<firmware type='efi'/>".
> But actually it's not supported in guest xml.
> # cat ovmf.xml | grep efi 
>   <os firmware='efi'>
>     <firmware type='efi'/>
> # virsh define ovmf.xml 
> Domain 'ovmf' defined from ovmf.xml
> # virsh dumpxml --inactive ovmf | grep /os -B5
>   <currentMemory unit='KiB'>2097152</currentMemory>
>   <vcpu placement='static'>2</vcpu>
>   <os firmware='efi'>
>     <type arch='x86_64' machine='pc-q35-6.0'>hvm</type>
>     <boot dev='hd'/>
>   </os>

This is correct. My original patches added support for <firmware type='efi'> but later it was dropped before libvirt-7.2.0 release as it was useless and confusing.

commit a9b1375d7d2f7d240dce09c5f8b62e568e386051
Author: Daniel P. Berrangé <berrange>
Date:   Mon Mar 29 19:00:05 2021 +0100

    conf: remove duplicated firmware type attribute

> Test Version:
> # rpm -q libvirt qemu-kvm
> libvirt-7.3.0-1.fc33.x86_64
> qemu-kvm-6.0.0-0.1.rc2.fc35.x86_64
> 
> 
> Thanks

Verified on libvirt-7.3.0-1.fc33.x86_64 and qemu-kvm-6.0.0-0.1.rc2.fc35.x86_64

S1: Boot OVMF guest with released image and different features combination

Test Matrix:
secure-boot	enrolled-key	Expected Result  	       Note
yes	        yes	        can boot	               the recommended combination
yes	        no	        can boot	
no	        no	        can boot	
no	        yes	        can't start with error	       no sense in this combination

S2: Boot OVMF guest with  latest image and different features combination

Test Matrix:
secure-boot	enrolled-key	Expected Result	                                     Note
yes	        yes	       can't boot with has invalid signature error	because the image don't include the enrolled key
yes	        no	       can't boot with has invalid signature error	
no	        no	       can boot	                                        the recommended combination
no	        yes	       can't start with error	                        no sense in this combination

Verified Steps:
1. Prepare an OVMF guest with the following xml and the above test matrix.
# virsh dumpxml --inactive ovmf | grep /os -B7
  <os firmware='efi'>
    <type arch='x86_64' machine='pc-q35-rhel8.4.0'>hvm</type>
    <firmware>
      <feature enabled='**yes**' name='enrolled-keys'/>
      <feature enabled='**yes**' name='secure-boot'/>
    </firmware>
    <boot dev='hd'/>
  </os>
2. Start the guest and check the boot process.
# virsh start ovmf
Domain 'ovmf' started

# virt-viewer ovmf

The test results are expected.

Hi, Pavel:

I still have two questions about this bug. Can you help to check it again?

1) As you said, without secure-boot the enrolled-keys would not be used. Does this mean that as long as the secure-boot is no, the guest does not support booting？
I was confused on this because I've got different test result in previous upstream test in comment 42 and in RHEL-AV 8.5 test.
libvirt-7.3.0-1.fc33.x86_64 secure-boot:no enrolled-keys:no   The guest can boot
libvirt-7.3.0-1.el8         secure-boot:no enrolled-keys:no   The guest can't start

2) Please also help to check if the test scenarios are enough to test this bug in comment 42.
Correct a test result in my previous test:
The guest with an unenrolled images can boot successfully when test with secure-boot=‘yes’ and enrolled-keys='no'.

Thank you very much.

(In reply to Meina Li from comment #42)
> Verified on libvirt-7.3.0-1.fc33.x86_64 and
> qemu-kvm-6.0.0-0.1.rc2.fc35.x86_64
> 
> S1: Boot OVMF guest with released image and different features combination
>
> Test Matrix:
> secure-boot	enrolled-key	Expected Result  	       Note
> yes	        yes	        can boot	               the recommended combination
> yes	        no	        can boot	
> no	        no	        can boot	
> no	        yes	        can't start with error	       no sense in this combination
> 
> S2: Boot OVMF guest with  latest image and different features combination
> 
> Test Matrix:
> secure-boot	enrolled-key	Expected Result	                                   
> Note
> yes	        yes	       can't boot with has invalid signature error	because the image don't include the enrolled key
> yes	        no	       can't boot with has invalid signature error	

This one should boot correctly.

> no	        no	       can boot	                                        the recommended combination

This is not true as libvirt should fail booting the VM as it should not by able to find suitable firmware.

> no	        yes	       can't start with error	                        no sense in this combination
> 
> Verified Steps:
> 1. Prepare an OVMF guest with the following xml and the above test matrix.

You should mention that in the guest you need to boot non-signed binary to trigger the invalid signature error.

> # virsh dumpxml --inactive ovmf | grep /os -B7
>   <os firmware='efi'>
>     <type arch='x86_64' machine='pc-q35-rhel8.4.0'>hvm</type>
>     <firmware>
>       <feature enabled='**yes**' name='enrolled-keys'/>
>       <feature enabled='**yes**' name='secure-boot'/>
>     </firmware>
>     <boot dev='hd'/>
>   </os>
> 2. Start the guest and check the boot process.
> # virsh start ovmf
> Domain 'ovmf' started
> 
> # virt-viewer ovmf
> 
> The test results are expected.

(In reply to Meina Li from comment #45)
> Hi, Pavel:
> 
> I still have two questions about this bug. Can you help to check it again?
> 
> 1) As you said, without secure-boot the enrolled-keys would not be used.
> Does this mean that as long as the secure-boot is no, the guest does not
> support booting？
> I was confused on this because I've got different test result in previous
> upstream test in comment 42 and in RHEL-AV 8.5 test.
> libvirt-7.3.0-1.fc33.x86_64 secure-boot:no enrolled-keys:no   The guest can
> boot
> libvirt-7.3.0-1.el8         secure-boot:no enrolled-keys:no   The guest
> can't start

The situation with firmware is getting a little bit complicated, let me describe what we currently have in RHEL:

RHEL-8.4 pakcage edk2-ovmf provides these firmware JSON files:

   /usr/share/qemu/firmware/40-edk2-ovmf-sb.json    secure-boot=yes  enrolled-keys=yes
   /usr/share/qemu/firmware/50-edk2-ovmf.json       secure-boot=yes  enrolled-keys=no

RHEL-8.5 with the most recent edk2-ovmf provides these firmware JSON files:

   /usr/share/qemu/firmware/40-edk2-ovmf-sb.json    secure-boot=yes  enrolled-keys=yes
   /usr/share/qemu/firmware/50-edk2-ovmf-cc.json    secure-boot=no   enrolled-keys=no
   /usr/share/qemu/firmware/50-edk2-ovmf.json       secure-boot=yes  enrolled-keys=no

Technically in RHEL-8.5 we have combination secure-boot=no and enrolled-keys=no but the 50-edk2-ovmf-cc.json has other limitations:

   it will require pc-q35-rhel8.5.0 machine type which will be available only on RHEL-AV-8.5.0, not in the basic RHEL-8.5
   it doesn't enable any other features and will is supposed to be used only with AMD SEV-ES

so that for regular usage that combination is invalid as well.

To update the matrix for RHEL-8.5.0 when booting non-signed OS inside the guest should look like this:

secure-boot	enrolled-key	Expected Result	                                     Note
yes	        yes	       can't boot with has invalid signature error	because the image don't include the enrolled key
yes	        no	       can boot
no	        no	       can't boot because libvirt fails with error	because there is no valid firmware to satisfy this combination
no	        yes	       can't start with error	                        no sense in this combination


> 2) Please also help to check if the test scenarios are enough to test this
> bug in comment 42.
> Correct a test result in my previous test:
> The guest with an unenrolled images can boot successfully when test with
> secure-boot=‘yes’ and enrolled-keys='no'.
> 
> Thank you very much.

Verified Version:
libvirt-7.3.0-1.scrmod+el8.5.0+11030+6b744098.x86_64
qemu-kvm-6.0.0-17.module+el8.5.0+11173+c9fce0bb.x86_64

Verified Steps:
1. Prepare an OVMF guest with the signed image and unsigned image respectively.
2. Update enabled features with different combination in guest xml.
# virsh dumpxml --inactive ovmf | grep /os -B7
  <os firmware='efi'>
    <type arch='x86_64' machine='pc-q35-rhel8.4.0'>hvm</type>
    <firmware>
      <feature enabled='**yes**' name='enrolled-keys'/>
      <feature enabled='**yes**' name='secure-boot'/>
    </firmware>
    <boot dev='hd'/>
  </os>
2. Start the guest and check the boot process.
# virsh start ovmf
Domain 'ovmf' started

# virt-viewer ovmf

Test Matrix:
S1: Boot OVMF guest with signed image and different features combination
Test Matrix:
secure-boot  enrolled-keys  Expected Result  	     
yes	          yes	          can boot	              
yes	          no	          can boot	
no	          no	          can't start guest with error
no	          yes	          can't start guest with error	     

S2: Boot OVMF guest with non-signed image and different features combination
secure-boot  enrolled-keys  Expected Result  	     
yes	          yes	          can't boot with has invalid signature error	              
yes	          no	          can boot
no	          no	          can't start guest with error
no	          yes	          can't start guest with error	

The test results are expected.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4684