Bug 1419191

Nova's code base currently contains this warning: 

"uefi support is without some kind of "
"functional testing and therefore "
"considered experimental."

Ensuring test coverage for existing Nova UEFI functionality is in effect the key aim of this bug. Ensuring that we can launch a UEFI guest using Nova, and that we can automate validation of same on future releases.

Created attachment 1268935 [details]
Using Openstack* UEFI Boot Support to Enhance Security in Multi-Tenant Cloud Applications

Intel pdf - Using Openstack* UEFI Boot Support to Enhance Security in Multi-Tenant Cloud Applications

(In reply to Edu Alcaniz from comment #4)
> Created attachment 1268935 [details]
> Using Openstack* UEFI Boot Support to Enhance Security in Multi-Tenant Cloud
> Applications
> 
> Intel pdf - Using Openstack* UEFI Boot Support to Enhance Security in
> Multi-Tenant Cloud Applications

NB, that PDF title is rather misleading. The use of UEFI as described does little to nothing to improve security of the host OS. The only supposed security improvement described in that document is guest related, and comes from the fact that they're booting clearlinux guest OS which has a cut down image foot print with minimal services running. The inference is that fewer services / less code in guest OS means greater security of the guest OS.

Now UEFI is capable of improving security more generally, if nova were to support its secure boot features, but that is not described in the doc and not implemented in Nova. 

IOW, as it stands today, UEFI support in Nova is only really compelling if the guest OS being used has dropped support for traditional BIOS boot.

Created attachment 1286183 [details]
Nova guest XML booted with Q35 machine type and OVMF

Kashyap, are you saying this works with openstack-nova-16.0.0-0.20170609171024.d147277.el7ost without changes? If not I'd expect the outcome of this work before closing to be filing bugs for the changes required to make it work OOTB.

Moving to Queens as it seems enablement work is outstanding.

[Sorry for the late reply, I missed the NEEDINFO e-mail.]

(In reply to Stephen Gordon from comment #23)
> Kashyap, are you saying this works with
> openstack-nova-16.0.0-0.20170609171024.d147277.el7ost without changes? 

Yes: Only for RHEL guests on RHEL hosts, it works without any changes on our part.

>If not I'd expect the outcome of this work before closing to be filing bugs for
> the changes required to make it work OOTB.

Talking to Kashyap to confirm, we can go ahead and validate RHEL guests with UEFI enabled on RHOSP 12/RHEL 7.4 hosts from a QE perspective. No code changes required.

To clarify, the documentation procedure to start Nova instances with OVMF and Q35 machine type: is the following:

(1) Ensure to have Q35 machine type in /etc/nova/nova.conf:

    $ grep q35 /etc/nova/nova.conf 
    hw_machine_type = x86_64=q35


(2) Get the RHEL 7.4 image from a trusted source.

(3) Import it into Glance:

    $ openstack image create --disk-format qcow2 \
        --container-format bare --public \
        --file ./rhel-guest-image-7.4-159.x86_64.qcow2 rhel74

(4) Set the metadata property for UEFI on the RHEL 7.4 image:

    $ openstack image set --property hw_firmware_type=uefi rhel74

(5) Boot an instance:

    $ openstack server create --image rhel74 --flavor m1.medium el74vm1

(6) Ensure it boots successfully:

    $ sudo virsh dumpxml 16748513-2836-45db-ae62-156ff7ea0f6b | grep -e q35 -e OVMF
        <type arch='x86_64' machine='pc-q35-2.7'>hvm</type>
        <loader readonly='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.fd</loader>

(In reply to Kashyap Chamarthy from comment #31)
> To clarify, the documentation procedure to start Nova instances with OVMF
> and Q35 machine type: is the following:
> 
> (1) Ensure to have Q35 machine type in /etc/nova/nova.conf:
> 
>     $ grep q35 /etc/nova/nova.conf 
>     hw_machine_type = x86_64=q35

We should also be highlighting the ability to set this per guest via the image properties as well.

(In reply to Kashyap Chamarthy from comment #26)
> [Sorry for the late reply, I missed the NEEDINFO e-mail.]
> 
> (In reply to Stephen Gordon from comment #23)
> > Kashyap, are you saying this works with
> > openstack-nova-16.0.0-0.20170609171024.d147277.el7ost without changes? 
> 
> Yes: Only for RHEL guests on RHEL hosts, it works without any changes on our
> part.

I was wrong, and made a mistake in realizing distinguishing the secure boot versus non-secure boot variants.

For RHEL, I take it that we build OVMF only with secure mode.  Therefore the OVMF code file is:

   /usr/share/OVMF/OVMF_CODE.secboot.fd.

As can be seen from extracting a RHEL 7.4 OVMF RPM:

$ tree 
.
├── OVMF-20170228-3.gitc325e41585e3.el7.noarch.rpm
└── usr
    └── share
        ├── doc
        │   └── OVMF
        │       ├── Licenses
        │       │   ├── CryptoPkg-License.txt
        │       │   ├── FatPkg-License.txt
        │       │   ├── IntelFrameworkModulePkg-License.txt
        │       │   ├── MdeModulePkg-License.txt
        │       │   ├── MdePkg-License.txt
        │       │   ├── OpensslLib-License.txt
        │       │   ├── OptionRomPkg-License.txt
        │       │   ├── OvmfPkg-License.txt
        │       │   ├── PcAtChipsetPkg-License.txt
        │       │   ├── SecurityPkg-License.txt
        │       │   ├── ShellPkg-License.txt
        │       │   └── UefiCpuPkg-License.txt
        │       ├── ovmf-whitepaper-c770f8c.txt
        │       └── README
        └── OVMF
            ├── OVMF_CODE.secboot.fd
            ├── OVMF_VARS.fd
            └── UefiShell.iso


And the current Nova libvirt driver is checking the path to be:

    [...]
    DEFAULT_UEFI_LOADER_PATH = {
        "x86_64": "/usr/share/OVMF/OVMF_CODE.fd",
        "aarch64": "/usr/share/AAVMF/AAVMF_CODE.fd"
    }
    [...]

Now I realize, without code changes to explicitly point to the secure
boot binary in Nova, and enabling the SMM (Secure Management Mode) it will not work.

Two things are required for the guest XML that Nova generates:

  (1) Enable SMM in the guest XML:


      <features>
        [...]
        <smm state='on'/>
      </features>

  (2) And the OVMF 'loader' related snippet looks As following, with Q35:

      <os>
        <type arch='x86_64' machine='pc-q35-2.7'>hvm</type>
        <loader readonly='yes' secure='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.secboot.fd</loader>
        <nvram>/var/lib/libvirt/qemu/nvram/instance-00000007_VARS.fd</nvram>
        <boot dev='hd'/>
        <smbios mode='sysinfo'/>
      </os>


So the procedure I outlined in comment#31 will NOT work with OVMF secure boot.

[...]

Given my comment#33, where the two things that are required from a *RHEL* perspective was:

(1) Make Nova to generate the guest XML with the 'smm' as part of the 
    'feature' attribute: <smm state='on'/>

(2) Use the *correct* OVMF loader variable path (for secure boot) for 
    RHEL: <loader readonly='yes' secure='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.secboot.fd</loader>

    NB: From RHEL 7.3 onwards we ship only the secure boot variant of OVMF

Therefore, I am putting this bug back to ASSIGNED state.

NOTE: We should also perhaps have a conversation with upstream Nova, as the OVMF binary names are different for different distributions (e.g. SUSE uses a different path than what RHEL has).

(In reply to Kashyap Chamarthy from comment #33)

> (2) And the OVMF 'loader' related snippet looks As following, with Q35:
>
>     <os>
>       <type arch='x86_64' machine='pc-q35-2.7'>hvm</type>
>       <loader
>        readonly='yes'
>        secure='yes'
>        type='pflash'
>       >/usr/share/OVMF/OVMF_CODE.secboot.fd</loader>
>       <nvram>/var/lib/libvirt/qemu/nvram/instance-00000007_VARS.fd</nvram>
>       <boot dev='hd'/>
>       <smbios mode='sysinfo'/>
>     </os>

Small update regarding the machine type:
- for downstream, you'll want to use pc-q35-rhel7.4.0 or later
- for upstream, you'll want to use pc-q35-2.9 or later
  (see <https://da.gd/mps4Q> why.)

(In reply to Kashyap Chamarthy from comment #34)

> NOTE: We should also perhaps have a conversation with upstream Nova, as
> the OVMF binary names are different for different distributions (e.g. SUSE
> uses a different path than what RHEL has).

I suggest conferring with developers of other virt tools (CC Cole, Pavel,
Rich) to see how they are handling this already. I vaguely recall that the
virt tools:
- have builtin pathnames,
- and can also query domcapabilities.

In the second case (domcapabilities), the following XPath expression returns
a list of firmware binaries:

  /domainCapabilities/os[supported='yes']/loader[supported='yes']/value

and the first entry thereof is used by the tools as "default UEFI firmware"
(I think?)

In turn the above <value> element list reflects the "nvram" stanza from
"/etc/libvirt/qemu.conf" -- or, if that stanza is missing, the default
firmware pathnames built into libvirt.

(In reply to Kashyap Chamarthy from comment #33)

>       <os>
>         <boot dev='hd'/>
>       </os>


Also, please use per-device <boot order='N'/> elements instead, like the the libvirt docs suggest. Thanks.

Here is the WIP specification:

    https://review.openstack.org/#/c/506720/ -- Add ability for OVMF Secure Boot

There's another critical item that we're missing here to actually make secure boot really *secure*, one must load the Secure Boot keys from Microsoft.  E.g. the steps described here:

 https://fedoraproject.org/wiki/Using_UEFI_with_QEMU#Run_EnrollDefaultKeys.efi


In a Fedora environment, the complete steps are as follows:
-----------------------------------------------------------------------
1) While the guest is shutdown, attach the UEFI shell ISO as a CD-ROM:

      $ virsh attach-disk f26-ovmf-vm2 \
          /usr/share/edk2/ovmf/UefiShell.iso vdc --type cdrom \
          --mode readonly --config --targetbus virtio
      Disk attached successfully

(2) Change the boot order of the devices so that the CD-ROM with
    'UefiShell.iso' is the first device (boot order starts from '1')
    that the guest boots into:

      $ virsh edit f26-ovmf-vm2
      [...]

    So that the fragment looks as below:

      <disk type='file' device='cdrom'>
        <driver name='qemu' type='raw'/>
        <source file='/usr/share/edk2/ovmf/UefiShell.iso'/>
        <target dev='vdc' bus='virtio'/>
        <readonly/>
      <boot order='1'/>

(3) Start the guest on Serial console:

      [root@taroxhost ~]# virsh start f26-ovmf-vm2 --console
      Domain f26-ovmf-vm2 started
      Connected to domain f26-ovmf-vm2
      Escape character is ^]
      UEFI Interactive Shell v2.1
      EDK II
      UEFI v2.60 (EDK II, 0x00010000)
      Mapping table
            FS0: Alias(s):HD0b:;BLK1:
                PciRoot(0x0)/Pci(0x1,0x2)/Pci(0x0,0x0)/HD(1,GPT,C76241E2-7F41-406E-8A6E-A73274B6DE1F,0x800,0x64000)
            FS1: Alias(s):CD1a:;BLK5:
                PciRoot(0x0)/Pci(0x1,0x5)/Pci(0x0,0x0)/CDROM(0x0)
           BLK0: Alias(s):
                PciRoot(0x0)/Pci(0x1,0x2)/Pci(0x0,0x0)
           BLK2: Alias(s):
                PciRoot(0x0)/Pci(0x1,0x2)/Pci(0x0,0x0)/HD(2,GPT,E17A8175-6CE8-4E21-801E-4D4D81D00B40,0x64800,0x200000)
           BLK3: Alias(s):
                PciRoot(0x0)/Pci(0x1,0x2)/Pci(0x0,0x0)/HD(3,GPT,8712A3D6-2835-49D0-BA37-41F05C14FBD2,0x264800,0x759B000)
           BLK4: Alias(s):
                PciRoot(0x0)/Pci(0x1,0x5)/Pci(0x0,0x0)



      Press ESC in 1 seconds to skip startup.nsh or any other key to continue.
      Shell>

(4) At the UEFI shell prompt, following the instructions from here [*],
    run the following commands ("fs0" -- change into the first device;
    "EnrollDefaultKeys.efi" -- which installs the Microsoft's KEK; and
    "reset" -- UEFI's way of saying reboot):

      [...]
      Shell> fs0:
      FS0:\> EnrollDefaultKeys.efi
      info: SetupMode=1 SecureBoot=0 SecureBootEnable=0 CustomMode=0 VendorKeys=1
      info: SetupMode=0 SecureBoot=1 SecureBootEnable=1 CustomMode=0 VendorKeys=0
      FS0:\> reset

(4.a)

    NOTE: If the UEFI Shell doesn't return cleanly after running
    "EnrollDefaultKeys.efi", just force power-off and power-on the guest
    from another shell.  E.g. to force power-off the guest, and start it
    again:

      $ sudo virsh destroy f26-ovmf-vm2
      $ sudo virsh start f26-ovmf-vm2 --console

      [This time boot into the Fedora 26 OS.]

    And on the guest serial console, you see Secure Boot enabled for
    *real*:

      ...
      EFI stub: UEFI Secure Boot is enabled.  
      ...
      [    0.000000] Secure boot enabled and kernel locked down

(5) To double-confirm, inside the guest, run `dmesg`:

      [root@f26-ovmf-vm2 ~]# dmesg | grep -i secure
      [    0.000000] Secure boot enabled and kernel locked down
      [    3.261277] EFI: Loaded cert 'Fedora Secure Boot CA: fde32599c2d61db1bf5807335d7b20e4cd963b42' linked to '.builtin_trusted_keys'
-----------------------------------------------------------------------

Created attachment 1359965 [details]
An example script that automatically creates OVMF variables ("VARS") file

The attached test program launches a QEMU guest, runs the UefiShell.iso, enrolls the default Secure Boot keys, and stores them in an OVMF VARS file called: output_VARS.fd.  This file then can then be used to boot guests, so that secure boot is enabled on all subsequent boots.

This just demonstrates a way to automate the Secure Boot key enrollment process on the UEFI shell.

---

To that end, Fedora Infrastructure developer Patrick Uiterwijk and myself worked towards creating a slightly more robust tool that will let you generate the OVMF variables ("VARS") file (along with a functional test):

https://github.com/puiterwijk/qemu-ovmf-secureboot/blob/master/ovmf-vars-generator.py

I'll be proposing this approach (of using a similar tool to generate the VARS file) in the Nova upstream spec to get further input.

Hi Kashyap,

(In reply to Kashyap Chamarthy from comment #46)
> Created attachment 1359965 [details]
> An example script that automatically creates OVMF variables ("VARS") file
> 
> The attached test program launches a QEMU guest, runs the UefiShell.iso,
> enrolls the default Secure Boot keys, and stores them in an OVMF VARS file
> called: output_VARS.fd.  This file then can then be used to boot guests, so
> that secure boot is enabled on all subsequent boots.
> 
> This just demonstrates a way to automate the Secure Boot key enrollment
> process on the UEFI shell.

This looks good. I can name one possible improvement (just for general QEMU-cleanliness): UefiShell.iso is supposed to be a CD-ROM image, so we should preferably expose it with a CD-ROM device, and not as a virtio-blk-pci device.

For that, there are two options. Both are equally functional, but the virtio variant is more performant:

(1) Use "-device ide-cd" rather than "-device virtio-blk-pci". This will place a SATA CD-ROM drive on Q35's built-in AHCI controller.

(2) Or else, use "-device virtio-scsi-pci,id=scsi0 -device scsi-cd,bus=scsi0.0,...". This will use a SCSI CD-ROM on a virtio-scsi controller.

... You could also drop the "virtio-rng-pci" device; it should not be necessary.

Just my two cents. Thanks.

The actual _RFE_ for support of OVMF with Secure Boot is being tracked here[*].

However, this bug itself is a test-only bug, and _technically_ can be closed as a duplicate of the RFE[*] bug. But there's a _ton_ of context and design discussion in this bug, and I am wary (because people don't normally bother to read "duplicate" bugs) of closing it just for the sake of bugzilla hygiene.

Maybe QE can use this bug for validating the functionality, once Nova wires it all up.


[*] https://bugzilla.redhat.com/show_bug.cgi?id=1369007 -- RFE: UEFI Secure Boot (SMM OVMF) of Nova Instances

Closing this out as a duplicate of 1369007 as it's coming up on our untriaged report due to missing target release and milestones etc, we can always link comments from this bug in the RFE.

*** This bug has been marked as a duplicate of bug 1369007 ***