Bug 1419191 - Validate Nova support for q35 and UEFI w/ OVMF
Summary: Validate Nova support for q35 and UEFI w/ OVMF
Keywords:
Status: CLOSED DUPLICATE of bug 1369007
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: ---
Assignee: Kashyap Chamarthy
QA Contact: Gabriel Szasz
URL:
Whiteboard:
Depends On: 1369007
Blocks: 1442136 1401758 1445659 1474725
TreeView+ depends on / blocked
 
Reported: 2017-02-03 20:38 UTC by Stephen Gordon
Modified: 2019-09-09 13:59 UTC (History)
19 users (show)

Fixed In Version: openstack-nova-16.0.0-0.20170609171024.d147277.el7ost
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-03-27 12:21:31 UTC
Target Upstream Version:


Attachments (Terms of Use)
Using Openstack* UEFI Boot Support to Enhance Security in Multi-Tenant Cloud Applications (77.93 KB, application/pdf)
2017-04-05 11:39 UTC, Edu Alcaniz
no flags Details
Nova guest XML booted with Q35 machine type and OVMF (6.02 KB, text/plain)
2017-06-08 15:31 UTC, Kashyap Chamarthy
no flags Details
An example script that automatically creates OVMF variables ("VARS") file (1.51 KB, text/plain)
2017-11-28 16:29 UTC, Kashyap Chamarthy
no flags Details

Description Stephen Gordon 2017-02-03 20:38:36 UTC
Description of problem:

We need to ensure that Nova, and our OOTB configuration of Nova, is able to support customers who wish to launch guests that use the q35 chipset, UEFI, and the OVMF that we provide.

The chipset and UEFI aspects are I believe already configurable via nova.conf and image properties respectively, the OVMF location is hardcoded which may be an issue (see also Bug # 1369007, Bug # 1401758).

Ideally the outcome of this bug is:

- Any changes to deployment config required to launch a q35 guest with UEFI enabled and OVMF (e.g. we may have to add a dependency on the OVMF package if it isn't already pulled in, configure the available machine types in nova.conf, etc.).

- Ensure we have one or more automation jobs that test this on an ongoing basis (ideally tempest should be testing this also!) to avoid regressions as seen in Bug # 1401758.

Comment 1 Stephen Gordon 2017-02-13 16:53:58 UTC
Nova's code base currently contains this warning: 

"uefi support is without some kind of "
"functional testing and therefore "
"considered experimental."

Ensuring test coverage for existing Nova UEFI functionality is in effect the key aim of this bug. Ensuring that we can launch a UEFI guest using Nova, and that we can automate validation of same on future releases.

Comment 4 Edu Alcaniz 2017-04-05 11:39:51 UTC
Created attachment 1268935 [details]
Using Openstack* UEFI Boot Support to Enhance Security in Multi-Tenant Cloud Applications

Intel pdf - Using Openstack* UEFI Boot Support to Enhance Security in Multi-Tenant Cloud Applications

Comment 6 Daniel Berrangé 2017-04-05 12:00:57 UTC
(In reply to Edu Alcaniz from comment #4)
> Created attachment 1268935 [details]
> Using Openstack* UEFI Boot Support to Enhance Security in Multi-Tenant Cloud
> Applications
> 
> Intel pdf - Using Openstack* UEFI Boot Support to Enhance Security in
> Multi-Tenant Cloud Applications

NB, that PDF title is rather misleading. The use of UEFI as described does little to nothing to improve security of the host OS. The only supposed security improvement described in that document is guest related, and comes from the fact that they're booting clearlinux guest OS which has a cut down image foot print with minimal services running. The inference is that fewer services / less code in guest OS means greater security of the guest OS.

Now UEFI is capable of improving security more generally, if nova were to support its secure boot features, but that is not described in the doc and not implemented in Nova. 

IOW, as it stands today, UEFI support in Nova is only really compelling if the guest OS being used has dropped support for traditional BIOS boot.

Comment 19 Kashyap Chamarthy 2017-06-08 15:31:41 UTC
Created attachment 1286183 [details]
Nova guest XML booted with Q35 machine type and OVMF

Comment 23 Stephen Gordon 2017-06-14 20:01:15 UTC
Kashyap, are you saying this works with openstack-nova-16.0.0-0.20170609171024.d147277.el7ost without changes? If not I'd expect the outcome of this work before closing to be filing bugs for the changes required to make it work OOTB.

Comment 24 Stephen Gordon 2017-06-15 17:42:21 UTC
Moving to Queens as it seems enablement work is outstanding.

Comment 26 Kashyap Chamarthy 2017-06-22 12:33:25 UTC
[Sorry for the late reply, I missed the NEEDINFO e-mail.]

(In reply to Stephen Gordon from comment #23)
> Kashyap, are you saying this works with
> openstack-nova-16.0.0-0.20170609171024.d147277.el7ost without changes? 

Yes: Only for RHEL guests on RHEL hosts, it works without any changes on our part.

>If not I'd expect the outcome of this work before closing to be filing bugs for
> the changes required to make it work OOTB.

Comment 28 Stephen Gordon 2017-06-22 14:55:33 UTC
Talking to Kashyap to confirm, we can go ahead and validate RHEL guests with UEFI enabled on RHOSP 12/RHEL 7.4 hosts from a QE perspective. No code changes required.

Comment 31 Kashyap Chamarthy 2017-07-24 14:43:32 UTC
To clarify, the documentation procedure to start Nova instances with OVMF and Q35 machine type: is the following:

(1) Ensure to have Q35 machine type in /etc/nova/nova.conf:

    $ grep q35 /etc/nova/nova.conf 
    hw_machine_type = x86_64=q35


(2) Get the RHEL 7.4 image from a trusted source.

(3) Import it into Glance:

    $ openstack image create --disk-format qcow2 \
        --container-format bare --public \
        --file ./rhel-guest-image-7.4-159.x86_64.qcow2 rhel74

(4) Set the metadata property for UEFI on the RHEL 7.4 image:

    $ openstack image set --property hw_firmware_type=uefi rhel74

(5) Boot an instance:

    $ openstack server create --image rhel74 --flavor m1.medium el74vm1

(6) Ensure it boots successfully:

    $ sudo virsh dumpxml 16748513-2836-45db-ae62-156ff7ea0f6b | grep -e q35 -e OVMF
        <type arch='x86_64' machine='pc-q35-2.7'>hvm</type>
        <loader readonly='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.fd</loader>

Comment 32 Stephen Gordon 2017-07-24 16:14:43 UTC
(In reply to Kashyap Chamarthy from comment #31)
> To clarify, the documentation procedure to start Nova instances with OVMF
> and Q35 machine type: is the following:
> 
> (1) Ensure to have Q35 machine type in /etc/nova/nova.conf:
> 
>     $ grep q35 /etc/nova/nova.conf 
>     hw_machine_type = x86_64=q35

We should also be highlighting the ability to set this per guest via the image properties as well.

Comment 33 Kashyap Chamarthy 2017-07-26 14:46:54 UTC
(In reply to Kashyap Chamarthy from comment #26)
> [Sorry for the late reply, I missed the NEEDINFO e-mail.]
> 
> (In reply to Stephen Gordon from comment #23)
> > Kashyap, are you saying this works with
> > openstack-nova-16.0.0-0.20170609171024.d147277.el7ost without changes? 
> 
> Yes: Only for RHEL guests on RHEL hosts, it works without any changes on our
> part.

I was wrong, and made a mistake in realizing distinguishing the secure boot versus non-secure boot variants.

For RHEL, I take it that we build OVMF only with secure mode.  Therefore the OVMF code file is:

   /usr/share/OVMF/OVMF_CODE.secboot.fd.

As can be seen from extracting a RHEL 7.4 OVMF RPM:

$ tree 
.
├── OVMF-20170228-3.gitc325e41585e3.el7.noarch.rpm
└── usr
    └── share
        ├── doc
        │   └── OVMF
        │       ├── Licenses
        │       │   ├── CryptoPkg-License.txt
        │       │   ├── FatPkg-License.txt
        │       │   ├── IntelFrameworkModulePkg-License.txt
        │       │   ├── MdeModulePkg-License.txt
        │       │   ├── MdePkg-License.txt
        │       │   ├── OpensslLib-License.txt
        │       │   ├── OptionRomPkg-License.txt
        │       │   ├── OvmfPkg-License.txt
        │       │   ├── PcAtChipsetPkg-License.txt
        │       │   ├── SecurityPkg-License.txt
        │       │   ├── ShellPkg-License.txt
        │       │   └── UefiCpuPkg-License.txt
        │       ├── ovmf-whitepaper-c770f8c.txt
        │       └── README
        └── OVMF
            ├── OVMF_CODE.secboot.fd
            ├── OVMF_VARS.fd
            └── UefiShell.iso


And the current Nova libvirt driver is checking the path to be:

    [...]
    DEFAULT_UEFI_LOADER_PATH = {
        "x86_64": "/usr/share/OVMF/OVMF_CODE.fd",
        "aarch64": "/usr/share/AAVMF/AAVMF_CODE.fd"
    }
    [...]

Now I realize, without code changes to explicitly point to the secure
boot binary in Nova, and enabling the SMM (Secure Management Mode) it will not work.

Two things are required for the guest XML that Nova generates:

  (1) Enable SMM in the guest XML:


      <features>
        [...]
        <smm state='on'/>
      </features>

  (2) And the OVMF 'loader' related snippet looks As following, with Q35:

      <os>
        <type arch='x86_64' machine='pc-q35-2.7'>hvm</type>
        <loader readonly='yes' secure='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.secboot.fd</loader>
        <nvram>/var/lib/libvirt/qemu/nvram/instance-00000007_VARS.fd</nvram>
        <boot dev='hd'/>
        <smbios mode='sysinfo'/>
      </os>


So the procedure I outlined in comment#31 will NOT work with OVMF secure boot.

[...]

Comment 34 Kashyap Chamarthy 2017-07-26 16:52:55 UTC
Given my comment#33, where the two things that are required from a *RHEL* perspective was:

(1) Make Nova to generate the guest XML with the 'smm' as part of the 
    'feature' attribute: <smm state='on'/>

(2) Use the *correct* OVMF loader variable path (for secure boot) for 
    RHEL: <loader readonly='yes' secure='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.secboot.fd</loader>

    NB: From RHEL 7.3 onwards we ship only the secure boot variant of OVMF

Therefore, I am putting this bug back to ASSIGNED state.

NOTE: We should also perhaps have a conversation with upstream Nova, as the OVMF binary names are different for different distributions (e.g. SUSE uses a different path than what RHEL has).

Comment 35 Laszlo Ersek 2017-07-26 19:01:16 UTC
(In reply to Kashyap Chamarthy from comment #33)

> (2) And the OVMF 'loader' related snippet looks As following, with Q35:
>
>     <os>
>       <type arch='x86_64' machine='pc-q35-2.7'>hvm</type>
>       <loader
>        readonly='yes'
>        secure='yes'
>        type='pflash'
>       >/usr/share/OVMF/OVMF_CODE.secboot.fd</loader>
>       <nvram>/var/lib/libvirt/qemu/nvram/instance-00000007_VARS.fd</nvram>
>       <boot dev='hd'/>
>       <smbios mode='sysinfo'/>
>     </os>

Small update regarding the machine type:
- for downstream, you'll want to use pc-q35-rhel7.4.0 or later
- for upstream, you'll want to use pc-q35-2.9 or later
  (see <https://da.gd/mps4Q> why.)

(In reply to Kashyap Chamarthy from comment #34)

> NOTE: We should also perhaps have a conversation with upstream Nova, as
> the OVMF binary names are different for different distributions (e.g. SUSE
> uses a different path than what RHEL has).

I suggest conferring with developers of other virt tools (CC Cole, Pavel,
Rich) to see how they are handling this already. I vaguely recall that the
virt tools:
- have builtin pathnames,
- and can also query domcapabilities.

In the second case (domcapabilities), the following XPath expression returns
a list of firmware binaries:

  /domainCapabilities/os[supported='yes']/loader[supported='yes']/value

and the first entry thereof is used by the tools as "default UEFI firmware"
(I think?)

In turn the above <value> element list reflects the "nvram" stanza from
"/etc/libvirt/qemu.conf" -- or, if that stanza is missing, the default
firmware pathnames built into libvirt.

Comment 36 Laszlo Ersek 2017-07-26 19:04:57 UTC
(In reply to Kashyap Chamarthy from comment #33)

>       <os>
>         <boot dev='hd'/>
>       </os>


Also, please use per-device <boot order='N'/> elements instead, like the the libvirt docs suggest. Thanks.

Comment 40 Kashyap Chamarthy 2017-09-25 08:21:47 UTC
Here is the WIP specification:

    https://review.openstack.org/#/c/506720/ -- Add ability for OVMF Secure Boot

Comment 44 Kashyap Chamarthy 2017-11-28 15:39:33 UTC
There's another critical item that we're missing here to actually make secure boot really *secure*, one must load the Secure Boot keys from Microsoft.  E.g. the steps described here:

 https://fedoraproject.org/wiki/Using_UEFI_with_QEMU#Run_EnrollDefaultKeys.efi


In a Fedora environment, the complete steps are as follows:
-----------------------------------------------------------------------
1) While the guest is shutdown, attach the UEFI shell ISO as a CD-ROM:

      $ virsh attach-disk f26-ovmf-vm2 \
          /usr/share/edk2/ovmf/UefiShell.iso vdc --type cdrom \
          --mode readonly --config --targetbus virtio
      Disk attached successfully

(2) Change the boot order of the devices so that the CD-ROM with
    'UefiShell.iso' is the first device (boot order starts from '1')
    that the guest boots into:

      $ virsh edit f26-ovmf-vm2
      [...]

    So that the fragment looks as below:

      <disk type='file' device='cdrom'>
        <driver name='qemu' type='raw'/>
        <source file='/usr/share/edk2/ovmf/UefiShell.iso'/>
        <target dev='vdc' bus='virtio'/>
        <readonly/>
      <boot order='1'/>

(3) Start the guest on Serial console:

      [root@taroxhost ~]# virsh start f26-ovmf-vm2 --console
      Domain f26-ovmf-vm2 started
      Connected to domain f26-ovmf-vm2
      Escape character is ^]
      UEFI Interactive Shell v2.1
      EDK II
      UEFI v2.60 (EDK II, 0x00010000)
      Mapping table
            FS0: Alias(s):HD0b:;BLK1:
                PciRoot(0x0)/Pci(0x1,0x2)/Pci(0x0,0x0)/HD(1,GPT,C76241E2-7F41-406E-8A6E-A73274B6DE1F,0x800,0x64000)
            FS1: Alias(s):CD1a:;BLK5:
                PciRoot(0x0)/Pci(0x1,0x5)/Pci(0x0,0x0)/CDROM(0x0)
           BLK0: Alias(s):
                PciRoot(0x0)/Pci(0x1,0x2)/Pci(0x0,0x0)
           BLK2: Alias(s):
                PciRoot(0x0)/Pci(0x1,0x2)/Pci(0x0,0x0)/HD(2,GPT,E17A8175-6CE8-4E21-801E-4D4D81D00B40,0x64800,0x200000)
           BLK3: Alias(s):
                PciRoot(0x0)/Pci(0x1,0x2)/Pci(0x0,0x0)/HD(3,GPT,8712A3D6-2835-49D0-BA37-41F05C14FBD2,0x264800,0x759B000)
           BLK4: Alias(s):
                PciRoot(0x0)/Pci(0x1,0x5)/Pci(0x0,0x0)



      Press ESC in 1 seconds to skip startup.nsh or any other key to continue.
      Shell>

(4) At the UEFI shell prompt, following the instructions from here [*],
    run the following commands ("fs0" -- change into the first device;
    "EnrollDefaultKeys.efi" -- which installs the Microsoft's KEK; and
    "reset" -- UEFI's way of saying reboot):

      [...]
      Shell> fs0:
      FS0:\> EnrollDefaultKeys.efi
      info: SetupMode=1 SecureBoot=0 SecureBootEnable=0 CustomMode=0 VendorKeys=1
      info: SetupMode=0 SecureBoot=1 SecureBootEnable=1 CustomMode=0 VendorKeys=0
      FS0:\> reset

(4.a)

    NOTE: If the UEFI Shell doesn't return cleanly after running
    "EnrollDefaultKeys.efi", just force power-off and power-on the guest
    from another shell.  E.g. to force power-off the guest, and start it
    again:

      $ sudo virsh destroy f26-ovmf-vm2
      $ sudo virsh start f26-ovmf-vm2 --console

      [This time boot into the Fedora 26 OS.]

    And on the guest serial console, you see Secure Boot enabled for
    *real*:

      ...
      EFI stub: UEFI Secure Boot is enabled.  
      ...
      [    0.000000] Secure boot enabled and kernel locked down

(5) To double-confirm, inside the guest, run `dmesg`:

      [root@f26-ovmf-vm2 ~]# dmesg | grep -i secure
      [    0.000000] Secure boot enabled and kernel locked down
      [    3.261277] EFI: Loaded cert 'Fedora Secure Boot CA: fde32599c2d61db1bf5807335d7b20e4cd963b42' linked to '.builtin_trusted_keys'
-----------------------------------------------------------------------

Comment 46 Kashyap Chamarthy 2017-11-28 16:29:41 UTC
Created attachment 1359965 [details]
An example script that automatically creates OVMF variables ("VARS") file

The attached test program launches a QEMU guest, runs the UefiShell.iso, enrolls the default Secure Boot keys, and stores them in an OVMF VARS file called: output_VARS.fd.  This file then can then be used to boot guests, so that secure boot is enabled on all subsequent boots.

This just demonstrates a way to automate the Secure Boot key enrollment process on the UEFI shell.

---

To that end, Fedora Infrastructure developer Patrick Uiterwijk and myself worked towards creating a slightly more robust tool that will let you generate the OVMF variables ("VARS") file (along with a functional test):

https://github.com/puiterwijk/qemu-ovmf-secureboot/blob/master/ovmf-vars-generator.py

I'll be proposing this approach (of using a similar tool to generate the VARS file) in the Nova upstream spec to get further input.

Comment 47 Laszlo Ersek 2017-11-28 17:49:27 UTC
Hi Kashyap,

(In reply to Kashyap Chamarthy from comment #46)
> Created attachment 1359965 [details]
> An example script that automatically creates OVMF variables ("VARS") file
> 
> The attached test program launches a QEMU guest, runs the UefiShell.iso,
> enrolls the default Secure Boot keys, and stores them in an OVMF VARS file
> called: output_VARS.fd.  This file then can then be used to boot guests, so
> that secure boot is enabled on all subsequent boots.
> 
> This just demonstrates a way to automate the Secure Boot key enrollment
> process on the UEFI shell.

This looks good. I can name one possible improvement (just for general QEMU-cleanliness): UefiShell.iso is supposed to be a CD-ROM image, so we should preferably expose it with a CD-ROM device, and not as a virtio-blk-pci device.

For that, there are two options. Both are equally functional, but the virtio variant is more performant:

(1) Use "-device ide-cd" rather than "-device virtio-blk-pci". This will place a SATA CD-ROM drive on Q35's built-in AHCI controller.

(2) Or else, use "-device virtio-scsi-pci,id=scsi0 -device scsi-cd,bus=scsi0.0,...". This will use a SCSI CD-ROM on a virtio-scsi controller.

... You could also drop the "virtio-rng-pci" device; it should not be necessary.

Just my two cents. Thanks.

Comment 49 Kashyap Chamarthy 2018-02-23 09:04:53 UTC
The actual _RFE_ for support of OVMF with Secure Boot is being tracked here[*].

However, this bug itself is a test-only bug, and _technically_ can be closed as a duplicate of the RFE[*] bug. But there's a _ton_ of context and design discussion in this bug, and I am wary (because people don't normally bother to read "duplicate" bugs) of closing it just for the sake of bugzilla hygiene.

Maybe QE can use this bug for validating the functionality, once Nova wires it all up.


[*] https://bugzilla.redhat.com/show_bug.cgi?id=1369007 -- RFE: UEFI Secure Boot (SMM OVMF) of Nova Instances

Comment 50 Lee Yarwood 2018-03-27 12:21:31 UTC
Closing this out as a duplicate of 1369007 as it's coming up on our untriaged report due to missing target release and milestones etc, we can always link comments from this bug in the RFE.

*** This bug has been marked as a duplicate of bug 1369007 ***


Note You need to log in before you can comment on or make changes to this bug.