Bug 1985924 - SecureBoot VM's fail to boot after edk2-ovmf update
Summary: SecureBoot VM's fail to boot after edk2-ovmf update
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: qemu-kvm
Version: CentOS Stream
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: beta
: ---
Assignee: Gerd Hoffmann
QA Contact: Xueqiang Wei
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-07-26 09:25 UTC by Rik Theys
Modified: 2021-08-18 05:29 UTC (History)
13 users (show)

Fixed In Version: qemu-kvm-6.0.0-20.el8.x86_64
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-08-18 05:28:55 UTC
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)
ovmf_debug.log (439.87 KB, text/plain)
2021-07-28 09:33 UTC, Rik Theys
no flags Details
first boot (918.29 KB, text/plain)
2021-08-04 12:21 UTC, Rik Theys
no flags Details
second boot (457.96 KB, text/plain)
2021-08-04 12:21 UTC, Rik Theys
no flags Details
log that shows the assert (3.32 MB, application/gzip)
2021-08-12 06:09 UTC, Rik Theys
no flags Details
debug logfile after upgrade of qemu-kvm and edk2-ovmf (502.20 KB, text/plain)
2021-08-12 10:51 UTC, Rik Theys
no flags Details

Description Rik Theys 2021-07-26 09:25:22 UTC
Description of problem:
After an upgrade of edk2-ovmf to edk2-ovmf-20200602gitca407c7246bf-5.el8.noarch on my oVirt hosts running CentOS 8 Stream, VM's that are configured with SecureBoot are no longer working correctly: it's not possible to access the console using spice or vnc and the qemu process is using 100% cpu on the host.


Version-Release number of selected component (if applicable):
edk2-ovmf-20200602gitca407c7246bf-5.el8.noarch

How reproducible:


Steps to Reproduce:
1. Create a VM with the Q35 Chipset with UEFI SecureBoot firmware type in oVirt
2. Boot it on a CentOS 8 Stream host with the latest package installed
3.

Actual results:
The VM boots, but the console is not accessible (black screen) and the qemu process is using 100% cpu on the host. VM is not accessible over ssh.

Expected results:
VM boots and console is accessible.

Additional info:
Downgrading only edk2-ovmf on the host resolves the issue.

Comment 1 Xueqiang Wei 2021-07-26 10:10:30 UTC
Similar issue is handling by Philippe.

Bug 1981196 - [RHEL-8.5] UEFI VM with static vars file doesn't boot anymore after edk2-ovmf update


According to https://bugzilla.redhat.com/show_bug.cgi?id=1981196#c8 and https://bugzilla.redhat.com/show_bug.cgi?id=1981196#c10, the VM secure boot successfully with the original vars file "/usr/share/OVMF/OVMF_VARS.secboot.fd". 

1. I want to confirm how to enable secure boot in your testing? Did you hit the message "CR has Bad Signature" in the edk2 log? Thanks.


According to https://bugzilla.redhat.com/show_bug.cgi?id=1981196#c11, hit the issue since qemu-kvm-4.2.0-52.module+el8.5.0+11386+ef5875dd.

2. Could you please provide the qemu-kvm version? Thanks.

Comment 2 Rik Theys 2021-07-26 10:20:41 UTC
Hi,

(In reply to Xueqiang Wei from comment #1)
> Similar issue is handling by Philippe.
> 
> Bug 1981196 - [RHEL-8.5] UEFI VM with static vars file doesn't boot anymore
> after edk2-ovmf update

I'm not authorized to see this bug. Can you add me to the CC list please?

> According to https://bugzilla.redhat.com/show_bug.cgi?id=1981196#c8 and
> https://bugzilla.redhat.com/show_bug.cgi?id=1981196#c10, the VM secure boot
> successfully with the original vars file
> "/usr/share/OVMF/OVMF_VARS.secboot.fd". 
> 
> 1. I want to confirm how to enable secure boot in your testing? Did you hit
> the message "CR has Bad Signature" in the edk2 log? Thanks.

Where can I find the edk2 log?

> 
> 
> According to https://bugzilla.redhat.com/show_bug.cgi?id=1981196#c11, hit
> the issue since qemu-kvm-4.2.0-52.module+el8.5.0+11386+ef5875dd.
> 
> 2. Could you please provide the qemu-kvm version? Thanks.

qemu-kvm-6.0.0-19.el8s.x86_64

Regards,
Rik

Comment 3 Xueqiang Wei 2021-07-26 12:11:25 UTC
(In reply to Rik Theys from comment #2)
> Hi,
> 
> (In reply to Xueqiang Wei from comment #1)
> > Similar issue is handling by Philippe.
> > 
> > Bug 1981196 - [RHEL-8.5] UEFI VM with static vars file doesn't boot anymore
> > after edk2-ovmf update
> 
> I'm not authorized to see this bug. Can you add me to the CC list please?

Done. Please take a look at the comment8, Comment10 and Comment11 in Bug 1981196.
And please check if you tested secure boot with the original vars file "/usr/share/OVMF/OVMF_VARS.secboot.fd"? If not, please retry the test with "/usr/share/OVMF/OVMF_VARS.secboot.fd". Thanks.

> 
> > According to https://bugzilla.redhat.com/show_bug.cgi?id=1981196#c8 and
> > https://bugzilla.redhat.com/show_bug.cgi?id=1981196#c10, the VM secure boot
> > successfully with the original vars file
> > "/usr/share/OVMF/OVMF_VARS.secboot.fd". 
> > 
> > 1. I want to confirm how to enable secure boot in your testing? Did you hit
> > the message "CR has Bad Signature" in the edk2 log? Thanks.
> 
> Where can I find the edk2 log?

You need to add the debugcon options, to check whether you also hit the message "CR has Bad Signature".

The first way: add debugcon options in guest xml:
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0' id='1'>
.....
  <qemu:commandline>
    <qemu:arg value='-global'/>
    <qemu:arg value='isa-debugcon.iobase=0x402'/>
    <qemu:arg value='-debugcon'/>
    <qemu:arg value='file:/tmp/ovmf_debug.log'/>
  </qemu:commandline>


The second way: add debugcon options to qemu command lines:
/usr/libexec/qemu-kvm \
......
-debugcon file:/tmp/ovmf_debug.log \
-global isa-debugcon.iobase=0x402 \


> 
> > 
> > 
> > According to https://bugzilla.redhat.com/show_bug.cgi?id=1981196#c11, hit
> > the issue since qemu-kvm-4.2.0-52.module+el8.5.0+11386+ef5875dd.
> > 
> > 2. Could you please provide the qemu-kvm version? Thanks.
> 
> qemu-kvm-6.0.0-19.el8s.x86_64
> 
> Regards,
> Rik

Comment 4 Rik Theys 2021-07-26 14:08:11 UTC
Hi,

(In reply to Xueqiang Wei from comment #3)
> (In reply to Rik Theys from comment #2)
> > Hi,
> > 
> > (In reply to Xueqiang Wei from comment #1)
> > > Similar issue is handling by Philippe.
> > > 
> > > Bug 1981196 - [RHEL-8.5] UEFI VM with static vars file doesn't boot anymore
> > > after edk2-ovmf update
> > 
> > I'm not authorized to see this bug. Can you add me to the CC list please?
> 
> Done. Please take a look at the comment8, Comment10 and Comment11 in Bug
> 1981196.

I can now see the bug, but I don't see the comments you mention. I only see comments 1,2,4 and 6.

> And please check if you tested secure boot with the original vars file
> "/usr/share/OVMF/OVMF_VARS.secboot.fd"? If not, please retry the test with
> "/usr/share/OVMF/OVMF_VARS.secboot.fd". Thanks.

According to the libvirt log, I'm using:

-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-31-stud-c8-15/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/usr/share/OVMF/OVMF_CODE.secboot.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/d9fa5850-c765-49c2-9933-eb0ad0d35bfc.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \

so I'm using /usr/share/OVMF/OVMF_CODE.secboot.fd? Should the second reference be /usr/share/OVMF/OVMF_VARS.secboot.fd??

I'm using oVirt, so the VM's are started by vdsm.

> > > 
> > > 1. I want to confirm how to enable secure boot in your testing? Did you hit
> > > the message "CR has Bad Signature" in the edk2 log? Thanks.
> > 
> > Where can I find the edk2 log?
> 
> You need to add the debugcon options, to check whether you also hit the
> message "CR has Bad Signature".
> 
> The first way: add debugcon options in guest xml:
> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'
> id='1'>
> .....
>   <qemu:commandline>
>     <qemu:arg value='-global'/>
>     <qemu:arg value='isa-debugcon.iobase=0x402'/>
>     <qemu:arg value='-debugcon'/>
>     <qemu:arg value='file:/tmp/ovmf_debug.log'/>
>   </qemu:commandline>
> 
> 
> The second way: add debugcon options to qemu command lines:
> /usr/libexec/qemu-kvm \
> ......
> -debugcon file:/tmp/ovmf_debug.log \
> -global isa-debugcon.iobase=0x402 \

I'm using oVirt. How can I add these flags?

Regards,
Rik

Comment 5 Xueqiang Wei 2021-07-27 03:38:42 UTC
(In reply to Rik Theys from comment #4)
> Hi,
> 
> (In reply to Xueqiang Wei from comment #3)
> > (In reply to Rik Theys from comment #2)
> > > Hi,
> > > 
> > > (In reply to Xueqiang Wei from comment #1)
> > > > Similar issue is handling by Philippe.
> > > > 
> > > > Bug 1981196 - [RHEL-8.5] UEFI VM with static vars file doesn't boot anymore
> > > > after edk2-ovmf update
> > > 
> > > I'm not authorized to see this bug. Can you add me to the CC list please?
> > 
> > Done. Please take a look at the comment8, Comment10 and Comment11 in Bug
> > 1981196.
> 
> I can now see the bug, but I don't see the comments you mention. I only see
> comments 1,2,4 and 6.
> 
> > And please check if you tested secure boot with the original vars file
> > "/usr/share/OVMF/OVMF_VARS.secboot.fd"? If not, please retry the test with
> > "/usr/share/OVMF/OVMF_VARS.secboot.fd". Thanks.
> 
> According to the libvirt log, I'm using:
> 
> -object
> '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/
> libvirt/qemu/domain-31-stud-c8-15/master-key.aes"}' \
> -blockdev
> '{"driver":"file","filename":"/usr/share/OVMF/OVMF_CODE.secboot.fd","node-
> name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
> -blockdev
> '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw",
> "file":"libvirt-pflash0-storage"}' \
> -blockdev
> '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/d9fa5850-c765-49c2-
> 9933-eb0ad0d35bfc.fd","node-name":"libvirt-pflash1-storage","auto-read-only":
> true,"discard":"unmap"}' \
> -blockdev
> '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw",
> "file":"libvirt-pflash1-storage"}' \
> 
> so I'm using /usr/share/OVMF/OVMF_CODE.secboot.fd? Should the second
> reference be /usr/share/OVMF/OVMF_VARS.secboot.fd??


I think if you using secure boot by default, the vars file should be copied from /usr/share/OVMF/OVMF_VARS.secboot.fd.

Since not using the original vars file, hit bug 1981196. I just want to double check the file "/var/lib/libvirt/qemu/nvram/d9fa5850-c765-49c2-9933-eb0ad0d35bfc.fd". If it was copied from "/usr/share/OVMF/OVMF_VARS.secboot.fd". 


The usage of qemu-kvm, please refer to https://bugzilla.redhat.com/show_bug.cgi?id=1981196#c7.
e.g. Tested with original vars file, enable secure boot.
# cp /usr/share/OVMF/OVMF_VARS.secboot.fd /home/kvm_autotest_root/images/avocado-vt-vm1_rhel840-64-virtio-scsi.qcow2_VARS.fd

The usage of libvirt, please refer to https://bugzilla.redhat.com/show_bug.cgi?id=1981196#c10.
e.g. 
steps:
1. cp /usr/share/OVMF/OVMF_VARS.secboot.fd  /var/lib/libvirt/qemu/nvram/GuestOne_VARS.secboot.fd 
2. set '<nvram template='/usr/share/edk2/ovmf/OVMF_VARS.fd'>/var/lib/libvirt/qemu/nvram/GuestOne_VARS.secboot.fd</nvram>' in guest xml
3. start guest,guest can boot up with secureboot enabled.

> 
> I'm using oVirt, so the VM's are started by vdsm.
> 
> > > > 
> > > > 1. I want to confirm how to enable secure boot in your testing? Did you hit
> > > > the message "CR has Bad Signature" in the edk2 log? Thanks.
> > > 
> > > Where can I find the edk2 log?
> > 
> > You need to add the debugcon options, to check whether you also hit the
> > message "CR has Bad Signature".
> > 
> > The first way: add debugcon options in guest xml:
> > <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'
> > id='1'>
> > .....
> >   <qemu:commandline>
> >     <qemu:arg value='-global'/>
> >     <qemu:arg value='isa-debugcon.iobase=0x402'/>
> >     <qemu:arg value='-debugcon'/>
> >     <qemu:arg value='file:/tmp/ovmf_debug.log'/>
> >   </qemu:commandline>
> > 
> > 
> > The second way: add debugcon options to qemu command lines:
> > /usr/libexec/qemu-kvm \
> > ......
> > -debugcon file:/tmp/ovmf_debug.log \
> > -global isa-debugcon.iobase=0x402 \
> 
> I'm using oVirt. How can I add these flags?

Unfortunately, I cannot help much with the oVirt. I exclusively use edk2 with qemu command lines.

> 
> Regards,
> Rik

Comment 6 Rik Theys 2021-07-27 07:03:12 UTC
Hi,

> > > Done. Please take a look at the comment8, Comment10 and Comment11 in Bug
> > > 1981196.
> > 
> > I can now see the bug, but I don't see the comments you mention. I only see
> > comments 1,2,4 and 6.
> > 
> > > And please check if you tested secure boot with the original vars file
> > > "/usr/share/OVMF/OVMF_VARS.secboot.fd"? If not, please retry the test with
> > > "/usr/share/OVMF/OVMF_VARS.secboot.fd". Thanks.
> > 
> > According to the libvirt log, I'm using:
> > 
> > -object
> > '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/
> > libvirt/qemu/domain-31-stud-c8-15/master-key.aes"}' \
> > -blockdev
> > '{"driver":"file","filename":"/usr/share/OVMF/OVMF_CODE.secboot.fd","node-
> > name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
> > -blockdev
> > '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw",
> > "file":"libvirt-pflash0-storage"}' \
> > -blockdev
> > '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/d9fa5850-c765-49c2-
> > 9933-eb0ad0d35bfc.fd","node-name":"libvirt-pflash1-storage","auto-read-only":
> > true,"discard":"unmap"}' \
> > -blockdev
> > '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw",
> > "file":"libvirt-pflash1-storage"}' \
> > 
> > so I'm using /usr/share/OVMF/OVMF_CODE.secboot.fd? Should the second
> > reference be /usr/share/OVMF/OVMF_VARS.secboot.fd??
> 
> 
> I think if you using secure boot by default, the vars file should be copied
> from /usr/share/OVMF/OVMF_VARS.secboot.fd.
> 
> Since not using the original vars file, hit bug 1981196. I just want to
> double check the file
> "/var/lib/libvirt/qemu/nvram/d9fa5850-c765-49c2-9933-eb0ad0d35bfc.fd". If it
> was copied from "/usr/share/OVMF/OVMF_VARS.secboot.fd". 

The file /var/lib/libvirt/qemu/nvram/d9fa5850-c765-49c2-9933-eb0ad0d35bfc.fd does not exist if the VM is not running. I assume oVirt creates the file when it starts up the VM.
When I start a VM and check the file, I don't see where it was copied from as it does not match any of the .fd files in /usr/share/OVMF:

[root@studvirt1 nvram]# sha1sum cfb362e4-3a38-42ff-85a1-24aa432aaac0.fd /usr/share/OVMF/*.fd
114a3afa1f60ae66af99a1510e23df46f1c83891  cfb362e4-3a38-42ff-85a1-24aa432aaac0.fd
633e46ebf28584f567ae7229786ee1f7afe359e0  /usr/share/OVMF/OVMF_CODE.secboot.fd
b62e676c9317a32aaa9ee1d76268d8c55663bd6e  /usr/share/OVMF/OVMF_VARS.fd
9c5aad3dcc7fc1ed7e136c178c26cbf316a9f746  /usr/share/OVMF/OVMF_VARS.secboot.fd

Note that I'm using edk2-ovmf-20200602gitca407c7246bf-5.el8.noarch as the new version prevents the VM's from booting and this is a production system.

Maybe oVirt is storing this information in it's database, and creating the .fd file upon boot?

> The usage of qemu-kvm, please refer to
> https://bugzilla.redhat.com/show_bug.cgi?id=1981196#c7.

I don't see any of the comments you are referring to. I assume they are marked as private?

> e.g. Tested with original vars file, enable secure boot.
> # cp /usr/share/OVMF/OVMF_VARS.secboot.fd
> /home/kvm_autotest_root/images/avocado-vt-vm1_rhel840-64-virtio-scsi.
> qcow2_VARS.fd
> 
> The usage of libvirt, please refer to
> https://bugzilla.redhat.com/show_bug.cgi?id=1981196#c10.
> e.g. 
> steps:
> 1. cp /usr/share/OVMF/OVMF_VARS.secboot.fd 
> /var/lib/libvirt/qemu/nvram/GuestOne_VARS.secboot.fd 
> 2. set '<nvram
> template='/usr/share/edk2/ovmf/OVMF_VARS.fd'>/var/lib/libvirt/qemu/nvram/
> GuestOne_VARS.secboot.fd</nvram>' in guest xml
> 3. start guest,guest can boot up with secureboot enabled.
> 

I don't have an easy way to test this since I can not run the commands by hand on this host as it's managed by oVirt.


> > 
> > I'm using oVirt, so the VM's are started by vdsm.
> > 
> > > > > 
> > > > > 1. I want to confirm how to enable secure boot in your testing? Did you hit
> > > > > the message "CR has Bad Signature" in the edk2 log? Thanks.
> > > > 
> > > > Where can I find the edk2 log?
> > > 
> > > You need to add the debugcon options, to check whether you also hit the
> > > message "CR has Bad Signature".
> > > 
> > > The first way: add debugcon options in guest xml:
> > > <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'
> > > id='1'>
> > > .....
> > >   <qemu:commandline>
> > >     <qemu:arg value='-global'/>
> > >     <qemu:arg value='isa-debugcon.iobase=0x402'/>
> > >     <qemu:arg value='-debugcon'/>
> > >     <qemu:arg value='file:/tmp/ovmf_debug.log'/>
> > >   </qemu:commandline>
> > > 
> > > 
> > > The second way: add debugcon options to qemu command lines:
> > > /usr/libexec/qemu-kvm \
> > > ......
> > > -debugcon file:/tmp/ovmf_debug.log \
> > > -global isa-debugcon.iobase=0x402 \
> > 
> > I'm using oVirt. How can I add these flags?
> 
> Unfortunately, I cannot help much with the oVirt. I exclusively use edk2
> with qemu command lines.

Comment 7 Xueqiang Wei 2021-07-27 20:07:21 UTC
(In reply to Rik Theys from comment #6)
> Hi,
> 
> > > > Done. Please take a look at the comment8, Comment10 and Comment11 in Bug
> > > > 1981196.
> > > 
> > > I can now see the bug, but I don't see the comments you mention. I only see
> > > comments 1,2,4 and 6.
> > > 
> > > > And please check if you tested secure boot with the original vars file
> > > > "/usr/share/OVMF/OVMF_VARS.secboot.fd"? If not, please retry the test with
> > > > "/usr/share/OVMF/OVMF_VARS.secboot.fd". Thanks.
> > > 
> > > According to the libvirt log, I'm using:
> > > 
> > > -object
> > > '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/
> > > libvirt/qemu/domain-31-stud-c8-15/master-key.aes"}' \
> > > -blockdev
> > > '{"driver":"file","filename":"/usr/share/OVMF/OVMF_CODE.secboot.fd","node-
> > > name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
> > > -blockdev
> > > '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw",
> > > "file":"libvirt-pflash0-storage"}' \
> > > -blockdev
> > > '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/d9fa5850-c765-49c2-
> > > 9933-eb0ad0d35bfc.fd","node-name":"libvirt-pflash1-storage","auto-read-only":
> > > true,"discard":"unmap"}' \
> > > -blockdev
> > > '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw",
> > > "file":"libvirt-pflash1-storage"}' \
> > > 
> > > so I'm using /usr/share/OVMF/OVMF_CODE.secboot.fd? Should the second
> > > reference be /usr/share/OVMF/OVMF_VARS.secboot.fd??
> > 
> > 
> > I think if you using secure boot by default, the vars file should be copied
> > from /usr/share/OVMF/OVMF_VARS.secboot.fd.
> > 
> > Since not using the original vars file, hit bug 1981196. I just want to
> > double check the file
> > "/var/lib/libvirt/qemu/nvram/d9fa5850-c765-49c2-9933-eb0ad0d35bfc.fd". If it
> > was copied from "/usr/share/OVMF/OVMF_VARS.secboot.fd". 
> 
> The file /var/lib/libvirt/qemu/nvram/d9fa5850-c765-49c2-9933-eb0ad0d35bfc.fd
> does not exist if the VM is not running. I assume oVirt creates the file
> when it starts up the VM.
> When I start a VM and check the file, I don't see where it was copied from
> as it does not match any of the .fd files in /usr/share/OVMF:
> 
> [root@studvirt1 nvram]# sha1sum cfb362e4-3a38-42ff-85a1-24aa432aaac0.fd
> /usr/share/OVMF/*.fd
> 114a3afa1f60ae66af99a1510e23df46f1c83891 
> cfb362e4-3a38-42ff-85a1-24aa432aaac0.fd
> 633e46ebf28584f567ae7229786ee1f7afe359e0 
> /usr/share/OVMF/OVMF_CODE.secboot.fd
> b62e676c9317a32aaa9ee1d76268d8c55663bd6e  /usr/share/OVMF/OVMF_VARS.fd
> 9c5aad3dcc7fc1ed7e136c178c26cbf316a9f746 
> /usr/share/OVMF/OVMF_VARS.secboot.fd
> 
> Note that I'm using edk2-ovmf-20200602gitca407c7246bf-5.el8.noarch as the
> new version prevents the VM's from booting and this is a production system.
> 
> Maybe oVirt is storing this information in it's database, and creating the
> .fd file upon boot?
> 
> > The usage of qemu-kvm, please refer to
> > https://bugzilla.redhat.com/show_bug.cgi?id=1981196#c7.
> 
> I don't see any of the comments you are referring to. I assume they are
> marked as private?

I add qemu command lines below.

> 
> > e.g. Tested with original vars file, enable secure boot.
> > # cp /usr/share/OVMF/OVMF_VARS.secboot.fd
> > /home/kvm_autotest_root/images/avocado-vt-vm1_rhel840-64-virtio-scsi.
> > qcow2_VARS.fd
> > 
> > The usage of libvirt, please refer to
> > https://bugzilla.redhat.com/show_bug.cgi?id=1981196#c10.
> > e.g. 
> > steps:
> > 1. cp /usr/share/OVMF/OVMF_VARS.secboot.fd 
> > /var/lib/libvirt/qemu/nvram/GuestOne_VARS.secboot.fd 
> > 2. set '<nvram
> > template='/usr/share/edk2/ovmf/OVMF_VARS.fd'>/var/lib/libvirt/qemu/nvram/
> > GuestOne_VARS.secboot.fd</nvram>' in guest xml
> > 3. start guest,guest can boot up with secureboot enabled.
> > 
> 
> I don't have an easy way to test this since I can not run the commands by
> hand on this host as it's managed by oVirt.

Can you login the host with ssh or console? If yes, it's easy to try the following steps.

1. Enable secure boot.
# cp /usr/share/OVMF/OVMF_VARS.secboot.fd /home/images/vm1_rhel840-64-virtio-scsi.qcow2_VARS.fd

2. Boot a rhel840 guest, and check secure boot inside guest.

# sh sec_boot_rhel840.sh
# cat sec_boot_rhel840.sh
usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -blockdev node-name=file_ovmf_code,driver=file,filename=/usr/share/OVMF/OVMF_CODE.secboot.fd,auto-read-only=on,discard=unmap \
    -blockdev node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code \
    -blockdev node-name=file_ovmf_vars,driver=file,filename=/home/images/vm1_rhel840-64-virtio-scsi.qcow2_VARS.fd,auto-read-only=on,discard=unmap \
    -blockdev node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars \
    -machine q35,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 15360  \
    -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
    -cpu 'Opteron_G5',+kvm_pv_unhalt \
    -chardev socket,wait=off,id=qmp_id_qmpmonitor1,path=/tmp/avocado_s1kqmcrc/monitor-qmpmonitor1-20210712-041534-ylCSGzB2,server=on  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,wait=off,id=qmp_id_catch_monitor,path=/tmp/avocado_s1kqmcrc/monitor-catch_monitor-20210712-041534-ylCSGzB2,server=on  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idlWZ9Is \
    -chardev socket,wait=off,id=chardev_serial0,path=/tmp/avocado_s1kqmcrc/serial-serial0-20210712-041534-ylCSGzB2,server=on \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20210712-041534-ylCSGzB2,path=/tmp/avocado_s1kqmcrc/seabios-20210712-041534-ylCSGzB2,server=on,wait=off \
    -device isa-debugcon,chardev=seabioslog_id_20210712-041534-ylCSGzB2,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/images/rhel840-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:34:31:1a:44:70,id=idU0eBKe,netdev=idgpW87T,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idgpW87T  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -monitor stdio \
    -debugcon file:/home/ovmf_debug.log \
    -global isa-debugcon.iobase=0x402 \

Check secure boot in the guest:
# dmesg | grep -i "Secure Boot"
[    0.000000] secureboot: Secure boot enabled
[    0.000000] Kernel is locked down from EFI secure boot; see man kernel_lockdown.7
[    3.476981] integrity: Loaded X.509 cert 'Red Hat Secure Boot CA 5: cc6fa5e72868ba494e939bbd680b9144769a9f8f'


3. Power off the guest, recopy the vars file and then start it again.
# cp /usr/share/OVMF/OVMF_VARS.secboot.fd /home/images/vm1_rhel840-64-virtio-scsi.qcow2_VARS.fd

The rhel840 guest should start successfully.

Check the ovmf debug log "/home/ovmf_debug.log", should not found "CR has Bad Signature" or other error message.

> 
> 
> > > 
> > > I'm using oVirt, so the VM's are started by vdsm.
> > > 
> > > > > > 
> > > > > > 1. I want to confirm how to enable secure boot in your testing? Did you hit
> > > > > > the message "CR has Bad Signature" in the edk2 log? Thanks.
> > > > > 
> > > > > Where can I find the edk2 log?
> > > > 
> > > > You need to add the debugcon options, to check whether you also hit the
> > > > message "CR has Bad Signature".
> > > > 
> > > > The first way: add debugcon options in guest xml:
> > > > <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'
> > > > id='1'>
> > > > .....
> > > >   <qemu:commandline>
> > > >     <qemu:arg value='-global'/>
> > > >     <qemu:arg value='isa-debugcon.iobase=0x402'/>
> > > >     <qemu:arg value='-debugcon'/>
> > > >     <qemu:arg value='file:/tmp/ovmf_debug.log'/>
> > > >   </qemu:commandline>
> > > > 
> > > > 
> > > > The second way: add debugcon options to qemu command lines:
> > > > /usr/libexec/qemu-kvm \
> > > > ......
> > > > -debugcon file:/tmp/ovmf_debug.log \
> > > > -global isa-debugcon.iobase=0x402 \
> > > 
> > > I'm using oVirt. How can I add these flags?
> > 
> > Unfortunately, I cannot help much with the oVirt. I exclusively use edk2
> > with qemu command lines.

Comment 8 Rik Theys 2021-07-28 07:23:55 UTC
> > 
> > I don't have an easy way to test this since I can not run the commands by
> > hand on this host as it's managed by oVirt.
> 
> Can you login the host with ssh or console? If yes, it's easy to try the
> following steps.
> 
> 1. Enable secure boot.
> # cp /usr/share/OVMF/OVMF_VARS.secboot.fd
> /home/images/vm1_rhel840-64-virtio-scsi.qcow2_VARS.fd
> 
> 2. Boot a rhel840 guest, and check secure boot inside guest.
> 
> # sh sec_boot_rhel840.sh
> # cat sec_boot_rhel840.sh
> usr/libexec/qemu-kvm \
>     -S  \
>     -name 'avocado-vt-vm1'  \
>     -sandbox on  \
>     -blockdev
> node-name=file_ovmf_code,driver=file,filename=/usr/share/OVMF/OVMF_CODE.
> secboot.fd,auto-read-only=on,discard=unmap \
>     -blockdev
> node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code \
>     -blockdev
> node-name=file_ovmf_vars,driver=file,filename=/home/images/vm1_rhel840-64-
> virtio-scsi.qcow2_VARS.fd,auto-read-only=on,discard=unmap \
>     -blockdev
> node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars \
>     -machine q35,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
>     -device
> pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,
> chassis=1 \
>     -device
> pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
>     -nodefaults \
>     -device VGA,bus=pcie.0,addr=0x2 \
>     -m 15360  \
>     -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
>     -cpu 'Opteron_G5',+kvm_pv_unhalt \
>     -chardev
> socket,wait=off,id=qmp_id_qmpmonitor1,path=/tmp/avocado_s1kqmcrc/monitor-
> qmpmonitor1-20210712-041534-ylCSGzB2,server=on  \
>     -mon chardev=qmp_id_qmpmonitor1,mode=control \
>     -chardev
> socket,wait=off,id=qmp_id_catch_monitor,path=/tmp/avocado_s1kqmcrc/monitor-
> catch_monitor-20210712-041534-ylCSGzB2,server=on  \
>     -mon chardev=qmp_id_catch_monitor,mode=control \
>     -device pvpanic,ioport=0x505,id=idlWZ9Is \
>     -chardev
> socket,wait=off,id=chardev_serial0,path=/tmp/avocado_s1kqmcrc/serial-serial0-
> 20210712-041534-ylCSGzB2,server=on \
>     -device isa-serial,id=serial0,chardev=chardev_serial0  \
>     -chardev
> socket,id=seabioslog_id_20210712-041534-ylCSGzB2,path=/tmp/avocado_s1kqmcrc/
> seabios-20210712-041534-ylCSGzB2,server=on,wait=off \
>     -device
> isa-debugcon,chardev=seabioslog_id_20210712-041534-ylCSGzB2,iobase=0x402 \
>     -device
> pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,
> chassis=2 \
>     -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
>     -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
>     -device
> pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,
> chassis=3 \
>     -device
> virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
>     -blockdev
> node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,
> aio=threads,filename=/home/images/rhel840-64-virtio-scsi.qcow2,cache.
> direct=on,cache.no-flush=off \
>     -blockdev
> node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-
> flush=off,file=file_image1 \
>     -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
>     -device
> pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,
> chassis=4 \
>     -device
> virtio-net-pci,mac=9a:34:31:1a:44:70,id=idU0eBKe,netdev=idgpW87T,bus=pcie-
> root-port-3,addr=0x0  \
>     -netdev tap,id=idgpW87T  \
>     -vnc :0  \
>     -rtc base=utc,clock=host,driftfix=slew  \
>     -boot menu=off,order=cdn,once=c,strict=off \
>     -enable-kvm \
>     -monitor stdio \
>     -debugcon file:/home/ovmf_debug.log \
>     -global isa-debugcon.iobase=0x402 \

I've modified the script to drop the network device and to use CentOS-8-GenericCloud-8.4.2105-20210603.0.x86_64.qcow2 instead.

[root@electa-vrtx-bl2 test]# ./sec_boot_rhel840.sh                                   
qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma [bit 12]     
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.abm [bit 5]  
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.sse4a [bit 6]      
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.misalignsse [bit 7]  
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.3dnowprefetch [bit 8]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.xop [bit 11] 
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.fma4 [bit 16]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.tbm [bit 21]
qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma [bit 12]     
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.abm [bit 5]  
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.sse4a [bit 6]      
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.misalignsse [bit 7]  
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.3dnowprefetch [bit 8]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.xop [bit 11] 
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.fma4 [bit 16]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.tbm [bit 21]
qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma [bit 12]     
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.abm [bit 5]  
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.sse4a [bit 6]      
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.misalignsse [bit 7]  
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.3dnowprefetch [bit 8]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.xop [bit 11] 
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.fma4 [bit 16]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.tbm [bit 21]
qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma [bit 12]     
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.abm [bit 5]  
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.sse4a [bit 6]      
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.misalignsse [bit 7]  
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.3dnowprefetch [bit 8]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.xop [bit 11] 
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.fma4 [bit 16]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.tbm [bit 21]
qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma [bit 12]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.abm [bit 5]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.sse4a [bit 6]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.misalignsse [bit 7]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.3dnowprefetch [bit 8]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.xop [bit 11]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.fma4 [bit 16]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.tbm [bit 21]
qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma [bit 12]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.abm [bit 5]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.sse4a [bit 6]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.misalignsse [bit 7]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.3dnowprefetch [bit 8]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.xop [bit 11]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.fma4 [bit 16]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.tbm [bit 21]
qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma [bit 12]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.abm [bit 5]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.sse4a [bit 6]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.misalignsse [bit 7]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.3dnowprefetch [bit 8]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.xop [bit 11]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.fma4 [bit 16]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.tbm [bit 21]
qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma [bit 12]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.abm [bit 5]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.sse4a [bit 6]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.misalignsse [bit 7]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.3dnowprefetch [bit 8]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.xop [bit 11]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.fma4 [bit 16]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.tbm [bit 21]
qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma [bit 12]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.abm [bit 5]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.sse4a [bit 6]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.misalignsse [bit 7]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.3dnowprefetch [bit 8]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.xop [bit 11]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.fma4 [bit 16]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.tbm [bit 21]
qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma [bit 12]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.abm [bit 5]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.sse4a [bit 6]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.misalignsse [bit 7]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.3dnowprefetch [bit 8]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.xop [bit 11]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.fma4 [bit 16]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.tbm [bit 21]
qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma [bit 12]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.abm [bit 5]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.sse4a [bit 6]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.misalignsse [bit 7]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.3dnowprefetch [bit 8]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.xop [bit 11]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.fma4 [bit 16]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.tbm [bit 21]
qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma [bit 12]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.abm [bit 5]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.sse4a [bit 6]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.misalignsse [bit 7]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.3dnowprefetch [bit 8]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.xop [bit 11]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.fma4 [bit 16]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.tbm [bit 21]

when I connect to the VNC console, it says: "Guest has not initialized the display (yet)."

> 
> Check secure boot in the guest:
> # dmesg | grep -i "Secure Boot"
> [    0.000000] secureboot: Secure boot enabled
> [    0.000000] Kernel is locked down from EFI secure boot; see man
> kernel_lockdown.7
> [    3.476981] integrity: Loaded X.509 cert 'Red Hat Secure Boot CA 5:
> cc6fa5e72868ba494e939bbd680b9144769a9f8f'
> 
I can not perform this step as the guest does not boot.

The /home/ovmf_debug.log is empty.

> 
> 3. Power off the guest, recopy the vars file and then start it again.
> # cp /usr/share/OVMF/OVMF_VARS.secboot.fd
> /home/images/vm1_rhel840-64-virtio-scsi.qcow2_VARS.fd

I've recopied the vars file (although it has not changed?)

> 
> The rhel840 guest should start successfully.

It still does not.

> 
> Check the ovmf debug log "/home/ovmf_debug.log", should not found "CR has
> Bad Signature" or other error message.

It's still empty.

Regards,

Comment 9 Xueqiang Wei 2021-07-28 08:38:57 UTC
(In reply to Rik Theys from comment #8)
> > > 
> > > I don't have an easy way to test this since I can not run the commands by
> > > hand on this host as it's managed by oVirt.
> > 
> > Can you login the host with ssh or console? If yes, it's easy to try the
> > following steps.
> > 
> > 1. Enable secure boot.
> > # cp /usr/share/OVMF/OVMF_VARS.secboot.fd
> > /home/images/vm1_rhel840-64-virtio-scsi.qcow2_VARS.fd
> > 
> > 2. Boot a rhel840 guest, and check secure boot inside guest.
> > 
> > # sh sec_boot_rhel840.sh
> > # cat sec_boot_rhel840.sh
> > usr/libexec/qemu-kvm \
> >     -S  \
> >     -name 'avocado-vt-vm1'  \
> >     -sandbox on  \
> >     -blockdev
> > node-name=file_ovmf_code,driver=file,filename=/usr/share/OVMF/OVMF_CODE.
> > secboot.fd,auto-read-only=on,discard=unmap \
> >     -blockdev
> > node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code \
> >     -blockdev
> > node-name=file_ovmf_vars,driver=file,filename=/home/images/vm1_rhel840-64-
> > virtio-scsi.qcow2_VARS.fd,auto-read-only=on,discard=unmap \
> >     -blockdev
> > node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars \
> >     -machine q35,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
> >     -device
> > pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,
> > chassis=1 \
> >     -device
> > pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
> >     -nodefaults \
> >     -device VGA,bus=pcie.0,addr=0x2 \
> >     -m 15360  \
> >     -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
> >     -cpu 'Opteron_G5',+kvm_pv_unhalt \
> >     -chardev
> > socket,wait=off,id=qmp_id_qmpmonitor1,path=/tmp/avocado_s1kqmcrc/monitor-
> > qmpmonitor1-20210712-041534-ylCSGzB2,server=on  \
> >     -mon chardev=qmp_id_qmpmonitor1,mode=control \
> >     -chardev
> > socket,wait=off,id=qmp_id_catch_monitor,path=/tmp/avocado_s1kqmcrc/monitor-
> > catch_monitor-20210712-041534-ylCSGzB2,server=on  \
> >     -mon chardev=qmp_id_catch_monitor,mode=control \
> >     -device pvpanic,ioport=0x505,id=idlWZ9Is \
> >     -chardev
> > socket,wait=off,id=chardev_serial0,path=/tmp/avocado_s1kqmcrc/serial-serial0-
> > 20210712-041534-ylCSGzB2,server=on \
> >     -device isa-serial,id=serial0,chardev=chardev_serial0  \
> >     -chardev
> > socket,id=seabioslog_id_20210712-041534-ylCSGzB2,path=/tmp/avocado_s1kqmcrc/
> > seabios-20210712-041534-ylCSGzB2,server=on,wait=off \
> >     -device
> > isa-debugcon,chardev=seabioslog_id_20210712-041534-ylCSGzB2,iobase=0x402 \
> >     -device
> > pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,
> > chassis=2 \
> >     -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
> >     -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
> >     -device
> > pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,
> > chassis=3 \
> >     -device
> > virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
> >     -blockdev
> > node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,
> > aio=threads,filename=/home/images/rhel840-64-virtio-scsi.qcow2,cache.
> > direct=on,cache.no-flush=off \
> >     -blockdev
> > node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-
> > flush=off,file=file_image1 \
> >     -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
> >     -device
> > pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,
> > chassis=4 \
> >     -device
> > virtio-net-pci,mac=9a:34:31:1a:44:70,id=idU0eBKe,netdev=idgpW87T,bus=pcie-
> > root-port-3,addr=0x0  \
> >     -netdev tap,id=idgpW87T  \
> >     -vnc :0  \
> >     -rtc base=utc,clock=host,driftfix=slew  \
> >     -boot menu=off,order=cdn,once=c,strict=off \
> >     -enable-kvm \
> >     -monitor stdio \
> >     -debugcon file:/home/ovmf_debug.log \
> >     -global isa-debugcon.iobase=0x402 \
> 
> I've modified the script to drop the network device and to use
> CentOS-8-GenericCloud-8.4.2105-20210603.0.x86_64.qcow2 instead.
> 
> [root@electa-vrtx-bl2 test]# ./sec_boot_rhel840.sh                          
> 
> qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma
> [bit 12]     
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.abm [bit 5]  
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.sse4a [bit 6]      
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.misalignsse [bit 7]  
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.3dnowprefetch [bit 8]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.xop [bit 11] 
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.fma4 [bit 16]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.tbm [bit 21]
> qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma
> [bit 12]     
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.abm [bit 5]  
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.sse4a [bit 6]      
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.misalignsse [bit 7]  
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.3dnowprefetch [bit 8]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.xop [bit 11] 
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.fma4 [bit 16]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.tbm [bit 21]
> qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma
> [bit 12]     
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.abm [bit 5]  
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.sse4a [bit 6]      
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.misalignsse [bit 7]  
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.3dnowprefetch [bit 8]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.xop [bit 11] 
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.fma4 [bit 16]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.tbm [bit 21]
> qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma
> [bit 12]     
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.abm [bit 5]  
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.sse4a [bit 6]      
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.misalignsse [bit 7]  
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.3dnowprefetch [bit 8]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.xop [bit 11] 
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.fma4 [bit 16]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.tbm [bit 21]
> qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma
> [bit 12]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.abm [bit 5]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.sse4a [bit 6]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.misalignsse [bit 7]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.3dnowprefetch [bit 8]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.xop [bit 11]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.fma4 [bit 16]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.tbm [bit 21]
> qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma
> [bit 12]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.abm [bit 5]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.sse4a [bit 6]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.misalignsse [bit 7]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.3dnowprefetch [bit 8]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.xop [bit 11]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.fma4 [bit 16]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.tbm [bit 21]
> qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma
> [bit 12]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.abm [bit 5]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.sse4a [bit 6]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.misalignsse [bit 7]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.3dnowprefetch [bit 8]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.xop [bit 11]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.fma4 [bit 16]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.tbm [bit 21]
> qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma
> [bit 12]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.abm [bit 5]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.sse4a [bit 6]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.misalignsse [bit 7]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.3dnowprefetch [bit 8]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.xop [bit 11]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.fma4 [bit 16]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.tbm [bit 21]
> qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma
> [bit 12]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.abm [bit 5]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.sse4a [bit 6]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.misalignsse [bit 7]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.3dnowprefetch [bit 8]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.xop [bit 11]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.fma4 [bit 16]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.tbm [bit 21]
> qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma
> [bit 12]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.abm [bit 5]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.sse4a [bit 6]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.misalignsse [bit 7]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.3dnowprefetch [bit 8]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.xop [bit 11]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.fma4 [bit 16]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.tbm [bit 21]
> qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma
> [bit 12]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.abm [bit 5]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.sse4a [bit 6]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.misalignsse [bit 7]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.3dnowprefetch [bit 8]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.xop [bit 11]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.fma4 [bit 16]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.tbm [bit 21]
> qemu-kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.fma
> [bit 12]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.abm [bit 5]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.sse4a [bit 6]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.misalignsse [bit 7]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.3dnowprefetch [bit 8]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.xop [bit 11]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.fma4 [bit 16]
> qemu-kvm: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.tbm [bit 21]
> 
> when I connect to the VNC console, it says: "Guest has not initialized the
> display (yet)."
> 
> > 
> > Check secure boot in the guest:
> > # dmesg | grep -i "Secure Boot"
> > [    0.000000] secureboot: Secure boot enabled
> > [    0.000000] Kernel is locked down from EFI secure boot; see man
> > kernel_lockdown.7
> > [    3.476981] integrity: Loaded X.509 cert 'Red Hat Secure Boot CA 5:
> > cc6fa5e72868ba494e939bbd680b9144769a9f8f'
> > 
> I can not perform this step as the guest does not boot.
> 
> The /home/ovmf_debug.log is empty.
> 
> > 
> > 3. Power off the guest, recopy the vars file and then start it again.
> > # cp /usr/share/OVMF/OVMF_VARS.secboot.fd
> > /home/images/vm1_rhel840-64-virtio-scsi.qcow2_VARS.fd
> 
> I've recopied the vars file (although it has not changed?)
> 
> > 
> > The rhel840 guest should start successfully.
> 
> It still does not.
> 
> > 
> > Check the ovmf debug log "/home/ovmf_debug.log", should not found "CR has
> > Bad Signature" or other error message.
> 
> It's still empty.

1. For the cpu warning message, you need to update the command line "-cpu 'Opteron_G5',+kvm_pv_unhalt \"
Your host cpu is not 'Opteron_G5'. You can use "-cpu host,+kvm_pv_unhalt \"

2. You need to con the guest, type "c" in hmp monitor. Please check the VM status by "info status".
QEMU 6.0.0 monitor - type 'help' for more information
(qemu) info status 
VM status: paused
(qemu) c
(qemu) info status 
VM status: running


> 
> Regards,

Comment 10 Rik Theys 2021-07-28 09:33:14 UTC
Created attachment 1806613 [details]
ovmf_debug.log

Hi,

I was able to start a VM this way now.

When I look at the VNC console it says "Guest has not initialized the display (yet)" until I enter 'c' in the monitor. Then I see the UEFI screen come up.

It fails to boot from the qcow2 image as it can't find the UEFI partition in it, but it at least shows the UEFI screen now.

When I stop the vm, copy the file again and restart the vm, the result is the same.

In attach the ovmf_debug.log file.

It seems I'm not able to reproduce the issue as I can with oVirt (as it does not show the UEFI screen there)?

I'm testing this on a different host as I can't upgrade the edk2-ovmf package on the production host right now. Could that make a difference?

Regards,
Rik

Comment 11 Xueqiang Wei 2021-07-30 08:20:32 UTC
(In reply to Rik Theys from comment #10)
> Created attachment 1806613 [details]
> ovmf_debug.log
> 
> Hi,
> 
> I was able to start a VM this way now.
> 
> When I look at the VNC console it says "Guest has not initialized the
> display (yet)" until I enter 'c' in the monitor. Then I see the UEFI screen
> come up.
> 
> It fails to boot from the qcow2 image as it can't find the UEFI partition in
> it, but it at least shows the UEFI screen now.
> 
> When I stop the vm, copy the file again and restart the vm, the result is
> the same.
> 
> In attach the ovmf_debug.log file.
> 
> It seems I'm not able to reproduce the issue as I can with oVirt (as it does
> not show the UEFI screen there)?
> 
> I'm testing this on a different host as I can't upgrade the edk2-ovmf
> package on the production host right now. Could that make a difference?
> 
> Regards,
> Rik

I think we could try it on a different host, make sure the environments are the same between them. (e.g. kernel version, qemu version, edk2 version, guest version).
 
I'm not sure whether the VM installed correctly before booting in your test. Please try to install your VM from cdrom first, and then secure boot it.
By the way, please test secure boot on a released guest.

1. create a new qcow2 image
# /usr/bin/qemu-img create -f qcow2 /home/images/rhel840-64-virtio-scsi.qcow2 20G

2. install a VM with iso file, add the following lines to qemu command lines(mentioned in Comment 7)

    -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,chassis=6 \
    -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie-root-port-5,addr=0x0 \
    -blockdev node-name=file_uefi_cd,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/iso/linux/RHEL8.4.0-BaseOS-x86_64.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_uefi_cd,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_uefi_cd \
    -device scsi-cd,id=ueficd,drive=drive_uefi_cd,write-cache=on \

3. enter 'c' in the monitor, and connect VNC
(qemu) c
$ remote-viewer vnc://your_host_ip:5900

4. Press ESC at the TianoCore splash screen. Navigate to Boot Manager | UEFI QEMU QEMU CD-ROM.
  install the VM from cdrom.

5.run test steps mentioned in Comment 7, check the debug log. Thanks.



Hi Philippe,

Please help check and analyze the attached ovmf debug log. Thanks.

Comment 12 Rik Theys 2021-08-04 12:19:34 UTC
Hi,

I've installed a CentOS 8.4 minimal OS in the qcow file and then reran the commands from comment 7.

The VM boots fine in this case, so it seems I can not reproduce the issue this way.

I will attach the ovmf_debug.log file from the first and second boot.

Regards,
Rik

Comment 13 Rik Theys 2021-08-04 12:21:27 UTC
Created attachment 1810835 [details]
first boot

Comment 14 Rik Theys 2021-08-04 12:21:51 UTC
Created attachment 1810836 [details]
second boot

Comment 15 Xueqiang Wei 2021-08-04 15:31:27 UTC
(In reply to Rik Theys from comment #12)
> Hi,
> 
> I've installed a CentOS 8.4 minimal OS in the qcow file and then reran the
> commands from comment 7.
> 
> The VM boots fine in this case, so it seems I can not reproduce the issue
> this way.
> 
> I will attach the ovmf_debug.log file from the first and second boot.
> 
> Regards,
> Rik

According to your debug log, the guest install successfully and boot successfully with the original vars file "/usr/share/OVMF/OVMF_VARS.secboot.fd".
So I think the vars file in this bug should be corrupted by uncertain operation. It should be the same issue with bug 1981196.


Hi Philippe,

Please help double check the ovmf debug log. Do you agree with me that tracking it by bug 1981196. Thanks.

Comment 16 Klaus Heinrich Kiwi 2021-08-06 15:22:56 UTC
Bug 1981196 is private, while this one is public and was reported earlier.

So I propose we dupe 1981196 into this one and bring the relevant information from that one here...

 -Klaus

Comment 17 Philippe Mathieu-Daudé 2021-08-09 11:44:39 UTC
(In reply to Xueqiang Wei from comment #15)
> (In reply to Rik Theys from comment #12)
> > Hi,
> > 
> > I've installed a CentOS 8.4 minimal OS in the qcow file and then reran the
> > commands from comment 7.
> > 
> > The VM boots fine in this case, so it seems I can not reproduce the issue
> > this way.
>
> Please help double check the ovmf debug log. Do you agree with me that
> tracking it by bug 1981196. Thanks.

Yes, likely the heap overflow in 'shim-x64' from RHEL8.4 reported in https://bugzilla.redhat.com/show_bug.cgi?id=1981196#c27,
since CentOS-8-GenericCloud-8.4.2105-20210603.0.x86_64 is based on it:
https://koji.mbox.centos.org/pkgs/packages/CentOS-8-GenericCloud/8.4.2105/20210603.0/data/logs/image/oz-x86_64.log

Is CentOS7 guest working?

Comment 18 Rik Theys 2021-08-10 06:51:10 UTC
Hi,

(In reply to Philippe Mathieu-Daudé from comment #17)
> (In reply to Xueqiang Wei from comment #15)
> > (In reply to Rik Theys from comment #12)
> > > Hi,
> > > 
> > > I've installed a CentOS 8.4 minimal OS in the qcow file and then reran the
> > > commands from comment 7.
> > > 
> > > The VM boots fine in this case, so it seems I can not reproduce the issue
> > > this way.
> >
> > Please help double check the ovmf debug log. Do you agree with me that
> > tracking it by bug 1981196. Thanks.
> 
> Yes, likely the heap overflow in 'shim-x64' from RHEL8.4 reported in
> https://bugzilla.redhat.com/show_bug.cgi?id=1981196#c27,
> since CentOS-8-GenericCloud-8.4.2105-20210603.0.x86_64 is based on it:
> https://koji.mbox.centos.org/pkgs/packages/CentOS-8-GenericCloud/8.4.2105/
> 20210603.0/data/logs/image/oz-x86_64.log
> 
> Is CentOS7 guest working?

No, a CentOS 7 guest has the same issue.

I've taken the following steps on the production host where we are experiencing this issue:

* Create a CentOS 7 VM with secureboot enabled and edk2-ovmf version 20200602gitca407c7246bf-5.el8 installed
  The VM boots fine and SecureBoot is enabled.

* Poweroff the VM

* Update edk2-ovmf to 20210527gite1999b264f1f-1.el8 and boot the VM.
  The spice console remains black and the VM does not seem to boot. When I connect to the VNC console it displays a message that it's waiting for the display to come up.

It would be strange if this is a shim bug as shim does not seem to have been loaded at that point. The VM does not even show the bios screen.

When I downgrade edk2-ovmf to the previous version and reboot the VM, it boots fine again.

Regards,
Rik

Comment 19 Gerd Hoffmann 2021-08-11 07:43:06 UTC
> I think we could try it on a different host, make sure the environments are
> the same between them. (e.g. kernel version, qemu version, edk2 version,
> guest version).

The guest version probably doesn't matter much given that the virtual machine
seems to not come very far at boot.  The display initialization happens before
ovmf tries to load something from disk.  So the shim failure (bug 1981196) is
probably unrelated.

Unless I missed something in the comments we didn't manage to get a ovmf.log
of a non-working boot yet.

Can you please try on the other host:
  (1) create a guest with a configuration as close as possibe to the one
      created by ovirt, with firmware logging added.
  (2) copy over the VARS file of the guest which doesn't boot and use it
      for the test guest.

Does that reproduce the failure you are seeing?

Comment 20 Rik Theys 2021-08-12 06:09:09 UTC
Created attachment 1813324 [details]
log that shows the assert

Hi,

I was finally able to reproduce it by using the qemu_cmdline vdsm hook and temporarily upgrade edk2-ovmf on the host where I'm seeing the issue.

In the results.tar.gz file are two directories ("works" and "fails") that both contain:

cmdline : the qemu cmdline as executed by vdsm/ovirt
ovmf_debug.log : the debug log
version: the edk2-ovmf package version installed during the test
the .fd files referenced in the qemu_kvm command line

the ovmf_debug.log file in the fails directory now shows the error:

NegotiateSmiFeatures: negotiation failed for feature bitmap 0x5
ASSERT /builddir/build/BUILD/edk2-e1999b264f1f/OvmfPkg/SmmControl2Dxe/SmiFeatures.c(191): ((BOOLEAN)(0==1))

I hope this helps in tracking down this issue.

Regards,
Rik

Comment 21 Xueqiang Wei 2021-08-12 08:35:51 UTC
(In reply to Gerd Hoffmann from comment #19)
> > I think we could try it on a different host, make sure the environments are
> > the same between them. (e.g. kernel version, qemu version, edk2 version,
> > guest version).
> 
> The guest version probably doesn't matter much given that the virtual machine
> seems to not come very far at boot.  The display initialization happens
> before
> ovmf tries to load something from disk.  So the shim failure (bug 1981196) is
> probably unrelated.
> 
> Unless I missed something in the comments we didn't manage to get a ovmf.log
> of a non-working boot yet.
> 
> Can you please try on the other host:
>   (1) create a guest with a configuration as close as possibe to the one
>       created by ovirt, with firmware logging added.
>   (2) copy over the VARS file of the guest which doesn't boot and use it
>       for the test guest.
> 
> Does that reproduce the failure you are seeing?


Let's check comment 20 first, it was reproduced by reporter. The attached logs should be useful. Thanks.

Comment 22 Gerd Hoffmann 2021-08-12 08:57:08 UTC
> the ovmf_debug.log file in the fails directory now shows the error:
> 
> NegotiateSmiFeatures: negotiation failed for feature bitmap 0x5
> ASSERT
> /builddir/build/BUILD/edk2-e1999b264f1f/OvmfPkg/SmmControl2Dxe/SmiFeatures.
> c(191): ((BOOLEAN)(0==1))

From distgit:

commit e0c5e50b37b43e1504b663c244e1eb561582dd5a
Author: Miroslav Rezanina <mrezanin@redhat.com>
Date:   Mon Nov 23 12:30:47 2020 +0100

    * Mon Nov 23 2020 Miroslav Rezanina <mrezanin@redhat.com> - 20200602gitca407c7246bf-4.el8
    - edk2-OvmfPkg-SmmControl2Dxe-negotiate-ICH9_LPC_SMI_F_CPU_.patch [bz#1849177]
    - edk2-OvmfPkg-CpuHotplugSmm-fix-CPU-hotplug-race-just-befo.patch [bz#1849177]
    - edk2-OvmfPkg-CpuHotplugSmm-fix-CPU-hotplug-race-just-afte.patch [bz#1849177]

I guess we have a very hot candidate now ;)

I see in rhel distgit that there is a rebase to edk2-stable202105.
Is that in c8s too?
If so: Does it fix the issue?
If not: Why not?

Comment 23 Rik Theys 2021-08-12 09:09:37 UTC
Hi,

(In reply to Gerd Hoffmann from comment #22)
> > the ovmf_debug.log file in the fails directory now shows the error:
> > 
> > NegotiateSmiFeatures: negotiation failed for feature bitmap 0x5
> > ASSERT
> > /builddir/build/BUILD/edk2-e1999b264f1f/OvmfPkg/SmmControl2Dxe/SmiFeatures.
> > c(191): ((BOOLEAN)(0==1))
> 
> From distgit:
> 
> commit e0c5e50b37b43e1504b663c244e1eb561582dd5a
> Author: Miroslav Rezanina <mrezanin@redhat.com>
> Date:   Mon Nov 23 12:30:47 2020 +0100
> 
>     * Mon Nov 23 2020 Miroslav Rezanina <mrezanin@redhat.com> -
> 20200602gitca407c7246bf-4.el8
>     - edk2-OvmfPkg-SmmControl2Dxe-negotiate-ICH9_LPC_SMI_F_CPU_.patch
> [bz#1849177]
>     - edk2-OvmfPkg-CpuHotplugSmm-fix-CPU-hotplug-race-just-befo.patch
> [bz#1849177]
>     - edk2-OvmfPkg-CpuHotplugSmm-fix-CPU-hotplug-race-just-afte.patch
> [bz#1849177]
> 
> I guess we have a very hot candidate now ;)

According to the rpm changelog, these fixes are already in 20200602gitca407c7246bf-5.el8, which is the version that still works for us:

* Mon Nov 23 2020 Miroslav Rezanina <mrezanin@redhat.com> - 20200602gitca407c7246bf-4.el8
- edk2-OvmfPkg-SmmControl2Dxe-negotiate-ICH9_LPC_SMI_F_CPU_.patch [bz#1849177]
- edk2-OvmfPkg-CpuHotplugSmm-fix-CPU-hotplug-race-just-befo.patch [bz#1849177]
- edk2-OvmfPkg-CpuHotplugSmm-fix-CPU-hotplug-race-just-afte.patch [bz#1849177]
- edk2-CryptoPkg-OpensslLib-Upgrade-OpenSSL-to-1.1.1g.patch [bz#1893806]
- edk2-redhat-bump-OpenSSL-dist-git-submodule-to-1.1.1g-RHE.patch [bz#1893806]
- Resolves: bz#1849177
  (OVMF: negotiate "SMI on VCPU hotplug" with QEMU)
- Resolves: bz#1893806
  (attempt advancing RHEL8 edk2's OpenSSL submodule to RHEL8 OpenSSL 1.1.1g (or later))

> 
> I see in rhel distgit that there is a rebase to edk2-stable202105.
> Is that in c8s too?

c8s has 20210527gite1999b264f1f, which has the issue.

> If so: Does it fix the issue?
No, in fact it seems to introduce it. We first triggered this issue when we upgraded our host from 20200602gitca407c7246bf-5.el8.noarch to 20210527gite1999b264f1f-1.el8

> If not: Why not?

Regards,
Rik

Comment 24 Gerd Hoffmann 2021-08-12 09:38:05 UTC
> > I see in rhel distgit that there is a rebase to edk2-stable202105.
> > Is that in c8s too?
> 
> c8s has 20210527gite1999b264f1f, which has the issue.
> 
> > If so: Does it fix the issue?
> No, in fact it seems to introduce it. We first triggered this issue when we
> upgraded our host from 20200602gitca407c7246bf-5.el8.noarch to
> 20210527gite1999b264f1f-1.el8

Ah, ok.  Thanks for the clarification.  I was thinking the upgrade *to* 20200602gitca407c7246bf-5.el8.noarch broke things, not the update *from* that version to newer.

So it is probably this upstream commit (picked up by the rebase).

commit f3bdfc41866edf7c256e689deb9d091a950c8fca
Author: Ankur Arora <ankur.a.arora@oracle.com>
Date:   Thu Mar 11 22:26:56 2021 -0800

    OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug

"feature bitmap 0x5" is BROADCAST + CPU_HOT_UNPLUG bits.

Feature CPU_HOT_UNPLUG without CPU_HOTPLUG doesn't make much sense indeed.
The only way I can see this happening is qemu advertizing CPU_HOT_UNPLUG
without CPU_HOTPLUG though.

Looks like I have to go dig on the qemu side of things, on a quick glance
it looks like ovmf is just the messenger here ...

Comment 25 Gerd Hoffmann 2021-08-12 10:00:44 UTC
> Looks like I have to go dig on the qemu side of things, on a quick glance
> it looks like ovmf is just the messenger here ...

Seems we run with ICH9-LPC.x-smi-cpu-hotplug=off and ICH9-LPC.x-smi-cpu-hotunplug=on
Upstream machine types look sane on a quick glance, probably something is messed up
with the rhel machine types.

Comment 26 Gerd Hoffmann 2021-08-12 10:18:38 UTC
> qemu-kvm-6.0.0-19.el8s.x86_64

6.0.0-20.el8 has some machine type changes, does the failure go away if you update to that version (or -21)?

Comment 28 Rik Theys 2021-08-12 10:49:15 UTC
(In reply to Gerd Hoffmann from comment #26)
> > qemu-kvm-6.0.0-19.el8s.x86_64
> 
> 6.0.0-20.el8 has some machine type changes, does the failure go away if you
> update to that version (or -21)?

I've updated the host to 

edk2-ovmf-20210527gite1999b264f1f-3.el8.noarch
qemu-kvm-6.0.0-26.el8s.x86_64

and the VM boots now! Thanks you all for your help!

I will attach the ovmf_debug.log from that boot.

Regards,
Rik

Comment 29 Rik Theys 2021-08-12 10:51:02 UTC
Created attachment 1813426 [details]
debug logfile after upgrade of qemu-kvm and edk2-ovmf

Comment 30 Yanan Fu 2021-08-17 06:39:14 UTC
Gating test with qemu-kvm-6.0.0-20.module+el8.5.0+11499+199527ef test pass.
Add 'Verified:Tested,SanityOnly' accordlingly.

Comment 31 Xueqiang Wei 2021-08-17 09:27:42 UTC
Tested edk2 test loop with qemu-kvm-6.0.0-27.module+el8.5.0+12121+c40c8708 and qemu-kvm-6.0.0-20.module+el8.5.0+11499+199527ef, no new bug found. And According to Comment 28, this bug has been fixed.


Versions:
kernel-4.18.0-325.el8.x86_64
qemu-kvm-6.0.0-27.module+el8.5.0+12121+c40c8708
qemu-kvm-6.0.0-20.module+el8.5.0+11499+199527ef
edk2-ovmf-20210527gite1999b264f1f-3.el8.noarch

Comment 32 Laszlo Ersek 2021-08-17 11:40:47 UTC
Excellent analysis, thank you, Gerd!

So the(main) missing RHEL-AV-8.4 QEMU bit was this hunk, now included in downstream commit 64c350696f1d ("x86: Add x86 rhel8.5 machine types", 2021-06-21), part of tag "qemu-kvm-6.0.0-20.module+el8.5.0+11499+199527ef":

diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index edc02a68caf1..0a374dec3931 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -369,6 +369,12 @@ GlobalProperty pc_rhel_compat[] = {
 };
 const size_t pc_rhel_compat_len = G_N_ELEMENTS(pc_rhel_compat);
 
+GlobalProperty pc_rhel_8_4_compat[] = {
+    /* pc_rhel_8_4_compat from pc_compat_5_2 */
+    { "ICH9-LPC", "x-smi-cpu-hotunplug", "off" },
+};
+const size_t pc_rhel_8_4_compat_len = G_N_ELEMENTS(pc_rhel_8_4_compat);
+
 GlobalProperty pc_rhel_8_3_compat[] = {
     /* pc_rhel_8_3_compat from pc_compat_5_1 */
     { "ICH9-LPC", "x-smi-cpu-hotplug", "off" },

Comment 33 Xueqiang Wei 2021-08-18 05:28:55 UTC
According to Comment 28 and Comment 31, set status to CURRENTRELEASE. Thanks.


Note You need to log in before you can comment on or make changes to this bug.