Bug 1961562

Summary: vm can not start with error as "internal error: unknown feature amd-sev-es"
Product: Red Hat Enterprise Linux 8 Reporter: yalzhang <yalzhang>
Component: libvirtAssignee: Pavel Hrdina <phrdina>
Status: CLOSED ERRATA QA Contact: Meina Li <meili>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 8.5CC: agurenko, atodorov, aybhalala, bxue, carl, coli, davide, dkaylor, dtantsur, eminguez, ernunes, gradde, jdenemar, jfrieben, jhughes, jinzhao, jsuchane, juzhang, knoel, meili, mmizuma, mpitt, nk, phrdina, pierre, rjones, troels, virt-maint, vpolasek, yoguo, zhilli
Target Milestone: betaKeywords: FutureFeature, Regression, Reopened, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-6.0.0-36.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-09 18:00:11 UTC Type: Feature Request
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1969483    

Description yalzhang@redhat.com 2021-05-18 09:24:52 UTC
Description of problem:
vm can not start with error as "internal error: unknown feature amd-sev-es"

Version-Release number of selected components (if applicable):
# rpm -q libvirt qemu-kvm kernel
libvirt-6.0.0-35.module+el8.5.0+10709+b3edb581.x86_64
qemu-kvm-4.2.0-50.module+el8.5.0+10875+d90dbc7e.x86_64
kernel-4.18.0-305.6.el8.x86_64

How reproducible:
100%

Steps to Reproduce:
1. provision the system to rhel 8.5 and install the virt:rhel module;
2. use virt-install to install a vm:
# virt-install -n rhel -r 1024 -f ./RHEL-8.5-x86_64-latest.qcow2  --import
WARNING  No operating system detected, VM performance may suffer. Specify an OS with --os-variant for optimal results.
WARNING  Unable to connect to graphical console: virt-viewer not installed. Please install the 'virt-viewer' package.
WARNING  No console to launch for the guest, defaulting to --wait -1
Starting install...
ERROR    internal error: unknown feature amd-sev-es
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect qemu:///system start rhel
otherwise, please restart your installation.

Actual results:
vm can not start with error as "internal error: unknown feature amd-sev-es"

Expected results:
vm should start successfully

Additional info:
# cat /var/log/libvirt/libvirtd.log | grep error
2021-05-18 08:44:48.522+0000: 22806: error : qemuFirmwareFeatureParse:595 : internal error: unknown feature amd-sev-es
2021-05-18 08:44:48.522+0000: 22804: error : qemuFirmwareFeatureParse:595 : internal error: unknown feature amd-sev-es
2021-05-18 08:44:48.525+0000: 22803: error : qemuFirmwareFeatureParse:595 : internal error: unknown feature amd-sev-es
2021-05-18 08:44:48.541+0000: 22804: error : qemuFirmwareFeatureParse:595 : internal error: unknown feature amd-sev-es

Comment 2 Richard W.M. Jones 2021-05-18 10:10:27 UTC
See bug 1961558 for similar issue on RHEL AV.

Note a simpler reproducer is:

# virsh domcapabilities
error: failed to get emulator capabilities
error: internal error: unknown feature amd-sev-es

Comment 3 yalzhang@redhat.com 2021-05-18 10:32:38 UTC
(In reply to Richard W.M. Jones from comment #2)
> See bug 1961558 for similar issue on RHEL AV.
> 
> Note a simpler reproducer is:
> 
> # virsh domcapabilities
> error: failed to get emulator capabilities
> error: internal error: unknown feature amd-sev-es

Yes, I think it is the same bug, so this one can be closed as duplicate.

*** This bug has been marked as a duplicate of bug 1961558 ***

Comment 4 Pavel Hrdina 2021-05-18 11:19:35 UTC
We should no close BZ as duplicate if the BZs are for RHEL and RHEL-AV as they have different code-base. Reopening the BZ as we will need to backport the following upstream commit:

commit 61d95a1073833ec4323c1ef28e71e913c55aa7b9
Author: Pavel Hrdina <phrdina>
Date:   Mon May 10 15:07:09 2021 +0200

    qemu_firmware: don't error out for unknown firmware features

Comment 9 Martin Pitt 2021-05-19 07:13:16 UTC
Bumping severity, as this completely breaks libvirt/qemu:

# cat /tmp/xml
<domain type='qemu'>
  <name>subVmTest1</name>
  <os>
    <type arch='x86_64'>hvm</type>
    <boot dev='hd'/>
    <boot dev='network'/>
  </os>
  <memory unit='MiB'>128</memory>
</domain>

# virsh define /tmp/xml
error: Failed to define domain from /tmp/xml
error: internal error: unknown feature amd-sev-es

Comment 11 Martin Pitt 2021-05-19 07:27:32 UTC
Is there any known workaround? Right now this completely blocks our package updates in RHEL 8.5 and our CI. Thanks!

Comment 13 yalzhang@redhat.com 2021-05-20 01:32:37 UTC
(In reply to Martin Pitt from comment #11)
> Is there any known workaround? Right now this completely blocks our package
> updates in RHEL 8.5 and our CI. Thanks!

You can try to download the package: edk2-20200602gitca407c7246bf-5.el8, refer to bug 1961558#c10

Comment 15 yalzhang@redhat.com 2021-05-20 01:46:34 UTC
In reply to yalzhang from comment #13)
> (In reply to Martin Pitt from comment #11)
> > Is there any known workaround? Right now this completely blocks our package
> > updates in RHEL 8.5 and our CI. Thanks!
> 
> You can try to download the package: edk2-20200602gitca407c7246bf-5.el8,
> refer to bug 1961558#c10

s/download/downgrade
s/edk2-20200602gitca407c7246bf-5.el8/edk2-ovmf-20200602gitca407c7246bf-5.el8.noarch

Sorry for the misunderstanding. I have tried and it works well. Just downgrade the current edk2-ovmf-20200602gitca407c7246bf-5.el8.noarch to edk2-ovmf-20200602gitca407c7246bf-4.el8.noarch which without the fix mentioned in bug 1961558#c10 will workaround the issue.

Comment 16 Pavel Hrdina 2021-05-20 11:40:03 UTC
(In reply to Martin Pitt from comment #11)
> Is there any known workaround? Right now this completely blocks our package
> updates in RHEL 8.5 and our CI. Thanks!

Yes, there is simple workaround:

  mkdir -p /etc/qemu/firmware
  touch /etc/qemu/firmware/50-edk2-ovmf-cc.json

This will create an empty file which can disable the new firmware, more details here [1], look for firmware description.

[1] <https://libvirt.org/formatdomain.html#operating-system-booting>

Comment 17 Martin Pitt 2021-05-20 12:47:03 UTC
Thanks Pavel! I'm trying that in https://github.com/cockpit-project/cockpit-machines/pull/177 and it seems to generally work. Great!

Comment 20 Alexander Todorov 2021-05-27 07:00:55 UTC
FTR I have been seeing this rather often during osbuild-composer testing after we switched to GitLab CI:
https://gitlab.com/osbuild/ci/osbuild-composer/-/jobs/1295926825 - this particular one is for CentOS but I've seen this on RHEL & Fedora as well.

Comment 21 Aditya Patel 2021-05-28 03:35:38 UTC
Faced the same issue in centos 8 after I updated all packages from cockpit and rebooted the server. Thanks to Paval, his patch worked for now.

Comment 22 Johnny Hughes 2021-05-29 11:02:53 UTC
This issue is also present in the current CentOS Stream 8.

# rpm -q libvirt-daemon qemu-kvm kernel edk2-ovmf
libvirt-daemon-6.0.0-35.module_el8.5.0+746+bbd5d70c.x86_64
qemu-kvm-4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64
kernel-4.18.0-305.el8.x86_64
edk2-ovmf-20200602gitca407c7246bf-5.el8.noarch

Comment 27 Meina Li 2021-06-04 07:24:07 UTC
Verified Version:
libvirt-6.0.0-36.module+el8.5.0+11222+c889b3f3.x86_64
qemu-kvm-4.2.0-51.module+el8.5.0+11141+9dff516f.x86_64

Verified Steps:
1. Prepare a guest xml:
# cat lmn.xml
...
<os>
    <type arch='x86_64' machine='pc-q35-rhel8.2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
...
 <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/lmn.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
...
2. Define and start the guest.
# virsh define lmn.xml 
Domain lmn defined from lmn.xml
# virsh start lmn
Domain lmn started

3. Check domcapabilities.
# virsh domcapabilities 
<domainCapabilities>
  <path>/usr/libexec/qemu-kvm</path>
  <domain>kvm</domain>
...
    <backup supported='no'/>
    <sev supported='no'/>
  </features>
</domainCapabilities>

Comment 28 Pierre Riteau 2021-06-07 20:35:16 UTC
Hello. Is there an estimate of when this issue may be fixed in CentOS Stream 8?

Comment 29 Carl George 🤠 2021-06-08 23:46:57 UTC
libvirt-6.0.0-36.el8 has been built [0] and released for CentOS Stream 8.


[0] https://koji.mbox.centos.org/koji/buildinfo?buildID=17918

Comment 30 Federico Iezzi 2021-06-09 06:52:21 UTC
Also CentOS 8 Advanced Virtualization module is broken (I didn't try the Stream 8 AV yet).
Any plans to fix this as well?

http://mirror.centos.org/centos/8/virt/x86_64/advanced-virtualization/Packages/l/

Comment 31 Pierre Riteau 2021-06-09 07:46:48 UTC
(In reply to Carl George 🤠 from comment #29)
> libvirt-6.0.0-36.el8 has been built [0] and released for CentOS Stream 8.
> 
> 
> [0] https://koji.mbox.centos.org/koji/buildinfo?buildID=17918

Great news, thank you Carl!

Comment 34 errata-xmlrpc 2021-11-09 18:00:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: virt:rhel and virt-devel:rhel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4191

Comment 35 Pavel Hrdina 2021-11-11 09:24:51 UTC
*** Bug 2022101 has been marked as a duplicate of this bug. ***