RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2184966 - Failed to migrate uefi vm with firmware XML config
Summary: Failed to migrate uefi vm with firmware XML config
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: libvirt
Version: 9.2
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Andrea Bolognani
QA Contact: Meina Li
URL:
Whiteboard:
Depends On:
Blocks: 2186383
TreeView+ depends on / blocked
 
Reported: 2023-04-06 10:19 UTC by Fangge Jin
Modified: 2023-11-07 09:40 UTC (History)
16 users (show)

Fixed In Version: libvirt-9.3.0-1.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2186383 (view as bug list)
Environment:
Last Closed: 2023-11-07 08:31:17 UTC
Type: Bug
Target Upstream Version: 9.3.0
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-154170 0 None None None 2023-04-06 10:20:48 UTC
Red Hat Product Errata RHSA-2023:6409 0 None None None 2023-11-07 08:31:49 UTC

Description Fangge Jin 2023-04-06 10:19:41 UTC 
Description of problem:
Migrate uefi vm with firmware feature configuration, it reports error:
error: cannot use feature-based firmware autoselection when firmware autoselection is disabled


Version-Release number of selected component (if applicable):
libvirt-9.0.0-10.el9_2.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Start a uefi vm with firmware feature:
<os firmware="efi">
  <type arch="x86_64" machine="pc-q35-rhel9.0.0">hvm</type>
  <firmware>
    <feature enabled="no" name="enrolled-keys"/>
    <feature enabled="yes" name="secure-boot"/>
  </firmware>
  <boot dev="hd"/>
</os>

2. Migrate vm
# virsh migrate vm11 qemu+tcp://{target_host}/system --live --p2p
error: cannot use feature-based firmware autoselection when firmware autoselection is disabled

3. Check the active xml and migratable xml:
# virsh dumpxml vm11 --xpath os 
<os>
  <type arch="x86_64" machine="pc-q35-rhel9.0.0">hvm</type>
  <firmware>
    <feature enabled="no" name="enrolled-keys"/>
    <feature enabled="yes" name="secure-boot"/>
  </firmware>
  <loader readonly="yes" secure="yes" type="pflash">/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader>
  <nvram template="/usr/share/edk2/ovmf/OVMF_VARS.fd">/var/lib/libvirt/qemu/nvram/vm11_VARS.fd</nvram>
  <boot dev="hd"/>
</os>

# virsh dumpxml vm11 --xpath os --migratable
<os>
  <type arch="x86_64" machine="pc-q35-rhel9.0.0">hvm</type>
  <firmware>
    <feature enabled="no" name="enrolled-keys"/>
    <feature enabled="yes" name="secure-boot"/>
  </firmware>
  <loader readonly="yes" secure="yes" type="pflash">/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader>
  <nvram template="/usr/share/edk2/ovmf/OVMF_VARS.fd">/var/lib/libvirt/qemu/nvram/vm11_VARS.fd</nvram>
  <boot dev="hd"/>
</os>



Actual results:
Migration failed

Expected results:
Migration succeeds

Additional info:
1. Migration can succeed with libvirt-8.5.0-7.4.el9_1.x86_64.
# virsh dumpxml vm11 --migratable --xpath //os
<os>
  <type arch="x86_64" machine="pc-q35-rhel9.0.0">hvm</type>
  <loader readonly="yes" type="pflash">/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader>
  <nvram template="/usr/share/edk2/ovmf/OVMF_VARS.fd">/var/lib/libvirt/qemu/nvram/vm11_VARS.fd</nvram>
  <boot dev="hd"/>
</os>
Comment 3 Fangge Jin 2023-04-10 09:48:05 UTC 
This seems has been fixed upstream:

commit a0065f97281f70d8c2017ca25c23f16120cee399
Author: Andrea Bolognani <abologna>
Date:   Fri Mar 17 11:12:31 2023 +0100

    conf: Don't format firmware type/features when migrating
    
    The combination of explicit firmware paths, which we now
    produce in all cases, and firmware autoselection knobs is
    explicitly rejected by libvirt 8.6.0 and newer.
    
    Right now we produce inherently migratable XML in all cases,
    since we always strip those bits, but that's going to change
    soon. To prepare for that, make sure that we always skip the
    problematic elements and attributes when preparing a
    migratable XML.
    
    The destination will simply receive a fully specified firmware
    configuration, which is indistinguishable from one that was
    manually provided by the user and is thus accepted by any old
    version of libvirt, regardless of whether or not firmware
    autoselection was used on the source host.
    
    Signed-off-by: Andrea Bolognani <abologna>
    Reviewed-by: Michal Privoznik <mprivozn>
Comment 4 Andrea Bolognani 2023-04-11 13:35:04 UTC 
This is pretty unfortunate.

I wish we'd have realized sooner that this migration issue existed,
but with RHEL 9.1 getting libvirt 8.5.0 and the changes in behavior
being introduced in libvirt 8.6.0 I guess the timing was just the
worst possible one :(

To summarize the extent of the problem, versions of libvirt >= 8.6.0
and < 9.2.0 produce, when feature-based firmware autoselection is in
use, migratable XML that's rejected by libvirt >= 8.6.0.

If we fixed libvirt 9.0.0 in RHEL 9.2 to produce more acceptable XML
we'd be good as far as RHEL is concerned, but from an upstream point
of view a much better solution would be to make it so that libvirt
accepts the incoming migration XML that it currently rejects.

I'm going to look into whether I can make that work.
Comment 5 Andrea Bolognani 2023-04-11 17:56:52 UTC 
Patches posted upstream.

  https://listman.redhat.com/archives/libvir-list/2023-April/239318.html

After applying the same changes to libvirt 9.0.0, I was able to
successfully migrate a domain from libvirt 8.6.0, which generates the
problematic XML.
Comment 10 Andrea Bolognani 2023-04-12 16:29:14 UTC 
Fix merged upstream.

  commit f9ad3023355bcbfc692bbe4997fdfa774866a980
  Author: Andrea Bolognani <abologna>
  Date:   Tue Apr 11 17:56:45 2023 +0200

    conf: Fix migration in some firmware autoselection scenarios
    
    Introduce a small kludge in the parser to avoid unnecessarily
    blocking incoming migration from a range of recent libvirt
    releases.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=2184966
    
    Signed-off-by: Andrea Bolognani <abologna>
    Reviewed-by: Ján Tomko <jtomko>

  v9.2.0-224-gf9ad302335
Comment 16 Meina Li 2023-05-11 05:47:25 UTC 
Test Version:
libvirt-9.3.0-1.el9.x86_64
qemu-kvm-8.0.0-2.el9.x86_64

Test Steps:
1. Prepare a running guest with the following os xml.
# virsh dumpxml rhel --xpath os
<os firmware="efi">
  <type arch="x86_64" machine="pc-q35-rhel9.2.0">hvm</type>
  <firmware>
    <feature enabled="yes" name="enrolled-keys"/>
    <feature enabled="yes" name="secure-boot"/>
  </firmware>
  <loader readonly="yes" secure="yes" type="pflash">/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader>
  <nvram template="/usr/share/edk2/ovmf/OVMF_VARS.secboot.fd">/var/lib/libvirt/qemu/nvram/rhel_VARS.fd</nvram>
  <boot dev="hd"/>
</os>
2. Check the migratable os xml.
# virsh dumpxml rhel --xpath os --migratable
<os>
  <type arch="x86_64" machine="pc-q35-rhel9.2.0">hvm</type>
  <loader readonly="yes" secure="yes" type="pflash">/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader>
  <nvram template="/usr/share/edk2/ovmf/OVMF_VARS.secboot.fd">/var/lib/libvirt/qemu/nvram/rhel_VARS.fd</nvram>
  <boot dev="hd"/>
</os>
3. Migrate the guest to target host.
# virsh migrate rhel qemu+ssh://dell-per740-04.dell2.lab.eng.bos.redhat.com/system --live --verbose --p2p --persistent
Migration: [100 %]
4. Check in the target host and the guest can work.
# virsh list --all
 Id   Name             State
---------------------------------
 1    rhel             running
# virsh dumpxml rhel --xpath os
<os firmware="efi">
  <type arch="x86_64" machine="pc-q35-rhel9.2.0">hvm</type>
  <firmware>
    <feature enabled="yes" name="enrolled-keys"/>
    <feature enabled="yes" name="secure-boot"/>
  </firmware>
  <loader readonly="yes" secure="yes" type="pflash">/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader>
  <nvram template="/usr/share/edk2/ovmf/OVMF_VARS.secboot.fd">/var/lib/libvirt/qemu/nvram/rhel_VARS.fd</nvram>
  <boot dev="hd"/>
</os>
5. Migrate back to the source host.
# virsh migrate rhel qemu+ssh://dell-per750-39.lab.eng.pek2.redhat.com/system --live --verbose --p2p --persistent
Migration: [100 %]
Comment 19 Meina Li 2023-05-23 02:08:33 UTC 
Test Version:
libvirt-9.3.0-2.el9.x86_64
qemu-kvm-8.0.0-3.el9.x86_64

Test Steps:
steps in comment 16

Test Result:
PASS
Comment 21 errata-xmlrpc 2023-11-07 08:31:17 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: libvirt security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6409
Note You need to log in before you can comment on or make changes to this bug.