Bug 2184966

This seems has been fixed upstream:

commit a0065f97281f70d8c2017ca25c23f16120cee399
Author: Andrea Bolognani <abologna>
Date:   Fri Mar 17 11:12:31 2023 +0100

    conf: Don't format firmware type/features when migrating
    
    The combination of explicit firmware paths, which we now
    produce in all cases, and firmware autoselection knobs is
    explicitly rejected by libvirt 8.6.0 and newer.
    
    Right now we produce inherently migratable XML in all cases,
    since we always strip those bits, but that's going to change
    soon. To prepare for that, make sure that we always skip the
    problematic elements and attributes when preparing a
    migratable XML.
    
    The destination will simply receive a fully specified firmware
    configuration, which is indistinguishable from one that was
    manually provided by the user and is thus accepted by any old
    version of libvirt, regardless of whether or not firmware
    autoselection was used on the source host.
    
    Signed-off-by: Andrea Bolognani <abologna>
    Reviewed-by: Michal Privoznik <mprivozn>

This is pretty unfortunate.

I wish we'd have realized sooner that this migration issue existed,
but with RHEL 9.1 getting libvirt 8.5.0 and the changes in behavior
being introduced in libvirt 8.6.0 I guess the timing was just the
worst possible one :(

To summarize the extent of the problem, versions of libvirt >= 8.6.0
and < 9.2.0 produce, when feature-based firmware autoselection is in
use, migratable XML that's rejected by libvirt >= 8.6.0.

If we fixed libvirt 9.0.0 in RHEL 9.2 to produce more acceptable XML
we'd be good as far as RHEL is concerned, but from an upstream point
of view a much better solution would be to make it so that libvirt
accepts the incoming migration XML that it currently rejects.

I'm going to look into whether I can make that work.

Patches posted upstream.

  https://listman.redhat.com/archives/libvir-list/2023-April/239318.html

After applying the same changes to libvirt 9.0.0, I was able to
successfully migrate a domain from libvirt 8.6.0, which generates the
problematic XML.

Fix merged upstream.

  commit f9ad3023355bcbfc692bbe4997fdfa774866a980
  Author: Andrea Bolognani <abologna>
  Date:   Tue Apr 11 17:56:45 2023 +0200

    conf: Fix migration in some firmware autoselection scenarios
    
    Introduce a small kludge in the parser to avoid unnecessarily
    blocking incoming migration from a range of recent libvirt
    releases.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=2184966
    
    Signed-off-by: Andrea Bolognani <abologna>
    Reviewed-by: Ján Tomko <jtomko>

  v9.2.0-224-gf9ad302335

Test Version:
libvirt-9.3.0-1.el9.x86_64
qemu-kvm-8.0.0-2.el9.x86_64

Test Steps:
1. Prepare a running guest with the following os xml.
# virsh dumpxml rhel --xpath os
<os firmware="efi">
  <type arch="x86_64" machine="pc-q35-rhel9.2.0">hvm</type>
  <firmware>
    <feature enabled="yes" name="enrolled-keys"/>
    <feature enabled="yes" name="secure-boot"/>
  </firmware>
  <loader readonly="yes" secure="yes" type="pflash">/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader>
  <nvram template="/usr/share/edk2/ovmf/OVMF_VARS.secboot.fd">/var/lib/libvirt/qemu/nvram/rhel_VARS.fd</nvram>
  <boot dev="hd"/>
</os>
2. Check the migratable os xml.
# virsh dumpxml rhel --xpath os --migratable
<os>
  <type arch="x86_64" machine="pc-q35-rhel9.2.0">hvm</type>
  <loader readonly="yes" secure="yes" type="pflash">/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader>
  <nvram template="/usr/share/edk2/ovmf/OVMF_VARS.secboot.fd">/var/lib/libvirt/qemu/nvram/rhel_VARS.fd</nvram>
  <boot dev="hd"/>
</os>
3. Migrate the guest to target host.
# virsh migrate rhel qemu+ssh://dell-per740-04.dell2.lab.eng.bos.redhat.com/system --live --verbose --p2p --persistent
Migration: [100 %]
4. Check in the target host and the guest can work.
# virsh list --all
 Id   Name             State
---------------------------------
 1    rhel             running
# virsh dumpxml rhel --xpath os
<os firmware="efi">
  <type arch="x86_64" machine="pc-q35-rhel9.2.0">hvm</type>
  <firmware>
    <feature enabled="yes" name="enrolled-keys"/>
    <feature enabled="yes" name="secure-boot"/>
  </firmware>
  <loader readonly="yes" secure="yes" type="pflash">/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd</loader>
  <nvram template="/usr/share/edk2/ovmf/OVMF_VARS.secboot.fd">/var/lib/libvirt/qemu/nvram/rhel_VARS.fd</nvram>
  <boot dev="hd"/>
</os>
5. Migrate back to the source host.
# virsh migrate rhel qemu+ssh://dell-per750-39.lab.eng.pek2.redhat.com/system --live --verbose --p2p --persistent
Migration: [100 %]

Test Version:
libvirt-9.3.0-2.el9.x86_64
qemu-kvm-8.0.0-3.el9.x86_64

Test Steps:
steps in comment 16

Test Result:
PASS

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: libvirt security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6409