scratch build
https://kojihub.stream.rdu2.redhat.com/koji/taskinfo?taskID=2142465

Hi Gerd,

From the scratch build of x86_64, I didn't find the qcow2 file:
# rpm -qa | grep edk2
edk2-ovmf-20230301gitf80f052277c8-2.el9.bz2186754.20230421.0950.noarch
# ll /usr/share/edk2/ovmf/
total 19868
-rw-r--r--. 1 root root   19008 Apr 21 03:56 EnrollDefaultKeys.efi
-rw-r--r--. 1 root root 4194304 Apr 21 03:57 OVMF.amdsev.fd
lrwxrwxrwx. 1 root root      12 Apr 21 03:57 OVMF_CODE.cc.fd -> OVMF_CODE.fd
-rw-r--r--. 1 root root 3653632 Apr 21 03:55 OVMF_CODE.fd
-rw-r--r--. 1 root root 3653632 Apr 21 03:56 OVMF_CODE.secboot.fd
-rw-r--r--. 1 root root 4194304 Apr 21 03:57 OVMF.inteltdx.fd
-rw-r--r--. 1 root root  540672 Apr 21 03:55 OVMF_VARS.fd
-rw-r--r--. 1 root root  540672 Apr 21 03:57 OVMF_VARS.secboot.fd
-rw-r--r--. 1 root root  949056 Apr 21 03:55 Shell.efi
-rw-r--r--. 1 root root 2594816 Apr 21 03:57 UefiShell.iso

So I'm just curious, is this bug only fixed for aarch64? Or do I have some installation errors somewhere that the file is not found? Because from bug 2161965 we can know that this feature is not only implemented on aarch.

Thanks.

When using qcow2 firmware images + smmuv3, the firmware is broken, qemu reports "Fail to update device iotlb" and the guest crashes.

MALLOC_PERTURB_=1  /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -blockdev '{"node-name": "file_aavmf_code", "driver": "file", "filename": "/usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.qcow2", "auto-read-only": true, "discard": "unmap"}' \
    -blockdev '{"node-name": "drive_aavmf_code", "driver": "qcow2", "read-only": true, "file": "file_aavmf_code"}' \
    -blockdev '{"node-name": "file_aavmf_vars", "driver": "file", "filename": "/root/avocado/data/avocado-vt/avocado-vt-vm1_rhel930-aarch64-virtio_qcow2_filesystem_VARS.fd", "auto-read-only": true, "discard": "unmap"}' \
    -blockdev '{"node-name": "drive_aavmf_vars", "driver": "qcow2", "read-only": false, "file": "file_aavmf_vars"}' \
    -machine virt,gic-version=host,its=on,ras=on,iommu=smmuv3,memory-backend=mem-machine_mem,pflash0=drive_aavmf_code,pflash1=drive_aavmf_vars \
    -device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
    -device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
    -nodefaults \
    -device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
    -device '{"driver": "virtio-gpu-pci", "bus": "pcie-root-port-1", "addr": "0x0", "iommu_platform": true}' \
    -m 8192 \
    -object '{"size": 8589934592, "id": "mem-machine_mem", "qom-type": "memory-backend-ram"}'  \
    -smp 4,maxcpus=4,cores=2,threads=1,clusters=1,sockets=2  \
    -cpu 'host' \
    -chardev socket,path=/var/tmp/avocado_q67d546k/monitor-qmpmonitor1-20230422-225332-rHmfjbmX,server=on,wait=off,id=qmp_id_qmpmonitor1  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,path=/var/tmp/avocado_q67d546k/monitor-catch_monitor-20230422-225332-rHmfjbmX,server=on,wait=off,id=qmp_id_catch_monitor  \
    -mon chardev=qmp_id_catch_monitor,mode=control  \
    -serial unix:'/var/tmp/avocado_q67d546k/serial-serial0-20230422-225332-rHmfjbmX',server=on,wait=off \
    -object '{"qom-type": "rng-random", "filename": "/dev/urandom", "id": "passthrough-PAyG66rC"}' \
    -device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
    -device '{"driver": "virtio-rng-pci", "id": "virtio-rng-JVgaQ1gS", "rng": "passthrough-PAyG66rC", "bus": "pcie-root-port-2", "addr": "0x0", "iommu_platform": true}' \
    -device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
    -device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-3", "addr": "0x0"}' \
    -device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
    -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/rhel930-aarch64-virtio.qcow2", "cache": {"direct": true, "no-flush": false}}' \
    -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
    -device '{"id": "pcie-root-port-4", "port": 4, "driver": "pcie-root-port", "addr": "0x1.0x4", "bus": "pcie.0", "chassis": 5}' \
    -device '{"driver": "virtio-blk-pci", "id": "image1", "drive": "drive_image1", "bootindex": 0, "write-cache": "on", "bus": "pcie-root-port-4", "addr": "0x0", "iommu_platform": true}' \
    -device '{"id": "pcie-root-port-5", "port": 5, "driver": "pcie-root-port", "addr": "0x1.0x5", "bus": "pcie.0", "chassis": 6}' \
    -device '{"driver": "virtio-net-pci", "mac": "9a:9c:d6:1d:1d:0c", "rombar": 0, "id": "idduALMG", "netdev": "idafZETG", "bus": "pcie-root-port-5", "addr": "0x0", "iommu_platform": true}'  \
    -netdev tap,id=idafZETG,vhost=on,vhostfd=16,fd=12  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew \
    -chardev socket,id=char_vtpm_avocado-vt-vm1_tpm0,path=/root/avocado/data/avocado-vt/swtpm/avocado-vt-vm1_tpm0_swtpm.sock \
    -tpmdev emulator,chardev=char_vtpm_avocado-vt-vm1_tpm0,id=emulator_vtpm_avocado-vt-vm1_tpm0 \
    -device '{"id": "tpm-tis-device_vtpm_avocado-vt-vm1_tpm0", "tpmdev": "emulator_vtpm_avocado-vt-vm1_tpm0", "driver": "tpm-tis-device"}' \
    -enable-kvm \
    -device '{"id": "pcie-root-port-6", "port": 6, "driver": "pcie-root-port", "addr": "0x1.0x6", "bus": "pcie.0", "chassis": 7}' \
    -device '{"driver": "virtio-balloon-pci", "id": "balloon0", "bus": "pcie-root-port-6", "addr": "0x0"}' \
    -device '{"id": "pcie_extra_root_port_0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x2", "chassis": 8}' \


[stdlog] 2023-04-22 22:54:58,610 avocado.virttest.qemu_vm INFO | [qemu output] qemu-kvm: Fail to update device iotlb
[stdlog] 2023-04-22 22:54:58,610 avocado.virttest.qemu_vm INFO | [qemu output] qemu-kvm: Fail to invalidate device iotlb

2023-04-22 22:54:58: [   65.652081] Internal error: Oops: 0000000096000004 [#1] SMP
2023-04-22 22:54:58: [   65.652083] Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables nfnetlink qrtr vfat fat virtio_balloon fuse xfs libcrc32c virtio_gpu virtio_dma_buf drm_shmem_helper drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_ce drm ghash_ce virtio_net sha2_ce sha256_arm64 net_failover failover sha1_ce virtio_blk nvme_tcp nvme_fabrics nvme nvme_core nvme_common virtio_mmio dm_multipath dm_mirror dm_region_hash dm_log dm_mod be2iscsi cxgb4i cxgb4 tls libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
2023-04-22 22:54:58: [   65.652121] CPU: 2 PID: 1090 Comm: kdumpctl Not tainted 5.14.0-302.el9.aarch64 #1
2023-04-22 22:54:58: [   65.652123] Hardware name: QEMU KVM Virtual Machine, BIOS edk2-20230301gitf80f052277c8-2.el9.bz2186754.20230421.0950 03/01/2023
2023-04-22 22:54:58: [   65.652124] pstate: 20400005 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
2023-04-22 22:54:58: [   65.652126] pc : kmem_cache_alloc+0xcc/0x2cc
2023-04-22 22:54:58: [   65.652131] lr : kmem_cache_alloc+0x94/0x2cc
2023-04-22 22:54:58: [   65.652132] sp : ffff800008cd3a00
2023-04-22 22:54:58: [   65.652133] x29: ffff800008cd3a00 x28: 0000000001200000 x27: 00000000001c0002
2023-04-22 22:54:58: [   65.652135] x26: ffff0000c3adaa00 x25: ffffadb6e157d890 x24: ffffadb6e157d890
2023-04-22 22:54:58: [   65.652137] x23: 0000000000000000 x22: 0000000000000cc0 x21: ffff0000c01d7000
2023-04-22 22:54:58: [   65.652139] x20: ffffadb6e2d1c250 x19: ffff0000c01d7000 x18: 0000000000000000
2023-04-22 22:54:58: [   65.652141] x17: 0000000000000000 x16: ffffadb6e2129b14 x15: 00000000000c8000
2023-04-22 22:54:58: [   65.652143] x14: 00000000000c8000 x13: 00000000000c8000 x12: 00000000000c8000
2023-04-22 22:54:58: [   65.652145] x11: ffffffffffffffff x10: ffffffffffffffff x9 : ffffadb6e1868cd4
2023-04-22 22:54:58: [   65.652147] x8 : ffff0000c054f838 x7 : 0000000000000000 x6 : 0000000000000000
2023-04-22 22:54:58: [   65.652148] x5 : 0000000007ffffff x4 : 42f09308dd9dde37 x3 : 0000000000000000
2023-04-22 22:54:58: [   65.652150] x2 : a0249dddcb099a1a x1 : 00000000000002d8 x0 : 1a9a09cbdd9d21c8
2023-04-22 22:54:58: [   65.652152] Call trace:
2023-04-22 22:54:58: [   65.652153]  kmem_cache_alloc+0xcc/0x2cc
2023-04-22 22:54:58: [   65.652155]  dup_mm+0x30/0x110
2023-04-22 22:54:58: [   65.652159]  copy_process+0x8f0/0x1150
2023-04-22 22:54:58: [   65.652161]  kernel_clone+0x90/0x454
2023-04-22 22:54:58: [   65.652162]  __do_sys_clone+0x6c/0xa0
2023-04-22 22:54:58: [   65.652164]  __arm64_sys_clone+0x24/0x2c
2023-04-22 22:54:58: [   65.652166]  invoke_syscall.constprop.0+0x7c/0xd0
2023-04-22 22:54:58: [   65.652169]  el0_svc_common.constprop.0+0x15c/0x174
2023-04-22 22:54:58: [   65.652171]  do_el0_svc+0x34/0x5c
2023-04-22 22:54:58: [   65.652173]  el0_svc+0x38/0x18c
2023-04-22 22:54:58: [   65.652176]  el0t_64_sync_handler+0xb4/0x130
2023-04-22 22:54:58: [   65.652178]  el0t_64_sync+0x178/0x17c
2023-04-22 22:54:58: [   65.652180] Code: f9405e64 8b010002 b9401343 dac00c42 (f8616814)
2023-04-22 22:54:58: [   65.652181] ---[ end trace a197b8f174f8dfa6 ]---
2023-04-22 22:54:58: [   65.652183] Kernel panic - not syncing: Oops: Fatal exception
2023-04-22 22:54:58: [   65.701368] SMP: stopping secondary CPUs
2023-04-22 22:54:58: [   65.701381] Kernel Offset: 0x2db6d94f0000 from 0xffff800008000000
2023-04-22 22:54:58: [   65.701383] PHYS_OFFSET: 0x40000000
2023-04-22 22:54:58: [   65.701384] CPU features: 0x00000,002e0108,cc01720b
2023-04-22 22:54:58: [   65.701385] Memory Limit: none

I have never met this failure in raw format, am I missing some parameters for the new format? Or let's open a new bug to fix it, since it only failed on smmuv3 currently. I will paste the tier1 result later, to see the full result.

> When using qcow2 firmware images + smmuv3, the firmware is broken, qemu
> reports "Fail to update device iotlb" and the guest crashes.

Hmm, strange error pattern.

> I have never met this failure in raw format, am I missing some parameters
> for the new format? Or let's open a new bug to fix it, since it only failed
> on smmuv3 currently. I will paste the tier1 result later, to see the full
> result.

Looks sane to me.  The only thing looking suspicious is the vars file:
/root/avocado/data/avocado-vt/avocado-vt-vm1_rhel930-aarch64-virtio_qcow2_filesystem_VARS.fd
Is that actually in qcow2 format (i.e. a copy of the vars-template-pflash.qcow2 file)?

I'm just using libvirt:

<domain type='kvm'>
  <name>fedora-bz2186754-qcow2-smmuv3</name>
  [ ... ]
  <os firmware='efi'>
    <type arch='aarch64' machine='virt-7.2'>hvm</type>
    <firmware>
      <feature enabled='no' name='enrolled-keys'/>
      <feature enabled='no' name='secure-boot'/>
    </firmware>
    <loader readonly='yes' type='pflash' format='qcow2'>/usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.qcow2</loader>
    <nvram template='/usr/share/edk2/aarch64/vars-template-pflash.qcow2' format='qcow2'>/home/kraxel/.config/libvirt/qemu/nvram/fedora-bz2186754-qcow2-smmuv3_VARS.qcow2</nvram>
    <boot dev='hd'/>
  </os>
  [ ... ]
      <iommu model='smmuv3'/>
  [ ... ]

Works fine here (fedora edk2 builds though).

edk2-aarch64.rpm has both raw and qcow2 firmware images.
What happens if you use the raw images of the same edk2 build?

(In reply to Gerd Hoffmann from comment #6)
> > When using qcow2 firmware images + smmuv3, the firmware is broken, qemu
> > reports "Fail to update device iotlb" and the guest crashes.
> 
> Hmm, strange error pattern.
> 
> > I have never met this failure in raw format, am I missing some parameters
> > for the new format? Or let's open a new bug to fix it, since it only failed
> > on smmuv3 currently. I will paste the tier1 result later, to see the full
> > result.
> 
> Looks sane to me.  The only thing looking suspicious is the vars file:
> /root/avocado/data/avocado-vt/avocado-vt-vm1_rhel930-aarch64-
> virtio_qcow2_filesystem_VARS.fd
> Is that actually in qcow2 format (i.e. a copy of the
> vars-template-pflash.qcow2 file)?
Yes, it's a qcow2 file.
> 
> I'm just using libvirt:
> 
> <domain type='kvm'>
>   <name>fedora-bz2186754-qcow2-smmuv3</name>
>   [ ... ]
>   <os firmware='efi'>
>     <type arch='aarch64' machine='virt-7.2'>hvm</type>
>     <firmware>
>       <feature enabled='no' name='enrolled-keys'/>
>       <feature enabled='no' name='secure-boot'/>
>     </firmware>
>     <loader readonly='yes' type='pflash'
> format='qcow2'>/usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.qcow2</loader>
>     <nvram template='/usr/share/edk2/aarch64/vars-template-pflash.qcow2'
> format='qcow2'>/home/kraxel/.config/libvirt/qemu/nvram/fedora-bz2186754-
> qcow2-smmuv3_VARS.qcow2</nvram>
>     <boot dev='hd'/>
>   </os>
>   [ ... ]
>       <iommu model='smmuv3'/>
>   [ ... ]
> 
> Works fine here (fedora edk2 builds though).
> 
> edk2-aarch64.rpm has both raw and qcow2 firmware images.
> What happens if you use the raw images of the same edk2 build?

I think I probably know what's the problem, I used kernel-redhat(linux 6.3) for this edk2 build test. When I using 5.14.0-300.el9.aarch64, the issue is gone. So please ignore my above comment, I will re-trigger a new job to test the scratch build, thanks.

> From the scratch build of x86_64, I didn't find the qcow2 file:
> # rpm -qa | grep edk2

> So I'm just curious, is this bug only fixed for aarch64? Or do I have some
> installation errors somewhere that the file is not found? Because from bug
> 2161965 we can know that this feature is not only implemented on aarch.

On aarch64 we have actually benefits from doing so (reduce memory footprint).

On x86_64 using qcow2 images would work too, but I can't see a good reason
to actually do the switch.  We would have to keep the raw images for backward
compatibility reasons, so the package size would double and the QE effort
needed would increase too.  I'm reluctant to do the switch unless there are
also some actual benefits for x86_64.

Move to "VERIFIED" based on comment 12.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: edk2 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6330