RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2126095 - [rhel9.2][intel_iommu]Booting guest with "-device intel-iommu,intremap=on,device-iotlb=on,caching-mode=on" causes kernel call trace
Summary: [rhel9.2][intel_iommu]Booting guest with "-device intel-iommu,intremap=on,dev...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: qemu-kvm
Version: 9.2
Hardware: Unspecified
OS: All
high
high
Target Milestone: rc
: 9.2
Assignee: Peter Xu
QA Contact: jinl
URL:
Whiteboard:
: 2126623 2127410 2135692 (view as bug list)
Depends On: 2135806
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-09-12 12:05 UTC by Lei Yang
Modified: 2023-05-09 07:46 UTC (History)
16 users (show)

Fixed In Version: qemu-kvm-7.1.0-4.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-05-09 07:20:41 UTC
Type: ---
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Gitlab redhat/centos-stream/src qemu-kvm merge_requests 121 0 None opened Revert "intel_iommu: Fix irqchip / X2APIC configuration checks" 2022-10-17 19:04:47 UTC
Red Hat Issue Tracker RHELPLAN-133740 0 None None None 2022-09-12 12:23:39 UTC
Red Hat Product Errata RHSA-2023:2162 0 None None None 2023-05-09 07:22:20 UTC

Description Lei Yang 2022-09-12 12:05:21 UTC 
Description of problem:
Booting guest with "-device intel-iommu,intremap=on,device-iotlb=on,caching-mode=on" causes kernel call trace

Version-Release number of selected component (if applicable):
qemu-kvm-7.1.0-1.el9.x86_64
kernel-5.14.0-162.el9.x86_64
edk2-ovmf-20220526git16779ede2d36-3.el9.noarch

How reproducible:
3/3

Steps to Reproduce:
1.Boot guest with "-device intel-iommu,intremap=on,device-iotlb=on,caching-mode=on"
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1'  \
-sandbox on  \
-blockdev node-name=file_ovmf_code,driver=file,filename=/usr/share/OVMF/OVMF_CODE.secboot.fd,auto-read-only=on,discard=unmap \
-blockdev node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code \
-blockdev node-name=file_ovmf_vars,driver=file,filename=/root/avocado/data/avocado-vt/avocado-vt-vm1_rhel920-64-virtio-scsi_avocado-vt-vm1_qcow2_filesystem_VARS.fd,auto-read-only=on,discard=unmap \
-blockdev node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars \
-machine q35,kernel-irqchip=split,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
-device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
-device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
-nodefaults \
-device intel-iommu,intremap=on,device-iotlb=on,caching-mode=on \
-device VGA,bus=pcie.0,addr=0x2 \
-m 30720 \
-object memory-backend-ram,size=30720M,id=mem-machine_mem  \
-smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2  \
-cpu 'Cascadelake-Server',ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,umip=on,pku=on,md-clear=on,stibp=on,arch-capabilities=on,xsaves=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,tsx-ctrl=on,hle=off,rtm=off,kvm_pv_unhalt=on \
-device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
-device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
-blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/rhel920-64-virtio-scsi_avocado-vt-vm1.qcow2,cache.direct=on,cache.no-flush=off \
-blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
-device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
-device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
-device virtio-net-pci,mac=9a:e1:ae:80:9c:f1,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on,id=id52t5JN,netdev=idlya5z2,bus=pcie-root-port-3,addr=0x0  \
-netdev tap,id=idlya5z2,vhost=on,vhostforce=on \
-vnc :10  \
-rtc base=utc,clock=host,driftfix=slew  \
-boot menu=off,order=cdn,once=c,strict=off \
-enable-kvm \
-device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5 \
-monitor stdio \

2.Guest can not boot up and causes kernel call trace.


Actual results:
Guest kernel call trace

Expected results:
Guest can boot up succeed and no error messages

Additional info:
1. qemu-kvm-7.0.0-12.el9.x86_64 can boot up succeed, so add keywords "Regression".
2. Guest kernel call trace log is added as an attachment.
Comment 2 Laurent Vivier 2022-09-21 08:50:28 UTC 
It looks like more like a iommu bug or a PCI bug than a virtio-net bug.

Peter, could you have a look to confirm that?
Comment 3 Laurent Vivier 2022-09-21 08:59:06 UTC 
This bug is potentially a duplicate of BZ2127410 as it also uses intel-iommu.
Comment 4 Peter Xu 2022-09-21 16:18:14 UTC 
Possible dup of bz2126623 too.  Can verify using "eim=off" attached to intel-iommu.
Comment 5 Laurent Vivier 2022-09-22 07:39:06 UTC 
Moving to qemu-kvm/Devices as it doesn't seem specific to virtio-networking
Comment 6 Lei Yang 2022-09-22 08:27:51 UTC 
(In reply to Peter Xu from comment #4)
> Possible dup of bz2126623 too.  Can verify using "eim=off" attached to
> intel-iommu.

Yep, I tried to add this parameter to intel_iommeu cmd line. There is no issue any more.

-device intel-iommu,intremap=on,device-iotlb=on,caching-mode=on,eim=off \
Comment 7 CongLi 2022-09-22 08:54:30 UTC 
*** Bug 2126623 has been marked as a duplicate of this bug. ***
Comment 8 CongLi 2022-09-22 08:57:05 UTC 
Copied from Peter's comment in https://bugzilla.redhat.com/show_bug.cgi?id=2126623#c6:

https://lore.kernel.org/qemu-devel/20220921161227.57259-1-peterx@redhat.com/T/#u
Comment 9 Yanghang Liu 2022-09-22 09:59:05 UTC 
*** Bug 2127410 has been marked as a duplicate of this bug. ***
Comment 10 menli@redhat.com 2022-09-29 05:32:16 UTC 
hit a similar issue with win11（both 21h2 and 22h2）.

packages:
kernel-5.14.0-167.el9.x86_64
qemu-kvm-7.1.0-1.el9.x86_64
edk2-ovmf-20220526git16779ede2d36-4.el9.noarch
seabios-bin-1.16.0-4.el9.noarch


Steps：
1. boot a win11 guest with the following command.

MALLOC_PERTURB_=1  /usr/libexec/qemu-kvm \
    -name 'win11'  \
    -machine q35,kernel-irqchip=split  \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 14336 \
    -smp 20,maxcpus=20,cores=10,threads=1,dies=1,sockets=2  \
    -cpu 'IvyBridge-IBRS',ss=on,vmx=on,pdcm=on,pcid=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,kvm_pv_unhalt=on \
    -device pvpanic,ioport=0x505,id=idGusuYv \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-3,addr=0x0,disable-legacy=on,disable-modern=false,iommu_platform=on,ats=on \
    -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/win11-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x1.0x4,bus=pcie.0,chassis=5 \
    -device virtio-net-pci,mac=9a:f0:66:a0:03:50,id=idL1tYyA,netdev=ide1YiHr,bus=pcie-root-port-4,addr=0x0,disable-legacy=on,disable-modern=false,iommu_platform=on,ats=on  \
    -netdev tap,id=ide1YiHr,vhost=on \
    -blockdev node-name=file_cd1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/iso/windows/winutils.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \
    -device scsi-cd,id=cd1,drive=drive_cd1,write-cache=on  \
    -vnc :10  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=6 \
    -enable-kvm \
    -qmp tcp:0:1231,server,nowait \
    -monitor stdio \
    -tpmdev emulator,id=tpm-tpm0,chardev=chrtpm \
    -chardev socket,id=chrtpm,path=/tmp/guest-swtpm.sock \
    -device tpm-crb,tpmdev=tpm-tpm0,id=tpm0 \
    -drive file=/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd,if=pflash,format=raw,unit=0,readonly=on  \
    -drive file=/home/OVMF_VARS.fd,if=pflash,format=raw,unit=1  \
    -device intel-iommu,intremap=on,device-iotlb=on,eim=on \


Actual results:
guest hang on the boot page with iommu.

Expected results:
The guest can boot normally.

Additional info:
1. Not reproduce on qemu-kvm-7.0.0-13.el9.x86_64, so it should be a qemu regression issue.
2. remove '-device intel-iommu,intremap=on,device-iotlb=on,eim=on ', guest can start normally

host info:
model name	: Intel(R) Xeon(R) CPU E7-4830 v2 @ 2.20GHz


Thanks
Menghuan
Comment 12 qing.wang 2022-10-18 08:53:59 UTC 
*** Bug 2135692 has been marked as a duplicate of this bug. ***
Comment 13 qing.wang 2022-10-18 08:58:50 UTC 
Hit same issue on

Red Hat Enterprise Linux release 9.2 Beta (Plow)
5.14.0-162.el9.x86_64
qemu-kvm-7.1.0-1.el9.x86_64
seabios-bin-1.16.0-4.el9.noarch
edk2-ovmf-20220526git16779ede2d36-3.el9.noarch
libvirt-8.5.0-6.el9.x86_64
python3-libvirt-8.5.0-2.el9.x86_64
virtio-win-prewhql-0.1-227.iso


How reproducible:
100%

Steps to Reproduce:
1. Boot VM with iommu enabled

/usr/libexec/qemu-kvm \
     -name 'avocado-vt-vm1'  \
     -sandbox on  \
     -machine q35,kernel-irqchip=split,memory-backend=mem-machine_mem \
     -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
     -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
     -nodefaults \
     -device intel-iommu,intremap=on,device-iotlb=on \
     -device VGA,bus=pcie.0,addr=0x2 \
     -m 8G \
     -object memory-backend-ram,size=8G,id=mem-machine_mem  \
     -smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2  \
     -cpu 'Cascadelake-Server',ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,umip=on,pku=on,md-clear=on,stibp=on,arch-capabilities=on,xsaves=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,tsx-ctrl=on,hle=off,rtm=off,kvm_pv_unhalt=on \
     -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
     -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
     -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
     -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
     -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on \
     -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/rhel910-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
     -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
     -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
     \
     -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
     \
     -vnc :5 \
     -monitor stdio \
     -rtc base=utc,clock=host,driftfix=slew  \
     -boot menu=off,order=cdn,once=c,strict=off \
     -enable-kvm \
     -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5


It will boot succeed if add eim=off on device intel-iommu

-device intel-iommu,intremap=on,device-iotlb=on,eim=off \
Comment 14 John Ferlan 2022-10-18 16:04:19 UTC 
This bug will be resolved by the qemu 7.2 rebase bug 2135806, I've changed the DTM/ITM and added the bug dependency.

Peter (or Mirek) - unless there's a desire for testing to have the patch for the existing tests using the 7.1 rebase, then I think the 9.2 downstream MR can be closed.
Comment 15 Peter Xu 2022-10-18 16:13:14 UTC 
(In reply to John Ferlan from comment #14)
> This bug will be resolved by the qemu 7.2 rebase bug 2135806, I've changed
> the DTM/ITM and added the bug dependency.
> 
> Peter (or Mirek) - unless there's a desire for testing to have the patch for
> the existing tests using the 7.1 rebase, then I think the 9.2 downstream MR
> can be closed.

What is the target date for the rebase?  As long as the QE is good with the delayed fix it'll be perfectly fine to me to close the MR.  Thanks.
Comment 18 Yanan Fu 2022-11-03 01:15:39 UTC 
QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.
Comment 25 errata-xmlrpc 2023-05-09 07:20:41 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2162
Note You need to log in before you can comment on or make changes to this bug.