Bug 1843852

Summary: qemu core dumped: qemu-kvm: /builddir/build/BUILD/qemu-4.2.0/memory.c:1928: memory_region_notify_one: Assertion `entry->iova >= notifier->start && entry_end <= notifier->end' failed.
Product: Red Hat Enterprise Linux 8 Reporter: yduan
Component: qemu-kvmAssignee: Eugenio Pérez Martín <eperezma>
qemu-kvm sub component: Networking QA Contact: Lei Yang <leiyang>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: aadam, coli, knoel, leiyang, virt-maint
Version: 8.3Keywords: Regression, Triaged
Target Milestone: rc   
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-4.2.0-41.module+el8.4.0+9504+ab2393e6 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1845758 (view as bug list) Environment:
Last Closed: 2021-05-18 15:21:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1845758    

Description yduan 2020-06-04 10:08:31 UTC
Description of problem:
(qemu) qemu-kvm: /builddir/build/BUILD/qemu-4.2.0/memory.c:1928: memory_region_notify_one: Assertion `entry->iova >= notifier->start && entry_end <= notifier->end' failed.
tmp.sh: line 28: 55965 Aborted                 (core dumped) /usr/libexec/qemu-kvm -S -name 'avocado-vt-vm1' -sandbox on -machine q35,kernel-irqchip=split -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 -nodefaults -device VGA,bus=pcie.0,addr=0x2 -m 30720 -smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2 -cpu 'Cascadelake-Server',+kvm_pv_unhalt -device pvpanic,ioport=0x505,id=idkY3DZB -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-3,addr=0x0 -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/rhel79-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 -device scsi-hd,id=image1,drive=drive_image1,write-cache=on -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x1.0x4,bus=pcie.0,chassis=5 -device virtio-net-pci,mac=9a:8c:11:ff:cc:c8,id=idrtC35V,netdev=idRY3gze,bus=pcie-root-port-4,addr=0x0,iommu_platform=on,ats=on -netdev tap,id=idRY3gze,vhost=on -vnc :0 -monitor stdio -rtc base=utc,clock=host,driftfix=slew -boot menu=off,order=cdn,once=c,strict=off -device intel-iommu,intremap=on,device-iotlb=on -enable-kvm

Version-Release number of selected component (if applicable):
Host:
4.18.0-203.el8.x86_64
qemu-kvm-core-4.2.0-19.module+el8.3.0+6478+69f490bb.x86_64
Guest:
3.10.0-1136.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Boot RHEL 7.9 guest with iommu-enabled virtio-net-pci device:
/usr/libexec/qemu-kvm \
 -machine q35,kernel-irqchip=split \
 -device intel-iommu,intremap=on,device-iotlb=on \
 -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x1.0x4,bus=pcie.0,chassis=5 \
 -device virtio-net-pci,mac=9a:8c:11:ff:cc:c8,id=idrtC35V,netdev=idRY3gze,bus=pcie-root-port-4,addr=0x0,iommu_platform=on,ats=on \
 -netdev tap,id=idRY3gze,vhost=on \
......

2.Update guest kernel option with "intel_iommu=on" and reboot:
[GUEST] # grubby --update-kernel=`grubby --default-kernel` --args="intel_iommu=on"
[GUEST] # reboot

Actual results:
qemu core dumped as in Description.

Expected results:
Guest can reboot successfully.

Additional info:
This can be reproduced with qemu-kvm-core-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64.
This cannot be reproduced with qemu-kvm-rhev-2.12.0-48.el7.x86_64.

Comment 2 Lei Yang 2020-06-05 04:07:50 UTC
I tried test on qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420.x86_64 and kernel-4.18.0-211.el8.x86_64.

Reproduce step
1. start a rhel7.9 guest.
qemu cli:
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1'  \
-sandbox on  \
-machine q35,kernel-irqchip=split \
-device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
-device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
-nodefaults \
-device VGA,bus=pcie.0,addr=0x2 \
-m 7168  \
-smp 6,maxcpus=6,cores=3,threads=1,dies=1,sockets=2  \
-cpu 'Haswell-noTSX',+kvm_pv_unhalt \
-device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
-device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on \
-blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/rhel79-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
-blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \
-device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
-device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
-device virtio-net-pci,mac=9a:26:bc:c0:16:57,id=idB6yM8C,netdev=idKJlj8B,bus=pcie-root-port-3,addr=0x0,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on  \
-netdev tap,id=idKJlj8B,vhost=on,vhostforce=on \
-vnc :0  \
-rtc base=utc,clock=host,driftfix=slew  \
-boot menu=off,order=cdn,once=c,strict=off \
-device intel-iommu,intremap=on,device-iotlb=on \
-enable-kvm \
-device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5 \

2.A core dump will occur after the guest starts,no action is required.The core dump info is the same as comment 1.

Comment 3 John Ferlan 2020-06-09 14:39:46 UTC
Assigned to Amnon for initial triage per bz process and age of bug created or assigned to virt-maint without triage

Looks to be network/iommu related

Comment 17 Lei Yang 2021-01-20 05:57:01 UTC
==steps
Test Version:
qemu-kvm-4.2.0-19.module+el8.3.0+6478+69f490bb.x86_64
4.18.0-203.el8.x86_64
Guest:
kernel-3.10.0-1160.el7.x86_64

1.Boot guest with iommu-enabled virtio-net-pci device
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1'  \
-sandbox on  \
-machine q35,kernel-irqchip=split \
-device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
-device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
-nodefaults \
-device intel-iommu,intremap=on,device-iotlb=on \
-device VGA,bus=pcie.0,addr=0x2 \
-m 26624  \
-smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
-cpu 'Skylake-Server',+kvm_pv_unhalt \
-device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
-device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on \
-blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/rhel79-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
-blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
-device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
-device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
-device virtio-net-pci,mac=9a:15:17:44:9a:ed,id=idHGyh18,netdev=idC7CARG,bus=pcie-root-port-3,addr=0x0,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on  \
-netdev tap,id=idC7CARG,vhost=on,vhostforce=on \
-vnc :0  \
-rtc base=utc,clock=host,driftfix=slew  \
-boot menu=off,order=cdn,once=c,strict=off \
-enable-kvm \
-device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5 \
-monitor stdio \

2.Update guest kernel option with "intel_iommu=on" and reboot:
# grubby --update-kernel=`grubby --default-kernel` --args="intel_iommu=on"
# reboot

3.qemu core dumped
qemu-kvm: /builddir/build/BUILD/qemu-4.2.0/memory.c:1928: memory_region_notify_one: Assertion `entry->iova >= notifier->start && entry_end <= notifier->end' failed.

==Reproduced with qemu-kvm-4.2.0-19.module+el8.3.0+6478+69f490bb.x86_64

==Verified with qemu-kvm-4.2.0-41.module+el8.4.0+9504+ab2393e6.x86_64
1.Boot guest with iommu-enabled virtio-net-pci device
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1'  \
-sandbox on  \
-machine q35,kernel-irqchip=split \
-device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
-device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
-nodefaults \
-device intel-iommu,intremap=on,device-iotlb=on \
-device VGA,bus=pcie.0,addr=0x2 \
-m 26624  \
-smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
-cpu 'Skylake-Server',+kvm_pv_unhalt \
-device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
-device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on \
-blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/rhel79-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
-blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
-device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
-device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
-device virtio-net-pci,mac=9a:43:da:db:d3:ec,id=idSUR4qx,netdev=idyfQh7e,bus=pcie-root-port-3,addr=0x0,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on  \
-netdev tap,id=idyfQh7e,vhost=on,vhostforce=on \
-vnc :0  \
-rtc base=utc,clock=host,driftfix=slew  \
-boot menu=off,order=cdn,once=c,strict=off \
-enable-kvm \
-device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5 \
-monitor stdio \

2.Update guest kernel option with "intel_iommu=on" and reboot:
# grubby --update-kernel=`grubby --default-kernel` --args="intel_iommu=on"
# reboot

3.ping out from guest
# ping 10.73.75.151  -c 10
PING 10.73.75.151 (10.73.75.151) 56(84) bytes of data.
64 bytes from 10.73.75.151: icmp_seq=1 ttl=64 time=0.449 ms
64 bytes from 10.73.75.151: icmp_seq=2 ttl=64 time=0.470 ms
64 bytes from 10.73.75.151: icmp_seq=3 ttl=64 time=0.439 ms
64 bytes from 10.73.75.151: icmp_seq=4 ttl=64 time=0.440 ms
64 bytes from 10.73.75.151: icmp_seq=5 ttl=64 time=0.543 ms
64 bytes from 10.73.75.151: icmp_seq=6 ttl=64 time=0.431 ms
64 bytes from 10.73.75.151: icmp_seq=7 ttl=64 time=0.497 ms
64 bytes from 10.73.75.151: icmp_seq=8 ttl=64 time=0.533 ms
64 bytes from 10.73.75.151: icmp_seq=9 ttl=64 time=0.484 ms
Updated HWADDR (08:9e:01:63:2c:6e)<->(10.73.73.251) IP pair into address cache
64 bytes from 10.73.75.151: icmp_seq=10 ttl=64 time=0.506 ms

--- 10.73.75.151 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9243ms
rtt min/avg/max/mdev = 0.431/0.479/0.543/0.040 ms

So this bug has been fixed very well. Move to 'VERIFIED'.

Comment 19 errata-xmlrpc 2021-05-18 15:21:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: virt:rhel and virt-devel:rhel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1762