Bug 1597482

The issue also happens when enable iommu for virtio disk:
#virsh edit iommu1
   <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' iommu='on' ats='on'/>
      <source file='/home/layer2.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>

The backtrace of the qemu process is as follows:
(gdb) bt
#0  0x00007fd8021751b7 in raise () at /lib64/libc.so.6
#1  0x00007fd8021768a8 in abort () at /lib64/libc.so.6
#2  0x000056343de9ed9a in qemu_get_ram_block (addr=<optimized out>) at /usr/src/debug/qemu-2.12.0/exec.c:1115
#3  0x000056343dea2b45 in qemu_map_ram_ptr (ram_block=<optimized out>, addr=140565389668856) at /usr/src/debug/qemu-2.12.0/exec.c:2288
#4  0x000056343dea8ff5 in lduw_le_phys_cached (cache=<optimized out>, cache=<optimized out>, endian=DEVICE_LITTLE_ENDIAN, result=0x0, attrs=..., addr=2)
    at /usr/src/debug/qemu-2.12.0/memory_ldst.inc.c:281
#5  0x000056343dea8ff5 in lduw_le_phys_cached (result=0x0, attrs=..., addr=2, cache=<optimized out>) at /usr/src/debug/qemu-2.12.0/memory_ldst.inc.c:315
#6  0x000056343dea8ff5 in lduw_le_phys_cached (cache=<optimized out>, addr=addr@entry=2) at /usr/src/debug/qemu-2.12.0/memory_ldst.inc.c:334
#7  0x000056343df3c3fc in virtio_queue_empty (vdev=<optimized out>, pa=2, cache=<optimized out>) at /usr/src/debug/qemu-2.12.0/include/hw/virtio/virtio-access.h:166
#8  0x000056343df3c3fc in virtio_queue_empty (vq=0x563442f1a000) at /usr/src/debug/qemu-2.12.0/hw/virtio/virtio.c:228
#9  0x000056343df3c3fc in virtio_queue_empty (vq=0x563442f1a000) at /usr/src/debug/qemu-2.12.0/hw/virtio/virtio.c:372
#10 0x000056343df3c5ac in virtio_queue_host_notifier_aio_poll (opaque=0x563442f1a068) at /usr/src/debug/qemu-2.12.0/hw/virtio/virtio.c:2423
#11 0x000056343e1ef83e in run_poll_handlers_once (ctx=ctx@entry=0x563440a2bb80) at util/aio-posix.c:497
#12 0x000056343e1f0285 in aio_poll (blocking=true, ctx=0x563440a2bb80) at util/aio-posix.c:573
#13 0x000056343e1f0285 in aio_poll (ctx=ctx@entry=0x563440a2bb80, blocking=<optimized out>) at util/aio-posix.c:602
#14 0x000056343e1ed94f in aio_wait_bh_oneshot (ctx=0x563440a2bb80, cb=cb@entry=0x56343df110e0 <virtio_blk_data_plane_stop_bh>, opaque=opaque@entry=0x563442f14580) at util/aio-wait.c:70
#15 0x000056343df11864 in virtio_blk_data_plane_stop (vdev=<optimized out>) at /usr/src/debug/qemu-2.12.0/hw/block/dataplane/virtio-blk.c:283
#16 0x000056343e0b6a15 in virtio_bus_stop_ioeventfd (bus=bus@entry=0x563442f0a0f8) at hw/virtio/virtio-bus.c:246
#17 0x000056343e0b3e51 in virtio_pci_common_write (proxy=0x563442f02000) at hw/virtio/virtio-pci.c:294
#18 0x000056343e0b3e51 in virtio_pci_common_write (opaque=0x563442f02000, addr=<optimized out>, val=0, size=<optimized out>) at hw/virtio/virtio-pci.c:1262
#19 0x000056343deee483 in memory_region_write_accessor (mr=<optimized out>, addr=<optimized out>, value=<optimized out>, size=<optimized out>, shift=<optimized out>, mask=<optimized out>, attrs=...) at /usr/src/debug/qemu-2.12.0/memory.c:530
#20 0x000056343deec199 in access_with_adjusted_size (addr=addr@entry=20, value=value@entry=0x7fd7ee1e7488, size=size@entry=1, access_size_min=<optimized out>, access_size_max=<optimized out>, access_fn=access_fn@entry=0x56343deee440 <memory_region_write_accessor>, mr=mr@entry=0x563442f029d0, attrs=attrs@entry=...) at /usr/src/debug/qemu-2.12.0/memory.c:597
#21 0x000056343def0235 in memory_region_dispatch_write (mr=mr@entry=0x563442f029d0, addr=addr@entry=20, data=0, size=size@entry=1, attrs=attrs@entry=...)
    at /usr/src/debug/qemu-2.12.0/memory.c:1474
#22 0x000056343dea018b in flatview_write (mr=0x563442f029d0, l=1, addr1=20, len=1, buf=0x7fd8089e4028 <Address 0x7fd8089e4028 out of bounds>, attrs=..., addr=4223664148, fv=0x563449c9d6c0) at /usr/src/debug/qemu-2.12.0/exec.c:3100
#23 0x000056343dea018b in flatview_write (fv=0x563449c9d6c0, addr=<optimized out>, attrs=..., buf=<optimized out>, len=<optimized out>) at /usr/src/debug/qemu-2.12.0/exec.c:3144
#24 0x000056343dea3bcf in address_space_write (as=<optimized out>, addr=<optimized out>, attrs=..., buf=<optimized out>, len=<optimized out>) at /usr/src/debug/qemu-2.12.0/exec.c:3260
#25 0x000056343dea3c75 in address_space_rw (as=<optimized out>, addr=<optimized out>, attrs=..., 
    attrs@entry=..., buf=buf@entry=0x7fd8089e4028 <Address 0x7fd8089e4028 out of bounds>, len=<optimized out>, is_write=<optimized out>) at /usr/src/debug/qemu-2.12.0/exec.c:3271
#26 0x000056343defe758 in kvm_cpu_exec (cpu=cpu@entry=0x563440de2000) at /usr/src/debug/qemu-2.12.0/accel/kvm/kvm-all.c:1992
#27 0x000056343dedc356 in qemu_kvm_cpu_thread_fn (arg=0x563440de2000) at /usr/src/debug/qemu-2.12.0/cpus.c:1215
#28 0x00007fd802513dd5 in start_thread () at /lib64/libpthread.so.0
#29 0x00007fd80223daed in clone () at /lib64/libc.so.6

The backtrace looks like the one in Bug 1547940, not sure whether it's a dup.

And I just remembered that Eric Anger just fixed a similar problem too:

commit a99761d3c85679da380c0f597468acd3dc1b53b3
Author: Eric Auger <eric.auger>
Date:   Wed Jun 13 15:19:06 2018 +0200

    exec: Fix MAP_RAM for cached access

Not sure it could fix either (or both) of the problem.

Peter

The bug can not be reproduced with qemu-kvm-rhev-2.10.0-21.el7_5.4.x86_64.

*** Bug 1599237 has been marked as a duplicate of this bug. ***

the issue occur also when start vm with iommu enabled for interface, both for vm using OVMF and seabios

or when attach interface with iommu enabled

libvirt-4.5.0-2.el7.x86_64
qemu-kvm-rhev-2.12.0-7.el7.x86_64

# virsh dumpxml rh | grep /interface  -B6
    <interface type='network'>
      <mac address='52:54:00:b9:54:d4'/>
      <source network='default'/>
      <model type='virtio'/>
      <driver iommu='on' ats='on'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>
...
 <iommu model='intel'>
      <driver intremap='on' caching_mode='on' eim='on' iotlb='on'/>
    </iommu>

(In reply to yalzhang from comment #8)
> the issue occur also when start vm with iommu enabled for interface, both
> for vm using OVMF and seabios
> 
> or when attach interface with iommu enabled
> 
> libvirt-4.5.0-2.el7.x86_64
> qemu-kvm-rhev-2.12.0-7.el7.x86_64
> 
> # virsh dumpxml rh | grep /interface  -B6
>     <interface type='network'>
>       <mac address='52:54:00:b9:54:d4'/>
>       <source network='default'/>
>       <model type='virtio'/>
>       <driver iommu='on' ats='on'/>
>       <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
> function='0x0'/>
>     </interface>
> ...
>  <iommu model='intel'>
>       <driver intremap='on' caching_mode='on' eim='on' iotlb='on'/>
>     </iommu>

sorry, this issue for interface is already covered in Bug 1520806

Fam,

This is a regression bug, I want to confirm with you if it will be fixed on rhel-7.6 or deferred to later version.


Many thanks

I agree this is basically bug 1547940, so I'll follow suit and defer to 7.6. But there is some good news: I reproduced the problem with both qemu-kvm-rhev 2.10 and 2.12, but not with upstream. So we can probably identify and backport from upstream to fix the core dump.

However, upstream has a different bug that QEMU hangs when guest kernel is initializing the vIOMMU enabled device. I've sent a series to fix that:

[PATCH 0/2] virtio-scsi: Fix QEMU hang with vIOMMU and ATS
[PATCH 1/2] virtio: Return true from virtio_queue_empty if broken
[PATCH 2/2] virtio-scsi/virtio-blk: Disable poll handlers when stopping vq handler

So maybe this BZ can be used to track those fixes as well.

I meant "defer to 7.7".

(In reply to Fam Zheng from comment #11)
> I agree this is basically bug 1547940, so I'll follow suit and defer to 7.6.
> But there is some good news: I reproduced the problem with both
> qemu-kvm-rhev 2.10 and 2.12, but not with upstream. So we can probably
> identify and backport from upstream to fix the core dump.
> 
> However, upstream has a different bug that QEMU hangs when guest kernel is
> initializing the vIOMMU enabled device. I've sent a series to fix that:
> 
> [PATCH 0/2] virtio-scsi: Fix QEMU hang with vIOMMU and ATS
> [PATCH 1/2] virtio: Return true from virtio_queue_empty if broken
> [PATCH 2/2] virtio-scsi/virtio-blk: Disable poll handlers when stopping vq
> handler
> 
> So maybe this BZ can be used to track those fixes as well.

Looks like these patches have not been merged, so we need to follow-up.

In speaking with Paolo, it seems that we want to commit additional fixes to patch #1 there, but that it can wait for the time being. Patch #2 was ruled incorrect upstream.

Until then, could you please do me a favor and attach the full libvirt XML (or even better, just show me the QEMU CLI?) That will make it easier for me to bisect upstream as I'm a bit new to configuring these devices and I'm running into a spot of trouble doing so.

Thank you,
--js

(In reply to John Snow from comment #16)
> In speaking with Paolo, it seems that we want to commit additional fixes to
> patch #1 there, but that it can wait for the time being. Patch #2 was ruled
> incorrect upstream.
> 
> Until then, could you please do me a favor and attach the full libvirt XML
> (or even better, just show me the QEMU CLI?) That will make it easier for me
> to bisect upstream as I'm a bit new to configuring these devices and I'm
> running into a spot of trouble doing so.
> 
> Thank you,
> --js


Please refer to "External Trackers" above. Polarion test case "RHEL7-11846" added before.

Apologies for missing it, thank you!

I found a series of backports from upstream that appear to rectify the issue, but further work appears to be required upstream, which I am not presently prepared to do for the 7.7 timeframe.

That said, If we backport:

1-4: https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg02443.html
5: https://lists.gnu.org/archive/html/qemu-devel/2018-06/msg03567.html
6: https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg03292.html

then the crash and the hang goes away, however, there are some lingering warnings:

qemu-system-x86_64: vtd_iommu_translate: detected translation failure (dev=00:02:00, iova=0x0)
qemu-system-x86_64: New fault is not recorded due to compression of faults
qemu-system-x86_64: virtio: zero sized buffers are not allowed

Since this has been tagged a regression... I am going to lean on the idea that some warnings are better than having it segfault, so I'll prepare a backport and see where the winds blow.

Fix included in qemu-kvm-rhev-2.12.0-24.el7

Reproduced it on qemu-kvm-rhev-2.12.0-23.el7.  Tested on qemu-kvm-rhev-2.12.0-24.el7, guest boot up successfully and qemu not crashed, but hit other issue in qemu monitor. 


Details as below:

Host:
kernel-3.10.0-1010.el7.x86_64
qemu-kvm-rhev-2.12.0-24.el7

Guest:
kernel-3.10.0-1010.el7.x86_64

1. In host, add "iommu=pt intel_iommu=on" to kernel line, then reboot host

2. Add "intel_iommu=on" to kernel line of q35 guest

3. boot virtio-scsi with iommu_platform=on 
/usr/libexec/qemu-kvm \ 
-M q35,kernel-irqchip=split \
-cpu host \ 
-enable-kvm \ 
-m 2G \ 
-smp 4 \ 
-nodefconfig \ 
-rtc base=localtime,driftfix=slew \ 
-device intel-iommu,device-iotlb=on,intremap \ 
-device virtio-scsi-pci,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on,id=scsi0,bus=pcie.0 \ 
-drive file=rhel76-64-virtio-scsi.qcow2,if=none,format=qcow2,cache=none,werror=stop,rerror=stop,id=drive-virtio-disk0,aio=native \ 
-device scsi-hd,bus=scsi0.0,drive=drive-virtio-disk0,id=virtio-disk0 \ 
-device piix3-usb-uhci,id=usb \ 
-device usb-tablet,id=tablet0 \ 
-vnc :0 \ 
-k en-us \ 
-vga std \ 
-qmp tcp:0:4444,server,nowait \ 
-boot menu=on \ 
-monitor stdio \ 
-device virtio-net-pci,mac=fa:f7:f8:5f:fa:5b,id=idn0VnaA,vectors=4,netdev=id8xJhp7,bus=pcie.0,addr=06 \ 
-netdev tap,id=id8xJhp7,vhost=on \

4. verify IOMMU enabled, in the guest 
# journalctl -k | grep -i "IOMMU enabled" 
Feb 28 11:13:59 localhost kernel: DMAR: IOMMU enabled
# journalctl -k | grep -i "DMAR: Intel(R) Virtualization Technology for Directed I/O"
Feb 28 11:13:59 localhost kernel: DMAR: Intel(R) Virtualization Technology for Directed I/O


Hit below issue during booting guest.

# sh bug_iommu.sh 
QEMU 2.12.0 monitor - type 'help' for more information
(qemu) qemu-kvm: vtd_iommu_translate: detected translation failure (dev=00:02:00, iova=0x0)
qemu-kvm: New fault is not recorded due to compression of faults
qemu-kvm: virtio: zero sized buffers are not allowed

(qemu) system_reset 
(qemu) qemu-kvm: Guest says index 29281 is available
qemu-kvm: virtio: zero sized buffers are not allowed

(qemu) system_powerdown


If track it by this bug or report a new bug?

A new bug. It's definitely related, but I believe this requires further work upstream as well and I need to coordinate with Paolo about it, so the timeframe here is different. Thank you!

(In reply to John Snow from comment #25)
> A new bug. It's definitely related, but I believe this requires further work
> upstream as well and I need to coordinate with Paolo about it, so the
> timeframe here is different. Thank you!


Track it by Bug 1686278 - qemu-kvm: vtd_iommu_translate: detected translation failure.

And verify this bug, do you agree with me? Thank you.

Yes; as long as the VM isn't crashing and works after the warning message, I would say this can be closed for now and we will revisit the spurious warnings in the near future.

According to Comment 24, Comment27,  set status to VERIFIED.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2553