Bug 1547940

Here is the abrt core dump file link:

http://fileshare.englab.nay.redhat.com/pub/section2/coredump/var/crash/pezhang/bug1547940/abrt/

Hi Pei,

(In reply to Pei Zhang from comment #2)
> Here is the abrt core dump file link:
> 
> http://fileshare.englab.nay.redhat.com/pub/section2/coredump/var/crash/
> pezhang/bug1547940/abrt/

It seems the link is broken, I get error 404.

The backtrace confirms the used ring index address isn't present in any of the RAM blocks (0x23ffb3002):

#0  0x00007f05b1314207 in ?? () from /lib64/libc.so.6
#1  0x00007f05b13158f8 in __GI_abort () at abort.c:99
#2  0x00005631234b1eba in qemu_get_ram_block (addr=<optimized out>) at /usr/src/debug/qemu-2.10.0/exec.c:1061
#3  0x00005631234b5aa5 in qemu_map_ram_ptr (ram_block=<optimized out>, addr=9663361026) at /usr/src/debug/qemu-2.10.0/exec.c:2193
#4  0x00005631234bc39b in address_space_lduw_internal_cached (cache=<optimized out>, cache=<optimized out>, endian=DEVICE_LITTLE_ENDIAN, result=0x0, attrs=..., addr=2) at /usr/src/debug/qemu-2.10.0/memory_ldst.inc.c:281
#5  address_space_lduw_le_cached (result=0x0, attrs=..., addr=2, cache=<optimized out>) at /usr/src/debug/qemu-2.10.0/memory_ldst.inc.c:315
#6  lduw_le_phys_cached (cache=<optimized out>, addr=addr@entry=2) at /usr/src/debug/qemu-2.10.0/memory_ldst.inc.c:334
#7  0x000056312353a707 in virtio_lduw_phys_cached (vdev=<optimized out>, pa=2, cache=<optimized out>) at /usr/src/debug/qemu-2.10.0/include/hw/virtio/virtio-access.h:166
#8  vring_used_idx (vq=vq@entry=0x563127de8000) at /usr/src/debug/qemu-2.10.0/hw/virtio/virtio.c:263
#9  0x000056312353cb59 in virtio_queue_update_used_idx (vdev=vdev@entry=0x563127ddc170, n=n@entry=0) at /usr/src/debug/qemu-2.10.0/hw/virtio/virtio.c:2346
#10 0x00005631235418d9 in vhost_virtqueue_stop (dev=dev@entry=0x563125d1ea80, vdev=vdev@entry=0x563127ddc170, vq=0x563125d1eca8, idx=0) at /usr/src/debug/qemu-2.10.0/hw/virtio/vhost.c:1148
#11 0x0000563123543574 in vhost_dev_stop (hdev=hdev@entry=0x563125d1ea80, vdev=vdev@entry=0x563127ddc170) at /usr/src/debug/qemu-2.10.0/hw/virtio/vhost.c:1598
#12 0x000056312352bb78 in vhost_net_stop_one (net=0x563125d1ea80, dev=0x563127ddc170) at /usr/src/debug/qemu-2.10.0/hw/net/vhost_net.c:289
#13 0x000056312352c06b in vhost_net_stop (dev=dev@entry=0x563127ddc170, ncs=<optimized out>, total_queues=total_queues@entry=1) at /usr/src/debug/qemu-2.10.0/hw/net/vhost_net.c:368
#14 0x0000563123529306 in virtio_net_vhost_status (status=<optimized out>, n=0x563127ddc170) at /usr/src/debug/qemu-2.10.0/hw/net/virtio-net.c:180
#15 virtio_net_set_status (vdev=0x563127ddc170, status=11 '\v') at /usr/src/debug/qemu-2.10.0/hw/net/virtio-net.c:254
#16 0x000056312353b65c in virtio_set_status (vdev=vdev@entry=0x563127ddc170, val=<optimized out>) at /usr/src/debug/qemu-2.10.0/hw/virtio/virtio.c:1147
#17 0x00005631236a82a0 in virtio_write_config (pci_dev=0x563127dd4000, address=<optimized out>, val=<optimized out>, len=<optimized out>) at hw/virtio/virtio-pci.c:664
#18 0x00005631234fa073 in memory_region_write_accessor (mr=<optimized out>, addr=<optimized out>, value=<optimized out>, size=<optimized out>, shift=<optimized out>, mask=<optimized out>, attrs=...)
    at /usr/src/debug/qemu-2.10.0/memory.c:560
#19 0x00005631234f7d89 in access_with_adjusted_size (addr=addr@entry=0, value=value@entry=0x7f05a84907b8, size=size@entry=2, access_size_min=<optimized out>, access_size_max=<optimized out>, 
    access=access@entry=0x5631234fa030 <memory_region_write_accessor>, mr=mr@entry=0x563126524400, attrs=attrs@entry=...) at /usr/src/debug/qemu-2.10.0/memory.c:626
#20 0x00005631234fbd05 in memory_region_dispatch_write (mr=mr@entry=0x563126524400, addr=addr@entry=0, data=1283, size=size@entry=2, attrs=attrs@entry=...) at /usr/src/debug/qemu-2.10.0/memory.c:1502
#21 0x00005631234b33f2 in flatview_write_continue (mr=0x563126524400, l=2, addr1=0, len=2, buf=0x7f05bba7b000 <Address 0x7f05bba7b000 out of bounds>, attrs=..., addr=3324, fv=0x56312cbec640) at /usr/src/debug/qemu-2.10.0/exec.c:2906
#22 flatview_write (fv=0x56312cbec640, addr=<optimized out>, attrs=..., buf=<optimized out>, len=<optimized out>) at /usr/src/debug/qemu-2.10.0/exec.c:2957
#23 0x00005631234b6a65 in flatview_rw (is_write=true, len=2, buf=0x7f05bba7b000 <Address 0x7f05bba7b000 out of bounds>, attrs=..., addr=3324, fv=<optimized out>) at /usr/src/debug/qemu-2.10.0/exec.c:3066
#24 address_space_rw (as=<optimized out>, addr=addr@entry=3324, attrs=attrs@entry=..., buf=buf@entry=0x7f05bba7b000 <Address 0x7f05bba7b000 out of bounds>, len=len@entry=2, is_write=is_write@entry=true)
    at /usr/src/debug/qemu-2.10.0/exec.c:3076
#25 0x0000563123509b4a in kvm_handle_io (count=1, size=2, direction=<optimized out>, data=<optimized out>, attrs=..., port=3324) at /usr/src/debug/qemu-2.10.0/accel/kvm/kvm-all.c:1807
#26 kvm_cpu_exec (cpu=cpu@entry=0x56312628e000) at /usr/src/debug/qemu-2.10.0/accel/kvm/kvm-all.c:2047
#27 0x00005631234e8cf2 in qemu_kvm_cpu_thread_fn (arg=0x56312628e000) at /usr/src/debug/qemu-2.10.0/cpus.c:1128
#28 0x00007f05b16b2dd5 in start_thread (arg=0x7f05a8491700) at pthread_create.c:308
#29 0x00007f05b13dcb3d in __GI_epoll_pwait (epfd=31206, events=0x7a19, maxevents=6, timeout=-1, set=0x7f05a8491700) at ../sysdeps/unix/sysv/linux/epoll_pwait.c:41
#30 0x0000000000000000 in ?? ()


I went manually through the list of RAM blocks, and I confirm there is no RAM block for this address.
Yet, we have to understand how that can happen.

Looking at the backtrace, something unexpected is happening on guest side.
The device gets stopped because the guest clear "bus master" bit in the PCI
command register of the Virtio device.
It can happen for example when guest is shutdown, or, in case of DPDK, when
then Virtio PMD is removed.
This is not expected in this test scenario at that point, as the guest should
be doing packets forwarding.

The coredump doesn't contain the guest memory, it could be good to have it to
try to understand what the guest is doing. The other alternative would be to
reproduce the problem manually.

Update:

1. Live migration over ovs+dpdk+vhost-user also hit this same issue. Bug title will be changed later.

2. This crash issue only happens when testing with vIOMMU support, no matter in pvp live migration or migration over ovs.

(In reply to Maxime Coquelin from comment #5)
> The backtrace confirms the used ring index address isn't present in any of
> the RAM blocks (0x23ffb3002):
> 
> #0  0x00007f05b1314207 in ?? () from /lib64/libc.so.6
> #1  0x00007f05b13158f8 in __GI_abort () at abort.c:99
> #2  0x00005631234b1eba in qemu_get_ram_block (addr=<optimized out>) at
> /usr/src/debug/qemu-2.10.0/exec.c:1061
> #3  0x00005631234b5aa5 in qemu_map_ram_ptr (ram_block=<optimized out>,
> addr=9663361026) at /usr/src/debug/qemu-2.10.0/exec.c:2193
> #4  0x00005631234bc39b in address_space_lduw_internal_cached
> (cache=<optimized out>, cache=<optimized out>, endian=DEVICE_LITTLE_ENDIAN,
> result=0x0, attrs=..., addr=2) at
> /usr/src/debug/qemu-2.10.0/memory_ldst.inc.c:281

Besides on knowing why a reset has happened inside guest, here I'm also curious whether the address space translation could have failed somehow.  The translation happened on memory_ldst.inc.c:264:

    mr = TRANSLATE(addr, &addr1, &l, false);

Since above stack shows that mr->ramblock must be NULL and addr1 is very possible a correct GPA (9663361026=0x23ffb3002, which falls into a valid mem range of a 8G guest).  To know that, maybe we can assert after above line with:

    assert(mr != &io_mem_unassigned);

If the assertion can be triggered, then it proves.

I'm still not sure whether this even worths a try, I'm just throwing this question out in case it would help.  I think if the translation failed, it means it's possible the page permission bits might be wrong somewhere (see the single failure tag translate_fail in flatview_do_translate()).

(Add Paolo in the loop too)

Peter

(In reply to Peter Xu from comment #12)
> (In reply to Maxime Coquelin from comment #5)
> > The backtrace confirms the used ring index address isn't present in any of
> > the RAM blocks (0x23ffb3002):
> > 
> > #0  0x00007f05b1314207 in ?? () from /lib64/libc.so.6
> > #1  0x00007f05b13158f8 in __GI_abort () at abort.c:99
> > #2  0x00005631234b1eba in qemu_get_ram_block (addr=<optimized out>) at
> > /usr/src/debug/qemu-2.10.0/exec.c:1061
> > #3  0x00005631234b5aa5 in qemu_map_ram_ptr (ram_block=<optimized out>,
> > addr=9663361026) at /usr/src/debug/qemu-2.10.0/exec.c:2193
> > #4  0x00005631234bc39b in address_space_lduw_internal_cached
> > (cache=<optimized out>, cache=<optimized out>, endian=DEVICE_LITTLE_ENDIAN,
> > result=0x0, attrs=..., addr=2) at
> > /usr/src/debug/qemu-2.10.0/memory_ldst.inc.c:281
> 
> Besides on knowing why a reset has happened inside guest, here I'm also
> curious whether the address space translation could have failed somehow. 
> The translation happened on memory_ldst.inc.c:264:
> 
>     mr = TRANSLATE(addr, &addr1, &l, false);
> 
> Since above stack shows that mr->ramblock must be NULL and addr1 is very
> possible a correct GPA (9663361026=0x23ffb3002, which falls into a valid mem
> range of a 8G guest).  To know that, maybe we can assert after above line
> with:
> 
>     assert(mr != &io_mem_unassigned);
> 
> If the assertion can be triggered, then it proves.

Yes, that's a good idea.
I can provide a brew build adding this assertion.

Thanks,
Maxime

Hi,

During debugging the nested device assignment breakage recently I found that the breakage might be closely related to (or even duplicate of) this bug too.  Currently I have posted two fixes already for either QEMU and kernel:

http://lists.gnu.org/archive/html/qemu-devel/2018-04/msg02609.html
https://lkml.org/lkml/2018/4/18/5

Basically they are all about incorrect mapped pages, or missing PSIs.  For this bug, if our above assumption stands, DPDK requested a page translation but it was missing (without correct permission flags set in PTEs).  That sounds quite like the nested problem.

I'll update in this bug again after everything is clear and settled upstream, then we can see whether those fixes can dissolve this bug too.

Peter

Update:

Tested with kernel[1] applying above patches Peter mentioned and upstream qemu-kvm[2], still hit crash issue. 


[1] git clone  git://git.app.eng.bos.redhat.com/rhel7.git
[2] git clone git://git.qemu.org/qemu.git

*** Bug 1552978 has been marked as a duplicate of this bug. ***

Hi Sitong,

Please ping me when you have time so that we see if we can reproduce this bug with RHEL 7.7.

Thanks,
Maxime

(In reply to Maxime Coquelin from comment #17)
> Hi Sitong,
> 
> Please ping me when you have time so that we see if we can reproduce this
> bug with RHEL 7.7.
> 
> Thanks,
> Maxime

Hi Maxime,

Thanks for reminder, I will ping you as soon as possible when test env is available.

Thanks,
Sitong

Hi Maxime,

This issue can not be reproduced with latest rhel7.7 testing. We tested 80 times nfv live migration with vIOMMU, including (60 with ovs and 20 with pvp), qemu keeps working well.


Versions:
qemu-kvm-rhev-2.12.0-31.el7.x86_64
3.10.0-1053.el7.x86_64
tuned-2.11.0-4.el7.noarch
openvswitch-2.9.0-106.el7fdp.x86_64
dpdk-18.11-4.el7_6.x86_64


Beaker job:
https://beaker.engineering.redhat.com/recipes/6963760#task94252573


Best regards,
Pei

In our latest several runs, qemu keeps working well during live migration testing with vIOMMU. So this bug can not be reproduced anymore in rhel7.7.

I would prefer to close it as 'CURRENTRELEASE'.

Maxime, feel free to let me know if you have any concerns. Thanks.