Bug 1541881

Description of problem:
This is doing pvp live migration. Boot dpdk's testpmd in src and dest host. Then start guest and start testpmd in guest, meanwhile start MoonGen in the third host.

When migrating guest from src to des host, sometimes dpdk show errors and qemu hang.
== dpdk errors, see below is related to bug:
Bug 1538953 - IOTLB entry size mismatch before/after migration during DPDK PVP testing

errors info:
..
VHOST_CONFIG: IOTLB pool empty, clear pending misses
VHOST_CONFIG: IOTLB pool still empty, failure
VHOST_CONFIG: IOTLB pool empty, clear pending misses
VHOST_CONFIG: IOTLB pool still empty, failure
VHOST_CONFIG: IOTLB pool empty, clear pending misses
VHOST_CONFIG: IOTLB pool still empty, failure
VHOST_CONFIG: IOTLB pool empty, clear pending misses
VHOST_CONFIG: IOTLB pool still empty, failure
VHOST_CONFIG: IOTLB pool empty, clear pending misses
VHOST_CONFIG: IOTLB pool still empty, failure
VHOST_CONFIG: IOTLB pool empty, clear pending misses
VHOST_CONFIG: IOTLB pool still empty, failure
VHOST_CONFIG: IOTLB pool empty, clear pending misses
VHOST_CONFIG: IOTLB pool empty, clear pending misses
VHOST_CONFIG: read message VHOST_USER_GET_VRING_BASE
PMD: Connection closed

== qemu hang should be a separate issue. qemu hang seems because it's main thread is waiting for dpdk's ack (recvmsg), however dpdk hit errors itself and can not ack.


Version-Release number of selected component (if applicable):
3.10.0-841.el7.x86_64
qemu-kvm-rhev-2.10.0-19.el7.x86_64
dpdk-17.11-7.el7.x86_64

How reproducible:
2/4

Steps to Reproduce:
1. Boot testpmd in src and des host, refer to [1]

2. Boot guest in src host, refer to[2]
 
3. Boot guest in des host, using same command line as [2] but adding "-incoming tcp:0:5555"

4. Start testpmd in guest, refer to[3]

5. Start MoonGen in another host. Guest dpdk's testpmd can receive packets.
 
6. Migrate guest from src to des
# migrate -d tcp:10.73.72.154:5555

7. Sometimes qemu in des host and guest will both become hang. Below is the gdb info of des qemu:

(gdb) thread apply all bt

Thread 7 (Thread 0x7fc1ecee8700 (LWP 45744)):
#0  0x00007fc1f4044e09 in syscall () at /lib64/libc.so.6
#1  0x0000558c4d3f4670 in qemu_event_wait ()
#2  0x0000558c4d40456e in call_rcu_thread ()
#3  0x00007fc1f4320dd5 in start_thread () at /lib64/libpthread.so.0
#4  0x00007fc1f404aaed in clone () at /lib64/libc.so.6

Thread 6 (Thread 0x7fc1ebee6700 (LWP 45812)):
#0  0x00007fc1f43274cd in __lll_lock_wait () at /lib64/libpthread.so.0
#1  0x00007fc1f4322dcb in _L_lock_812 () at /lib64/libpthread.so.0
#2  0x00007fc1f4322c98 in pthread_mutex_lock () at /lib64/libpthread.so.0
#3  0x0000558c4d3f3faf in qemu_mutex_lock ()
#4  0x0000558c4d10b9dc in qemu_mutex_lock_iothread ()
#5  0x0000558c4d0d525d in prepare_mmio_access.isra.23 ()
#6  0x0000558c4d0d61e0 in flatview_write ()
#7  0x0000558c4d0d9955 in address_space_rw ()
#8  0x0000558c4d12c978 in kvm_cpu_exec ()
#9  0x0000558c4d10bbe2 in qemu_kvm_cpu_thread_fn ()
#10 0x00007fc1f4320dd5 in start_thread () at /lib64/libpthread.so.0
#11 0x00007fc1f404aaed in clone () at /lib64/libc.so.6

Thread 5 (Thread 0x7fc1eb6e5700 (LWP 45813)):
---Type <return> to continue, or q <return> to quit---
#0  0x00007fc1f4041517 in ioctl () at /lib64/libc.so.6
#1  0x0000558c4d12c7f5 in kvm_vcpu_ioctl ()
#2  0x0000558c4d12c8c3 in kvm_cpu_exec ()
#3  0x0000558c4d10bbe2 in qemu_kvm_cpu_thread_fn ()
#4  0x00007fc1f4320dd5 in start_thread () at /lib64/libpthread.so.0
#5  0x00007fc1f404aaed in clone () at /lib64/libc.so.6

Thread 4 (Thread 0x7fc1eaee4700 (LWP 45814)):
#0  0x00007fc1f4041517 in ioctl () at /lib64/libc.so.6
#1  0x0000558c4d12c7f5 in kvm_vcpu_ioctl ()
#2  0x0000558c4d12c8c3 in kvm_cpu_exec ()
#3  0x0000558c4d10bbe2 in qemu_kvm_cpu_thread_fn ()
#4  0x00007fc1f4320dd5 in start_thread () at /lib64/libpthread.so.0
#5  0x00007fc1f404aaed in clone () at /lib64/libc.so.6

Thread 3 (Thread 0x7fc1ea2df700 (LWP 45815)):
#0  0x00007fc1f4041517 in ioctl () at /lib64/libc.so.6
#1  0x0000558c4d12c7f5 in kvm_vcpu_ioctl ()
#2  0x0000558c4d12c8c3 in kvm_cpu_exec ()
#3  0x0000558c4d10bbe2 in qemu_kvm_cpu_thread_fn ()
#4  0x00007fc1f4320dd5 in start_thread () at /lib64/libpthread.so.0
#5  0x00007fc1f404aaed in clone () at /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 2 (Thread 0x7fc1e7dff700 (LWP 45816)):
#0  0x00007fc1f4324945 in pthread_cond_wait@@GLIBC_2.3.2 ()
    at /lib64/libpthread.so.0
#1  0x0000558c4d3f4240 in qemu_cond_wait ()
#2  0x0000558c4d32593b in vnc_worker_thread_loop ()
#3  0x0000558c4d325e78 in vnc_worker_thread ()
#4  0x00007fc1f4320dd5 in start_thread () at /lib64/libpthread.so.0
#5  0x00007fc1f404aaed in clone () at /lib64/libc.so.6

Thread 1 (Thread 0x7fc1fe6bbd00 (LWP 45743)):
#0  0x00007fc1f4327bfd in recvmsg () at /lib64/libpthread.so.0
#1  0x0000558c4d3ad7cc in qio_channel_socket_readv ()
#2  0x0000558c4d39ef12 in tcp_chr_recv ()
#3  0x0000558c4d3a0a31 in tcp_chr_sync_read ()
#4  0x0000558c4d39b171 in qemu_chr_fe_read_all ()
#5  0x0000558c4d167577 in vhost_user_read.isra.1 ()
#6  0x0000558c4d1679ad in process_message_reply.part.2 ()
#7  0x0000558c4d167acb in vhost_user_send_device_iotlb_msg ()
#8  0x0000558c4d166b8a in vhost_backend_update_device_iotlb ()
#9  0x0000558c4d164e33 in vhost_device_iotlb_miss ()
#10 0x0000558c4d166ed5 in slave_read ()
#11 0x0000558c4d3f1a78 in aio_dispatch_handlers ()
#12 0x0000558c4d3f2318 in aio_dispatch ()
---Type <return> to continue, or q <return> to quit---
#13 0x0000558c4d3ef47e in aio_ctx_dispatch ()
#14 0x00007fc1f5bc28f9 in g_main_context_dispatch () at /lib64/libglib-2.0.so.0
#15 0x0000558c4d3f15ac in main_loop_wait ()
#16 0x0000558c4d0d18da in main ()


Actual results:
qemu and guest become hang.


Expected results:
qemu and guest should not become hang.


Additional info:
1. Peter suggested QE to file this bug with setting a low priority, as it's caused by dpdk's errors, and this situation may not very common

Hi Peter, 

Please feel free to add comments if needed. Thanks for your analyzing for QE to report this bug.


Reference:
[1]
/usr/bin/testpmd \
-l 2,4,6,8,10 \
--socket-mem 1024,1024 \
-n 4 \
--vdev net_vhost0,iface=/tmp/vhost-user1,client=0,iommu-support=1 \
--vdev net_vhost1,iface=/tmp/vhost-user2,client=0,iommu-support=1 \
-- \
--portmask=f \
--disable-hw-vlan \
-i \
--rxq=1 --txq=1 \
--nb-cores=4 \
--forward-mode=io

testpmd> set portlist 0,2,1,3
testpmd> start 

[2]
/usr/libexec/qemu-kvm \
-name rhel7.5_nonrt \
-M q35,kernel-irqchip=split \
-cpu host -m 8G \
-device intel-iommu,intremap=on,caching-mode=on,device-iotlb=on \
-object memory-backend-file,id=mem,size=8G,mem-path=/dev/hugepages,share=on \
-numa node,memdev=mem -mem-prealloc \
-smp 4,sockets=1,cores=4,threads=1 \
-device pcie-root-port,id=root.1,chassis=1 \
-device pcie-root-port,id=root.2,chassis=2 \
-device pcie-root-port,id=root.3,chassis=3 \
-device pcie-root-port,id=root.4,chassis=4 \
-drive file=/mnt/nfv/rhel7.5_nonrt.qcow2,format=qcow2,if=none,id=drive-virtio-blk0,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=drive-virtio-blk0,id=virtio-blk0,bus=root.1,iommu_platform=on,ats=on \
-netdev tap,id=hostnet0,vhost=on \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=18:66:da:5f:dd:01,bus=root.2 \
-chardev socket,id=charnet1,path=/tmp/vhost-user1 \
-netdev vhost-user,chardev=charnet1,id=hostnet1 \
-device virtio-net-pci,netdev=hostnet1,id=net1,mac=18:66:da:5f:dd:02,iommu_platform=on,ats=on,bus=root.3 \
-chardev socket,id=charnet2,path=/tmp/vhost-user2 \
-netdev vhost-user,chardev=charnet2,id=hostnet2 \
-device virtio-net-pci,netdev=hostnet2,id=net2,mac=18:66:da:5f:dd:03,iommu_platform=on,ats=on,bus=root.4 \
-vnc :2 \
-monitor stdio \


[3]
# modprobe vfio
# modprobe vfio-pci
# dpdk-devbind --bind=vfio-pci 0000:03:00.0
# dpdk-devbind --bind=vfio-pci 0000:04:00.0

# /usr/bin/testpmd \
-l 1,2,3 \
-n 4 \
-d /usr/lib64/librte_pmd_virtio.so.1 \
-w 0000:03:00.0 -w 0000:04:00.0 \
-- \
--nb-cores=2 \
--disable-hw-vlan \
-i \
--disable-rss \
--rxq=1 --txq=1