Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
This is doing pvp live migration. Boot dpdk's testpmd in src and dest host. Then start guest and start testpmd in guest, meanwhile start MoonGen in the third host.
When migrating guest from src to des host, sometimes dpdk show errors and qemu hang.
== dpdk errors, see below is related to bug:
Bug 1538953 - IOTLB entry size mismatch before/after migration during DPDK PVP testing
errors info:
..
VHOST_CONFIG: IOTLB pool empty, clear pending misses
VHOST_CONFIG: IOTLB pool still empty, failure
VHOST_CONFIG: IOTLB pool empty, clear pending misses
VHOST_CONFIG: IOTLB pool still empty, failure
VHOST_CONFIG: IOTLB pool empty, clear pending misses
VHOST_CONFIG: IOTLB pool still empty, failure
VHOST_CONFIG: IOTLB pool empty, clear pending misses
VHOST_CONFIG: IOTLB pool still empty, failure
VHOST_CONFIG: IOTLB pool empty, clear pending misses
VHOST_CONFIG: IOTLB pool still empty, failure
VHOST_CONFIG: IOTLB pool empty, clear pending misses
VHOST_CONFIG: IOTLB pool still empty, failure
VHOST_CONFIG: IOTLB pool empty, clear pending misses
VHOST_CONFIG: IOTLB pool empty, clear pending misses
VHOST_CONFIG: read message VHOST_USER_GET_VRING_BASE
PMD: Connection closed
== qemu hang should be a separate issue. qemu hang seems because it's main thread is waiting for dpdk's ack (recvmsg), however dpdk hit errors itself and can not ack.
Version-Release number of selected component (if applicable):
3.10.0-841.el7.x86_64
qemu-kvm-rhev-2.10.0-19.el7.x86_64
dpdk-17.11-7.el7.x86_64
How reproducible:
2/4
Steps to Reproduce:
1. Boot testpmd in src and des host, refer to [1]
2. Boot guest in src host, refer to[2]
3. Boot guest in des host, using same command line as [2] but adding "-incoming tcp:0:5555"
4. Start testpmd in guest, refer to[3]
5. Start MoonGen in another host. Guest dpdk's testpmd can receive packets.
6. Migrate guest from src to des
# migrate -d tcp:10.73.72.154:5555
7. Sometimes qemu in des host and guest will both become hang. Below is the gdb info of des qemu:
(gdb) thread apply all bt
Thread 7 (Thread 0x7fc1ecee8700 (LWP 45744)):
#0 0x00007fc1f4044e09 in syscall () at /lib64/libc.so.6
#1 0x0000558c4d3f4670 in qemu_event_wait ()
#2 0x0000558c4d40456e in call_rcu_thread ()
#3 0x00007fc1f4320dd5 in start_thread () at /lib64/libpthread.so.0
#4 0x00007fc1f404aaed in clone () at /lib64/libc.so.6
Thread 6 (Thread 0x7fc1ebee6700 (LWP 45812)):
#0 0x00007fc1f43274cd in __lll_lock_wait () at /lib64/libpthread.so.0
#1 0x00007fc1f4322dcb in _L_lock_812 () at /lib64/libpthread.so.0
#2 0x00007fc1f4322c98 in pthread_mutex_lock () at /lib64/libpthread.so.0
#3 0x0000558c4d3f3faf in qemu_mutex_lock ()
#4 0x0000558c4d10b9dc in qemu_mutex_lock_iothread ()
#5 0x0000558c4d0d525d in prepare_mmio_access.isra.23 ()
#6 0x0000558c4d0d61e0 in flatview_write ()
#7 0x0000558c4d0d9955 in address_space_rw ()
#8 0x0000558c4d12c978 in kvm_cpu_exec ()
#9 0x0000558c4d10bbe2 in qemu_kvm_cpu_thread_fn ()
#10 0x00007fc1f4320dd5 in start_thread () at /lib64/libpthread.so.0
#11 0x00007fc1f404aaed in clone () at /lib64/libc.so.6
Thread 5 (Thread 0x7fc1eb6e5700 (LWP 45813)):
---Type <return> to continue, or q <return> to quit---
#0 0x00007fc1f4041517 in ioctl () at /lib64/libc.so.6
#1 0x0000558c4d12c7f5 in kvm_vcpu_ioctl ()
#2 0x0000558c4d12c8c3 in kvm_cpu_exec ()
#3 0x0000558c4d10bbe2 in qemu_kvm_cpu_thread_fn ()
#4 0x00007fc1f4320dd5 in start_thread () at /lib64/libpthread.so.0
#5 0x00007fc1f404aaed in clone () at /lib64/libc.so.6
Thread 4 (Thread 0x7fc1eaee4700 (LWP 45814)):
#0 0x00007fc1f4041517 in ioctl () at /lib64/libc.so.6
#1 0x0000558c4d12c7f5 in kvm_vcpu_ioctl ()
#2 0x0000558c4d12c8c3 in kvm_cpu_exec ()
#3 0x0000558c4d10bbe2 in qemu_kvm_cpu_thread_fn ()
#4 0x00007fc1f4320dd5 in start_thread () at /lib64/libpthread.so.0
#5 0x00007fc1f404aaed in clone () at /lib64/libc.so.6
Thread 3 (Thread 0x7fc1ea2df700 (LWP 45815)):
#0 0x00007fc1f4041517 in ioctl () at /lib64/libc.so.6
#1 0x0000558c4d12c7f5 in kvm_vcpu_ioctl ()
#2 0x0000558c4d12c8c3 in kvm_cpu_exec ()
#3 0x0000558c4d10bbe2 in qemu_kvm_cpu_thread_fn ()
#4 0x00007fc1f4320dd5 in start_thread () at /lib64/libpthread.so.0
#5 0x00007fc1f404aaed in clone () at /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
Thread 2 (Thread 0x7fc1e7dff700 (LWP 45816)):
#0 0x00007fc1f4324945 in pthread_cond_wait@@GLIBC_2.3.2 ()
at /lib64/libpthread.so.0
#1 0x0000558c4d3f4240 in qemu_cond_wait ()
#2 0x0000558c4d32593b in vnc_worker_thread_loop ()
#3 0x0000558c4d325e78 in vnc_worker_thread ()
#4 0x00007fc1f4320dd5 in start_thread () at /lib64/libpthread.so.0
#5 0x00007fc1f404aaed in clone () at /lib64/libc.so.6
Thread 1 (Thread 0x7fc1fe6bbd00 (LWP 45743)):
#0 0x00007fc1f4327bfd in recvmsg () at /lib64/libpthread.so.0
#1 0x0000558c4d3ad7cc in qio_channel_socket_readv ()
#2 0x0000558c4d39ef12 in tcp_chr_recv ()
#3 0x0000558c4d3a0a31 in tcp_chr_sync_read ()
#4 0x0000558c4d39b171 in qemu_chr_fe_read_all ()
#5 0x0000558c4d167577 in vhost_user_read.isra.1 ()
#6 0x0000558c4d1679ad in process_message_reply.part.2 ()
#7 0x0000558c4d167acb in vhost_user_send_device_iotlb_msg ()
#8 0x0000558c4d166b8a in vhost_backend_update_device_iotlb ()
#9 0x0000558c4d164e33 in vhost_device_iotlb_miss ()
#10 0x0000558c4d166ed5 in slave_read ()
#11 0x0000558c4d3f1a78 in aio_dispatch_handlers ()
#12 0x0000558c4d3f2318 in aio_dispatch ()
---Type <return> to continue, or q <return> to quit---
#13 0x0000558c4d3ef47e in aio_ctx_dispatch ()
#14 0x00007fc1f5bc28f9 in g_main_context_dispatch () at /lib64/libglib-2.0.so.0
#15 0x0000558c4d3f15ac in main_loop_wait ()
#16 0x0000558c4d0d18da in main ()
Actual results:
qemu and guest become hang.
Expected results:
qemu and guest should not become hang.
Additional info:
1. Peter suggested QE to file this bug with setting a low priority, as it's caused by dpdk's errors, and this situation may not very common
Hi Peter,
Please feel free to add comments if needed. Thanks for your analyzing for QE to report this bug.
Reference:
[1]
/usr/bin/testpmd \
-l 2,4,6,8,10 \
--socket-mem 1024,1024 \
-n 4 \
--vdev net_vhost0,iface=/tmp/vhost-user1,client=0,iommu-support=1 \
--vdev net_vhost1,iface=/tmp/vhost-user2,client=0,iommu-support=1 \
-- \
--portmask=f \
--disable-hw-vlan \
-i \
--rxq=1 --txq=1 \
--nb-cores=4 \
--forward-mode=io
testpmd> set portlist 0,2,1,3
testpmd> start
[2]
/usr/libexec/qemu-kvm \
-name rhel7.5_nonrt \
-M q35,kernel-irqchip=split \
-cpu host -m 8G \
-device intel-iommu,intremap=on,caching-mode=on,device-iotlb=on \
-object memory-backend-file,id=mem,size=8G,mem-path=/dev/hugepages,share=on \
-numa node,memdev=mem -mem-prealloc \
-smp 4,sockets=1,cores=4,threads=1 \
-device pcie-root-port,id=root.1,chassis=1 \
-device pcie-root-port,id=root.2,chassis=2 \
-device pcie-root-port,id=root.3,chassis=3 \
-device pcie-root-port,id=root.4,chassis=4 \
-drive file=/mnt/nfv/rhel7.5_nonrt.qcow2,format=qcow2,if=none,id=drive-virtio-blk0,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=drive-virtio-blk0,id=virtio-blk0,bus=root.1,iommu_platform=on,ats=on \
-netdev tap,id=hostnet0,vhost=on \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=18:66:da:5f:dd:01,bus=root.2 \
-chardev socket,id=charnet1,path=/tmp/vhost-user1 \
-netdev vhost-user,chardev=charnet1,id=hostnet1 \
-device virtio-net-pci,netdev=hostnet1,id=net1,mac=18:66:da:5f:dd:02,iommu_platform=on,ats=on,bus=root.3 \
-chardev socket,id=charnet2,path=/tmp/vhost-user2 \
-netdev vhost-user,chardev=charnet2,id=hostnet2 \
-device virtio-net-pci,netdev=hostnet2,id=net2,mac=18:66:da:5f:dd:03,iommu_platform=on,ats=on,bus=root.4 \
-vnc :2 \
-monitor stdio \
[3]
# modprobe vfio
# modprobe vfio-pci
# dpdk-devbind --bind=vfio-pci 0000:03:00.0
# dpdk-devbind --bind=vfio-pci 0000:04:00.0
# /usr/bin/testpmd \
-l 1,2,3 \
-n 4 \
-d /usr/lib64/librte_pmd_virtio.so.1 \
-w 0000:03:00.0 -w 0000:04:00.0 \
-- \
--nb-cores=2 \
--disable-hw-vlan \
-i \
--disable-rss \
--rxq=1 --txq=1
Patches posted upstream and merged in to DPDK master, will be in upcoming v18.02 release, and queued for next v17.11 LTS release:
commit 82b9c1540348b6be7996203065e10421e953cea9
Author: Maxime Coquelin <maxime.coquelin>
Date: Mon Feb 5 16:04:57 2018 +0100
vhost: remove pending IOTLB entry if miss request failed
In case vhost_user_iotlb_miss returns an error, the pending IOTLB
entry has to be removed from the list as no IOTLB update will be
received.
Fixes: fed67a20ac94 ("vhost: introduce guest IOVA to backend VA helper")
Cc: stable
Suggested-by: Tiwei Bie <tiwei.bie>
Signed-off-by: Maxime Coquelin <maxime.coquelin>
commit 37771844a05c7b0a7b039dcae1b4b0a69b4acced
Author: Maxime Coquelin <maxime.coquelin>
Date: Mon Feb 5 16:04:56 2018 +0100
vhost: fix IOTLB pool out-of-memory handling
In the unlikely case the IOTLB memory pool runs out of memory,
an issue may happen if all entries are used by the IOTLB cache,
and an IOTLB miss happen. If the iotlb pending list is empty,
then no memory is freed and allocation fails a second time.
This patch fixes this by doing an IOTLB cache random evict if
the IOTLB pending list is empty, ensuring the second allocation
try will succeed.
In the same spirit, the opposite is done when inserting an
IOTLB entry in the IOTLB cache fails due to out of memory. In
this case, the IOTLB pending is flushed if the IOTLB cache is
empty to ensure the new entry can be inserted.
Fixes: d012d1f293f4 ("vhost: add IOTLB helper functions")
Fixes: f72c2ad63aeb ("vhost: add pending IOTLB miss request list and helpers")
Cc: stable
Signed-off-by: Maxime Coquelin <maxime.coquelin>
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2018:1267
Comment 10Timothy Redaelli
2018-05-03 15:02:21 UTC
*** Bug 1533408 has been marked as a duplicate of this bug. ***