Description of problem: [OVS} OVS daemon got segfault when virsh start <guest> [root@netqe9 ~]# gdb -p 108362 GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Attaching to process 108362 Reading symbols from /usr/sbin/ovs-vswitchd...Reading symbols from /usr/lib/debug/usr/sbin/ovs-vswitchd.debug...done. done. Reading symbols from /usr/lib64/openvswitch/librte_bitratestats.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_bitratestats.so.2.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_bitratestats.so.2 Reading symbols from /usr/lib64/openvswitch/librte_bus_pci.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_bus_pci.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_bus_pci.so.1 Reading symbols from /usr/lib64/openvswitch/librte_bus_vdev.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_bus_vdev.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_bus_vdev.so.1 Reading symbols from /usr/lib64/openvswitch/librte_cmdline.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_cmdline.so.2.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_cmdline.so.2 Reading symbols from /usr/lib64/openvswitch/librte_eal.so.6...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_eal.so.6.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_eal.so.6 Reading symbols from /usr/lib64/openvswitch/librte_efd.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_efd.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_efd.so.1 Reading symbols from /usr/lib64/openvswitch/librte_ethdev.so.8...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_ethdev.so.8.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_ethdev.so.8 Reading symbols from /usr/lib64/openvswitch/librte_eventdev.so.3...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_eventdev.so.3.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_eventdev.so.3 Reading symbols from /usr/lib64/openvswitch/librte_gro.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_gro.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_gro.so.1 Reading symbols from /usr/lib64/openvswitch/librte_gso.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_gso.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_gso.so.1 Reading symbols from /usr/lib64/openvswitch/librte_hash.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_hash.so.2.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_hash.so.2 Reading symbols from /usr/lib64/openvswitch/librte_ip_frag.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_ip_frag.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_ip_frag.so.1 Reading symbols from /usr/lib64/openvswitch/librte_kvargs.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_kvargs.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_kvargs.so.1 Reading symbols from /usr/lib64/openvswitch/librte_latencystats.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_latencystats.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_latencystats.so.1 Reading symbols from /usr/lib64/openvswitch/librte_mbuf.so.3...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_mbuf.so.3.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_mbuf.so.3 Reading symbols from /usr/lib64/openvswitch/librte_member.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_member.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_member.so.1 Reading symbols from /usr/lib64/openvswitch/librte_mempool.so.3...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_mempool.so.3.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_mempool.so.3 Reading symbols from /usr/lib64/openvswitch/librte_mempool_octeontx.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_mempool_octeontx.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_mempool_octeontx.so.1 Reading symbols from /usr/lib64/openvswitch/librte_mempool_ring.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_mempool_ring.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_mempool_ring.so.1 Reading symbols from /usr/lib64/openvswitch/librte_mempool_stack.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_mempool_stack.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_mempool_stack.so.1 Reading symbols from /usr/lib64/openvswitch/librte_meter.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_meter.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_meter.so.1 Reading symbols from /usr/lib64/openvswitch/librte_metrics.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_metrics.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_metrics.so.1 Reading symbols from /usr/lib64/openvswitch/librte_net.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_net.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_net.so.1 Reading symbols from /usr/lib64/openvswitch/librte_pci.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pci.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_pci.so.1 Reading symbols from /usr/lib64/openvswitch/librte_pdump.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pdump.so.2.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_pdump.so.2 Reading symbols from /usr/lib64/openvswitch/librte_pmd_bnxt.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_bnxt.so.2.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_pmd_bnxt.so.2 Reading symbols from /usr/lib64/openvswitch/librte_pmd_e1000.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_e1000.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_pmd_e1000.so.1 Reading symbols from /usr/lib64/openvswitch/librte_pmd_enic.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_enic.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_pmd_enic.so.1 Reading symbols from /usr/lib64/openvswitch/librte_pmd_failsafe.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_failsafe.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_pmd_failsafe.so.1 Reading symbols from /usr/lib64/openvswitch/librte_pmd_i40e.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_i40e.so.2.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_pmd_i40e.so.2 Reading symbols from /usr/lib64/openvswitch/librte_pmd_ixgbe.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_ixgbe.so.2.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_pmd_ixgbe.so.2 Reading symbols from /usr/lib64/openvswitch/librte_pmd_mlx4.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_mlx4.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_pmd_mlx4.so.1 Reading symbols from /usr/lib64/openvswitch/librte_pmd_mlx5.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_mlx5.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_pmd_mlx5.so.1 Reading symbols from /usr/lib64/openvswitch/librte_pmd_nfp.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_nfp.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_pmd_nfp.so.1 Reading symbols from /usr/lib64/openvswitch/librte_pmd_qede.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_qede.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_pmd_qede.so.1 Reading symbols from /usr/lib64/openvswitch/librte_pmd_ring.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_ring.so.2.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_pmd_ring.so.2 Reading symbols from /usr/lib64/openvswitch/librte_pmd_virtio.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_virtio.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_pmd_virtio.so.1 Reading symbols from /usr/lib64/openvswitch/librte_ring.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_ring.so.1.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_ring.so.1 Reading symbols from /usr/lib64/openvswitch/librte_vhost.so.4...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_vhost.so.4.debug...done. done. Loaded symbols for /usr/lib64/openvswitch/librte_vhost.so.4 Reading symbols from /lib64/libofproto-2.10.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libofproto-2.10.so.0.0.0.debug...done. done. Loaded symbols for /lib64/libofproto-2.10.so.0 Reading symbols from /lib64/libsflow-2.10.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libsflow-2.10.so.0.0.0.debug...done. done. Loaded symbols for /lib64/libsflow-2.10.so.0 Reading symbols from /lib64/libopenvswitch-2.10.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libopenvswitch-2.10.so.0.0.0.debug...done. done. Loaded symbols for /lib64/libopenvswitch-2.10.so.0 Reading symbols from /lib64/libssl.so.10...Reading symbols from /lib64/libssl.so.10...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libssl.so.10 Reading symbols from /lib64/libcrypto.so.10...Reading symbols from /lib64/libcrypto.so.10...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libcrypto.so.10 Reading symbols from /lib64/libcap-ng.so.0...Reading symbols from /lib64/libcap-ng.so.0...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libcap-ng.so.0 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [New LWP 108767] [New LWP 108766] [New LWP 108765] [New LWP 108764] [New LWP 108763] [New LWP 108762] [New LWP 108761] [New LWP 108760] [New LWP 108759] [New LWP 108758] [New LWP 108757] [New LWP 108756] [New LWP 108755] [New LWP 108754] [New LWP 108753] [New LWP 108752] [New LWP 108751] [New LWP 108750] [New LWP 108749] [New LWP 108748] [New LWP 108747] [New LWP 108746] [New LWP 108745] [New LWP 108744] [New LWP 108641] [New LWP 108640] [New LWP 108413] [New LWP 108412] [New LWP 108380] [New LWP 108375] [New LWP 108372] [New LWP 108370] [New LWP 108364] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/libunbound.so.2...Reading symbols from /lib64/libunbound.so.2...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libunbound.so.2 Reading symbols from /lib64/libnuma.so.1...Reading symbols from /lib64/libnuma.so.1...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libnuma.so.1 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /lib64/libgssapi_krb5.so.2...Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libgssapi_krb5.so.2 Reading symbols from /lib64/libkrb5.so.3...Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /lib64/libk5crypto.so.3...Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libk5crypto.so.3 Reading symbols from /lib64/libz.so.1...Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /lib64/libevent-2.0.so.5...Reading symbols from /lib64/libevent-2.0.so.5...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libevent-2.0.so.5 Reading symbols from /lib64/libpython2.7.so.1.0...Reading symbols from /lib64/libpython2.7.so.1.0...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libpython2.7.so.1.0 Reading symbols from /lib64/libkrb5support.so.0...Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libutil.so.1 Reading symbols from /lib64/libselinux.so.1...Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /lib64/libpcre.so.1...Reading symbols from /lib64/libpcre.so.1...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libpcre.so.1 Reading symbols from /usr/lib64/openvswitch/librte_pmd_mlx4_glue.so.17.11.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/openvswitch/librte_pmd_mlx4_glue.so.17.11.1 Reading symbols from /lib64/libibverbs.so.1...Reading symbols from /lib64/libibverbs.so.1...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libibverbs.so.1 Reading symbols from /lib64/libmlx4.so.1...Reading symbols from /lib64/libmlx4.so.1...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libmlx4.so.1 Reading symbols from /lib64/libnl-route-3.so.200...Reading symbols from /lib64/libnl-route-3.so.200...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libnl-route-3.so.200 Reading symbols from /lib64/libnl-3.so.200...Reading symbols from /lib64/libnl-3.so.200...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libnl-3.so.200 Reading symbols from /usr/lib64/openvswitch/librte_pmd_mlx5_glue.so.17.11.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/openvswitch/librte_pmd_mlx5_glue.so.17.11.1 Reading symbols from /lib64/libmlx5.so.1...Reading symbols from /lib64/libmlx5.so.1...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libmlx5.so.1 Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 0x00007f6cf4a8820d in poll () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install glibc-2.17-260.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-33.el7.x86_64 libcap-ng-0.7.5-4.el7.x86_64 libcom_err-1.42.9-13.el7.x86_64 libevent-2.0.21-4.el7.x86_64 libgcc-4.8.5-36.el7.x86_64 libibverbs-17.2-3.el7.x86_64 libnl3-3.2.28-4.el7.x86_64 libselinux-2.5-14.1.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64 openssl-libs-1.0.2k-15.el7.x86_64 pcre-8.32-17.el7.x86_64 python-libs-2.7.5-75.el7.x86_64 unbound-libs-1.6.6-1.el7.x86_64 zlib-1.2.7-18.el7.x86_64 (gdb) c Continuing. [New Thread 0x7f6cfc1f8c80 (LWP 109070)] [Thread 0x7f6cfc1f8c80 (LWP 109070) exited] [New Thread 0x7f6cfc1f8c80 (LWP 109148)] [Thread 0x7f6cfc1f8c80 (LWP 109148) exited] [New Thread 0x7f6cfc1f8c80 (LWP 109153)] [Thread 0x7f6cfc1f8c80 (LWP 109153) exited] [New Thread 0x7f6cfc1f8c80 (LWP 109160)] Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7f6cb5ffb700 (LWP 108413)] 0x00007f6cf6b95f53 in qva_to_vva (dev=0x7f69bfffe740, qva=0, len=0x7f6cb5ffa4f0) at /usr/src/debug/openvswitch-2.10.0/dpdk-17.11/lib/librte_vhost/vhost_user.c:373 373 for (i = 0; i < dev->mem->nregions; i++) { (gdb) bt #0 0x00007f6cf6b95f53 in qva_to_vva (dev=0x7f69bfffe740, qva=0, len=0x7f6cb5ffa4f0) at /usr/src/debug/openvswitch-2.10.0/dpdk-17.11/lib/librte_vhost/vhost_user.c:373 #1 0x00007f6cf6b96253 in translate_ring_addresses (dev=dev@entry=0x7f69bfffe740, vq_index=vq_index@entry=0) at /usr/src/debug/openvswitch-2.10.0/dpdk-17.11/lib/librte_vhost/vhost_user.c:427 #2 0x00007f6cf6b96cae in vhost_user_set_vring_kick (pmsg=0x7f6cb5ffa7a0, pdev=<synthetic pointer>) at /usr/src/debug/openvswitch-2.10.0/dpdk-17.11/lib/librte_vhost/vhost_user.c:832 #3 vhost_user_msg_handler (vid=<optimized out>, fd=fd@entry=78) at /usr/src/debug/openvswitch-2.10.0/dpdk-17.11/lib/librte_vhost/vhost_user.c:1445 #4 0x00007f6cf6b9424f in vhost_user_read_cb (connfd=78, dat=0x7f6c480008c0, remove=0x7f6cb5ffa990) at /usr/src/debug/openvswitch-2.10.0/dpdk-17.11/lib/librte_vhost/socket.c:292 #5 0x00007f6cf6b92c18 in fdset_event_dispatch (arg=0x7f6cf6da51c0 <vhost_user+8192>) at /usr/src/debug/openvswitch-2.10.0/dpdk-17.11/lib/librte_vhost/fd_man.c:276 #6 0x00007f6cf5780dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6cf4a92ead in clone () from /lib64/libc.so.6 (gdb) quit A debugging session is active. Inferior 1 [process 108362] will be detached. Quit anyway? (y or n) y Detaching from program: /usr/sbin/ovs-vswitchd, process 108362 [root@netqe9 ~]# NOTE: OVS-dpdk is running in dpdkvhostuserclient mode. Version-Release number of selected component (if applicable): [root@netqe9 ~]# rpm -q openvswitch2.10 openvswitch2.10-2.10.0-1.el7fdp.x86_64 [root@netqe9 ~]# rpm -qa | grep qemu libvirt-daemon-driver-qemu-4.5.0-6.el7.x86_64 qemu-kvm-rhev-2.10.0-21.el7.x86_64 qemu-kvm-common-rhev-2.10.0-21.el7.x86_64 qemu-img-rhev-2.10.0-21.el7.x86_64 ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch [root@netqe9 ~]# rpm -q libvirt libvirt-4.5.0-6.el7.x86_64 [root@netqe9 ~]# !un uname -a Linux netqe9.knqe.lab.eng.bos.redhat.com 3.10.0-933.el7.x86_64 #1 SMP Sat Aug 11 11:32:57 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux [root@netqe9 ~]# How reproducible: Reproducible Steps to Reproduce: 1. Config an OVS-dpdk bridge with a vhost port in dpdkvhostuserclient mode 2. virsh start the guest 3. Actual results: Expected results: Additional info:
Hi Jean-Tsung, Could you please share the OVS logs so that I can check the vhost-user request received? Thanks, Maxime
(In reply to Maxime Coquelin from comment #3) > Hi Jean-Tsung, > > Could you please share the OVS logs so that I can check the vhost-user > request received? > > Thanks, > Maxime Hi Maxime, Just reproduced it. Got the following piece of log when "virsh start <guest>"; and, the daemon got segfaulted. 2018-08-23T23:57:31.175Z|00108|netdev_linux|INFO|ioctl(SIOCGIFINDEX) on vxlan_sys_4789 device failed: No such device 2018-08-23T23:57:31.216Z|00109|netdev_linux|INFO|ioctl(SIOCGIFINDEX) on vxlan_sys_4789 device failed: No such device 2018-08-23T23:57:31.471Z|00001|dpdk|INFO|VHOST_CONFIG: /tmp/vhost0: connected 2018-08-23T23:57:31.471Z|00002|dpdk|INFO|VHOST_CONFIG: new device, handle is 0 2018-08-23T23:57:31.495Z|00001|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_FEATURES 2018-08-23T23:57:31.495Z|00002|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES 2018-08-23T23:57:31.495Z|00003|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES 2018-08-23T23:57:31.495Z|00004|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_QUEUE_NUM 2018-08-23T23:57:31.495Z|00005|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_SLAVE_REQ_FD 2018-08-23T23:57:31.495Z|00006|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_OWNER 2018-08-23T23:57:31.495Z|00007|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_FEATURES 2018-08-23T23:57:31.496Z|00008|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL 2018-08-23T23:57:31.496Z|00009|dpdk|INFO|VHOST_CONFIG: vring call idx:0 file:81 2018-08-23T23:57:31.496Z|00010|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL 2018-08-23T23:57:31.496Z|00011|dpdk|INFO|VHOST_CONFIG: vring call idx:1 file:82 2018-08-23T23:57:31.503Z|00110|netdev_linux|INFO|ioctl(SIOCGIFINDEX) on vxlan_sys_4789 device failed: No such device 2018-08-23T23:57:31.548Z|00111|netdev_linux|INFO|ioctl(SIOCGIFINDEX) on vxlan_sys_4789 device failed: No such device 2018-08-23T23:57:33.197Z|00112|netdev_linux|INFO|ioctl(SIOCGIFINDEX) on vxlan_sys_4789 device failed: No such device 2018-08-23T23:57:35.196Z|00113|netdev_linux|INFO|ioctl(SIOCGIFINDEX) on vxlan_sys_4789 device failed: No such device 2018-08-23T23:57:35.242Z|00114|netdev_linux|INFO|ioctl(SIOCGIFINDEX) on vxlan_sys_4789 device failed: No such device 2018-08-23T23:57:40.691Z|00012|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE 2018-08-23T23:57:40.693Z|00013|dpdk|INFO|VHOST_CONFIG: set queue enable: 1 to qp idx: 0 2018-08-23T23:57:40.693Z|00014|netdev_dpdk|INFO|State of queue 0 ( tx_qid 0 ) of vhost device '/tmp/vhost0'changed to 'enabled' 2018-08-23T23:57:40.693Z|00015|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE 2018-08-23T23:57:40.693Z|00016|dpdk|INFO|VHOST_CONFIG: set queue enable: 1 to qp idx: 1 2018-08-23T23:57:40.693Z|00017|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE 2018-08-23T23:57:40.693Z|00018|dpdk|INFO|VHOST_CONFIG: set queue enable: 1 to qp idx: 0 2018-08-23T23:57:40.693Z|00019|netdev_dpdk|INFO|State of queue 0 ( tx_qid 0 ) of vhost device '/tmp/vhost0'changed to 'enabled' 2018-08-23T23:57:40.693Z|00020|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE 2018-08-23T23:57:40.693Z|00021|dpdk|INFO|VHOST_CONFIG: set queue enable: 1 to qp idx: 1
(In reply to Jean-Tsung Hsiao from comment #4) > (In reply to Maxime Coquelin from comment #3) > > Hi Jean-Tsung, > > > > Could you please share the OVS logs so that I can check the vhost-user > > request received? > > > > Thanks, > > Maxime > > Hi Maxime, > Just reproduced it. Got the following piece of log when "virsh start > <guest>"; and, the daemon got segfaulted. Thanks Jean-Tsung, It seems the trace wasn't fully evacuated into the journal. Any chance I can connect to your setup?
So I managed to reproduce on Jean's setup. The crash happens because mmap of guest memory fails due to SElinux policy: ovs-vswitchd[60356]: ovs|00024|dpdk|ERR|VHOST_CONFIG: mmap region 0 failed. and: type=AVC msg=audit(1535118626.626:515): avc: denied { map } for pid=60356 comm="ovs-vswitchd" path=2F6465762F6875676570616765732F6C6962766972742F71656D752F332D6D712D7668752D74756E6E656C2D7365727665722F71656D755F6261636B5F6D656D2E5F6F626A656374735F72616D2D6E6F6465302E79427A6E3162202864656C6574656429 dev="hugetlbfs" ino=4125881 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:svirt_image_t:s0 tclass=file permissive=0 Jean already filed a Bz for this SELinux policy issue (Bz1620257). However, the vhost-user library should be more robust, so that it handles gracefully SELinux policy issues. I'll keep this Bz to make vhost-user more robust, but lower the priority.
Hi Aaron, See Comment #6 for the same AVC listed in bug 1620257. Thanks! Jean
*** Bug 1601134 has been marked as a duplicate of this bug. ***
Patch posted upstream: http://patches.dpdk.org/patch/48346/
The patch is part of v18.11.1 upstream. So making it depending on Bz1713698 (Rebase to 18.11.2) and set FDP release to 19.E.
Have verified and passed the fix with OVS-2.11.0-18.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2522