The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.
Bug 1621626 - [OVS} OVS daemon got segfault when virsh start <guest>
Summary: [OVS} OVS daemon got segfault when virsh start <guest>
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Fast Datapath
Classification: Red Hat
Component: openvswitch2.11
Version: FDP 19.C
Hardware: x86_64
OS: Linux
low
low
Target Milestone: ---
: ---
Assignee: Open vSwitch development team
QA Contact: Jean-Tsung Hsiao
URL:
Whiteboard:
: 1601134 (view as bug list)
Depends On: 1713698
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-08-23 17:14 UTC by Jean-Tsung Hsiao
Modified: 2019-09-11 17:32 UTC (History)
10 users (show)

Fixed In Version: openvswitch2.10-2.10.0-49.el7fdn
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-19 16:20:36 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1620257 0 unspecified CLOSED Ping failed between guests over OVS-dpdk tunnel when Selinux=Enforcing 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHBA-2019:2522 0 None None None 2019-08-19 16:20:43 UTC

Internal Links: 1620257

Description Jean-Tsung Hsiao 2018-08-23 17:14:41 UTC
Description of problem: [OVS} OVS daemon got segfault when virsh start <guest>

[root@netqe9 ~]# gdb -p 108362
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 108362
Reading symbols from /usr/sbin/ovs-vswitchd...Reading symbols from /usr/lib/debug/usr/sbin/ovs-vswitchd.debug...done.
done.
Reading symbols from /usr/lib64/openvswitch/librte_bitratestats.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_bitratestats.so.2.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_bitratestats.so.2
Reading symbols from /usr/lib64/openvswitch/librte_bus_pci.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_bus_pci.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_bus_pci.so.1
Reading symbols from /usr/lib64/openvswitch/librte_bus_vdev.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_bus_vdev.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_bus_vdev.so.1
Reading symbols from /usr/lib64/openvswitch/librte_cmdline.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_cmdline.so.2.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_cmdline.so.2
Reading symbols from /usr/lib64/openvswitch/librte_eal.so.6...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_eal.so.6.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_eal.so.6
Reading symbols from /usr/lib64/openvswitch/librte_efd.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_efd.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_efd.so.1
Reading symbols from /usr/lib64/openvswitch/librte_ethdev.so.8...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_ethdev.so.8.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_ethdev.so.8
Reading symbols from /usr/lib64/openvswitch/librte_eventdev.so.3...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_eventdev.so.3.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_eventdev.so.3
Reading symbols from /usr/lib64/openvswitch/librte_gro.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_gro.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_gro.so.1
Reading symbols from /usr/lib64/openvswitch/librte_gso.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_gso.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_gso.so.1
Reading symbols from /usr/lib64/openvswitch/librte_hash.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_hash.so.2.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_hash.so.2
Reading symbols from /usr/lib64/openvswitch/librte_ip_frag.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_ip_frag.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_ip_frag.so.1
Reading symbols from /usr/lib64/openvswitch/librte_kvargs.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_kvargs.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_kvargs.so.1
Reading symbols from /usr/lib64/openvswitch/librte_latencystats.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_latencystats.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_latencystats.so.1
Reading symbols from /usr/lib64/openvswitch/librte_mbuf.so.3...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_mbuf.so.3.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_mbuf.so.3
Reading symbols from /usr/lib64/openvswitch/librte_member.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_member.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_member.so.1
Reading symbols from /usr/lib64/openvswitch/librte_mempool.so.3...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_mempool.so.3.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_mempool.so.3
Reading symbols from /usr/lib64/openvswitch/librte_mempool_octeontx.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_mempool_octeontx.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_mempool_octeontx.so.1
Reading symbols from /usr/lib64/openvswitch/librte_mempool_ring.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_mempool_ring.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_mempool_ring.so.1
Reading symbols from /usr/lib64/openvswitch/librte_mempool_stack.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_mempool_stack.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_mempool_stack.so.1
Reading symbols from /usr/lib64/openvswitch/librte_meter.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_meter.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_meter.so.1
Reading symbols from /usr/lib64/openvswitch/librte_metrics.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_metrics.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_metrics.so.1
Reading symbols from /usr/lib64/openvswitch/librte_net.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_net.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_net.so.1
Reading symbols from /usr/lib64/openvswitch/librte_pci.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pci.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_pci.so.1
Reading symbols from /usr/lib64/openvswitch/librte_pdump.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pdump.so.2.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_pdump.so.2
Reading symbols from /usr/lib64/openvswitch/librte_pmd_bnxt.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_bnxt.so.2.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_pmd_bnxt.so.2
Reading symbols from /usr/lib64/openvswitch/librte_pmd_e1000.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_e1000.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_pmd_e1000.so.1
Reading symbols from /usr/lib64/openvswitch/librte_pmd_enic.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_enic.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_pmd_enic.so.1
Reading symbols from /usr/lib64/openvswitch/librte_pmd_failsafe.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_failsafe.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_pmd_failsafe.so.1
Reading symbols from /usr/lib64/openvswitch/librte_pmd_i40e.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_i40e.so.2.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_pmd_i40e.so.2
Reading symbols from /usr/lib64/openvswitch/librte_pmd_ixgbe.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_ixgbe.so.2.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_pmd_ixgbe.so.2
Reading symbols from /usr/lib64/openvswitch/librte_pmd_mlx4.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_mlx4.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_pmd_mlx4.so.1
Reading symbols from /usr/lib64/openvswitch/librte_pmd_mlx5.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_mlx5.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_pmd_mlx5.so.1
Reading symbols from /usr/lib64/openvswitch/librte_pmd_nfp.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_nfp.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_pmd_nfp.so.1
Reading symbols from /usr/lib64/openvswitch/librte_pmd_qede.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_qede.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_pmd_qede.so.1
Reading symbols from /usr/lib64/openvswitch/librte_pmd_ring.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_ring.so.2.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_pmd_ring.so.2
Reading symbols from /usr/lib64/openvswitch/librte_pmd_virtio.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_pmd_virtio.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_pmd_virtio.so.1
Reading symbols from /usr/lib64/openvswitch/librte_ring.so.1...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_ring.so.1.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_ring.so.1
Reading symbols from /usr/lib64/openvswitch/librte_vhost.so.4...Reading symbols from /usr/lib/debug/usr/lib64/openvswitch/librte_vhost.so.4.debug...done.
done.
Loaded symbols for /usr/lib64/openvswitch/librte_vhost.so.4
Reading symbols from /lib64/libofproto-2.10.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libofproto-2.10.so.0.0.0.debug...done.
done.
Loaded symbols for /lib64/libofproto-2.10.so.0
Reading symbols from /lib64/libsflow-2.10.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libsflow-2.10.so.0.0.0.debug...done.
done.
Loaded symbols for /lib64/libsflow-2.10.so.0
Reading symbols from /lib64/libopenvswitch-2.10.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libopenvswitch-2.10.so.0.0.0.debug...done.
done.
Loaded symbols for /lib64/libopenvswitch-2.10.so.0
Reading symbols from /lib64/libssl.so.10...Reading symbols from /lib64/libssl.so.10...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libssl.so.10
Reading symbols from /lib64/libcrypto.so.10...Reading symbols from /lib64/libcrypto.so.10...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypto.so.10
Reading symbols from /lib64/libcap-ng.so.0...Reading symbols from /lib64/libcap-ng.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcap-ng.so.0
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[New LWP 108767]
[New LWP 108766]
[New LWP 108765]
[New LWP 108764]
[New LWP 108763]
[New LWP 108762]
[New LWP 108761]
[New LWP 108760]
[New LWP 108759]
[New LWP 108758]
[New LWP 108757]
[New LWP 108756]
[New LWP 108755]
[New LWP 108754]
[New LWP 108753]
[New LWP 108752]
[New LWP 108751]
[New LWP 108750]
[New LWP 108749]
[New LWP 108748]
[New LWP 108747]
[New LWP 108746]
[New LWP 108745]
[New LWP 108744]
[New LWP 108641]
[New LWP 108640]
[New LWP 108413]
[New LWP 108412]
[New LWP 108380]
[New LWP 108375]
[New LWP 108372]
[New LWP 108370]
[New LWP 108364]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libunbound.so.2...Reading symbols from /lib64/libunbound.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libunbound.so.2
Reading symbols from /lib64/libnuma.so.1...Reading symbols from /lib64/libnuma.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnuma.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libgssapi_krb5.so.2...Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libevent-2.0.so.5...Reading symbols from /lib64/libevent-2.0.so.5...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libevent-2.0.so.5
Reading symbols from /lib64/libpython2.7.so.1.0...Reading symbols from /lib64/libpython2.7.so.1.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libpython2.7.so.1.0
Reading symbols from /lib64/libkrb5support.so.0...Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libselinux.so.1...Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libpcre.so.1...Reading symbols from /lib64/libpcre.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libpcre.so.1
Reading symbols from /usr/lib64/openvswitch/librte_pmd_mlx4_glue.so.17.11.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/openvswitch/librte_pmd_mlx4_glue.so.17.11.1
Reading symbols from /lib64/libibverbs.so.1...Reading symbols from /lib64/libibverbs.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libibverbs.so.1
Reading symbols from /lib64/libmlx4.so.1...Reading symbols from /lib64/libmlx4.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libmlx4.so.1
Reading symbols from /lib64/libnl-route-3.so.200...Reading symbols from /lib64/libnl-route-3.so.200...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnl-route-3.so.200
Reading symbols from /lib64/libnl-3.so.200...Reading symbols from /lib64/libnl-3.so.200...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnl-3.so.200
Reading symbols from /usr/lib64/openvswitch/librte_pmd_mlx5_glue.so.17.11.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/openvswitch/librte_pmd_mlx5_glue.so.17.11.1
Reading symbols from /lib64/libmlx5.so.1...Reading symbols from /lib64/libmlx5.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libmlx5.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
0x00007f6cf4a8820d in poll () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.17-260.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-33.el7.x86_64 libcap-ng-0.7.5-4.el7.x86_64 libcom_err-1.42.9-13.el7.x86_64 libevent-2.0.21-4.el7.x86_64 libgcc-4.8.5-36.el7.x86_64 libibverbs-17.2-3.el7.x86_64 libnl3-3.2.28-4.el7.x86_64 libselinux-2.5-14.1.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64 openssl-libs-1.0.2k-15.el7.x86_64 pcre-8.32-17.el7.x86_64 python-libs-2.7.5-75.el7.x86_64 unbound-libs-1.6.6-1.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) c
Continuing.
[New Thread 0x7f6cfc1f8c80 (LWP 109070)]
[Thread 0x7f6cfc1f8c80 (LWP 109070) exited]
[New Thread 0x7f6cfc1f8c80 (LWP 109148)]
[Thread 0x7f6cfc1f8c80 (LWP 109148) exited]
[New Thread 0x7f6cfc1f8c80 (LWP 109153)]
[Thread 0x7f6cfc1f8c80 (LWP 109153) exited]
[New Thread 0x7f6cfc1f8c80 (LWP 109160)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f6cb5ffb700 (LWP 108413)]
0x00007f6cf6b95f53 in qva_to_vva (dev=0x7f69bfffe740, qva=0, len=0x7f6cb5ffa4f0)
    at /usr/src/debug/openvswitch-2.10.0/dpdk-17.11/lib/librte_vhost/vhost_user.c:373
373		for (i = 0; i < dev->mem->nregions; i++) {
(gdb) bt
#0  0x00007f6cf6b95f53 in qva_to_vva (dev=0x7f69bfffe740, qva=0, len=0x7f6cb5ffa4f0)
    at /usr/src/debug/openvswitch-2.10.0/dpdk-17.11/lib/librte_vhost/vhost_user.c:373
#1  0x00007f6cf6b96253 in translate_ring_addresses (dev=dev@entry=0x7f69bfffe740, 
    vq_index=vq_index@entry=0)
    at /usr/src/debug/openvswitch-2.10.0/dpdk-17.11/lib/librte_vhost/vhost_user.c:427
#2  0x00007f6cf6b96cae in vhost_user_set_vring_kick (pmsg=0x7f6cb5ffa7a0, 
    pdev=<synthetic pointer>)
    at /usr/src/debug/openvswitch-2.10.0/dpdk-17.11/lib/librte_vhost/vhost_user.c:832
#3  vhost_user_msg_handler (vid=<optimized out>, fd=fd@entry=78)
    at /usr/src/debug/openvswitch-2.10.0/dpdk-17.11/lib/librte_vhost/vhost_user.c:1445
#4  0x00007f6cf6b9424f in vhost_user_read_cb (connfd=78, dat=0x7f6c480008c0, 
    remove=0x7f6cb5ffa990)
    at /usr/src/debug/openvswitch-2.10.0/dpdk-17.11/lib/librte_vhost/socket.c:292
#5  0x00007f6cf6b92c18 in fdset_event_dispatch (arg=0x7f6cf6da51c0 <vhost_user+8192>)
    at /usr/src/debug/openvswitch-2.10.0/dpdk-17.11/lib/librte_vhost/fd_man.c:276
#6  0x00007f6cf5780dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f6cf4a92ead in clone () from /lib64/libc.so.6
(gdb) quit
A debugging session is active.

	Inferior 1 [process 108362] will be detached.

Quit anyway? (y or n) y
Detaching from program: /usr/sbin/ovs-vswitchd, process 108362
[root@netqe9 ~]# 

NOTE: OVS-dpdk is running in dpdkvhostuserclient mode.

Version-Release number of selected component (if applicable):
[root@netqe9 ~]# rpm -q openvswitch2.10
openvswitch2.10-2.10.0-1.el7fdp.x86_64
[root@netqe9 ~]# rpm -qa | grep qemu
libvirt-daemon-driver-qemu-4.5.0-6.el7.x86_64
qemu-kvm-rhev-2.10.0-21.el7.x86_64
qemu-kvm-common-rhev-2.10.0-21.el7.x86_64
qemu-img-rhev-2.10.0-21.el7.x86_64
ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
[root@netqe9 ~]# rpm -q libvirt
libvirt-4.5.0-6.el7.x86_64
[root@netqe9 ~]# !un
uname -a
Linux netqe9.knqe.lab.eng.bos.redhat.com 3.10.0-933.el7.x86_64 #1 SMP Sat Aug 11 11:32:57 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@netqe9 ~]# 


How reproducible: Reproducible


Steps to Reproduce:
1. Config an OVS-dpdk bridge with a vhost port in dpdkvhostuserclient mode
2. virsh start the guest
3.

Actual results:


Expected results:


Additional info:

Comment 3 Maxime Coquelin 2018-08-23 21:20:05 UTC
Hi Jean-Tsung,

Could you please share the OVS logs so that I can check the vhost-user
request received?

Thanks,
Maxime

Comment 4 Jean-Tsung Hsiao 2018-08-24 00:03:21 UTC
(In reply to Maxime Coquelin from comment #3)
> Hi Jean-Tsung,
> 
> Could you please share the OVS logs so that I can check the vhost-user
> request received?
> 
> Thanks,
> Maxime

Hi Maxime,
Just reproduced it. Got the following piece of log when "virsh start <guest>"; and, the daemon got segfaulted.

2018-08-23T23:57:31.175Z|00108|netdev_linux|INFO|ioctl(SIOCGIFINDEX) on vxlan_sys_4789 device failed: No such device
2018-08-23T23:57:31.216Z|00109|netdev_linux|INFO|ioctl(SIOCGIFINDEX) on vxlan_sys_4789 device failed: No such device
2018-08-23T23:57:31.471Z|00001|dpdk|INFO|VHOST_CONFIG: /tmp/vhost0: connected
2018-08-23T23:57:31.471Z|00002|dpdk|INFO|VHOST_CONFIG: new device, handle is 0
2018-08-23T23:57:31.495Z|00001|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
2018-08-23T23:57:31.495Z|00002|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES
2018-08-23T23:57:31.495Z|00003|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES
2018-08-23T23:57:31.495Z|00004|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_QUEUE_NUM
2018-08-23T23:57:31.495Z|00005|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_SLAVE_REQ_FD
2018-08-23T23:57:31.495Z|00006|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_OWNER
2018-08-23T23:57:31.495Z|00007|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
2018-08-23T23:57:31.496Z|00008|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
2018-08-23T23:57:31.496Z|00009|dpdk|INFO|VHOST_CONFIG: vring call idx:0 file:81
2018-08-23T23:57:31.496Z|00010|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
2018-08-23T23:57:31.496Z|00011|dpdk|INFO|VHOST_CONFIG: vring call idx:1 file:82
2018-08-23T23:57:31.503Z|00110|netdev_linux|INFO|ioctl(SIOCGIFINDEX) on vxlan_sys_4789 device failed: No such device
2018-08-23T23:57:31.548Z|00111|netdev_linux|INFO|ioctl(SIOCGIFINDEX) on vxlan_sys_4789 device failed: No such device
2018-08-23T23:57:33.197Z|00112|netdev_linux|INFO|ioctl(SIOCGIFINDEX) on vxlan_sys_4789 device failed: No such device
2018-08-23T23:57:35.196Z|00113|netdev_linux|INFO|ioctl(SIOCGIFINDEX) on vxlan_sys_4789 device failed: No such device
2018-08-23T23:57:35.242Z|00114|netdev_linux|INFO|ioctl(SIOCGIFINDEX) on vxlan_sys_4789 device failed: No such device
2018-08-23T23:57:40.691Z|00012|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
2018-08-23T23:57:40.693Z|00013|dpdk|INFO|VHOST_CONFIG: set queue enable: 1 to qp idx: 0
2018-08-23T23:57:40.693Z|00014|netdev_dpdk|INFO|State of queue 0 ( tx_qid 0 ) of vhost device '/tmp/vhost0'changed to 'enabled'
2018-08-23T23:57:40.693Z|00015|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
2018-08-23T23:57:40.693Z|00016|dpdk|INFO|VHOST_CONFIG: set queue enable: 1 to qp idx: 1
2018-08-23T23:57:40.693Z|00017|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
2018-08-23T23:57:40.693Z|00018|dpdk|INFO|VHOST_CONFIG: set queue enable: 1 to qp idx: 0
2018-08-23T23:57:40.693Z|00019|netdev_dpdk|INFO|State of queue 0 ( tx_qid 0 ) of vhost device '/tmp/vhost0'changed to 'enabled'
2018-08-23T23:57:40.693Z|00020|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
2018-08-23T23:57:40.693Z|00021|dpdk|INFO|VHOST_CONFIG: set queue enable: 1 to qp idx: 1

Comment 5 Maxime Coquelin 2018-08-24 07:31:17 UTC
(In reply to Jean-Tsung Hsiao from comment #4)
> (In reply to Maxime Coquelin from comment #3)
> > Hi Jean-Tsung,
> > 
> > Could you please share the OVS logs so that I can check the vhost-user
> > request received?
> > 
> > Thanks,
> > Maxime
> 
> Hi Maxime,
> Just reproduced it. Got the following piece of log when "virsh start
> <guest>"; and, the daemon got segfaulted.

Thanks Jean-Tsung,

It seems the trace wasn't fully evacuated into the journal.

Any chance I can connect to your setup?

Comment 6 Maxime Coquelin 2018-08-24 14:30:02 UTC
So I managed to reproduce on Jean's setup.

The crash happens because mmap of guest memory fails due to SElinux policy:

ovs-vswitchd[60356]: ovs|00024|dpdk|ERR|VHOST_CONFIG: mmap region 0 failed.

and:

type=AVC msg=audit(1535118626.626:515): avc:  denied  { map } for  pid=60356 comm="ovs-vswitchd" path=2F6465762F6875676570616765732F6C6962766972742F71656D752F332D6D712D7668752D74756E6E656C2D7365727665722F71656D755F6261636B5F6D656D2E5F6F626A656374735F72616D2D6E6F6465302E79427A6E3162202864656C6574656429 dev="hugetlbfs" ino=4125881 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:svirt_image_t:s0 tclass=file permissive=0

Jean already filed a Bz for this SELinux policy issue (Bz1620257).

However, the vhost-user library should be more robust, so that it handles
gracefully SELinux policy issues.
I'll keep this Bz to make vhost-user more robust, but lower the priority.

Comment 7 Jean-Tsung Hsiao 2018-08-28 14:15:17 UTC
Hi Aaron,
See Comment #6 for the same AVC listed in bug 1620257.
Thanks!
Jean

Comment 8 Jean-Tsung Hsiao 2018-08-29 02:13:13 UTC
*** Bug 1601134 has been marked as a duplicate of this bug. ***

Comment 10 Maxime Coquelin 2018-11-27 09:24:52 UTC
Patch posted upstream:
http://patches.dpdk.org/patch/48346/

Comment 12 Maxime Coquelin 2019-07-04 14:51:05 UTC
The patch is part of v18.11.1 upstream.

So making it depending on Bz1713698 (Rebase to 18.11.2) and set FDP release to 19.E.

Comment 14 Jean-Tsung Hsiao 2019-07-24 15:07:40 UTC
Have verified and passed the fix with OVS-2.11.0-18.

Comment 16 errata-xmlrpc 2019-08-19 16:20:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2522


Note You need to log in before you can comment on or make changes to this bug.