Red Hat Bugzilla – Bug 1494616
OVS daemon got seqfaults inside qede set mtu
Last modified: 2018-06-21 09:37:50 EDT
Description of problem: OVS daemon got seqfaults inside qede set mtu aconole collects the following dump trace: Zone 36: name:<1e4bf7ed4b264>, phys:0xcffb8b3c0, len:0x8000, virt:0x7ffcbfb8b3c0, socket_id:0, flags:0 [New Thread 0x7ffff3164700 (LWP 31408)] [New Thread 0x7ffff2963700 (LWP 31409)] [New Thread 0x7ffff1961700 (LWP 31411)] [New Thread 0x7ffff2162700 (LWP 31410)] [New Thread 0x7ffff0cbf700 (LWP 31412)] [New Thread 0x7fffe3fff700 (LWP 31413)] [New Thread 0x7fffe37fe700 (LWP 31414)] [New Thread 0x7fffe2ffd700 (LWP 31415)] [New Thread 0x7fffe27fc700 (LWP 31416)] [New Thread 0x7fffe1ffb700 (LWP 31417)] [New Thread 0x7fffe17fa700 (LWP 31418)] [New Thread 0x7fffe0ff9700 (LWP 31419)] [New Thread 0x7fffc3fff700 (LWP 31420)] [New Thread 0x7fffc37fe700 (LWP 31421)] [New Thread 0x7fffc2ffd700 (LWP 31422)] [New Thread 0x7fffc27fc700 (LWP 31423)] [New Thread 0x7fffc1ffb700 (LWP 31424)] [New Thread 0x7fffc17fa700 (LWP 31425)] [New Thread 0x7fffc0ff9700 (LWP 31426)] [New Thread 0x7fffa3fff700 (LWP 31427)] [New Thread 0x7fffa37fe700 (LWP 31428)] [New Thread 0x7fffa2ffd700 (LWP 31429)] [New Thread 0x7fffa27fc700 (LWP 31430)] [New Thread 0x7fffa1ffb700 (LWP 31431)] [New Thread 0x7fffa17fa700 (LWP 31432)] [New Thread 0x7fffa0ff9700 (LWP 31433)] [New Thread 0x7fff83fff700 (LWP 31434)] [New Thread 0x7fff837fe700 (LWP 31435)] [New Thread 0x7fff82ffd700 (LWP 31436)] [New Thread 0x7fff827fc700 (LWP 31437)] [New Thread 0x7fff81ffb700 (LWP 31438)] [New Thread 0x7fff817fa700 (LWP 31439)] Program received signal SIGSEGV, Segmentation fault. 0x0000555555657234 in qede_set_mtu () (gdb) bt #0 0x0000555555657234 in qede_set_mtu () #1 0x00005555555e4991 in rte_eth_dev_set_mtu () #2 0x00005555558b6815 in dpdk_eth_dev_queue_setup () #3 0x00005555558b91c6 in netdev_dpdk_reconfigure () #4 0x00005555557e2bac in reconfigure_datapath () #5 0x00005555557e3657 in do_add_port () #6 0x00005555557e37dd in dpif_netdev_port_add () #7 0x00005555557e972e in dpif_port_add () #8 0x000055555579ca60 in port_add () #9 0x0000555555793431 in ofproto_port_add () #10 0x00005555557813e5 in bridge_add_ports__ () #11 0x0000555555783048 in bridge_reconfigure () #12 0x00005555557865a9 in bridge_run () #13 0x0000555555595155 in main () Version-Release number of selected component (if applicable): [root@netqe9 jhsiao]# uname -a Linux netqe9.knqe.lab.eng.bos.redhat.com 3.10.0-693.el7.x86_64 #1 SMP Thu Jul 6 19:56:57 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux [root@netqe9 jhsiao]# rpm -q openvswitch openvswitch-2.8.0-0.3.20170920git436b6fe.el7fdb.x86_64 Ethernet controller: QLogic Corp. FastLinQ QL45000 Series 25GbE Controller (rev 10) How reproducible:Reproducible Steps to Reproduce: 1.Add 25 Gb qede to OVS-dpdk bridge 2. 3. Actual results: Expected results: Additional info:
Can you let me know where I can find the core file?
With ovs-2.7.2-10, on adding qede to OVS-dpdk bridge, got the following error: 2017-10-11T03:00:23.105Z|00065|dpdk|WARN|EAL: Requested device 0000:04:00.0 cannot be used 2017-10-11T03:00:23.105Z|00066|dpdk|ERR|EAL: Driver cannot attach the device (0000:04:00.0) 2017-10-11T03:00:23.105Z|00067|netdev_dpdk|WARN|Error attaching device '0000:04:00.0' to DPDK 2017-10-11T03:00:23.105Z|00068|netdev|WARN|dpdk-10: could not set configuration (Invalid argument)
Created attachment 1337364 [details] ovs-vswitchd core file Reproduced the issue with ovs-2.8.0-1 --- crashed set mtu.
[root@netqe9 coredump]# coredumpctl gdb 20680 PID: 20680 (ovs-vswitchd) UID: 0 (root) GID: 0 (root) Signal: 11 (SEGV) Timestamp: Wed 2017-10-11 13:26:16 EDT (11min ago) Command Line: ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user root:root --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach Executable: /usr/sbin/ovs-vswitchd Control Group: /system.slice/ovs-vswitchd.service Unit: ovs-vswitchd.service Slice: system.slice Boot ID: b2efb7498a3f435294ab8ec1860aa8d4 Machine ID: 3074afd2ce4e4151bdfd6f8ea64c722f Hostname: netqe9.knqe.lab.eng.bos.redhat.com Coredump: /var/lib/systemd/coredump/core.ovs-vswitchd.0.b2efb7498a3f435294ab8ec1860aa8d4.20680.1507742776000000.xz Message: Process 20680 (ovs-vswitchd) of user 0 dumped core. Stack trace of thread 20680: #0 0x0000559464396534 qede_set_mtu (ovs-vswitchd) #1 0x0000559464323c41 rte_eth_dev_set_mtu (ovs-vswitchd) #2 0x00005594645f5e85 dpdk_eth_dev_queue_setup (ovs-vswitchd) #3 0x00005594645f8ae6 netdev_dpdk_reconfigure (ovs-vswitchd) #4 0x000055946452225c reconfigure_datapath (ovs-vswitchd) #5 0x0000559464522d07 do_add_port (ovs-vswitchd) #6 0x0000559464522e8d dpif_netdev_port_add (ovs-vswitchd) #7 0x0000559464528dde dpif_port_add (ovs-vswitchd) #8 0x00005594644dc0e0 port_add (ovs-vswitchd) #9 0x00005594644d2ab1 ofproto_port_add (ovs-vswitchd) #10 0x00005594644c0a85 bridge_add_ports__ (ovs-vswitchd) #11 0x00005594644c26e8 bridge_reconfigure (ovs-vswitchd) #12 0x00005594644c5c49 bridge_run (ovs-vswitchd) #13 0x00005594642d4155 main (ovs-vswitchd) #14 0x00007f0e1444bc05 __libc_start_main (libc.so.6) #15 0x00005594642d8328 _start (ovs-vswitchd) Stack trace of thread 20683: #0 0x00007f0e14522923 epoll_wait (libc.so.6) #1 0x0000559464420e3f eal_intr_thread_main (ovs-vswitchd) #2 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #3 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20712: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x0000559464525c6b pmd_thread_main (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20714: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x0000559464525c6b pmd_thread_main (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20684: #0 0x00007f0e14f46b03 recvfrom (libpthread.so.0) #1 0x00005594643b5f81 pdump_thread_main (ovs-vswitchd) #2 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #3 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20713: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x0000559464525c6b pmd_thread_main (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20685: #0 0x00007f0e144e91ad __nanosleep (libc.so.6) #1 0x00007f0e144e9044 sleep (libc.so.6) #2 0x00005594645d45ef xsleep (ovs-vswitchd) #3 0x00005594645f8096 dpdk_watchdog (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20686: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x0000559464623ea3 clean_thread_main (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20687: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x000055946459edf3 ovsrcu_postpone_thread (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20715: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x0000559464525c6b pmd_thread_main (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20688: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594644f4741 udpif_upcall_handler (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20689: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594644f4741 udpif_upcall_handler (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20681: #0 0x00007f0e14f4698d accept (libpthread.so.0) #1 0x000055946441d6b1 vfio_mp_sync_thread (ovs-vswitchd) #2 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #3 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20690: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594644f4741 udpif_upcall_handler (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20691: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594644f4741 udpif_upcall_handler (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20692: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594644f4741 udpif_upcall_handler (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20693: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594644f4741 udpif_upcall_handler (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20694: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594644f4741 udpif_upcall_handler (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20695: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594644f4741 udpif_upcall_handler (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20696: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594644f4741 udpif_upcall_handler (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20697: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594644f4741 udpif_upcall_handler (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20698: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594644f4741 udpif_upcall_handler (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20699: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594644f4741 udpif_upcall_handler (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20700: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594644f4741 udpif_upcall_handler (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20701: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594644f4741 udpif_upcall_handler (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20702: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594644f4741 udpif_upcall_handler (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20703: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594644f4741 udpif_upcall_handler (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20704: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594644f4741 udpif_upcall_handler (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20705: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594644f5164 udpif_revalidator (ovs-vswitchd) #4 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #5 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #6 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20706: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594645a0d88 ovs_barrier_block (ovs-vswitchd) #4 0x00005594644f501b udpif_revalidator (ovs-vswitchd) #5 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #6 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #7 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20707: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594645a0d88 ovs_barrier_block (ovs-vswitchd) #4 0x00005594644f501b udpif_revalidator (ovs-vswitchd) #5 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #6 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #7 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20708: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594645a0d88 ovs_barrier_block (ovs-vswitchd) #4 0x00005594644f501b udpif_revalidator (ovs-vswitchd) #5 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #6 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #7 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20709: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594645a0d88 ovs_barrier_block (ovs-vswitchd) #4 0x00005594644f501b udpif_revalidator (ovs-vswitchd) #5 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #6 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #7 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20710: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594645a0d88 ovs_barrier_block (ovs-vswitchd) #4 0x00005594644f501b udpif_revalidator (ovs-vswitchd) #5 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #6 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #7 0x00007f0e1452234d __clone (libc.so.6) Stack trace of thread 20711: #0 0x00007f0e14517a3d poll (libc.so.6) #1 0x00005594645cae84 time_poll (ovs-vswitchd) #2 0x00005594645b4d4c poll_block (ovs-vswitchd) #3 0x00005594645a0d88 ovs_barrier_block (ovs-vswitchd) #4 0x00005594644f501b udpif_revalidator (ovs-vswitchd) #5 0x00005594645a0256 ovsthread_wrapper (ovs-vswitchd) #6 0x00007f0e14f3fe25 start_thread (libpthread.so.0) #7 0x00007f0e1452234d __clone (libc.so.6) GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/ovs-vswitchd...Reading symbols from /usr/lib/debug/usr/sbin/ovs-vswitchd.debug...done. done. [New LWP 20680] [New LWP 20683] [New LWP 20712] [New LWP 20714] [New LWP 20684] [New LWP 20713] [New LWP 20685] [New LWP 20686] [New LWP 20687] [New LWP 20715] [New LWP 20688] [New LWP 20689] [New LWP 20681] [New LWP 20690] [New LWP 20691] [New LWP 20692] [New LWP 20693] [New LWP 20694] [New LWP 20695] [New LWP 20696] [New LWP 20697] [New LWP 20698] [New LWP 20699] [New LWP 20700] [New LWP 20701] [New LWP 20702] [New LWP 20703] [New LWP 20704] [New LWP 20705] [New LWP 20706] [New LWP 20707] [New LWP 20708] [New LWP 20709] [New LWP 20710] [New LWP 20711] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfi'. Program terminated with signal 11, Segmentation fault. #0 qede_set_mtu (dev=0x559464c99740 <rte_eth_devices>, mtu=<optimized out>) at /usr/src/debug/openvswitch-2.8.0/dpdk-stable-17.05.2/drivers/net/qede/qede_ethdev.c:1763 1763 bufsz = (uint16_t)rte_pktmbuf_data_room_size( Missing separate debuginfos, use: debuginfo-install glibc-2.17-196.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-8.el7.x86_64 libcap-ng-0.7.5-4.el7.x86_64 libcom_err-1.42.9-10.el7.x86_64 libgcc-4.8.5-16.el7.x86_64 libpcap-1.5.3-9.el7.x86_64 libselinux-2.5-11.el7.x86_64 numactl-libs-2.0.9-6.el7_2.x86_64 openssl-libs-1.0.2k-8.el7.x86_64 pcre-8.32-17.el7.x86_64 zlib-1.2.7-17.el7.x86_64 (gdb) p qdev $1 = (struct qede_dev *) 0x7f0afffcd200 (gdb) p *qdev Cannot access memory at address 0x7f0afffcd200 (gdb) p edev $2 = (struct ecore_dev *) 0x7f0afffcd200 (gdb) p *edev Cannot access memory at address 0x7f0afffcd200 (gdb) p dev_info $3 = {pci_dev = 0x5594662637b0, driver_name = 0x0, if_index = 0, min_rx_bufsize = 1024, max_rx_pktlen = 9672, max_rx_queues = 32, max_tx_queues = 32, max_mac_addrs = 512, max_hash_mac_addrs = 0, max_vfs = 0, max_vmdq_pools = 0, rx_offload_capa = 95, tx_offload_capa = 687, reta_size = 128, hash_key_size = 40 '(', flow_type_rss_offloads = 527668, default_rxconf = {rx_thresh = { pthresh = 0 '\000', hthresh = 0 '\000', wthresh = 0 '\000'}, rx_free_thresh = 0, rx_drop_en = 0 '\000', rx_deferred_start = 0 '\000'}, default_txconf = {tx_thresh = {pthresh = 0 '\000', hthresh = 0 '\000', wthresh = 0 '\000'}, tx_rs_thresh = 0, tx_free_thresh = 0, txq_flags = 1, tx_deferred_start = 0 '\000'}, vmdq_queue_base = 0, vmdq_queue_num = 0, vmdq_pool_base = 0, rx_desc_lim = {nb_max = 65535, nb_min = 128, nb_align = 128, nb_seg_max = 0, nb_mtu_seg_max = 0}, tx_desc_lim = {nb_max = 65535, nb_min = 256, nb_align = 256, nb_seg_max = 255, nb_mtu_seg_max = 18}, speed_capa = 1280, nb_rx_queues = 0, nb_tx_queues = 0} (gdb) p dev->data $4 = (struct rte_eth_dev_data *) 0x7f0afffced40 (gdb) p *dev->data Cannot access memory at address 0x7f0afffced40 (gdb) p i $5 = <optimized out> (gdb) p fp $6 = (struct qede_fastpath *) 0x7f0ad0605700 (gdb) p *fp Cannot access memory at address 0x7f0ad0605700 (gdb) bt #0 qede_set_mtu (dev=0x559464c99740 <rte_eth_devices>, mtu=<optimized out>) at /usr/src/debug/openvswitch-2.8.0/dpdk-stable-17.05.2/drivers/net/qede/qede_ethdev.c:1763 #1 0x0000559464323c41 in rte_eth_dev_set_mtu (port_id=<optimized out>, mtu=<optimized out>) at /usr/src/debug/openvswitch-2.8.0/dpdk-stable-17.05.2/lib/librte_ether/rte_ethdev.c:1967 #2 0x00005594645f5e85 in dpdk_eth_dev_queue_setup ( dev=dev@entry=0x7f0affb881c0, n_rxq=n_rxq@entry=1, n_txq=n_txq@entry=5) at lib/netdev-dpdk.c:699 #3 0x00005594645f8ae6 in dpdk_eth_dev_init (dev=0x7f0affb881c0) at lib/netdev-dpdk.c:784 #4 netdev_dpdk_reconfigure (netdev=0x7f0affb881c0) at lib/netdev-dpdk.c:3191 #5 0x000055946452225c in port_reconfigure (port=0x5594662bdc80) at lib/dpif-netdev.c:3161 #6 reconfigure_datapath (dp=dp@entry=0x7f0e15f7e010) at lib/dpif-netdev.c:3547 #7 0x0000559464522d07 in do_add_port (dp=dp@entry=0x7f0e15f7e010, devname=devname@entry=0x5594662bdc20 "dpdk-10", type=0x55946465a524 "dpdk", port_no=port_no@entry=1) at lib/dpif-netdev.c:1506 #8 0x0000559464522e8d in dpif_netdev_port_add (dpif=<optimized out>, netdev=0x7f0affb881c0, port_nop=0x7ffd5c57553c) at lib/dpif-netdev.c:1532 ---Type <return> to continue, or q <return> to quit--- #9 0x0000559464528dde in dpif_port_add (dpif=0x55946625aa90, netdev=netdev@entry=0x7f0affb881c0, port_nop=port_nop@entry=0x7ffd5c57559c) at lib/dpif.c:564 #10 0x00005594644dc0e0 in port_add (ofproto_=0x55946625d110, netdev=0x7f0affb881c0) at ofproto/ofproto-dpif.c:3667 #11 0x00005594644d2ab1 in ofproto_port_add (ofproto=0x55946625d110, netdev=0x7f0affb881c0, ofp_portp=0x7ffd5c575698) at ofproto/ofproto.c:2003 #12 0x00005594644c0a85 in bridge_add_ports__ (br=br@entry=0x55946625caf0, wanted_ports=wanted_ports@entry=0x55946625cbd0, with_requested_port=with_requested_port@entry=true) at vswitchd/bridge.c:1799 #13 0x00005594644c26e8 in bridge_add_ports (wanted_ports=0x55946625cbd0, br=0x55946625caf0) at vswitchd/bridge.c:942 #14 bridge_reconfigure (ovs_cfg=0x559466281270) at vswitchd/bridge.c:663 #15 0x00005594644c5c49 in bridge_run () at vswitchd/bridge.c:2992 #16 0x00005594642d4155 in main (argc=12, argv=0x7ffd5c575ba8) at vswitchd/ovs-vswitchd.c:119 (gdb) f 4 #4 netdev_dpdk_reconfigure (netdev=0x7f0affb881c0) at lib/netdev-dpdk.c:3191 3191 err = dpdk_eth_dev_init(dev); (gdb) p dev $7 = <optimized out> (gdb) p *dev value has been optimized out (gdb) p dev $8 = <optimized out> (gdb) p *netdev Cannot access memory at address 0x7f0affb881c0 (gdb) My guess is that something isn't setup correctly when the reconfigure request is issued.
The problem is because qede_set_mtu() assumes that qdev->fp_array[*]->rxq are already allocated, but buffers are allocated in qede_init_fp_queue(). This worked until backport of Author: Mark Kavanagh <mark.b.kavanagh@intel.com> Date: Tue Aug 1 15:03:08 2017 -0700 netdev-dpdk: use rte_eth_dev_set_mtu. which added a call to rte_eth_dev_set_mtu() in dpdk_eth_dev_queue_setup() which result in a crash. A possile fix could be moving the rte_eth_dev_set_mtu() just after the rte_eth_dev_start() call, which allocates the fastpath descriptors
Created attachment 1350621 [details] proposed fix
Upstream patch: https://mail.openvswitch.org/pipermail/ovs-dev/2017-November/340983.html
Hi Christian, Based on you comment to this other bz ( https://bugzilla.redhat.com/show_bug.cgi?id=1531145#c10 ) I assume that this one will also fixed by the new package?
Hi Matteo, We should verify on the Q Logic 45000 series as that was what the bug was originally reported on. Bug 1531145 was effecting both cards (41000 and 45000) but I'd just like to run it on a 45000x to confirm. We are having trouble with our systems right now with the 45000x so once we have fixed those systems we will double check with that RPM on that card. Thanks. Jean once those systems are functional again please verify this bug is no longer present with the rpm provided in bug 1531145.
NOTE: With openvswitch-2.9.0-0.4.20180124git26cdc33.el7fdp did not see Ramrod error, and the daemon did not crash 2018-02-09T00:58:55.998Z|00046|memory|INFO|50240 kB peak resident set size after 55.0 seconds 2018-02-09T00:58:56.059Z|00047|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports recirculation 2018-02-09T00:58:56.059Z|00048|ofproto_dpif|INFO|netdev@ovs-netdev: VLAN header stack length probed as 1 2018-02-09T00:58:56.059Z|00049|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS label stack length probed as 3 2018-02-09T00:58:56.059Z|00050|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports truncate action 2018-02-09T00:58:56.059Z|00051|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports unique flow ids 2018-02-09T00:58:56.059Z|00052|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports clone action 2018-02-09T00:58:56.059Z|00053|ofproto_dpif|INFO|netdev@ovs-netdev: Max sample nesting level probed as 10 2018-02-09T00:58:56.059Z|00054|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports eventmask in conntrack action 2018-02-09T00:58:56.059Z|00055|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_clear action 2018-02-09T00:58:56.059Z|00056|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_state 2018-02-09T00:58:56.059Z|00057|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_zone 2018-02-09T00:58:56.059Z|00058|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_mark 2018-02-09T00:58:56.059Z|00059|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_label 2018-02-09T00:58:56.059Z|00060|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_state_nat 2018-02-09T00:58:56.059Z|00061|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_orig_tuple 2018-02-09T00:58:56.059Z|00062|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_orig_tuple6 2018-02-09T00:58:56.159Z|00063|bridge|INFO|bridge ovsbr0: added interface ovsbr0 on port 65534 2018-02-09T00:58:56.169Z|00064|bridge|INFO|bridge ovsbr0: using datapath ID 0000ce46cff6ec4a 2018-02-09T00:58:56.169Z|00065|connmgr|INFO|ovsbr0: added service controller "punix:/var/run/openvswitch/ovsbr0.mgmt" 2018-02-09T00:58:56.327Z|00066|dpdk|ERR|EAL: Driver cannot attach the device (0000:04:00.0) 2018-02-09T00:58:56.327Z|00067|dpdk|ERR|EAL: No port found for device (0000:04:00.0) 2018-02-09T00:58:56.327Z|00068|netdev_dpdk|WARN|Error attaching device '0000:04:00.0' to DPDK 2018-02-09T00:58:56.327Z|00069|netdev|WARN|dpdk-10: could not set configuration (Invalid argument) 2018-02-09T00:58:56.403Z|00070|dpdk|ERR|EAL: Driver cannot attach the device (0000:04:00.0) 2018-02-09T00:58:56.404Z|00071|dpdk|ERR|EAL: No port found for device (0000:04:00.0) 2018-02-09T00:58:56.404Z|00072|netdev_dpdk|WARN|Error attaching device '0000:04:00.0' to DPDK 2018-02-09T00:58:56.404Z|00073|netdev|WARN|dpdk-10: could not set configuration (Invalid argument) 2018-02-09T00:58:56.467Z|00074|dpdk|ERR|EAL: Driver cannot attach the device (0000:04:00.1) 2018-02-09T00:58:56.467Z|00075|dpdk|ERR|EAL: No port found for device (0000:04:00.1) 2018-02-09T00:58:56.467Z|00076|netdev_dpdk|WARN|Error attaching device '0000:04:00.1' to DPDK 2018-02-09T00:58:56.468Z|00077|netdev|WARN|dpdk-11: could not set configuration (Invalid argument) 2018-02-09T00:58:56.542Z|00078|dpdk|ERR|EAL: Driver cannot attach the device (0000:04:00.0) 2018-02-09T00:58:56.542Z|00079|dpdk|ERR|EAL: No port found for device (0000:04:00.0) 2018-02-09T00:58:56.542Z|00080|netdev_dpdk|WARN|Error attaching device '0000:04:00.0' to DPDK 2018-02-09T00:58:56.542Z|00081|netdev|WARN|dpdk-10: could not set configuration (Invalid argument) 2018-02-09T00:58:56.542Z|00082|dpdk|INFO|VHOST_CONFIG: vhost-user server: socket created, fd: 55 2018-02-09T00:58:56.542Z|00083|netdev_dpdk|INFO|Socket /var/run/openvswitch/vhost0 created for vhost-user port vhost0 2018-02-09T00:58:56.545Z|00084|dpdk|INFO|VHOST_CONFIG: bind to /var/run/openvswitch/vhost0 2018-02-09T00:58:56.545Z|00085|netdev_dpdk|WARN|dpdkvhostuser ports are considered deprecated; please migrate to dpdkvhostuserclient ports. 2018-02-09T00:58:56.550Z|00086|dpif_netdev|INFO|PMD thread on numa_id: 0, core id: 10 created. 2018-02-09T00:58:56.555Z|00087|dpif_netdev|INFO|PMD thread on numa_id: 0, core id: 22 created. 2018-02-09T00:58:56.555Z|00088|dpif_netdev|INFO|There are 2 pmd threads on numa node 0 2018-02-09T00:58:56.570Z|00089|dpif_netdev|INFO|Core 10 on numa node 0 assigned port 'vhost0' rx queue 0 (measured processing cycles 0). 2018-02-09T00:58:56.571Z|00090|bridge|INFO|bridge ovsbr0: added interface vhost0 on port 20 2018-02-09T00:58:56.624Z|00091|dpdk|ERR|EAL: Driver cannot attach the device (0000:04:00.1) 2018-02-09T00:58:56.624Z|00092|dpdk|ERR|EAL: No port found for device (0000:04:00.1) 2018-02-09T00:58:56.624Z|00093|netdev_dpdk|WARN|Error attaching device '0000:04:00.1' to DPDK 2018-02-09T00:58:56.624Z|00094|netdev|WARN|dpdk-11: could not set configuration (Invalid argument) 2018-02-09T00:58:56.682Z|00095|dpdk|ERR|EAL: Driver cannot attach the device (0000:04:00.0) 2018-02-09T00:58:56.682Z|00096|dpdk|ERR|EAL: No port found for device (0000:04:00.0) 2018-02-09T00:58:56.682Z|00097|netdev_dpdk|WARN|Error attaching device '0000:04:00.0' to DPDK 2018-02-09T00:58:56.682Z|00098|netdev|WARN|dpdk-10: could not set configuration (Invalid argument) 2018-02-09T00:58:56.682Z|00099|dpdk|INFO|VHOST_CONFIG: vhost-user server: socket created, fd: 64 2018-02-09T00:58:56.683Z|00100|netdev_dpdk|INFO|Socket /var/run/openvswitch/vhost1 created for vhost-user port vhost1 2018-02-09T00:58:56.683Z|00101|dpdk|INFO|VHOST_CONFIG: bind to /var/run/openvswitch/vhost1 2018-02-09T00:58:56.697Z|00102|dpif_netdev|INFO|Core 10 on numa node 0 assigned port 'vhost0' rx queue 0 (measured processing cycles 0). 2018-02-09T00:58:56.697Z|00103|dpif_netdev|INFO|Core 22 on numa node 0 assigned port 'vhost1' rx queue 0 (measured processing cycles 0). 2018-02-09T00:58:56.697Z|00104|bridge|INFO|bridge ovsbr0: added interface vhost1 on port 21 2018-02-09T00:58:56.747Z|00105|dpdk|ERR|EAL: Driver cannot attach the device (0000:04:00.1) 2018-02-09T00:58:56.747Z|00106|dpdk|ERR|EAL: No port found for device (0000:04:00.1) 2018-02-09T00:58:56.747Z|00107|netdev_dpdk|WARN|Error attaching device '0000:04:00.1' to DPDK 2018-02-09T00:58:56.747Z|00108|netdev|WARN|dpdk-11: could not set configuration (Invalid argument) 2018-02-09T00:58:56.805Z|00109|dpdk|ERR|EAL: Driver cannot attach the device (0000:04:00.0) 2018-02-09T00:58:56.805Z|00110|dpdk|ERR|EAL: No port found for device (0000:04:00.0) 2018-02-09T00:58:56.805Z|00111|netdev_dpdk|WARN|Error attaching device '0000:04:00.0' to DPDK 2018-02-09T00:58:56.805Z|00112|netdev|WARN|dpdk-10: could not set configuration (Invalid argument) 2018-02-09T00:58:56.852Z|00113|dpdk|ERR|EAL: Driver cannot attach the device (0000:04:00.1) 2018-02-09T00:58:56.852Z|00114|dpdk|ERR|EAL: No port found for device (0000:04:00.1) 2018-02-09T00:58:56.852Z|00115|netdev_dpdk|WARN|Error attaching device '0000:04:00.1' to DPDK 2018-02-09T00:58:56.852Z|00116|netdev|WARN|dpdk-11: could not set configuration (Invalid argument) 2018-02-09T00:58:56.909Z|00117|dpdk|ERR|EAL: Driver cannot attach the device (0000:04:00.0) 2018-02-09T00:58:56.909Z|00118|dpdk|ERR|EAL: No port found for device (0000:04:00.0) 2018-02-09T00:58:56.909Z|00119|netdev_dpdk|WARN|Error attaching device '0000:04:00.0' to DPDK 2018-02-09T00:58:56.909Z|00120|netdev|WARN|dpdk-10: could not set configuration (Invalid argument) 2018-02-09T00:58:56.956Z|00121|dpdk|ERR|EAL: Driver cannot attach the device (0000:04:00.1) 2018-02-09T00:58:56.956Z|00122|dpdk|ERR|EAL: No port found for device (0000:04:00.1) 2018-02-09T00:58:56.956Z|00123|netdev_dpdk|WARN|Error attaching device '0000:04:00.1' to DPDK 2018-02-09T00:58:56.956Z|00124|netdev|WARN|dpdk-11: could not set configuration (Invalid argument) 2018-02-09T00:58:56.974Z|00125|connmgr|INFO|ovsbr0<->unix#1: 1 flow_mods in the last 0 s (1 deletes) 2018-02-09T00:58:56.980Z|00126|connmgr|INFO|ovsbr0<->unix#3: 1 flow_mods in the last 0 s (1 adds) 2018-02-09T00:58:56.987Z|00127|connmgr|INFO|ovsbr0<->unix#5: 1 flow_mods in the last 0 s (1 adds) 2018-02-09T00:58:56.994Z|00128|connmgr|INFO|ovsbr0<->unix#7: 1 flow_mods in the last 0 s (1 adds) 2018-02-09T00:58:57.000Z|00129|connmgr|INFO|ovsbr0<->unix#9: 1 flow_mods in the last 0 s (1 adds) 2018-02-09T00:59:06.000Z|00130|memory|INFO|peak resident set size grew 487% in last 10.0 seconds, from 50240 kB to 295116 kB 2018-02-09T00:59:06.000Z|00131|memory|INFO|handlers:17 ports:3 revalidators:7 rules:8
*** Bug 1531145 has been marked as a duplicate of this bug. ***
The fix has been verified. Was able to add qede to OVS. Below is part of "ovs-vsctl list interface dpdk-10" output showing mtu setting was correctly done: mtu : 1500 mtu_request : [] name : "dpdk-10" Also, tested the mtu_request: root@netqe9 ~]# vctl set interface dpdk-10 mtu_request=9000 [root@netqe9 ~]# vctl list interface dpdk-10 _uuid : 1994940c-bfef-499c-b55c-0ec12cf972db admin_state : up bfd : {} bfd_status : {} cfm_fault : [] cfm_fault_status : [] cfm_flap_count : [] cfm_health : [] cfm_mpid : [] cfm_remote_mpids : [] cfm_remote_opstate : [] duplex : full error : [] external_ids : {} ifindex : 1776385 ingress_policing_burst: 0 ingress_policing_rate: 0 lacp_current : [] link_resets : 2 link_speed : 1000000000000 link_state : up lldp : {} mac : [] mac_in_use : "00:0e:1e:d3:f7:1c" mtu : 9000 mtu_request : 9000 name : "dpdk-10"
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1962