Description of problem: When the openvswitch codebase was changed to use fewer file descriptors, it allowed for the kernel to wake more threads simultaneously, which exposed a thundering herd issue in the poll_block() mechanism. Reported upstream by David Ahern. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Dcbw indicated that this bug is needed to reduce / eliminate the packet out of ordering problem. So escalating it here. Related bugs are https://bugzilla.redhat.com/show_bug.cgi?id=1844576 https://bugzilla.redhat.com/show_bug.cgi?id=1834444 Cambria and Dcbw can provide the details. Out of order packets make 3rd party apps like E/// choke. So we need to get packets in order as much as possible before handing them off to pods and containers and VMs.
https://mail.openvswitch.org/pipermail/ovs-dev/2020-July/373146.html Latest version submitted.
https://mail.openvswitch.org/pipermail/ovs-dev/2020-September/375326.html Is the latest version.
This is now at v5 for user space patches: https://patchwork.ozlabs.org/project/openvswitch/list/?series=252282 v1 for kernel space patches: https://marc.info/?l=linux-netdev&m=162504684016825&w=2
Background on testing: Setup two NS to ping back and forth ip netns add left ip netns add right ip link add center-left type veth peer name left0 netns left ip link add center-right type veth peer name right0 netns right ip link set center-left up ip link set center-right up ip -n left ip link set left0 up ip -n left ip addr add 172.31.110.10/24 dev left0 ip -n right ip link set right0 up ip -n right ip addr add 172.31.110.11/24 dev right0 ovs-vsctl add-br br0 ovs-vsctl add-port br0 center-right ovs-vsctl add-port br0 center-left Start recording the wakeups due to packet arrival # in one terminal perf record -e sched:sched_wakeup,irq:softirq_entry -ag Generate ARP request messages # in a separate terminal ip netns exec left arping -I left0 -c 1 172.31.110.11 Use the following to look at the number of handler threads woken # in the perf terminal after exiting perf script This is a simple way of reproduction. Without the entire packge (kernel and userspace changes) we should see multiple handler threads woken up during execution. With the kernel and userspace changes applied, we should see only a single handler thread woken.
It still has the issue that we can see multiple handler threads woken_up modprobe -r veth ip netns del left ip netns del right ip netns add left ip netns add right ip link add center-left type veth peer name left0 netns left ip link add center-right type veth peer name right0 netns right ip link set center-left up ip link set center-right up ip netns exec left ip link set left0 up ip netns exec left ip addr add 172.31.110.10/24 dev left0 ip netns exec right ip link set right0 up ip netns exec right ip addr add 172.31.110.11/24 dev right0 systemctl start openvswitch ovs-vsctl --if-exists del-br br0 ovs-vsctl add-br br0 ovs-vsctl add-port br0 center-right ovs-vsctl add-port br0 center-left perf record -e sched:sched_wakeup,irq:softirq_entry -ag ip netns exec left arping -I left0 -c 1 172.31.110.11 without kernel update --------------------------------------------------------------------- [root@dell-per740-40 ~]# perf script | grep handler ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) swapper 0 [022] 376.257398: sched:sched_wakeup: comm=handler1 pid=20693 prio=110 target_cpu=064 swapper 0 [022] 376.257418: sched:sched_wakeup: comm=handler2 pid=20694 prio=110 target_cpu=038 swapper 0 [022] 376.257431: sched:sched_wakeup: comm=handler4 pid=20696 prio=110 target_cpu=044 swapper 0 [022] 376.257440: sched:sched_wakeup: comm=handler5 pid=20697 prio=110 target_cpu=018 swapper 0 [022] 376.257448: sched:sched_wakeup: comm=handler7 pid=20699 prio=110 target_cpu=066 swapper 0 [022] 376.257456: sched:sched_wakeup: comm=handler8 pid=20700 prio=110 target_cpu=058 swapper 0 [022] 376.257468: sched:sched_wakeup: comm=handler9 pid=20701 prio=110 target_cpu=046 swapper 0 [022] 376.257480: sched:sched_wakeup: comm=handler10 pid=20702 prio=110 target_cpu=004 swapper 0 [022] 376.257494: sched:sched_wakeup: comm=handler12 pid=20704 prio=110 target_cpu=024 swapper 0 [041] 376.257502: sched:sched_wakeup: comm=handler3 pid=20695 prio=110 target_cpu=041 swapper 0 [022] 376.257506: sched:sched_wakeup: comm=handler13 pid=20705 prio=110 target_cpu=016 swapper 0 [022] 376.257518: sched:sched_wakeup: comm=handler14 pid=20706 prio=110 target_cpu=056 swapper 0 [022] 376.257530: sched:sched_wakeup: comm=handler15 pid=20707 prio=110 target_cpu=020 swapper 0 [022] 376.257543: sched:sched_wakeup: comm=handler16 pid=20708 prio=110 target_cpu=070 swapper 0 [045] 376.257544: sched:sched_wakeup: comm=handler11 pid=20703 prio=110 target_cpu=045 swapper 0 [022] 376.257553: sched:sched_wakeup: comm=handler17 pid=20709 prio=110 target_cpu=026 swapper 0 [022] 376.257562: sched:sched_wakeup: comm=handler18 pid=20710 prio=110 target_cpu=028 swapper 0 [022] 376.257570: sched:sched_wakeup: comm=handler19 pid=20711 prio=110 target_cpu=052 swapper 0 [022] 376.257577: sched:sched_wakeup: comm=handler20 pid=20712 prio=110 target_cpu=008 swapper 0 [022] 376.257585: sched:sched_wakeup: comm=handler21 pid=20713 prio=110 target_cpu=040 swapper 0 [022] 376.257594: sched:sched_wakeup: comm=handler22 pid=20714 prio=110 target_cpu=002 swapper 0 [022] 376.257603: sched:sched_wakeup: comm=handler23 pid=20715 prio=110 target_cpu=054 swapper 0 [022] 376.257612: sched:sched_wakeup: comm=handler24 pid=20716 prio=110 target_cpu=022 swapper 0 [022] 376.257620: sched:sched_wakeup: comm=handler25 pid=20717 prio=110 target_cpu=010 swapper 0 [022] 376.257628: sched:sched_wakeup: comm=handler26 pid=20718 prio=110 target_cpu=060 swapper 0 [022] 376.257637: sched:sched_wakeup: comm=handler27 pid=20719 prio=110 target_cpu=012 swapper 0 [022] 376.257645: sched:sched_wakeup: comm=handler28 pid=20720 prio=110 target_cpu=014 swapper 0 [022] 376.257652: sched:sched_wakeup: comm=handler29 pid=20721 prio=110 target_cpu=036 swapper 0 [022] 376.257660: sched:sched_wakeup: comm=handler30 pid=20722 prio=110 target_cpu=068 swapper 0 [022] 376.257668: sched:sched_wakeup: comm=handler31 pid=20723 prio=110 target_cpu=034 handler1 20693 [064] 376.257674: irq:softirq_entry: vec=3 [action=NET_RX] swapper 0 [022] 376.257676: sched:sched_wakeup: comm=handler32 pid=20724 prio=110 target_cpu=000 swapper 0 [022] 376.257684: sched:sched_wakeup: comm=handler33 pid=20725 prio=110 target_cpu=042 swapper 0 [022] 376.257692: sched:sched_wakeup: comm=handler34 pid=20726 prio=110 target_cpu=048 swapper 0 [022] 376.257700: sched:sched_wakeup: comm=handler35 pid=20727 prio=110 target_cpu=044 swapper 0 [022] 376.257708: sched:sched_wakeup: comm=handler36 pid=20728 prio=110 target_cpu=038 swapper 0 [022] 376.257716: sched:sched_wakeup: comm=handler37 pid=20729 prio=110 target_cpu=006 swapper 0 [022] 376.257724: sched:sched_wakeup: comm=handler38 pid=20730 prio=110 target_cpu=050 swapper 0 [022] 376.257732: sched:sched_wakeup: comm=handler39 pid=20731 prio=110 target_cpu=062 swapper 0 [022] 376.257740: sched:sched_wakeup: comm=handler40 pid=20732 prio=110 target_cpu=028 swapper 0 [022] 376.257748: sched:sched_wakeup: comm=handler41 pid=20733 prio=110 target_cpu=002 swapper 0 [022] 376.257756: sched:sched_wakeup: comm=handler42 pid=20734 prio=110 target_cpu=008 swapper 0 [022] 376.257763: sched:sched_wakeup: comm=handler43 pid=20735 prio=110 target_cpu=038 swapper 0 [022] 376.257771: sched:sched_wakeup: comm=handler44 pid=20736 prio=110 target_cpu=040 swapper 0 [022] 376.257778: sched:sched_wakeup: comm=handler45 pid=20737 prio=110 target_cpu=038 swapper 0 [022] 376.257786: sched:sched_wakeup: comm=handler46 pid=20738 prio=110 target_cpu=068 swapper 0 [022] 376.257793: sched:sched_wakeup: comm=handler47 pid=20739 prio=110 target_cpu=024 swapper 0 [022] 376.257801: sched:sched_wakeup: comm=handler48 pid=20740 prio=110 target_cpu=008 swapper 0 [022] 376.257808: sched:sched_wakeup: comm=handler49 pid=20741 prio=110 target_cpu=026 swapper 0 [022] 376.257815: sched:sched_wakeup: comm=handler50 pid=20742 prio=110 target_cpu=052 swapper 0 [022] 376.257823: sched:sched_wakeup: comm=handler51 pid=20743 prio=110 target_cpu=058 swapper 0 [022] 376.257833: sched:sched_wakeup: comm=handler52 pid=20744 prio=110 target_cpu=028 swapper 0 [022] 376.257845: sched:sched_wakeup: comm=handler53 pid=20745 prio=110 target_cpu=016 swapper 0 [022] 376.257854: sched:sched_wakeup: comm=handler54 pid=20746 prio=110 target_cpu=038 ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) arping 23089 [048] 380.056380: sched:sched_wakeup: comm=handler1 pid=20693 prio=110 target_cpu=064 arping 23089 [048] 380.056397: sched:sched_wakeup: comm=handler2 pid=20694 prio=110 target_cpu=038 arping 23089 [048] 380.056409: sched:sched_wakeup: comm=handler4 pid=20696 prio=110 target_cpu=044 arping 23089 [048] 380.056418: sched:sched_wakeup: comm=handler5 pid=20697 prio=110 target_cpu=018 arping 23089 [048] 380.056426: sched:sched_wakeup: comm=handler7 pid=20699 prio=110 target_cpu=066 arping 23089 [048] 380.056434: sched:sched_wakeup: comm=handler8 pid=20700 prio=110 target_cpu=058 arping 23089 [048] 380.056442: sched:sched_wakeup: comm=handler9 pid=20701 prio=110 target_cpu=046 arping 23089 [048] 380.056451: sched:sched_wakeup: comm=handler10 pid=20702 prio=110 target_cpu=004 arping 23089 [048] 380.056461: sched:sched_wakeup: comm=handler12 pid=20704 prio=110 target_cpu=024 arping 23089 [048] 380.056469: sched:sched_wakeup: comm=handler13 pid=20705 prio=110 target_cpu=016 swapper 0 [041] 380.056473: sched:sched_wakeup: comm=handler3 pid=20695 prio=110 target_cpu=041 arping 23089 [048] 380.056477: sched:sched_wakeup: comm=handler14 pid=20706 prio=110 target_cpu=056 arping 23089 [048] 380.056485: sched:sched_wakeup: comm=handler15 pid=20707 prio=110 target_cpu=020 arping 23089 [048] 380.056493: sched:sched_wakeup: comm=handler16 pid=20708 prio=110 target_cpu=070 arping 23089 [048] 380.056502: sched:sched_wakeup: comm=handler17 pid=20709 prio=110 target_cpu=026 arping 23089 [048] 380.056510: sched:sched_wakeup: comm=handler18 pid=20710 prio=110 target_cpu=028 swapper 0 [045] 380.056512: sched:sched_wakeup: comm=handler11 pid=20703 prio=110 target_cpu=045 arping 23089 [048] 380.056518: sched:sched_wakeup: comm=handler19 pid=20711 prio=110 target_cpu=052 arping 23089 [048] 380.056526: sched:sched_wakeup: comm=handler20 pid=20712 prio=110 target_cpu=008 arping 23089 [048] 380.056534: sched:sched_wakeup: comm=handler21 pid=20713 prio=110 target_cpu=040 arping 23089 [048] 380.056543: sched:sched_wakeup: comm=handler22 pid=20714 prio=110 target_cpu=002 arping 23089 [048] 380.056551: sched:sched_wakeup: comm=handler23 pid=20715 prio=110 target_cpu=054 arping 23089 [048] 380.056559: sched:sched_wakeup: comm=handler24 pid=20716 prio=110 target_cpu=022 arping 23089 [048] 380.056566: sched:sched_wakeup: comm=handler25 pid=20717 prio=110 target_cpu=010 arping 23089 [048] 380.056574: sched:sched_wakeup: comm=handler26 pid=20718 prio=110 target_cpu=060 arping 23089 [048] 380.056583: sched:sched_wakeup: comm=handler27 pid=20719 prio=110 target_cpu=012 arping 23089 [048] 380.056594: sched:sched_wakeup: comm=handler28 pid=20720 prio=110 target_cpu=014 arping 23089 [048] 380.056606: sched:sched_wakeup: comm=handler29 pid=20721 prio=110 target_cpu=036 arping 23089 [048] 380.056619: sched:sched_wakeup: comm=handler30 pid=20722 prio=110 target_cpu=068 arping 23089 [048] 380.056627: sched:sched_wakeup: comm=handler31 pid=20723 prio=110 target_cpu=034 arping 23089 [048] 380.056635: sched:sched_wakeup: comm=handler32 pid=20724 prio=110 target_cpu=000 arping 23089 [048] 380.056643: sched:sched_wakeup: comm=handler33 pid=20725 prio=110 target_cpu=042 arping 23089 [048] 380.056653: sched:sched_wakeup: comm=handler34 pid=20726 prio=110 target_cpu=050 arping 23089 [048] 380.056663: sched:sched_wakeup: comm=handler35 pid=20727 prio=110 target_cpu=046 handler4 20696 [044] 380.056664: irq:softirq_entry: vec=3 [action=NET_RX] arping 23089 [048] 380.056671: sched:sched_wakeup: comm=handler36 pid=20728 prio=110 target_cpu=038 arping 23089 [048] 380.056679: sched:sched_wakeup: comm=handler37 pid=20729 prio=110 target_cpu=006 arping 23089 [048] 380.056687: sched:sched_wakeup: comm=handler38 pid=20730 prio=110 target_cpu=050 arping 23089 [048] 380.056695: sched:sched_wakeup: comm=handler39 pid=20731 prio=110 target_cpu=062 arping 23089 [048] 380.056703: sched:sched_wakeup: comm=handler40 pid=20732 prio=110 target_cpu=028 arping 23089 [048] 380.056711: sched:sched_wakeup: comm=handler41 pid=20733 prio=110 target_cpu=002 arping 23089 [048] 380.056719: sched:sched_wakeup: comm=handler42 pid=20734 prio=110 target_cpu=008 handler4 20696 [044] 380.056722: sched:sched_wakeup: comm=handler1 pid=20693 prio=110 target_cpu=064 arping 23089 [048] 380.056727: sched:sched_wakeup: comm=handler43 pid=20735 prio=110 target_cpu=038 arping 23089 [048] 380.056735: sched:sched_wakeup: comm=handler44 pid=20736 prio=110 target_cpu=040 arping 23089 [048] 380.056742: sched:sched_wakeup: comm=handler45 pid=20737 prio=110 target_cpu=038 handler4 20696 [044] 380.056749: sched:sched_wakeup: comm=handler2 pid=20694 prio=110 target_cpu=046 arping 23089 [048] 380.056750: sched:sched_wakeup: comm=handler46 pid=20738 prio=110 target_cpu=068 arping 23089 [048] 380.056758: sched:sched_wakeup: comm=handler47 pid=20739 prio=110 target_cpu=024 arping 23089 [048] 380.056766: sched:sched_wakeup: comm=handler48 pid=20740 prio=110 target_cpu=008 handler4 20696 [044] 380.056768: sched:sched_wakeup: comm=handler5 pid=20697 prio=110 target_cpu=018 swapper 0 [041] 380.056771: sched:sched_wakeup: comm=handler3 pid=20695 prio=110 target_cpu=041 arping 23089 [048] 380.056773: sched:sched_wakeup: comm=handler49 pid=20741 prio=110 target_cpu=026 arping 23089 [048] 380.056781: sched:sched_wakeup: comm=handler50 pid=20742 prio=110 target_cpu=052 handler4 20696 [044] 380.056784: sched:sched_wakeup: comm=handler7 pid=20699 prio=110 target_cpu=066 arping 23089 [048] 380.056789: sched:sched_wakeup: comm=handler51 pid=20743 prio=110 target_cpu=058 arping 23089 [048] 380.056796: sched:sched_wakeup: comm=handler52 pid=20744 prio=110 target_cpu=028 handler4 20696 [044] 380.056798: sched:sched_wakeup: comm=handler8 pid=20700 prio=110 target_cpu=060 arping 23089 [048] 380.056804: sched:sched_wakeup: comm=handler53 pid=20745 prio=110 target_cpu=016 handler4 20696 [044] 380.056809: sched:sched_wakeup: comm=handler9 pid=20701 prio=110 target_cpu=046 arping 23089 [048] 380.056812: sched:sched_wakeup: comm=handler54 pid=20746 prio=110 target_cpu=038 handler4 20696 [044] 380.056819: sched:sched_wakeup: comm=handler10 pid=20702 prio=110 target_cpu=004 handler4 20696 [044] 380.056832: sched:sched_wakeup: comm=handler12 pid=20704 prio=110 target_cpu=024 handler4 20696 [044] 380.056843: sched:sched_wakeup: comm=handler13 pid=20705 prio=110 target_cpu=016 handler12 20704 [024] 380.056855: irq:softirq_entry: vec=1 [action=TIMER] handler1 20693 [064] 380.056855: irq:softirq_entry: vec=9 [action=RCU] handler4 20696 [044] 380.056858: sched:sched_wakeup: comm=handler14 pid=20706 prio=110 target_cpu=056 handler4 20696 [044] 380.056870: sched:sched_wakeup: comm=handler15 pid=20707 prio=110 target_cpu=020 handler12 20704 [024] 380.056875: sched:sched_wakeup: comm=kworker/24:1 pid=556 prio=120 target_cpu=024 handler4 20696 [044] 380.056881: sched:sched_wakeup: comm=handler16 pid=20708 prio=110 target_cpu=070 handler12 20704 [024] 380.056883: irq:softirq_entry: vec=9 [action=RCU] swapper 0 [045] 380.056888: sched:sched_wakeup: comm=handler11 pid=20703 prio=110 target_cpu=045 handler4 20696 [044] 380.056891: sched:sched_wakeup: comm=handler17 pid=20709 prio=110 target_cpu=026 handler4 20696 [044] 380.056901: sched:sched_wakeup: comm=handler18 pid=20710 prio=110 target_cpu=028 handler4 20696 [044] 380.056911: sched:sched_wakeup: comm=handler19 pid=20711 prio=110 target_cpu=052 handler4 20696 [044] 380.056921: sched:sched_wakeup: comm=handler20 pid=20712 prio=110 target_cpu=008 handler1 20693 [064] 380.056928: irq:softirq_entry: vec=3 [action=NET_RX] handler4 20696 [044] 380.056934: sched:sched_wakeup: comm=handler21 pid=20713 prio=110 target_cpu=040 handler4 20696 [044] 380.056950: sched:sched_wakeup: comm=handler22 pid=20714 prio=110 target_cpu=002 handler1 20693 [064] 380.056952: sched:sched_wakeup: comm=arping pid=23089 prio=120 target_cpu=048 handler4 20696 [044] 380.056968: sched:sched_wakeup: comm=handler23 pid=20715 prio=110 target_cpu=054 handler4 20696 [044] 380.056981: sched:sched_wakeup: comm=handler24 pid=20716 prio=110 target_cpu=022 handler4 20696 [044] 380.056991: sched:sched_wakeup: comm=handler25 pid=20717 prio=110 target_cpu=010 handler4 20696 [044] 380.057002: sched:sched_wakeup: comm=handler26 pid=20718 prio=110 target_cpu=060 handler4 20696 [044] 380.057016: sched:sched_wakeup: comm=handler27 pid=20719 prio=110 target_cpu=012 handler4 20696 [044] 380.057031: sched:sched_wakeup: comm=handler28 pid=20720 prio=110 target_cpu=014 handler4 20696 [044] 380.057046: sched:sched_wakeup: comm=handler29 pid=20721 prio=110 target_cpu=036 handler4 20696 [044] 380.057062: sched:sched_wakeup: comm=handler30 pid=20722 prio=110 target_cpu=068 handler4 20696 [044] 380.057079: sched:sched_wakeup: comm=handler31 pid=20723 prio=110 target_cpu=034 handler4 20696 [044] 380.057091: sched:sched_wakeup: comm=handler32 pid=20724 prio=110 target_cpu=000 handler4 20696 [044] 380.057102: sched:sched_wakeup: comm=handler33 pid=20725 prio=110 target_cpu=042 handler4 20696 [044] 380.057113: sched:sched_wakeup: comm=handler34 pid=20726 prio=110 target_cpu=050 handler4 20696 [044] 380.057123: sched:sched_wakeup: comm=handler35 pid=20727 prio=110 target_cpu=046 handler4 20696 [044] 380.057133: sched:sched_wakeup: comm=handler36 pid=20728 prio=110 target_cpu=038 handler4 20696 [044] 380.057143: sched:sched_wakeup: comm=handler37 pid=20729 prio=110 target_cpu=006 handler4 20696 [044] 380.057153: sched:sched_wakeup: comm=handler38 pid=20730 prio=110 target_cpu=050 handler4 20696 [044] 380.057164: sched:sched_wakeup: comm=handler39 pid=20731 prio=110 target_cpu=062 handler4 20696 [044] 380.057175: sched:sched_wakeup: comm=handler40 pid=20732 prio=110 target_cpu=028 handler4 20696 [044] 380.057185: sched:sched_wakeup: comm=handler41 pid=20733 prio=110 target_cpu=002 handler4 20696 [044] 380.057195: sched:sched_wakeup: comm=handler42 pid=20734 prio=110 target_cpu=008 handler4 20696 [044] 380.057208: sched:sched_wakeup: comm=handler43 pid=20735 prio=110 target_cpu=038 handler4 20696 [044] 380.057222: sched:sched_wakeup: comm=handler44 pid=20736 prio=110 target_cpu=040 handler4 20696 [044] 380.057232: sched:sched_wakeup: comm=handler45 pid=20737 prio=110 target_cpu=038 handler4 20696 [044] 380.057242: sched:sched_wakeup: comm=handler46 pid=20738 prio=110 target_cpu=068 handler4 20696 [044] 380.057253: sched:sched_wakeup: comm=handler47 pid=20739 prio=110 target_cpu=024 handler4 20696 [044] 380.057262: sched:sched_wakeup: comm=handler48 pid=20740 prio=110 target_cpu=008 handler4 20696 [044] 380.057276: sched:sched_wakeup: comm=handler49 pid=20741 prio=110 target_cpu=026 handler4 20696 [044] 380.057289: sched:sched_wakeup: comm=handler50 pid=20742 prio=110 target_cpu=052 handler4 20696 [044] 380.057300: sched:sched_wakeup: comm=handler51 pid=20743 prio=110 target_cpu=058 handler4 20696 [044] 380.057309: sched:sched_wakeup: comm=handler52 pid=20744 prio=110 target_cpu=028 handler4 20696 [044] 380.057320: sched:sched_wakeup: comm=handler53 pid=20745 prio=110 target_cpu=016 handler4 20696 [044] 380.057330: sched:sched_wakeup: comm=handler54 pid=20746 prio=110 target_cpu=038 handler4 20696 [044] 380.057350: irq:softirq_entry: vec=1 [action=TIMER] handler4 20696 [044] 380.057360: sched:sched_wakeup: comm=kworker/44:2 pid=20487 prio=120 target_cpu=044 handler4 20696 [044] 380.057367: irq:softirq_entry: vec=9 [action=RCU] ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) swapper 0 [030] 380.352968: sched:sched_wakeup: comm=handler1 pid=20693 prio=110 target_cpu=064 swapper 0 [030] 380.352982: sched:sched_wakeup: comm=handler2 pid=20694 prio=110 target_cpu=046 swapper 0 [030] 380.352994: sched:sched_wakeup: comm=handler4 pid=20696 prio=110 target_cpu=044 swapper 0 [030] 380.353001: sched:sched_wakeup: comm=handler5 pid=20697 prio=110 target_cpu=018 swapper 0 [030] 380.353008: sched:sched_wakeup: comm=handler7 pid=20699 prio=110 target_cpu=066 swapper 0 [030] 380.353018: sched:sched_wakeup: comm=handler8 pid=20700 prio=110 target_cpu=060 swapper 0 [030] 380.353030: sched:sched_wakeup: comm=handler9 pid=20701 prio=110 target_cpu=030 swapper 0 [030] 380.353040: sched:sched_wakeup: comm=handler10 pid=20702 prio=110 target_cpu=004 swapper 0 [030] 380.353054: sched:sched_wakeup: comm=handler12 pid=20704 prio=110 target_cpu=024 swapper 0 [030] 380.353066: sched:sched_wakeup: comm=handler13 pid=20705 prio=110 target_cpu=016 swapper 0 [030] 380.353077: sched:sched_wakeup: comm=handler14 pid=20706 prio=110 target_cpu=056 swapper 0 [041] 380.353077: sched:sched_wakeup: comm=handler3 pid=20695 prio=110 target_cpu=041 swapper 0 [030] 380.353087: sched:sched_wakeup: comm=handler15 pid=20707 prio=110 target_cpu=020 swapper 0 [030] 380.353099: sched:sched_wakeup: comm=handler16 pid=20708 prio=110 target_cpu=070 swapper 0 [045] 380.353108: sched:sched_wakeup: comm=handler11 pid=20703 prio=110 target_cpu=045 swapper 0 [030] 380.353111: sched:sched_wakeup: comm=handler17 pid=20709 prio=110 target_cpu=026 swapper 0 [030] 380.353123: sched:sched_wakeup: comm=handler18 pid=20710 prio=110 target_cpu=028 swapper 0 [030] 380.353134: sched:sched_wakeup: comm=handler19 pid=20711 prio=110 target_cpu=052 swapper 0 [030] 380.353145: sched:sched_wakeup: comm=handler20 pid=20712 prio=110 target_cpu=008 swapper 0 [030] 380.353156: sched:sched_wakeup: comm=handler21 pid=20713 prio=110 target_cpu=040 swapper 0 [030] 380.353168: sched:sched_wakeup: comm=handler22 pid=20714 prio=110 target_cpu=002 handler7 20699 [066] 380.353177: irq:softirq_entry: vec=3 [action=NET_RX] swapper 0 [030] 380.353178: sched:sched_wakeup: comm=handler23 pid=20715 prio=110 target_cpu=054 swapper 0 [030] 380.353190: sched:sched_wakeup: comm=handler24 pid=20716 prio=110 target_cpu=022 swapper 0 [030] 380.353204: sched:sched_wakeup: comm=handler25 pid=20717 prio=110 target_cpu=010 swapper 0 [030] 380.353214: sched:sched_wakeup: comm=handler26 pid=20718 prio=110 target_cpu=060 swapper 0 [030] 380.353225: sched:sched_wakeup: comm=handler27 pid=20719 prio=110 target_cpu=012 swapper 0 [030] 380.353236: sched:sched_wakeup: comm=handler28 pid=20720 prio=110 target_cpu=014 swapper 0 [030] 380.353247: sched:sched_wakeup: comm=handler29 pid=20721 prio=110 target_cpu=036 swapper 0 [030] 380.353259: sched:sched_wakeup: comm=handler30 pid=20722 prio=110 target_cpu=068 swapper 0 [030] 380.353270: sched:sched_wakeup: comm=handler31 pid=20723 prio=110 target_cpu=034 swapper 0 [030] 380.353280: sched:sched_wakeup: comm=handler32 pid=20724 prio=110 target_cpu=000 swapper 0 [030] 380.353288: sched:sched_wakeup: comm=handler33 pid=20725 prio=110 target_cpu=042 swapper 0 [030] 380.353295: sched:sched_wakeup: comm=handler34 pid=20726 prio=110 target_cpu=050 swapper 0 [030] 380.353302: sched:sched_wakeup: comm=handler35 pid=20727 prio=110 target_cpu=046 swapper 0 [030] 380.353309: sched:sched_wakeup: comm=handler36 pid=20728 prio=110 target_cpu=038 swapper 0 [030] 380.353317: sched:sched_wakeup: comm=handler37 pid=20729 prio=110 target_cpu=006 swapper 0 [030] 380.353326: sched:sched_wakeup: comm=handler38 pid=20730 prio=110 target_cpu=024 swapper 0 [030] 380.353333: sched:sched_wakeup: comm=handler39 pid=20731 prio=110 target_cpu=062 swapper 0 [030] 380.353340: sched:sched_wakeup: comm=handler40 pid=20732 prio=110 target_cpu=028 swapper 0 [030] 380.353348: sched:sched_wakeup: comm=handler41 pid=20733 prio=110 target_cpu=002 swapper 0 [030] 380.353354: sched:sched_wakeup: comm=handler42 pid=20734 prio=110 target_cpu=008 swapper 0 [030] 380.353361: sched:sched_wakeup: comm=handler43 pid=20735 prio=110 target_cpu=038 swapper 0 [030] 380.353368: sched:sched_wakeup: comm=handler44 pid=20736 prio=110 target_cpu=040 swapper 0 [030] 380.353375: sched:sched_wakeup: comm=handler45 pid=20737 prio=110 target_cpu=038 swapper 0 [030] 380.353382: sched:sched_wakeup: comm=handler46 pid=20738 prio=110 target_cpu=068 swapper 0 [030] 380.353389: sched:sched_wakeup: comm=handler47 pid=20739 prio=110 target_cpu=024 swapper 0 [030] 380.353396: sched:sched_wakeup: comm=handler48 pid=20740 prio=110 target_cpu=008 swapper 0 [030] 380.353403: sched:sched_wakeup: comm=handler49 pid=20741 prio=110 target_cpu=026 swapper 0 [030] 380.353410: sched:sched_wakeup: comm=handler50 pid=20742 prio=110 target_cpu=052 swapper 0 [030] 380.353417: sched:sched_wakeup: comm=handler51 pid=20743 prio=110 target_cpu=058 swapper 0 [030] 380.353424: sched:sched_wakeup: comm=handler52 pid=20744 prio=110 target_cpu=028 swapper 0 [030] 380.353431: sched:sched_wakeup: comm=handler53 pid=20745 prio=110 target_cpu=016 swapper 0 [030] 380.353438: sched:sched_wakeup: comm=handler54 pid=20746 prio=110 target_cpu=038 ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) 7f7496318b0c __run_exit_handlers+0x13c (/usr/lib64/libc-2.28.so) ffffffff88f331a0 search_binary_handler+0x90 ([kernel.kallsyms]) ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms]) [root@dell-per740-40 ~]# [root@dell-per740-40 ~]# uname -a Linux dell-per740-40.rhts.eng.pek2.redhat.com 4.18.0-323.el8.x86_64 #1 SMP Wed Jul 14 12:52:14 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux [root@dell-per740-40 ~]# rpm -qa | grep openv python3-openvswitch2.15-2.15.0-32.el8fdp.x86_64 openvswitch2.15-2.15.0-32.el8fdp.x86_64 openvswitch-selinux-extra-policy-1.0-28.el8fdp.noarch openvswitch2.15-test-2.15.0-32.el8fdp.noarch [root@dell-per740-40 ~]# with kernel update-------------------------------------------------------------------------------------------------------- [root@dell-per740-40 ~]# perf script | grep arping arping 9172 [017] 479.735555: irq:softirq_entry: vec=9 [action=RCU] swapper 0 [017] 479.735739: sched:sched_wakeup: comm=arping pid=9172 prio=120 target_cpu=017 arping 9172 [017] 479.737554: irq:softirq_entry: vec=7 [action=SCHED] arping 9172 [017] 479.738554: irq:softirq_entry: vec=9 [action=RCU] arping 9172 [017] 479.738699: sched:sched_wakeup: comm=kworker/u146:0 pid=8905 prio=120 target_cpu=041 arping 9172 [017] 479.738730: irq:softirq_entry: vec=3 [action=NET_RX] arping 9172 [017] 479.738757: sched:sched_wakeup: comm=handler290 pid=9086 prio=110 target_cpu=043 arping 9172 [017] 479.738777: sched:sched_wakeup: comm=handler291 pid=9087 prio=110 target_cpu=011 arping 9172 [017] 479.738788: sched:sched_wakeup: comm=handler294 pid=9088 prio=110 target_cpu=049 arping 9172 [017] 479.738797: sched:sched_wakeup: comm=handler293 pid=9089 prio=110 target_cpu=029 arping 9172 [017] 479.738806: sched:sched_wakeup: comm=handler295 pid=9090 prio=110 target_cpu=069 arping 9172 [017] 479.738816: sched:sched_wakeup: comm=handler296 pid=9091 prio=110 target_cpu=045 arping 9172 [017] 479.738824: sched:sched_wakeup: comm=handler292 pid=9092 prio=110 target_cpu=051 arping 9172 [017] 479.738832: sched:sched_wakeup: comm=handler299 pid=9093 prio=110 target_cpu=059 arping 9172 [017] 479.738840: sched:sched_wakeup: comm=handler300 pid=9094 prio=110 target_cpu=055 arping 9172 [017] 479.738849: sched:sched_wakeup: comm=handler297 pid=9095 prio=110 target_cpu=057 arping 9172 [017] 479.738858: sched:sched_wakeup: comm=handler298 pid=9096 prio=110 target_cpu=025 arping 9172 [017] 479.738865: sched:sched_wakeup: comm=handler305 pid=9097 prio=110 target_cpu=063 arping 9172 [017] 479.738873: sched:sched_wakeup: comm=handler301 pid=9098 prio=110 target_cpu=019 arping 9172 [017] 479.738881: sched:sched_wakeup: comm=handler302 pid=9099 prio=110 target_cpu=003 arping 9172 [017] 479.738890: sched:sched_wakeup: comm=handler303 pid=9100 prio=110 target_cpu=067 arping 9172 [017] 479.738899: sched:sched_wakeup: comm=handler304 pid=9101 prio=110 target_cpu=035 arping 9172 [017] 479.738907: sched:sched_wakeup: comm=handler306 pid=9102 prio=110 target_cpu=037 arping 9172 [017] 479.738920: sched:sched_wakeup: comm=handler307 pid=9103 prio=110 target_cpu=041 arping 9172 [017] 479.738928: sched:sched_wakeup: comm=handler308 pid=9104 prio=110 target_cpu=001 arping 9172 [017] 479.738936: sched:sched_wakeup: comm=handler309 pid=9105 prio=110 target_cpu=045 arping 9172 [017] 479.738944: sched:sched_wakeup: comm=handler310 pid=9106 prio=110 target_cpu=039 arping 9172 [017] 479.738953: sched:sched_wakeup: comm=handler311 pid=9107 prio=110 target_cpu=013 arping 9172 [017] 479.738961: sched:sched_wakeup: comm=handler312 pid=9108 prio=110 target_cpu=071 arping 9172 [017] 479.738969: sched:sched_wakeup: comm=handler313 pid=9109 prio=110 target_cpu=005 arping 9172 [017] 479.738978: sched:sched_wakeup: comm=handler314 pid=9110 prio=110 target_cpu=021 arping 9172 [017] 479.738986: sched:sched_wakeup: comm=handler315 pid=9111 prio=110 target_cpu=061 arping 9172 [017] 479.738995: sched:sched_wakeup: comm=handler316 pid=9112 prio=110 target_cpu=047 arping 9172 [017] 479.739004: sched:sched_wakeup: comm=handler317 pid=9113 prio=110 target_cpu=065 arping 9172 [017] 479.739012: sched:sched_wakeup: comm=handler318 pid=9114 prio=110 target_cpu=015 arping 9172 [017] 479.739020: sched:sched_wakeup: comm=handler319 pid=9115 prio=110 target_cpu=027 arping 9172 [017] 479.739029: sched:sched_wakeup: comm=handler320 pid=9116 prio=110 target_cpu=023 arping 9172 [017] 479.739038: sched:sched_wakeup: comm=handler321 pid=9117 prio=110 target_cpu=031 arping 9172 [017] 479.739047: sched:sched_wakeup: comm=handler322 pid=9118 prio=110 target_cpu=041 arping 9172 [017] 479.739055: sched:sched_wakeup: comm=handler323 pid=9119 prio=110 target_cpu=069 arping 9172 [017] 479.739063: sched:sched_wakeup: comm=handler324 pid=9120 prio=110 target_cpu=053 arping 9172 [017] 479.739075: sched:sched_wakeup: comm=handler325 pid=9121 prio=110 target_cpu=005 arping 9172 [017] 479.739088: sched:sched_wakeup: comm=handler326 pid=9122 prio=110 target_cpu=009 arping 9172 [017] 479.739101: sched:sched_wakeup: comm=handler327 pid=9123 prio=110 target_cpu=035 arping 9172 [017] 479.739110: sched:sched_wakeup: comm=handler328 pid=9124 prio=110 target_cpu=041 arping 9172 [017] 479.739119: sched:sched_wakeup: comm=handler329 pid=9125 prio=110 target_cpu=035 arping 9172 [017] 479.739129: sched:sched_wakeup: comm=handler330 pid=9126 prio=110 target_cpu=015 arping 9172 [017] 479.739138: sched:sched_wakeup: comm=handler331 pid=9127 prio=110 target_cpu=037 arping 9172 [017] 479.739146: sched:sched_wakeup: comm=handler332 pid=9128 prio=110 target_cpu=065 arping 9172 [017] 479.739156: sched:sched_wakeup: comm=handler333 pid=9129 prio=110 target_cpu=035 arping 9172 [017] 479.739164: sched:sched_wakeup: comm=handler334 pid=9130 prio=110 target_cpu=051 arping 9172 [017] 479.739173: sched:sched_wakeup: comm=handler335 pid=9131 prio=110 target_cpu=067 arping 9172 [017] 479.739181: sched:sched_wakeup: comm=handler336 pid=9132 prio=110 target_cpu=027 arping 9172 [017] 479.739199: sched:sched_wakeup: comm=handler341 pid=9137 prio=110 target_cpu=023 arping 9172 [017] 479.739207: sched:sched_wakeup: comm=handler342 pid=9138 prio=110 target_cpu=045 handler291 9087 [011] 479.739276: sched:sched_wakeup: comm=arping pid=9172 prio=120 target_cpu=017 arping 9172 [017] 479.739313: sched:sched_wakeup: comm=kworker/u146:0 pid=8905 prio=120 target_cpu=041 swapper 0 [017] 480.739220: sched:sched_wakeup: comm=arping pid=9172 prio=120 target_cpu=017 arping 9172 [017] 480.739272: sched:sched_wakeup: comm=kworker/u146:0 pid=8905 prio=120 target_cpu=041 arping 9172 [017] 480.739470: irq:softirq_entry: vec=7 [action=SCHED] arping 9172 [017] 480.739483: irq:softirq_entry: vec=9 [action=RCU] swapper 0 [017] 480.745556: sched:sched_wakeup: comm=arping pid=9172 prio=120 target_cpu=017 ksoftirqd/17 116 [017] 480.753587: sched:sched_wakeup: comm=arping pid=9172 prio=120 target_cpu=017 arping 9172 [017] 480.753686: sched:sched_wakeup: comm=bash pid=7890 prio=120 target_cpu=051 [root@dell-per740-40 ~]# uname -a Linux dell-per740-40.rhts.eng.pek2.redhat.com 4.18.0-328.el8.mr1064_210805_1629.x86_64 #1 SMP Thu Aug 5 16:42:01 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux [root@dell-per740-40 ~]# uname -r 4.18.0-328.el8.mr1064_210805_1629.x86_64 [root@dell-per740-40 ~]# cat /proc/cmdline BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-328.el8.mr1064_210805_1629.x86_64 root=/dev/mapper/rhel_dell--per740--40-root ro crashkernel=auto resume=/dev/mapper/rhel_dell--per740--40-swap rd.lvm.lv=rhel_dell-per740-40/root rd.lvm.lv=rhel_dell-per740-40/swap console=ttyS0,115200n81 [root@dell-per740-40 ~]#
Can you confirm per-cpu dispatch is enabled? `ovs-appctl dpif-netlink/dispatch-mode`
(In reply to Mark Gray from comment #9) > Can you confirm per-cpu dispatch is enabled? > > `ovs-appctl dpif-netlink/dispatch-mode` Hi Mark Both old kernel (4.18.0-323.el8.x86_64) and new build kernel 4.18.0-328.el8.mr1064_210805_1629.x86_64 do not have this command [root@dell-per740-40 ~]# ovs-appctl list-commands | grep dpif dpif-netdev/bond-show [dp] dpif-netdev/pmd-perf-log-set on|off [-b before] [-a after] [-e|-ne] [-us usec] [-q qlen] dpif-netdev/pmd-perf-show [-nh] [-it iter-history-len] [-ms ms-history-len] [-pmd core] [dp] dpif-netdev/pmd-rxq-rebalance [dp] dpif-netdev/pmd-rxq-show [-pmd core] [dp] dpif-netdev/pmd-stats-clear [-pmd core] [dp] dpif-netdev/pmd-stats-show [-pmd core] [dp] dpif-netdev/subtable-lookup-prio-get dpif-netdev/subtable-lookup-prio-set [lookup_func] [prio] dpif/dump-dps dpif/dump-flows [-m] [--names | --no-names] bridge dpif/set-dp-features bridge dpif/show dpif/show-dp-features bridge [root@dell-per740-40 ~]# ovs-appctl dpif-netlink/dispatch-mode "dpif-netlink/dispatch-mode" is not a valid command (use "list-commands" to see a list of valid commands) ovs-appctl: ovs-vswitchd: server returned an error Am i missing something ？ Thanks.
Yes, I see you are missing the userspace component. This fix requires an updated kernel *and* an updated userspace package. The patch will land in the userspace openvswitch2.16 package which is not available yet as far as I know. @tredaelli might know when this will be available.
modprobe -r veth ip netns del left ip netns del right ip netns add left ip netns add right ip link add center-left type veth peer name left0 netns left ip link add center-right type veth peer name right0 netns right ip link set center-left up ip link set center-right up ip netns exec left ip link set left0 up ip netns exec left ip addr add 172.31.110.10/24 dev left0 ip netns exec right ip link set right0 up ip netns exec right ip addr add 172.31.110.11/24 dev right0 systemctl start openvswitch ovs-vsctl --if-exists del-br br0 ovs-vsctl add-br br0 ovs-vsctl add-port br0 center-right ovs-vsctl add-port br0 center-left perf record -e sched:sched_wakeup,irq:softirq_entry -ag ip netns exec left arping -I left0 -c 1 172.31.110.11 with old kernel [root@dell-per740-40 ~]# uname -a Linux dell-per740-40.rhts.eng.pek2.redhat.com 4.18.0-323.el8.x86_64 #1 SMP Wed Jul 14 12:52:14 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux ovs version [root@dell-per740-40 ~]# rpm -qa | grep openv openvswitch2.16-2.16.0-1.el8fdp.x86_64 openvswitch-selinux-extra-policy-1.0-28.el8fdp.noarch python3-openvswitch2.16-2.16.0-1.el8fdp.x86_64 openvswitch2.16-test-2.16.0-1.el8fdp.noarch perf script [root@dell-per740-40 ~]# perf script | grep arp swapper 0 [002] 1983.763394: sched:sched_wakeup: comm=arping pid=24006 prio=120 target_cpu=002 arping 24006 [002] 1983.764254: irq:softirq_entry: vec=7 [action=SCHED] arping 24006 [002] 1983.765253: irq:softirq_entry: vec=9 [action=RCU] arping 24006 [002] 1983.765273: irq:softirq_entry: vec=9 [action=RCU] arping 24006 [002] 1983.765283: irq:softirq_entry: vec=9 [action=RCU] arping 24006 [002] 1983.766382: sched:sched_wakeup: comm=kworker/u145:16 pid=461 prio=120 target_cpu=062 arping 24006 [002] 1983.766413: irq:softirq_entry: vec=3 [action=NET_RX] arping 24006 [002] 1983.766498: sched:sched_wakeup: comm=handler171 pid=21281 prio=110 target_cpu=048 arping 24006 [002] 1983.766525: sched:sched_wakeup: comm=handler175 pid=21285 prio=110 target_cpu=008 arping 24006 [002] 1983.766536: sched:sched_wakeup: comm=handler176 pid=21286 prio=110 target_cpu=056 arping 24006 [002] 1983.766546: sched:sched_wakeup: comm=handler177 pid=21287 prio=110 target_cpu=004 arping 24006 [002] 1983.766555: sched:sched_wakeup: comm=handler178 pid=21288 prio=110 target_cpu=032 arping 24006 [002] 1983.766564: sched:sched_wakeup: comm=handler179 pid=21289 prio=110 target_cpu=058 arping 24006 [002] 1983.766572: sched:sched_wakeup: comm=handler180 pid=21290 prio=110 target_cpu=060 arping 24006 [002] 1983.766581: sched:sched_wakeup: comm=handler181 pid=21291 prio=110 target_cpu=062 arping 24006 [002] 1983.766590: sched:sched_wakeup: comm=handler182 pid=21292 prio=110 target_cpu=064 arping 24006 [002] 1983.766599: sched:sched_wakeup: comm=handler183 pid=21293 prio=110 target_cpu=066 arping 24006 [002] 1983.766623: sched:sched_wakeup: comm=handler190 pid=21300 prio=110 target_cpu=034 arping 24006 [002] 1983.766635: sched:sched_wakeup: comm=handler191 pid=21301 prio=110 target_cpu=006 arping 24006 [002] 1983.766650: sched:sched_wakeup: comm=handler192 pid=21302 prio=110 target_cpu=010 arping 24006 [002] 1983.766660: sched:sched_wakeup: comm=handler193 pid=21303 prio=110 target_cpu=014 arping 24006 [002] 1983.766676: sched:sched_wakeup: comm=handler194 pid=21304 prio=110 target_cpu=016 arping 24006 [002] 1983.766703: sched:sched_wakeup: comm=handler195 pid=21305 prio=110 target_cpu=000 arping 24006 [002] 1983.766718: sched:sched_wakeup: comm=handler196 pid=21306 prio=110 target_cpu=018 arping 24006 [002] 1983.766730: sched:sched_wakeup: comm=handler198 pid=21308 prio=110 target_cpu=038 handler152 21261 [067] 1983.767836: sched:sched_wakeup: comm=arping pid=24006 prio=120 target_cpu=069 arping 24006 [069] 1983.767906: sched:sched_wakeup: comm=kworker/u146:26 pid=495 prio=120 target_cpu=027 swapper 0 [069] 1984.766785: sched:sched_wakeup: comm=arping pid=24006 prio=120 target_cpu=069 arping 24006 [069] 1984.766828: sched:sched_wakeup: comm=kworker/u146:26 pid=495 prio=120 target_cpu=027 arping 24006 [069] 1984.767170: irq:softirq_entry: vec=7 [action=SCHED] arping 24006 [069] 1984.767184: irq:softirq_entry: vec=9 [action=RCU] swapper 0 [069] 1984.775247: sched:sched_wakeup: comm=arping pid=24006 prio=120 target_cpu=069 ksoftirqd/69 428 [069] 1984.783303: sched:sched_wakeup: comm=arping pid=24006 prio=120 target_cpu=069 with new kernel [root@dell-per740-40 ~]# uname -a Linux dell-per740-40.rhts.eng.pek2.redhat.com 4.18.0-328.el8.mr1064_210805_1629.x86_64 #1 SMP Thu Aug 5 16:42:01 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux [root@dell-per740-40 ~]# uname -r 4.18.0-328.el8.mr1064_210805_1629.x86_64 [root@dell-per740-40 ~]# cat /proc/cmdline BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-328.el8.mr1064_210805_1629.x86_64 root=/dev/mapper/rhel_dell--per740--40-root ro crashkernel=auto resume=/dev/mapper/rhel_dell--per740--40-swap rd.lvm.lv=rhel_dell-per740-40/root rd.lvm.lv=rhel_dell-per740-40/swap console=ttyS0,115200n81 [root@dell-per740-40 ~]# ovs-appctl dpif-netlink/dispatch-mode ovs-system: per-cpu dispatch mode [root@dell-per740-40 ~]# perf script | grep arping swapper 0 [013] 82.841579: sched:sched_wakeup: comm=arping pid=9077 prio=120 target_cpu=013 arping 9077 [013] 82.841960: irq:softirq_entry: vec=9 [action=RCU] arping 9077 [013] 82.843960: irq:softirq_entry: vec=7 [action=SCHED] arping 9077 [013] 82.843978: irq:softirq_entry: vec=9 [action=RCU] arping 9077 [013] 82.843992: irq:softirq_entry: vec=9 [action=RCU] arping 9077 [013] 82.844001: irq:softirq_entry: vec=9 [action=RCU] arping 9077 [013] 82.844625: sched:sched_wakeup: comm=kworker/u146:3 pid=464 prio=120 target_cpu=011 arping 9077 [013] 82.844652: irq:softirq_entry: vec=3 [action=NET_RX] arping 9077 [013] 82.844677: sched:sched_wakeup: comm=handler379 pid=8972 prio=110 target_cpu=047 handler413 9006 [043] 82.845255: sched:sched_wakeup: comm=arping pid=9077 prio=120 target_cpu=013 arping 9077 [013] 82.845293: sched:sched_wakeup: comm=kworker/u146:3 pid=464 prio=120 target_cpu=011 swapper 0 [013] 83.844694: sched:sched_wakeup: comm=arping pid=9077 prio=120 target_cpu=013 arping 9077 [013] 83.844743: sched:sched_wakeup: comm=kworker/u146:3 pid=464 prio=120 target_cpu=011 arping 9077 [013] 83.844874: irq:softirq_entry: vec=7 [action=SCHED] arping 9077 [013] 83.844889: irq:softirq_entry: vec=9 [action=RCU] swapper 0 [013] 83.850967: sched:sched_wakeup: comm=arping pid=9077 prio=120 target_cpu=013 ksoftirqd/13 92 [013] 83.858998: sched:sched_wakeup: comm=arping pid=9077 prio=120 target_cpu=013 arping 9077 [013] 83.859099: sched:sched_wakeup: comm=bash pid=5254 prio=120 target_cpu=047 So , The result is bug was fixed on new ovs2.16 new kernel version .
clearing needinfo as this fix has been closed and released.
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days