Bug 1834444 - File descriptor changes exposed a thundering herd issue [NEEDINFO]
Summary: File descriptor changes exposed a thundering herd issue
Keywords:
Status: POST
Alias: None
Product: Red Hat Enterprise Linux Fast Datapath
Classification: Red Hat
Component: openvswitch2.11
Version: FDP 20.A
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: ---
Assignee: Open vSwitch development team
QA Contact: Hekai Wang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-11 17:48 UTC by Aaron Conole
Modified: 2021-11-14 22:29 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
hewang: needinfo? (aconole)
hewang: needinfo? (ctrautma)
mark.d.gray: needinfo? (tredaelli)


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker FD-647 0 None None None 2021-08-12 13:33:09 UTC

Internal Links: 1992773

Description Aaron Conole 2020-05-11 17:48:43 UTC
Description of problem:

When the openvswitch codebase was changed to use fewer file descriptors, it allowed for the kernel to wake more threads simultaneously, which exposed a thundering herd issue in the poll_block() mechanism.  Reported upstream by David Ahern.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Rashid Khan 2020-07-20 21:37:36 UTC
Dcbw indicated that this bug is needed to reduce / eliminate the packet out of ordering problem. 
So escalating it here. 

Related bugs are 
https://bugzilla.redhat.com/show_bug.cgi?id=1844576
https://bugzilla.redhat.com/show_bug.cgi?id=1834444

Cambria and Dcbw can provide the details. 

Out of order packets make 3rd party apps like E/// choke. So we need to get packets in order as much as possible before handing them off to pods and containers and VMs.

Comment 2 Aaron Conole 2020-07-22 18:59:02 UTC
https://mail.openvswitch.org/pipermail/ovs-dev/2020-July/373146.html

Latest version submitted.

Comment 3 Aaron Conole 2020-10-23 20:08:59 UTC
https://mail.openvswitch.org/pipermail/ovs-dev/2020-September/375326.html

Is the latest version.

Comment 4 Mark Gray 2021-07-08 16:17:49 UTC
This is now at v5 for user space patches: https://patchwork.ozlabs.org/project/openvswitch/list/?series=252282
v1 for kernel space patches: https://marc.info/?l=linux-netdev&m=162504684016825&w=2

Comment 7 Aaron Conole 2021-08-12 13:30:58 UTC
Background on testing:

Setup two NS to ping back and forth
  ip netns add left
  ip netns add right
  ip link add center-left type veth peer name left0 netns left
  ip link add center-right type veth peer name right0 netns right
  ip link set center-left up
  ip link set center-right up
  ip -n left ip link set left0 up
  ip -n left ip addr add 172.31.110.10/24 dev left0
  ip -n right ip link set right0 up
  ip -n right ip addr add 172.31.110.11/24 dev right0

  ovs-vsctl add-br br0
  ovs-vsctl add-port br0 center-right
  ovs-vsctl add-port br0 center-left

Start recording the wakeups due to packet arrival
  # in one terminal
  perf record -e sched:sched_wakeup,irq:softirq_entry -ag

Generate ARP request messages
  # in a separate terminal
  ip netns exec left arping -I left0 -c 1 172.31.110.11

Use the following to look at the number of handler threads woken
  # in the perf terminal after exiting
  perf script

This is a simple way of reproduction.  Without the entire packge (kernel and userspace changes) we should
see multiple handler threads woken up during execution.  With the kernel and userspace changes applied, we
should see only a single handler thread woken.

Comment 8 Hekai Wang 2021-08-16 07:50:59 UTC
It still has the issue that we can see multiple handler threads woken_up

modprobe -r veth
ip netns del left
ip netns del right
ip netns add left
ip netns add right
ip link add center-left type veth peer name left0 netns left
ip link add center-right type veth peer name right0 netns right
ip link set center-left up
ip link set center-right up
ip netns exec left ip link set left0 up
ip netns exec left ip addr add 172.31.110.10/24 dev left0
ip netns exec right ip link set right0 up
ip netns exec right ip addr add 172.31.110.11/24 dev right0

systemctl start openvswitch
ovs-vsctl --if-exists del-br br0 
ovs-vsctl add-br br0
ovs-vsctl add-port br0 center-right
ovs-vsctl add-port br0 center-left


perf record -e sched:sched_wakeup,irq:softirq_entry -ag

ip netns exec left arping -I left0 -c 1 172.31.110.11



without kernel update ---------------------------------------------------------------------
[root@dell-per740-40 ~]# perf script | grep handler
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
swapper     0 [022]   376.257398: sched:sched_wakeup: comm=handler1 pid=20693 prio=110 target_cpu=064
swapper     0 [022]   376.257418: sched:sched_wakeup: comm=handler2 pid=20694 prio=110 target_cpu=038
swapper     0 [022]   376.257431: sched:sched_wakeup: comm=handler4 pid=20696 prio=110 target_cpu=044
swapper     0 [022]   376.257440: sched:sched_wakeup: comm=handler5 pid=20697 prio=110 target_cpu=018
swapper     0 [022]   376.257448: sched:sched_wakeup: comm=handler7 pid=20699 prio=110 target_cpu=066
swapper     0 [022]   376.257456: sched:sched_wakeup: comm=handler8 pid=20700 prio=110 target_cpu=058
swapper     0 [022]   376.257468: sched:sched_wakeup: comm=handler9 pid=20701 prio=110 target_cpu=046
swapper     0 [022]   376.257480: sched:sched_wakeup: comm=handler10 pid=20702 prio=110 target_cpu=004
swapper     0 [022]   376.257494: sched:sched_wakeup: comm=handler12 pid=20704 prio=110 target_cpu=024
swapper     0 [041]   376.257502: sched:sched_wakeup: comm=handler3 pid=20695 prio=110 target_cpu=041
swapper     0 [022]   376.257506: sched:sched_wakeup: comm=handler13 pid=20705 prio=110 target_cpu=016
swapper     0 [022]   376.257518: sched:sched_wakeup: comm=handler14 pid=20706 prio=110 target_cpu=056
swapper     0 [022]   376.257530: sched:sched_wakeup: comm=handler15 pid=20707 prio=110 target_cpu=020
swapper     0 [022]   376.257543: sched:sched_wakeup: comm=handler16 pid=20708 prio=110 target_cpu=070
swapper     0 [045]   376.257544: sched:sched_wakeup: comm=handler11 pid=20703 prio=110 target_cpu=045
swapper     0 [022]   376.257553: sched:sched_wakeup: comm=handler17 pid=20709 prio=110 target_cpu=026
swapper     0 [022]   376.257562: sched:sched_wakeup: comm=handler18 pid=20710 prio=110 target_cpu=028
swapper     0 [022]   376.257570: sched:sched_wakeup: comm=handler19 pid=20711 prio=110 target_cpu=052
swapper     0 [022]   376.257577: sched:sched_wakeup: comm=handler20 pid=20712 prio=110 target_cpu=008
swapper     0 [022]   376.257585: sched:sched_wakeup: comm=handler21 pid=20713 prio=110 target_cpu=040
swapper     0 [022]   376.257594: sched:sched_wakeup: comm=handler22 pid=20714 prio=110 target_cpu=002
swapper     0 [022]   376.257603: sched:sched_wakeup: comm=handler23 pid=20715 prio=110 target_cpu=054
swapper     0 [022]   376.257612: sched:sched_wakeup: comm=handler24 pid=20716 prio=110 target_cpu=022
swapper     0 [022]   376.257620: sched:sched_wakeup: comm=handler25 pid=20717 prio=110 target_cpu=010
swapper     0 [022]   376.257628: sched:sched_wakeup: comm=handler26 pid=20718 prio=110 target_cpu=060
swapper     0 [022]   376.257637: sched:sched_wakeup: comm=handler27 pid=20719 prio=110 target_cpu=012
swapper     0 [022]   376.257645: sched:sched_wakeup: comm=handler28 pid=20720 prio=110 target_cpu=014
swapper     0 [022]   376.257652: sched:sched_wakeup: comm=handler29 pid=20721 prio=110 target_cpu=036
swapper     0 [022]   376.257660: sched:sched_wakeup: comm=handler30 pid=20722 prio=110 target_cpu=068
swapper     0 [022]   376.257668: sched:sched_wakeup: comm=handler31 pid=20723 prio=110 target_cpu=034
handler1 20693 [064]   376.257674:  irq:softirq_entry: vec=3 [action=NET_RX]
swapper     0 [022]   376.257676: sched:sched_wakeup: comm=handler32 pid=20724 prio=110 target_cpu=000
swapper     0 [022]   376.257684: sched:sched_wakeup: comm=handler33 pid=20725 prio=110 target_cpu=042
swapper     0 [022]   376.257692: sched:sched_wakeup: comm=handler34 pid=20726 prio=110 target_cpu=048
swapper     0 [022]   376.257700: sched:sched_wakeup: comm=handler35 pid=20727 prio=110 target_cpu=044
swapper     0 [022]   376.257708: sched:sched_wakeup: comm=handler36 pid=20728 prio=110 target_cpu=038
swapper     0 [022]   376.257716: sched:sched_wakeup: comm=handler37 pid=20729 prio=110 target_cpu=006
swapper     0 [022]   376.257724: sched:sched_wakeup: comm=handler38 pid=20730 prio=110 target_cpu=050
swapper     0 [022]   376.257732: sched:sched_wakeup: comm=handler39 pid=20731 prio=110 target_cpu=062
swapper     0 [022]   376.257740: sched:sched_wakeup: comm=handler40 pid=20732 prio=110 target_cpu=028
swapper     0 [022]   376.257748: sched:sched_wakeup: comm=handler41 pid=20733 prio=110 target_cpu=002
swapper     0 [022]   376.257756: sched:sched_wakeup: comm=handler42 pid=20734 prio=110 target_cpu=008
swapper     0 [022]   376.257763: sched:sched_wakeup: comm=handler43 pid=20735 prio=110 target_cpu=038
swapper     0 [022]   376.257771: sched:sched_wakeup: comm=handler44 pid=20736 prio=110 target_cpu=040
swapper     0 [022]   376.257778: sched:sched_wakeup: comm=handler45 pid=20737 prio=110 target_cpu=038
swapper     0 [022]   376.257786: sched:sched_wakeup: comm=handler46 pid=20738 prio=110 target_cpu=068
swapper     0 [022]   376.257793: sched:sched_wakeup: comm=handler47 pid=20739 prio=110 target_cpu=024
swapper     0 [022]   376.257801: sched:sched_wakeup: comm=handler48 pid=20740 prio=110 target_cpu=008
swapper     0 [022]   376.257808: sched:sched_wakeup: comm=handler49 pid=20741 prio=110 target_cpu=026
swapper     0 [022]   376.257815: sched:sched_wakeup: comm=handler50 pid=20742 prio=110 target_cpu=052
swapper     0 [022]   376.257823: sched:sched_wakeup: comm=handler51 pid=20743 prio=110 target_cpu=058
swapper     0 [022]   376.257833: sched:sched_wakeup: comm=handler52 pid=20744 prio=110 target_cpu=028
swapper     0 [022]   376.257845: sched:sched_wakeup: comm=handler53 pid=20745 prio=110 target_cpu=016
swapper     0 [022]   376.257854: sched:sched_wakeup: comm=handler54 pid=20746 prio=110 target_cpu=038
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
arping 23089 [048]   380.056380: sched:sched_wakeup: comm=handler1 pid=20693 prio=110 target_cpu=064
arping 23089 [048]   380.056397: sched:sched_wakeup: comm=handler2 pid=20694 prio=110 target_cpu=038
arping 23089 [048]   380.056409: sched:sched_wakeup: comm=handler4 pid=20696 prio=110 target_cpu=044
arping 23089 [048]   380.056418: sched:sched_wakeup: comm=handler5 pid=20697 prio=110 target_cpu=018
arping 23089 [048]   380.056426: sched:sched_wakeup: comm=handler7 pid=20699 prio=110 target_cpu=066
arping 23089 [048]   380.056434: sched:sched_wakeup: comm=handler8 pid=20700 prio=110 target_cpu=058
arping 23089 [048]   380.056442: sched:sched_wakeup: comm=handler9 pid=20701 prio=110 target_cpu=046
arping 23089 [048]   380.056451: sched:sched_wakeup: comm=handler10 pid=20702 prio=110 target_cpu=004
arping 23089 [048]   380.056461: sched:sched_wakeup: comm=handler12 pid=20704 prio=110 target_cpu=024
arping 23089 [048]   380.056469: sched:sched_wakeup: comm=handler13 pid=20705 prio=110 target_cpu=016
swapper     0 [041]   380.056473: sched:sched_wakeup: comm=handler3 pid=20695 prio=110 target_cpu=041
arping 23089 [048]   380.056477: sched:sched_wakeup: comm=handler14 pid=20706 prio=110 target_cpu=056
arping 23089 [048]   380.056485: sched:sched_wakeup: comm=handler15 pid=20707 prio=110 target_cpu=020
arping 23089 [048]   380.056493: sched:sched_wakeup: comm=handler16 pid=20708 prio=110 target_cpu=070
arping 23089 [048]   380.056502: sched:sched_wakeup: comm=handler17 pid=20709 prio=110 target_cpu=026
arping 23089 [048]   380.056510: sched:sched_wakeup: comm=handler18 pid=20710 prio=110 target_cpu=028
swapper     0 [045]   380.056512: sched:sched_wakeup: comm=handler11 pid=20703 prio=110 target_cpu=045
arping 23089 [048]   380.056518: sched:sched_wakeup: comm=handler19 pid=20711 prio=110 target_cpu=052
arping 23089 [048]   380.056526: sched:sched_wakeup: comm=handler20 pid=20712 prio=110 target_cpu=008
arping 23089 [048]   380.056534: sched:sched_wakeup: comm=handler21 pid=20713 prio=110 target_cpu=040
arping 23089 [048]   380.056543: sched:sched_wakeup: comm=handler22 pid=20714 prio=110 target_cpu=002
arping 23089 [048]   380.056551: sched:sched_wakeup: comm=handler23 pid=20715 prio=110 target_cpu=054
arping 23089 [048]   380.056559: sched:sched_wakeup: comm=handler24 pid=20716 prio=110 target_cpu=022
arping 23089 [048]   380.056566: sched:sched_wakeup: comm=handler25 pid=20717 prio=110 target_cpu=010
arping 23089 [048]   380.056574: sched:sched_wakeup: comm=handler26 pid=20718 prio=110 target_cpu=060
arping 23089 [048]   380.056583: sched:sched_wakeup: comm=handler27 pid=20719 prio=110 target_cpu=012
arping 23089 [048]   380.056594: sched:sched_wakeup: comm=handler28 pid=20720 prio=110 target_cpu=014
arping 23089 [048]   380.056606: sched:sched_wakeup: comm=handler29 pid=20721 prio=110 target_cpu=036
arping 23089 [048]   380.056619: sched:sched_wakeup: comm=handler30 pid=20722 prio=110 target_cpu=068
arping 23089 [048]   380.056627: sched:sched_wakeup: comm=handler31 pid=20723 prio=110 target_cpu=034
arping 23089 [048]   380.056635: sched:sched_wakeup: comm=handler32 pid=20724 prio=110 target_cpu=000
arping 23089 [048]   380.056643: sched:sched_wakeup: comm=handler33 pid=20725 prio=110 target_cpu=042
arping 23089 [048]   380.056653: sched:sched_wakeup: comm=handler34 pid=20726 prio=110 target_cpu=050
arping 23089 [048]   380.056663: sched:sched_wakeup: comm=handler35 pid=20727 prio=110 target_cpu=046
handler4 20696 [044]   380.056664:  irq:softirq_entry: vec=3 [action=NET_RX]
arping 23089 [048]   380.056671: sched:sched_wakeup: comm=handler36 pid=20728 prio=110 target_cpu=038
arping 23089 [048]   380.056679: sched:sched_wakeup: comm=handler37 pid=20729 prio=110 target_cpu=006
arping 23089 [048]   380.056687: sched:sched_wakeup: comm=handler38 pid=20730 prio=110 target_cpu=050
arping 23089 [048]   380.056695: sched:sched_wakeup: comm=handler39 pid=20731 prio=110 target_cpu=062
arping 23089 [048]   380.056703: sched:sched_wakeup: comm=handler40 pid=20732 prio=110 target_cpu=028
arping 23089 [048]   380.056711: sched:sched_wakeup: comm=handler41 pid=20733 prio=110 target_cpu=002
arping 23089 [048]   380.056719: sched:sched_wakeup: comm=handler42 pid=20734 prio=110 target_cpu=008
handler4 20696 [044]   380.056722: sched:sched_wakeup: comm=handler1 pid=20693 prio=110 target_cpu=064
arping 23089 [048]   380.056727: sched:sched_wakeup: comm=handler43 pid=20735 prio=110 target_cpu=038
arping 23089 [048]   380.056735: sched:sched_wakeup: comm=handler44 pid=20736 prio=110 target_cpu=040
arping 23089 [048]   380.056742: sched:sched_wakeup: comm=handler45 pid=20737 prio=110 target_cpu=038
handler4 20696 [044]   380.056749: sched:sched_wakeup: comm=handler2 pid=20694 prio=110 target_cpu=046
arping 23089 [048]   380.056750: sched:sched_wakeup: comm=handler46 pid=20738 prio=110 target_cpu=068
arping 23089 [048]   380.056758: sched:sched_wakeup: comm=handler47 pid=20739 prio=110 target_cpu=024
arping 23089 [048]   380.056766: sched:sched_wakeup: comm=handler48 pid=20740 prio=110 target_cpu=008
handler4 20696 [044]   380.056768: sched:sched_wakeup: comm=handler5 pid=20697 prio=110 target_cpu=018
swapper     0 [041]   380.056771: sched:sched_wakeup: comm=handler3 pid=20695 prio=110 target_cpu=041
arping 23089 [048]   380.056773: sched:sched_wakeup: comm=handler49 pid=20741 prio=110 target_cpu=026
arping 23089 [048]   380.056781: sched:sched_wakeup: comm=handler50 pid=20742 prio=110 target_cpu=052
handler4 20696 [044]   380.056784: sched:sched_wakeup: comm=handler7 pid=20699 prio=110 target_cpu=066
arping 23089 [048]   380.056789: sched:sched_wakeup: comm=handler51 pid=20743 prio=110 target_cpu=058
arping 23089 [048]   380.056796: sched:sched_wakeup: comm=handler52 pid=20744 prio=110 target_cpu=028
handler4 20696 [044]   380.056798: sched:sched_wakeup: comm=handler8 pid=20700 prio=110 target_cpu=060
arping 23089 [048]   380.056804: sched:sched_wakeup: comm=handler53 pid=20745 prio=110 target_cpu=016
handler4 20696 [044]   380.056809: sched:sched_wakeup: comm=handler9 pid=20701 prio=110 target_cpu=046
arping 23089 [048]   380.056812: sched:sched_wakeup: comm=handler54 pid=20746 prio=110 target_cpu=038
handler4 20696 [044]   380.056819: sched:sched_wakeup: comm=handler10 pid=20702 prio=110 target_cpu=004
handler4 20696 [044]   380.056832: sched:sched_wakeup: comm=handler12 pid=20704 prio=110 target_cpu=024
handler4 20696 [044]   380.056843: sched:sched_wakeup: comm=handler13 pid=20705 prio=110 target_cpu=016
handler12 20704 [024]   380.056855:  irq:softirq_entry: vec=1 [action=TIMER]
handler1 20693 [064]   380.056855:  irq:softirq_entry: vec=9 [action=RCU]
handler4 20696 [044]   380.056858: sched:sched_wakeup: comm=handler14 pid=20706 prio=110 target_cpu=056
handler4 20696 [044]   380.056870: sched:sched_wakeup: comm=handler15 pid=20707 prio=110 target_cpu=020
handler12 20704 [024]   380.056875: sched:sched_wakeup: comm=kworker/24:1 pid=556 prio=120 target_cpu=024
handler4 20696 [044]   380.056881: sched:sched_wakeup: comm=handler16 pid=20708 prio=110 target_cpu=070
handler12 20704 [024]   380.056883:  irq:softirq_entry: vec=9 [action=RCU]
swapper     0 [045]   380.056888: sched:sched_wakeup: comm=handler11 pid=20703 prio=110 target_cpu=045
handler4 20696 [044]   380.056891: sched:sched_wakeup: comm=handler17 pid=20709 prio=110 target_cpu=026
handler4 20696 [044]   380.056901: sched:sched_wakeup: comm=handler18 pid=20710 prio=110 target_cpu=028
handler4 20696 [044]   380.056911: sched:sched_wakeup: comm=handler19 pid=20711 prio=110 target_cpu=052
handler4 20696 [044]   380.056921: sched:sched_wakeup: comm=handler20 pid=20712 prio=110 target_cpu=008
handler1 20693 [064]   380.056928:  irq:softirq_entry: vec=3 [action=NET_RX]
handler4 20696 [044]   380.056934: sched:sched_wakeup: comm=handler21 pid=20713 prio=110 target_cpu=040
handler4 20696 [044]   380.056950: sched:sched_wakeup: comm=handler22 pid=20714 prio=110 target_cpu=002
handler1 20693 [064]   380.056952: sched:sched_wakeup: comm=arping pid=23089 prio=120 target_cpu=048
handler4 20696 [044]   380.056968: sched:sched_wakeup: comm=handler23 pid=20715 prio=110 target_cpu=054
handler4 20696 [044]   380.056981: sched:sched_wakeup: comm=handler24 pid=20716 prio=110 target_cpu=022
handler4 20696 [044]   380.056991: sched:sched_wakeup: comm=handler25 pid=20717 prio=110 target_cpu=010
handler4 20696 [044]   380.057002: sched:sched_wakeup: comm=handler26 pid=20718 prio=110 target_cpu=060
handler4 20696 [044]   380.057016: sched:sched_wakeup: comm=handler27 pid=20719 prio=110 target_cpu=012
handler4 20696 [044]   380.057031: sched:sched_wakeup: comm=handler28 pid=20720 prio=110 target_cpu=014
handler4 20696 [044]   380.057046: sched:sched_wakeup: comm=handler29 pid=20721 prio=110 target_cpu=036
handler4 20696 [044]   380.057062: sched:sched_wakeup: comm=handler30 pid=20722 prio=110 target_cpu=068
handler4 20696 [044]   380.057079: sched:sched_wakeup: comm=handler31 pid=20723 prio=110 target_cpu=034
handler4 20696 [044]   380.057091: sched:sched_wakeup: comm=handler32 pid=20724 prio=110 target_cpu=000
handler4 20696 [044]   380.057102: sched:sched_wakeup: comm=handler33 pid=20725 prio=110 target_cpu=042
handler4 20696 [044]   380.057113: sched:sched_wakeup: comm=handler34 pid=20726 prio=110 target_cpu=050
handler4 20696 [044]   380.057123: sched:sched_wakeup: comm=handler35 pid=20727 prio=110 target_cpu=046
handler4 20696 [044]   380.057133: sched:sched_wakeup: comm=handler36 pid=20728 prio=110 target_cpu=038
handler4 20696 [044]   380.057143: sched:sched_wakeup: comm=handler37 pid=20729 prio=110 target_cpu=006
handler4 20696 [044]   380.057153: sched:sched_wakeup: comm=handler38 pid=20730 prio=110 target_cpu=050
handler4 20696 [044]   380.057164: sched:sched_wakeup: comm=handler39 pid=20731 prio=110 target_cpu=062
handler4 20696 [044]   380.057175: sched:sched_wakeup: comm=handler40 pid=20732 prio=110 target_cpu=028
handler4 20696 [044]   380.057185: sched:sched_wakeup: comm=handler41 pid=20733 prio=110 target_cpu=002
handler4 20696 [044]   380.057195: sched:sched_wakeup: comm=handler42 pid=20734 prio=110 target_cpu=008
handler4 20696 [044]   380.057208: sched:sched_wakeup: comm=handler43 pid=20735 prio=110 target_cpu=038
handler4 20696 [044]   380.057222: sched:sched_wakeup: comm=handler44 pid=20736 prio=110 target_cpu=040
handler4 20696 [044]   380.057232: sched:sched_wakeup: comm=handler45 pid=20737 prio=110 target_cpu=038
handler4 20696 [044]   380.057242: sched:sched_wakeup: comm=handler46 pid=20738 prio=110 target_cpu=068
handler4 20696 [044]   380.057253: sched:sched_wakeup: comm=handler47 pid=20739 prio=110 target_cpu=024
handler4 20696 [044]   380.057262: sched:sched_wakeup: comm=handler48 pid=20740 prio=110 target_cpu=008
handler4 20696 [044]   380.057276: sched:sched_wakeup: comm=handler49 pid=20741 prio=110 target_cpu=026
handler4 20696 [044]   380.057289: sched:sched_wakeup: comm=handler50 pid=20742 prio=110 target_cpu=052
handler4 20696 [044]   380.057300: sched:sched_wakeup: comm=handler51 pid=20743 prio=110 target_cpu=058
handler4 20696 [044]   380.057309: sched:sched_wakeup: comm=handler52 pid=20744 prio=110 target_cpu=028
handler4 20696 [044]   380.057320: sched:sched_wakeup: comm=handler53 pid=20745 prio=110 target_cpu=016
handler4 20696 [044]   380.057330: sched:sched_wakeup: comm=handler54 pid=20746 prio=110 target_cpu=038
handler4 20696 [044]   380.057350:  irq:softirq_entry: vec=1 [action=TIMER]
handler4 20696 [044]   380.057360: sched:sched_wakeup: comm=kworker/44:2 pid=20487 prio=120 target_cpu=044
handler4 20696 [044]   380.057367:  irq:softirq_entry: vec=9 [action=RCU]
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
swapper     0 [030]   380.352968: sched:sched_wakeup: comm=handler1 pid=20693 prio=110 target_cpu=064
swapper     0 [030]   380.352982: sched:sched_wakeup: comm=handler2 pid=20694 prio=110 target_cpu=046
swapper     0 [030]   380.352994: sched:sched_wakeup: comm=handler4 pid=20696 prio=110 target_cpu=044
swapper     0 [030]   380.353001: sched:sched_wakeup: comm=handler5 pid=20697 prio=110 target_cpu=018
swapper     0 [030]   380.353008: sched:sched_wakeup: comm=handler7 pid=20699 prio=110 target_cpu=066
swapper     0 [030]   380.353018: sched:sched_wakeup: comm=handler8 pid=20700 prio=110 target_cpu=060
swapper     0 [030]   380.353030: sched:sched_wakeup: comm=handler9 pid=20701 prio=110 target_cpu=030
swapper     0 [030]   380.353040: sched:sched_wakeup: comm=handler10 pid=20702 prio=110 target_cpu=004
swapper     0 [030]   380.353054: sched:sched_wakeup: comm=handler12 pid=20704 prio=110 target_cpu=024
swapper     0 [030]   380.353066: sched:sched_wakeup: comm=handler13 pid=20705 prio=110 target_cpu=016
swapper     0 [030]   380.353077: sched:sched_wakeup: comm=handler14 pid=20706 prio=110 target_cpu=056
swapper     0 [041]   380.353077: sched:sched_wakeup: comm=handler3 pid=20695 prio=110 target_cpu=041
swapper     0 [030]   380.353087: sched:sched_wakeup: comm=handler15 pid=20707 prio=110 target_cpu=020
swapper     0 [030]   380.353099: sched:sched_wakeup: comm=handler16 pid=20708 prio=110 target_cpu=070
swapper     0 [045]   380.353108: sched:sched_wakeup: comm=handler11 pid=20703 prio=110 target_cpu=045
swapper     0 [030]   380.353111: sched:sched_wakeup: comm=handler17 pid=20709 prio=110 target_cpu=026
swapper     0 [030]   380.353123: sched:sched_wakeup: comm=handler18 pid=20710 prio=110 target_cpu=028
swapper     0 [030]   380.353134: sched:sched_wakeup: comm=handler19 pid=20711 prio=110 target_cpu=052
swapper     0 [030]   380.353145: sched:sched_wakeup: comm=handler20 pid=20712 prio=110 target_cpu=008
swapper     0 [030]   380.353156: sched:sched_wakeup: comm=handler21 pid=20713 prio=110 target_cpu=040
swapper     0 [030]   380.353168: sched:sched_wakeup: comm=handler22 pid=20714 prio=110 target_cpu=002
handler7 20699 [066]   380.353177:  irq:softirq_entry: vec=3 [action=NET_RX]
swapper     0 [030]   380.353178: sched:sched_wakeup: comm=handler23 pid=20715 prio=110 target_cpu=054
swapper     0 [030]   380.353190: sched:sched_wakeup: comm=handler24 pid=20716 prio=110 target_cpu=022
swapper     0 [030]   380.353204: sched:sched_wakeup: comm=handler25 pid=20717 prio=110 target_cpu=010
swapper     0 [030]   380.353214: sched:sched_wakeup: comm=handler26 pid=20718 prio=110 target_cpu=060
swapper     0 [030]   380.353225: sched:sched_wakeup: comm=handler27 pid=20719 prio=110 target_cpu=012
swapper     0 [030]   380.353236: sched:sched_wakeup: comm=handler28 pid=20720 prio=110 target_cpu=014
swapper     0 [030]   380.353247: sched:sched_wakeup: comm=handler29 pid=20721 prio=110 target_cpu=036
swapper     0 [030]   380.353259: sched:sched_wakeup: comm=handler30 pid=20722 prio=110 target_cpu=068
swapper     0 [030]   380.353270: sched:sched_wakeup: comm=handler31 pid=20723 prio=110 target_cpu=034
swapper     0 [030]   380.353280: sched:sched_wakeup: comm=handler32 pid=20724 prio=110 target_cpu=000
swapper     0 [030]   380.353288: sched:sched_wakeup: comm=handler33 pid=20725 prio=110 target_cpu=042
swapper     0 [030]   380.353295: sched:sched_wakeup: comm=handler34 pid=20726 prio=110 target_cpu=050
swapper     0 [030]   380.353302: sched:sched_wakeup: comm=handler35 pid=20727 prio=110 target_cpu=046
swapper     0 [030]   380.353309: sched:sched_wakeup: comm=handler36 pid=20728 prio=110 target_cpu=038
swapper     0 [030]   380.353317: sched:sched_wakeup: comm=handler37 pid=20729 prio=110 target_cpu=006
swapper     0 [030]   380.353326: sched:sched_wakeup: comm=handler38 pid=20730 prio=110 target_cpu=024
swapper     0 [030]   380.353333: sched:sched_wakeup: comm=handler39 pid=20731 prio=110 target_cpu=062
swapper     0 [030]   380.353340: sched:sched_wakeup: comm=handler40 pid=20732 prio=110 target_cpu=028
swapper     0 [030]   380.353348: sched:sched_wakeup: comm=handler41 pid=20733 prio=110 target_cpu=002
swapper     0 [030]   380.353354: sched:sched_wakeup: comm=handler42 pid=20734 prio=110 target_cpu=008
swapper     0 [030]   380.353361: sched:sched_wakeup: comm=handler43 pid=20735 prio=110 target_cpu=038
swapper     0 [030]   380.353368: sched:sched_wakeup: comm=handler44 pid=20736 prio=110 target_cpu=040
swapper     0 [030]   380.353375: sched:sched_wakeup: comm=handler45 pid=20737 prio=110 target_cpu=038
swapper     0 [030]   380.353382: sched:sched_wakeup: comm=handler46 pid=20738 prio=110 target_cpu=068
swapper     0 [030]   380.353389: sched:sched_wakeup: comm=handler47 pid=20739 prio=110 target_cpu=024
swapper     0 [030]   380.353396: sched:sched_wakeup: comm=handler48 pid=20740 prio=110 target_cpu=008
swapper     0 [030]   380.353403: sched:sched_wakeup: comm=handler49 pid=20741 prio=110 target_cpu=026
swapper     0 [030]   380.353410: sched:sched_wakeup: comm=handler50 pid=20742 prio=110 target_cpu=052
swapper     0 [030]   380.353417: sched:sched_wakeup: comm=handler51 pid=20743 prio=110 target_cpu=058
swapper     0 [030]   380.353424: sched:sched_wakeup: comm=handler52 pid=20744 prio=110 target_cpu=028
swapper     0 [030]   380.353431: sched:sched_wakeup: comm=handler53 pid=20745 prio=110 target_cpu=016
swapper     0 [030]   380.353438: sched:sched_wakeup: comm=handler54 pid=20746 prio=110 target_cpu=038
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
	    7f7496318b0c __run_exit_handlers+0x13c (/usr/lib64/libc-2.28.so)
	ffffffff88f331a0 search_binary_handler+0x90 ([kernel.kallsyms])
	ffffffff89118008 cursor_timer_handler+0x28 ([kernel.kallsyms])
[root@dell-per740-40 ~]# 



[root@dell-per740-40 ~]# uname -a
Linux dell-per740-40.rhts.eng.pek2.redhat.com 4.18.0-323.el8.x86_64 #1 SMP Wed Jul 14 12:52:14 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux


[root@dell-per740-40 ~]# rpm -qa | grep openv
python3-openvswitch2.15-2.15.0-32.el8fdp.x86_64
openvswitch2.15-2.15.0-32.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-28.el8fdp.noarch
openvswitch2.15-test-2.15.0-32.el8fdp.noarch
[root@dell-per740-40 ~]# 


with kernel update--------------------------------------------------------------------------------------------------------

[root@dell-per740-40 ~]# perf script | grep arping
arping  9172 [017]   479.735555:  irq:softirq_entry: vec=9 [action=RCU]
swapper     0 [017]   479.735739: sched:sched_wakeup: comm=arping pid=9172 prio=120 target_cpu=017
arping  9172 [017]   479.737554:  irq:softirq_entry: vec=7 [action=SCHED]
arping  9172 [017]   479.738554:  irq:softirq_entry: vec=9 [action=RCU]
arping  9172 [017]   479.738699: sched:sched_wakeup: comm=kworker/u146:0 pid=8905 prio=120 target_cpu=041
arping  9172 [017]   479.738730:  irq:softirq_entry: vec=3 [action=NET_RX]
arping  9172 [017]   479.738757: sched:sched_wakeup: comm=handler290 pid=9086 prio=110 target_cpu=043
arping  9172 [017]   479.738777: sched:sched_wakeup: comm=handler291 pid=9087 prio=110 target_cpu=011
arping  9172 [017]   479.738788: sched:sched_wakeup: comm=handler294 pid=9088 prio=110 target_cpu=049
arping  9172 [017]   479.738797: sched:sched_wakeup: comm=handler293 pid=9089 prio=110 target_cpu=029
arping  9172 [017]   479.738806: sched:sched_wakeup: comm=handler295 pid=9090 prio=110 target_cpu=069
arping  9172 [017]   479.738816: sched:sched_wakeup: comm=handler296 pid=9091 prio=110 target_cpu=045
arping  9172 [017]   479.738824: sched:sched_wakeup: comm=handler292 pid=9092 prio=110 target_cpu=051
arping  9172 [017]   479.738832: sched:sched_wakeup: comm=handler299 pid=9093 prio=110 target_cpu=059
arping  9172 [017]   479.738840: sched:sched_wakeup: comm=handler300 pid=9094 prio=110 target_cpu=055
arping  9172 [017]   479.738849: sched:sched_wakeup: comm=handler297 pid=9095 prio=110 target_cpu=057
arping  9172 [017]   479.738858: sched:sched_wakeup: comm=handler298 pid=9096 prio=110 target_cpu=025
arping  9172 [017]   479.738865: sched:sched_wakeup: comm=handler305 pid=9097 prio=110 target_cpu=063
arping  9172 [017]   479.738873: sched:sched_wakeup: comm=handler301 pid=9098 prio=110 target_cpu=019
arping  9172 [017]   479.738881: sched:sched_wakeup: comm=handler302 pid=9099 prio=110 target_cpu=003
arping  9172 [017]   479.738890: sched:sched_wakeup: comm=handler303 pid=9100 prio=110 target_cpu=067
arping  9172 [017]   479.738899: sched:sched_wakeup: comm=handler304 pid=9101 prio=110 target_cpu=035
arping  9172 [017]   479.738907: sched:sched_wakeup: comm=handler306 pid=9102 prio=110 target_cpu=037
arping  9172 [017]   479.738920: sched:sched_wakeup: comm=handler307 pid=9103 prio=110 target_cpu=041
arping  9172 [017]   479.738928: sched:sched_wakeup: comm=handler308 pid=9104 prio=110 target_cpu=001
arping  9172 [017]   479.738936: sched:sched_wakeup: comm=handler309 pid=9105 prio=110 target_cpu=045
arping  9172 [017]   479.738944: sched:sched_wakeup: comm=handler310 pid=9106 prio=110 target_cpu=039
arping  9172 [017]   479.738953: sched:sched_wakeup: comm=handler311 pid=9107 prio=110 target_cpu=013
arping  9172 [017]   479.738961: sched:sched_wakeup: comm=handler312 pid=9108 prio=110 target_cpu=071
arping  9172 [017]   479.738969: sched:sched_wakeup: comm=handler313 pid=9109 prio=110 target_cpu=005
arping  9172 [017]   479.738978: sched:sched_wakeup: comm=handler314 pid=9110 prio=110 target_cpu=021
arping  9172 [017]   479.738986: sched:sched_wakeup: comm=handler315 pid=9111 prio=110 target_cpu=061
arping  9172 [017]   479.738995: sched:sched_wakeup: comm=handler316 pid=9112 prio=110 target_cpu=047
arping  9172 [017]   479.739004: sched:sched_wakeup: comm=handler317 pid=9113 prio=110 target_cpu=065
arping  9172 [017]   479.739012: sched:sched_wakeup: comm=handler318 pid=9114 prio=110 target_cpu=015
arping  9172 [017]   479.739020: sched:sched_wakeup: comm=handler319 pid=9115 prio=110 target_cpu=027
arping  9172 [017]   479.739029: sched:sched_wakeup: comm=handler320 pid=9116 prio=110 target_cpu=023
arping  9172 [017]   479.739038: sched:sched_wakeup: comm=handler321 pid=9117 prio=110 target_cpu=031
arping  9172 [017]   479.739047: sched:sched_wakeup: comm=handler322 pid=9118 prio=110 target_cpu=041
arping  9172 [017]   479.739055: sched:sched_wakeup: comm=handler323 pid=9119 prio=110 target_cpu=069
arping  9172 [017]   479.739063: sched:sched_wakeup: comm=handler324 pid=9120 prio=110 target_cpu=053
arping  9172 [017]   479.739075: sched:sched_wakeup: comm=handler325 pid=9121 prio=110 target_cpu=005
arping  9172 [017]   479.739088: sched:sched_wakeup: comm=handler326 pid=9122 prio=110 target_cpu=009
arping  9172 [017]   479.739101: sched:sched_wakeup: comm=handler327 pid=9123 prio=110 target_cpu=035
arping  9172 [017]   479.739110: sched:sched_wakeup: comm=handler328 pid=9124 prio=110 target_cpu=041
arping  9172 [017]   479.739119: sched:sched_wakeup: comm=handler329 pid=9125 prio=110 target_cpu=035
arping  9172 [017]   479.739129: sched:sched_wakeup: comm=handler330 pid=9126 prio=110 target_cpu=015
arping  9172 [017]   479.739138: sched:sched_wakeup: comm=handler331 pid=9127 prio=110 target_cpu=037
arping  9172 [017]   479.739146: sched:sched_wakeup: comm=handler332 pid=9128 prio=110 target_cpu=065
arping  9172 [017]   479.739156: sched:sched_wakeup: comm=handler333 pid=9129 prio=110 target_cpu=035
arping  9172 [017]   479.739164: sched:sched_wakeup: comm=handler334 pid=9130 prio=110 target_cpu=051
arping  9172 [017]   479.739173: sched:sched_wakeup: comm=handler335 pid=9131 prio=110 target_cpu=067
arping  9172 [017]   479.739181: sched:sched_wakeup: comm=handler336 pid=9132 prio=110 target_cpu=027
arping  9172 [017]   479.739199: sched:sched_wakeup: comm=handler341 pid=9137 prio=110 target_cpu=023
arping  9172 [017]   479.739207: sched:sched_wakeup: comm=handler342 pid=9138 prio=110 target_cpu=045
handler291  9087 [011]   479.739276: sched:sched_wakeup: comm=arping pid=9172 prio=120 target_cpu=017
arping  9172 [017]   479.739313: sched:sched_wakeup: comm=kworker/u146:0 pid=8905 prio=120 target_cpu=041
swapper     0 [017]   480.739220: sched:sched_wakeup: comm=arping pid=9172 prio=120 target_cpu=017
arping  9172 [017]   480.739272: sched:sched_wakeup: comm=kworker/u146:0 pid=8905 prio=120 target_cpu=041
arping  9172 [017]   480.739470:  irq:softirq_entry: vec=7 [action=SCHED]
arping  9172 [017]   480.739483:  irq:softirq_entry: vec=9 [action=RCU]
swapper     0 [017]   480.745556: sched:sched_wakeup: comm=arping pid=9172 prio=120 target_cpu=017
ksoftirqd/17   116 [017]   480.753587: sched:sched_wakeup: comm=arping pid=9172 prio=120 target_cpu=017
arping  9172 [017]   480.753686: sched:sched_wakeup: comm=bash pid=7890 prio=120 target_cpu=051
[root@dell-per740-40 ~]# uname -a
Linux dell-per740-40.rhts.eng.pek2.redhat.com 4.18.0-328.el8.mr1064_210805_1629.x86_64 #1 SMP Thu Aug 5 16:42:01 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
[root@dell-per740-40 ~]# uname -r
4.18.0-328.el8.mr1064_210805_1629.x86_64
[root@dell-per740-40 ~]# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-328.el8.mr1064_210805_1629.x86_64 root=/dev/mapper/rhel_dell--per740--40-root ro crashkernel=auto resume=/dev/mapper/rhel_dell--per740--40-swap rd.lvm.lv=rhel_dell-per740-40/root rd.lvm.lv=rhel_dell-per740-40/swap console=ttyS0,115200n81
[root@dell-per740-40 ~]#

Comment 9 Mark Gray 2021-08-16 09:57:48 UTC
Can you confirm per-cpu dispatch is enabled?

`ovs-appctl dpif-netlink/dispatch-mode`

Comment 10 Hekai Wang 2021-08-17 02:36:53 UTC
(In reply to Mark Gray from comment #9)
> Can you confirm per-cpu dispatch is enabled?
> 
> `ovs-appctl dpif-netlink/dispatch-mode`

Hi Mark
Both old kernel (4.18.0-323.el8.x86_64) and new build kernel 4.18.0-328.el8.mr1064_210805_1629.x86_64 do not have this command 

[root@dell-per740-40 ~]# ovs-appctl list-commands | grep dpif
  dpif-netdev/bond-show   [dp]
  dpif-netdev/pmd-perf-log-set on|off [-b before] [-a after] [-e|-ne] [-us usec] [-q qlen]
  dpif-netdev/pmd-perf-show [-nh] [-it iter-history-len] [-ms ms-history-len] [-pmd core] [dp]
  dpif-netdev/pmd-rxq-rebalance [dp]
  dpif-netdev/pmd-rxq-show [-pmd core] [dp]
  dpif-netdev/pmd-stats-clear [-pmd core] [dp]
  dpif-netdev/pmd-stats-show [-pmd core] [dp]
  dpif-netdev/subtable-lookup-prio-get 
  dpif-netdev/subtable-lookup-prio-set [lookup_func] [prio]
  dpif/dump-dps           
  dpif/dump-flows         [-m] [--names | --no-names] bridge
  dpif/set-dp-features    bridge
  dpif/show               
  dpif/show-dp-features   bridge
[root@dell-per740-40 ~]# ovs-appctl dpif-netlink/dispatch-mode
"dpif-netlink/dispatch-mode" is not a valid command (use "list-commands" to see a list of valid commands)
ovs-appctl: ovs-vswitchd: server returned an error



Am i missing something ?

Thanks.

Comment 11 Mark Gray 2021-08-17 07:31:12 UTC
Yes, I see you are missing the userspace component. This fix requires an updated kernel *and* an updated userspace package. The patch will land in the userspace openvswitch2.16 package which is not available yet as far as I know. @tredaelli might know when this will be available.

Comment 12 Hekai Wang 2021-08-20 08:43:21 UTC
modprobe -r veth
ip netns del left
ip netns del right
ip netns add left
ip netns add right
ip link add center-left type veth peer name left0 netns left
ip link add center-right type veth peer name right0 netns right
ip link set center-left up
ip link set center-right up
ip netns exec left ip link set left0 up
ip netns exec left ip addr add 172.31.110.10/24 dev left0
ip netns exec right ip link set right0 up
ip netns exec right ip addr add 172.31.110.11/24 dev right0

systemctl start openvswitch
ovs-vsctl --if-exists del-br br0 
ovs-vsctl add-br br0
ovs-vsctl add-port br0 center-right
ovs-vsctl add-port br0 center-left


perf record -e sched:sched_wakeup,irq:softirq_entry -ag

ip netns exec left arping -I left0 -c 1 172.31.110.11


with old kernel 
[root@dell-per740-40 ~]# uname -a
Linux dell-per740-40.rhts.eng.pek2.redhat.com 4.18.0-323.el8.x86_64 #1 SMP Wed Jul 14 12:52:14 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux

ovs version 
[root@dell-per740-40 ~]# rpm -qa | grep openv
openvswitch2.16-2.16.0-1.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-28.el8fdp.noarch
python3-openvswitch2.16-2.16.0-1.el8fdp.x86_64
openvswitch2.16-test-2.16.0-1.el8fdp.noarch


perf script 

[root@dell-per740-40 ~]# perf script | grep arp
swapper     0 [002]  1983.763394: sched:sched_wakeup: comm=arping pid=24006 prio=120 target_cpu=002
arping 24006 [002]  1983.764254:  irq:softirq_entry: vec=7 [action=SCHED]
arping 24006 [002]  1983.765253:  irq:softirq_entry: vec=9 [action=RCU]
arping 24006 [002]  1983.765273:  irq:softirq_entry: vec=9 [action=RCU]
arping 24006 [002]  1983.765283:  irq:softirq_entry: vec=9 [action=RCU]
arping 24006 [002]  1983.766382: sched:sched_wakeup: comm=kworker/u145:16 pid=461 prio=120 target_cpu=062
arping 24006 [002]  1983.766413:  irq:softirq_entry: vec=3 [action=NET_RX]
arping 24006 [002]  1983.766498: sched:sched_wakeup: comm=handler171 pid=21281 prio=110 target_cpu=048
arping 24006 [002]  1983.766525: sched:sched_wakeup: comm=handler175 pid=21285 prio=110 target_cpu=008
arping 24006 [002]  1983.766536: sched:sched_wakeup: comm=handler176 pid=21286 prio=110 target_cpu=056
arping 24006 [002]  1983.766546: sched:sched_wakeup: comm=handler177 pid=21287 prio=110 target_cpu=004
arping 24006 [002]  1983.766555: sched:sched_wakeup: comm=handler178 pid=21288 prio=110 target_cpu=032
arping 24006 [002]  1983.766564: sched:sched_wakeup: comm=handler179 pid=21289 prio=110 target_cpu=058
arping 24006 [002]  1983.766572: sched:sched_wakeup: comm=handler180 pid=21290 prio=110 target_cpu=060
arping 24006 [002]  1983.766581: sched:sched_wakeup: comm=handler181 pid=21291 prio=110 target_cpu=062
arping 24006 [002]  1983.766590: sched:sched_wakeup: comm=handler182 pid=21292 prio=110 target_cpu=064
arping 24006 [002]  1983.766599: sched:sched_wakeup: comm=handler183 pid=21293 prio=110 target_cpu=066
arping 24006 [002]  1983.766623: sched:sched_wakeup: comm=handler190 pid=21300 prio=110 target_cpu=034
arping 24006 [002]  1983.766635: sched:sched_wakeup: comm=handler191 pid=21301 prio=110 target_cpu=006
arping 24006 [002]  1983.766650: sched:sched_wakeup: comm=handler192 pid=21302 prio=110 target_cpu=010
arping 24006 [002]  1983.766660: sched:sched_wakeup: comm=handler193 pid=21303 prio=110 target_cpu=014
arping 24006 [002]  1983.766676: sched:sched_wakeup: comm=handler194 pid=21304 prio=110 target_cpu=016
arping 24006 [002]  1983.766703: sched:sched_wakeup: comm=handler195 pid=21305 prio=110 target_cpu=000
arping 24006 [002]  1983.766718: sched:sched_wakeup: comm=handler196 pid=21306 prio=110 target_cpu=018
arping 24006 [002]  1983.766730: sched:sched_wakeup: comm=handler198 pid=21308 prio=110 target_cpu=038
handler152 21261 [067]  1983.767836: sched:sched_wakeup: comm=arping pid=24006 prio=120 target_cpu=069
arping 24006 [069]  1983.767906: sched:sched_wakeup: comm=kworker/u146:26 pid=495 prio=120 target_cpu=027
swapper     0 [069]  1984.766785: sched:sched_wakeup: comm=arping pid=24006 prio=120 target_cpu=069
arping 24006 [069]  1984.766828: sched:sched_wakeup: comm=kworker/u146:26 pid=495 prio=120 target_cpu=027
arping 24006 [069]  1984.767170:  irq:softirq_entry: vec=7 [action=SCHED]
arping 24006 [069]  1984.767184:  irq:softirq_entry: vec=9 [action=RCU]
swapper     0 [069]  1984.775247: sched:sched_wakeup: comm=arping pid=24006 prio=120 target_cpu=069
ksoftirqd/69   428 [069]  1984.783303: sched:sched_wakeup: comm=arping pid=24006 prio=120 target_cpu=069



with new kernel



[root@dell-per740-40 ~]# uname -a
Linux dell-per740-40.rhts.eng.pek2.redhat.com 4.18.0-328.el8.mr1064_210805_1629.x86_64 #1 SMP Thu Aug 5 16:42:01 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
[root@dell-per740-40 ~]# uname -r
4.18.0-328.el8.mr1064_210805_1629.x86_64
[root@dell-per740-40 ~]# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-328.el8.mr1064_210805_1629.x86_64 root=/dev/mapper/rhel_dell--per740--40-root ro crashkernel=auto resume=/dev/mapper/rhel_dell--per740--40-swap rd.lvm.lv=rhel_dell-per740-40/root rd.lvm.lv=rhel_dell-per740-40/swap console=ttyS0,115200n81


[root@dell-per740-40 ~]# ovs-appctl dpif-netlink/dispatch-mode
ovs-system: per-cpu dispatch mode

[root@dell-per740-40 ~]# perf script | grep arping
swapper     0 [013]    82.841579: sched:sched_wakeup: comm=arping pid=9077 prio=120 target_cpu=013
arping  9077 [013]    82.841960:  irq:softirq_entry: vec=9 [action=RCU]
arping  9077 [013]    82.843960:  irq:softirq_entry: vec=7 [action=SCHED]
arping  9077 [013]    82.843978:  irq:softirq_entry: vec=9 [action=RCU]
arping  9077 [013]    82.843992:  irq:softirq_entry: vec=9 [action=RCU]
arping  9077 [013]    82.844001:  irq:softirq_entry: vec=9 [action=RCU]
arping  9077 [013]    82.844625: sched:sched_wakeup: comm=kworker/u146:3 pid=464 prio=120 target_cpu=011
arping  9077 [013]    82.844652:  irq:softirq_entry: vec=3 [action=NET_RX]
arping  9077 [013]    82.844677: sched:sched_wakeup: comm=handler379 pid=8972 prio=110 target_cpu=047
handler413  9006 [043]    82.845255: sched:sched_wakeup: comm=arping pid=9077 prio=120 target_cpu=013
arping  9077 [013]    82.845293: sched:sched_wakeup: comm=kworker/u146:3 pid=464 prio=120 target_cpu=011
swapper     0 [013]    83.844694: sched:sched_wakeup: comm=arping pid=9077 prio=120 target_cpu=013
arping  9077 [013]    83.844743: sched:sched_wakeup: comm=kworker/u146:3 pid=464 prio=120 target_cpu=011
arping  9077 [013]    83.844874:  irq:softirq_entry: vec=7 [action=SCHED]
arping  9077 [013]    83.844889:  irq:softirq_entry: vec=9 [action=RCU]
swapper     0 [013]    83.850967: sched:sched_wakeup: comm=arping pid=9077 prio=120 target_cpu=013
ksoftirqd/13    92 [013]    83.858998: sched:sched_wakeup: comm=arping pid=9077 prio=120 target_cpu=013
arping  9077 [013]    83.859099: sched:sched_wakeup: comm=bash pid=5254 prio=120 target_cpu=047


So , The result is bug was fixed on new ovs2.16 new kernel version .


Note You need to log in before you can comment on or make changes to this bug.