Bug 2222860

Summary:	setting n-handler-threads didn't work
Product:	Red Hat Enterprise Linux Fast Datapath	Reporter:	mhou <mhou>
Component:	openvswitch	Assignee:	Michael Santana <msantana>
openvswitch sub component:	daemons and tools	QA Contact:	mhou <mhou>
Status:	NEW ---	Docs Contact:
Severity:	high
Priority:	unspecified	CC:	ctrautma, fleitner, jhsiao, kzhang, msantana
Version:	RHEL 8.0
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description mhou 2023-07-14 07:49:10 UTC

Description of problem:
when change n-handler-threads and n-revalidator-threads through ovs-vsctl, configuration didn't effected.

Version-Release number of selected component (if applicable):
openvswitch2.17-2.17.0-105.el8fdp/openvswitch3.1-3.1.0-39.el8fdp

How reproducible: 100%


Steps to Reproduce:
1.keep numa0 cpus except core0 as isolation cpus, any other cpu of numa1 and core0 of numa0 as housekeeping. The Housekeeping cpu have 29. 

# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-499.rt7.288.el8.x86_64 root=/dev/mapper/rhel_dell--per750--03-root ro ksdevice=bootif pci=realloc resume=/dev/mapper/rhel_dell--per750--03-swap rd.lvm.lv=rhel_dell-per750-03/root rd.lvm.lv=rhel_dell-per750-03/swap console=ttyS0,115200n81 skew_tick=1 isolcpus=managed_irq,domain,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54 intel_pstate=disable nosoftlockup tsc=reliable nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54 irqaffinity=0,1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55 crashkernel=1G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G rcutree.kthread_prio=51 default_hugepagesz=1G hugepagesz=1G hugepages=48 intel_iommu=on iommu=pt intel_idle.max_cstate=0 processor.max_cstate=0 intel_pstate=disable

# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54
node 0 size: 31257 MB
node 0 free: 951 MB
node 1 cpus: 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55
node 1 size: 31178 MB
node 1 free: 2283 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 
2. configure n-revalidator-threads=6. and other_config:n-handler-threads=24. OVS still configure handler to 31.
ovs-vsctl set Open_vSwitch . other_config:n-revalidator-threads=6
ovs-vsctl set Open_vSwitch . other_config:n-handler-threads=24

3. check the n-revalidator-threads change to 6, but handler threads still 31.
2023-07-11T02:07:13.277Z|00049|ofproto_dpif_upcall|INFO|Overriding n-handler-threads to 31, setting n-revalidator-threads to 6
2023-07-11T02:07:13.277Z|00050|ofproto_dpif_upcall|INFO|Starting 37 threads

Actual results:
1. n-handler-threads still 31

Expected results:
1. n-handler-threads shoule be 24

Additional info:

Comment 1 Michael Santana 2023-07-14 21:01:25 UTC

Hi @mhou,

n-handler-threads is not available on the newest openvswitch and kernel when using per-CPU dispatch mode. Openvswitch defaults to per-CPU dispatch in the latest kernel and OVS versions, and there is no way to change it. In effect n-handler-threads is only intended to be used in per-vport mode in older kernels. You can check which dispatch mode your system is running via `ovs-appctl dpif-netlink/dispatch-mode`. It is most likely in per-CPU dispatch mode which ignores n-handler-threads.

The reason you have 31 Handlers threads is to balance the upcalls that could happen on all the CPUs. The way that is calculated is via you will have a minimum of what ever number of active CPUs, 29 in this case, plus an additional extra threads calculated via the next prime number from 29, which is 31 in this case

I am working on ensuring that setting this parameter warns the user that making any changes has no effect and updating the documenttation if it's not clear enough that this setting is only intended to be used in per-vport mode and will not work in per-CPU mode

Comment 2 mhou 2023-07-15 09:04:46 UTC

Hello Michael

Current mode actually is per-cpu mode on my test bed.

# ovs-appctl dpif-netlink/dispatch-mode
ovs-system: per-cpu dispatch mode

The current ovs document didn't contain any section to describe what difference between per-CPU and per-vport.

http://www.openvswitch.org/support/dist-docs/ovs-vswitchd.conf.db.5.txt

       other_config  : n-handler-threads: optional string, containing an inte‐
       ger, at least 1
              Attempts to specify the number of threads for software datapaths
              to  use for handling new flows. Some datapaths may choose to ig‐
              nore this and it will be set to a sensible option for the  data‐
              path type.

              This  configuration  is  per datapath. If you have more than one
              software datapath (e.g. some  system  bridges  and  some  netdev
              bridges),  then the total number of threads is n-handler-threads
              times the number of software datapaths.

....
         other_config : n-handler-threads
                                     optional string, containing  an  integer,
                                     at least 1