Bug 2222860 - setting n-handler-threads didn't work
Summary: setting n-handler-threads didn't work
Keywords:
Status: NEW
Alias: None
Product: Red Hat Enterprise Linux Fast Datapath
Classification: Red Hat
Component: openvswitch
Version: RHEL 8.0
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: Michael Santana
QA Contact: mhou
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-07-14 07:49 UTC by mhou
Modified: 2023-07-15 09:04 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker FD-3009 0 None None None 2023-07-14 07:49:40 UTC

Description mhou 2023-07-14 07:49:10 UTC
Description of problem:
when change n-handler-threads and n-revalidator-threads through ovs-vsctl, configuration didn't effected.

Version-Release number of selected component (if applicable):
openvswitch2.17-2.17.0-105.el8fdp/openvswitch3.1-3.1.0-39.el8fdp

How reproducible: 100%


Steps to Reproduce:
1.keep numa0 cpus except core0 as isolation cpus, any other cpu of numa1 and core0 of numa0 as housekeeping. The Housekeeping cpu have 29. 

# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-499.rt7.288.el8.x86_64 root=/dev/mapper/rhel_dell--per750--03-root ro ksdevice=bootif pci=realloc resume=/dev/mapper/rhel_dell--per750--03-swap rd.lvm.lv=rhel_dell-per750-03/root rd.lvm.lv=rhel_dell-per750-03/swap console=ttyS0,115200n81 skew_tick=1 isolcpus=managed_irq,domain,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54 intel_pstate=disable nosoftlockup tsc=reliable nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54 irqaffinity=0,1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55 crashkernel=1G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G rcutree.kthread_prio=51 default_hugepagesz=1G hugepagesz=1G hugepages=48 intel_iommu=on iommu=pt intel_idle.max_cstate=0 processor.max_cstate=0 intel_pstate=disable

# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54
node 0 size: 31257 MB
node 0 free: 951 MB
node 1 cpus: 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55
node 1 size: 31178 MB
node 1 free: 2283 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 
2. configure n-revalidator-threads=6. and other_config:n-handler-threads=24. OVS still configure handler to 31.
ovs-vsctl set Open_vSwitch . other_config:n-revalidator-threads=6
ovs-vsctl set Open_vSwitch . other_config:n-handler-threads=24

3. check the n-revalidator-threads change to 6, but handler threads still 31.
2023-07-11T02:07:13.277Z|00049|ofproto_dpif_upcall|INFO|Overriding n-handler-threads to 31, setting n-revalidator-threads to 6
2023-07-11T02:07:13.277Z|00050|ofproto_dpif_upcall|INFO|Starting 37 threads

Actual results:
1. n-handler-threads still 31

Expected results:
1. n-handler-threads shoule be 24

Additional info:

Comment 1 Michael Santana 2023-07-14 21:01:25 UTC
Hi @mhou,

n-handler-threads is not available on the newest openvswitch and kernel when using per-CPU dispatch mode. Openvswitch defaults to per-CPU dispatch in the latest kernel and OVS versions, and there is no way to change it. In effect n-handler-threads is only intended to be used in per-vport mode in older kernels. You can check which dispatch mode your system is running via `ovs-appctl dpif-netlink/dispatch-mode`. It is most likely in per-CPU dispatch mode which ignores n-handler-threads.

The reason you have 31 Handlers threads is to balance the upcalls that could happen on all the CPUs. The way that is calculated is via you will have a minimum of what ever number of active CPUs, 29 in this case, plus an additional extra threads calculated via the next prime number from 29, which is 31 in this case

I am working on ensuring that setting this parameter warns the user that making any changes has no effect and updating the documenttation if it's not clear enough that this setting is only intended to be used in per-vport mode and will not work in per-CPU mode

Comment 2 mhou 2023-07-15 09:04:46 UTC
Hello Michael

Current mode actually is per-cpu mode on my test bed.

# ovs-appctl dpif-netlink/dispatch-mode
ovs-system: per-cpu dispatch mode

The current ovs document didn't contain any section to describe what difference between per-CPU and per-vport.

http://www.openvswitch.org/support/dist-docs/ovs-vswitchd.conf.db.5.txt

       other_config  : n-handler-threads: optional string, containing an inte‐
       ger, at least 1
              Attempts to specify the number of threads for software datapaths
              to  use for handling new flows. Some datapaths may choose to ig‐
              nore this and it will be set to a sensible option for the  data‐
              path type.

              This  configuration  is  per datapath. If you have more than one
              software datapath (e.g. some  system  bridges  and  some  netdev
              bridges),  then the total number of threads is n-handler-threads
              times the number of software datapaths.

....
         other_config : n-handler-threads
                                     optional string, containing  an  integer,
                                     at least 1


Note You need to log in before you can comment on or make changes to this bug.