Bug 2143694

Summary:	Revalidator and handler threads are not updated when cpu affinity changes
Product:	Red Hat Enterprise Linux Fast Datapath	Reporter:	Adrián Moreno <amorenoz>
Component:	openvswitch2.13	Assignee:	Adrián Moreno <amorenoz>
Status:	VERIFIED ---	QA Contact:	liting <tli>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	FDP 20.E	CC:	ctrautma, echaudro, hewang, jhsiao, qding, ralongi, tli, tredaelli
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	OVS:PM OVS:RFE	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Adrián Moreno 2022-11-17 14:45:26 UTC

Description of problem:

When ovs-vswitchd starts, it reads the number of available CPUs and determines the number of revalidator / handler threads accordingly.

However, if the affinity mask is changed dynamically (e.g: via taskset) OVS does not update the number of threads and this can lead to performance problems.


Version-Release number of selected component (if applicable):

All versions but this is specially problematic with >= 2.16 because per-cpu dispatch requires the number of handler to be equal to the number of available cpus. Besides, in older versions there's still the possible workaround of manually setting the number of threads via OVSDB. In >= 2.16 it's not possible.


How reproducible:

100%

Steps to Reproduce:
1. Run OVS and check the number of threads.
2. Change OVS affinity using taskset.
3. Verify the number of threads again

Actual results:

The number of threads have not changed.

Expected results:

The number of threads should be adjusted automatically.

Additional info:

Comment 1 Eelco Chaudron 2022-11-25 07:34:58 UTC

Assigning this BZ to Adrian, as he already has a patch out trying to fix this:

https://patchwork.ozlabs.org/project/openvswitch/patch/20221104220230.781507-1-amorenoz@redhat.com/

Comment 2 Adrián Moreno 2022-11-25 12:40:44 UTC

Yep, sorry, I meant to assign it to myself.

Comment 3 Eelco Chaudron 2023-03-02 08:07:25 UTC

Looks like this got accepted into OVS:

https://patchwork.ozlabs.org/project/openvswitch/patch/20221219182906.183791-1-amorenoz@redhat.com/

0d23948a598a ("ovs-thread: Detect changes in number of CPUs.")


As it's part of v3.1, we should be able to close this bz.

$ git tag --contains 0d23948a598a
v3.1.0

Adrian, can we close this BZ?

Comment 4 liting 2023-03-03 07:18:42 UTC

Hi Eelco,
 
 I reproduce this issue with openvswitch2.17-2.17.0-65.el9fdp, and verify pass with openvswitch3.1-3.1.0-5.el9fdp.x86_64. Please help me have a look. Thanks
 
[root@dell-per730-53 ~]# rpm -qa|grep openvswitch
openvswitch-selinux-extra-policy-1.0-31.el9fdp.noarch
openvswitch2.17-2.17.0-65.el9fdp.x86_64
[root@dell-per730-53 ~]# ovs-vsctl add-br ovsbr0
[root@dell-per730-53 ~]# top -H -p $(pgrep ovs-vswitchd)
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                P 
  92567 openvsw+  10 -10 2863300 241448  31020 S   0.3   0.4   0:00.03 revalidator33                                         51 
  92568 openvsw+  10 -10 2863300 241448  31020 S   0.3   0.4   0:00.03 revalidator34                                          1 
  92518 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.12 ovs-vswitchd                                          53 
  92535 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.00 handler1                                               3 
  92536 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.00 handler2                                               5 
  92537 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.00 handler3                                              17 
  92538 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.00 handler4                                              19 
  92539 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.00 urcu5                                                  3 
  92540 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.00 handler6                                              43 
  92541 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.00 handler7                                              11 
  92542 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.00 handler8                                              37 
  92543 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.00 handler9                                               1 
  92544 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.00 handler10                                             35 

[root@dell-per730-53 ~]# taskset -cp 2 92518
pid 92518's current affinity list: 0,1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55
pid 92518's new affinity list: 2

[root@dell-per730-53 ~]# top -H -p $(pgrep ovs-vswitchd)
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                P 
  92567 openvsw+  10 -10 2863300 241448  31020 S   0.3   0.4   0:00.07 revalidator33                                         51 
  92570 openvsw+  10 -10 2863300 241448  31020 S   0.3   0.4   0:00.06 revalidator36                                         37 
  92518 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.14 ovs-vswitchd                                           2 
  92535 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.00 handler1                                               3 
  92536 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.00 handler2                                               5 
  92537 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.00 handler3                                              17 
  92538 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.00 handler4                                              19 
  92539 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.00 urcu5                                                  3 
  92540 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.00 handler6                                              43 
  92541 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.00 handler7                                              11 
  92542 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.00 handler8                                              37 
  92543 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.00 handler9                                               1 
  92544 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.00 handler10                                             35 
  92545 openvsw+  10 -10 2863300 241448  31020 S   0.0   0.4   0:00.00 handler11                                             51
  
  
  [root@dell-per730-52 ~]# rpm -qa|grep openvswitch
openvswitch-selinux-extra-policy-1.0-31.el9fdp.noarch
openvswitch3.1-3.1.0-5.el9fdp.x86_64
  [root@dell-per730-52 ~]#  ovs-vsctl add-br ovsbr0
  [root@dell-per730-52 ~]# top -H -p $(pgrep ovs-vswitchd)
      PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                P 
  60311 openvsw+  10 -10 2865088 243224  32088 S   0.3   0.4   0:00.01 revalidator33                                         35 
  60312 openvsw+  10 -10 2865088 243224  32088 S   0.3   0.4   0:00.01 revalidator34                                          1 
  60263 openvsw+  10 -10 2865088 243224  32088 S   0.0   0.4   0:00.11 ovs-vswitchd                                          33 
  60279 openvsw+  10 -10 2865088 243224  32088 S   0.0   0.4   0:00.00 handler1                                              43 
  60280 openvsw+  10 -10 2865088 243224  32088 S   0.0   0.4   0:00.00 handler2                                              45 
  60281 openvsw+  10 -10 2865088 243224  32088 S   0.0   0.4   0:00.00 handler3                                               1 
  60282 openvsw+  10 -10 2865088 243224  32088 S   0.0   0.4   0:00.00 handler4                                              23 
  60283 openvsw+  10 -10 2865088 243224  32088 S   0.0   0.4   0:00.00 urcu5                                                 43 
  60284 openvsw+  10 -10 2865088 243224  32088 S   0.0   0.4   0:00.00 handler6                                               3 
  60285 openvsw+  10 -10 2865088 243224  32088 S   0.0   0.4   0:00.00 handler7                                              47 
  60286 openvsw+  10 -10 2865088 243224  32088 S   0.0   0.4   0:00.00 handler8                                              29 
  60287 openvsw+  10 -10 2865088 243224  32088 S   0.0   0.4   0:00.00 handler9                                              37 
  60288 openvsw+  10 -10 2865088 243224  32088 S   0.0   0.4   0:00.00 handler10                                             25 
  60289 openvsw+  10 -10 2865088 243224  32088 S   0.0   0.4   0:00.00 handler11                                             35 

[root@dell-per730-52 ~]# taskset -cp 2 60263
pid 60263's current affinity list: 0,1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55
pid 60263's new affinity list: 2

[root@dell-per730-52 ~]# top -H -p $(pgrep ovs-vswitchd)
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                P 
  60263 openvsw+  10 -10 2824120 207676  32088 S   0.7   0.3   0:00.16 ovs-vswitchd                                           2 
  60283 openvsw+  10 -10 2824120 207676  32088 S   0.0   0.3   0:00.00 urcu5                                                 43 
  60331 openvsw+  10 -10 2824120 207676  32088 S   0.0   0.3   0:00.00 handler43                                              2 
  60332 openvsw+  10 -10 2824120 207676  32088 S   0.0   0.3   0:00.00 handler42                                              2 
  60333 openvsw+  10 -10 2824120 207676  32088 S   0.0   0.3   0:00.00 revalidator41                                          2

Comment 6 Adrián Moreno 2023-03-03 12:58:04 UTC

Yep. Test looks good to me.

Comment 7 liting 2023-04-04 08:56:09 UTC

Verified this issue pass on openvswitch3.1-3.1.0-6.el9fdp.x86_64, so close it.
[root@dell-per730-53 ~]# rpm -qa|grep openvswitch
openvswitch-selinux-extra-policy-1.0-31.el9fdp.noarch
openvswitch3.1-3.1.0-6.el9fdp.x86_64

[root@dell-per730-53 ~]# top -H -p $(pgrep ovs-vswitchd)
%Cpu(s):  4.4 us,  1.2 sy,  0.0 ni, 94.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  63787.3 total,  30098.8 free,  26139.1 used,   7994.6 buff/cache
MiB Swap:  28560.0 total,  28560.0 free,      0.0 used.  37648.2 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                P 
 211615 openvsw+  10 -10  268.5g 353708  32244 S   0.3   0.5   0:00.47 revalidator56                                         51 
 211616 openvsw+  10 -10  268.5g 353708  32244 S   0.3   0.5   0:00.46 revalidator57                                          9 
 210954 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:01.04 ovs-vswitchd                                          29 
 210985 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.11 eal-intr-thread                                       39 
 211492 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 iavf-event-thre                                       49 
 211524 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 telemetry-v2                                          50 
 211526 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 dpdk_watchdog1                                        29 
 211527 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 urcu2                                                  3 
 211529 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.04 ovs_vhost3                                            33 
 211534 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 ct_clean4                                             29 
 211535 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 ipf_clean5                                             7 
 211542 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 handler6                                              55 
 211544 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 handler7                                               1 
 211545 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 handler8                                               5 
 211547 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 handler9                                               1 
 211548 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 handler10                                              1 
 211549 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 handler11                                             51 
 211551 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 handler12                                             39 
 211552 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 handler13                                             41 
 211554 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 handler14                                             33 
 211555 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 handler15                                              1 
 211557 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 handler16                                             39 
 211558 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 handler17                                              1 
 211560 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 handler18                                             29 
 211561 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 handler19                                             15 
 211563 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 handler20                                             33 
 211564 openvsw+  10 -10  268.5g 353708  32244 S   0.0   0.5   0:00.00 handler21                                              1 
    
  
  
[root@dell-per730-53 ~]# taskset -cp 2 210954
pid 210954's current affinity list: 0-55
pid 210954's new affinity list: 2

[root@dell-per730-53 ~]# top -H -p $(pgrep ovs-vswitchd)
Threads:  17 total,   2 running,  15 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.4 us,  1.2 sy,  0.0 ni, 94.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  63787.3 total,  30172.8 free,  26065.2 used,   7994.7 buff/cache
MiB Swap:  28560.0 total,  28560.0 free,      0.0 used.  37722.2 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                P 
 211632 openvsw+  10 -10  268.5g 285464  32244 R  99.9   0.4  12:10.21 pmd-c26/id:63                                         26 
 211629 openvsw+  10 -10  268.5g 285464  32244 R  99.7   0.4  12:11.22 pmd-c24/id:62                                         24 
 210954 openvsw+  10 -10  268.5g 285464  32244 S   0.0   0.4   0:01.08 ovs-vswitchd                                           2 
 210985 openvsw+  10 -10  268.5g 285464  32244 S   0.0   0.4   0:00.12 eal-intr-thread                                       39 
 211492 openvsw+  10 -10  268.5g 285464  32244 S   0.0   0.4   0:00.00 iavf-event-thre                                       49 
 211524 openvsw+  10 -10  268.5g 285464  32244 S   0.0   0.4   0:00.00 telemetry-v2                                          50 
 211526 openvsw+  10 -10  268.5g 285464  32244 S   0.0   0.4   0:00.00 dpdk_watchdog1                                        29 
 211527 openvsw+  10 -10  268.5g 285464  32244 S   0.0   0.4   0:00.00 urcu2                                                 53 
 211529 openvsw+  10 -10  268.5g 285464  32244 S   0.0   0.4   0:00.06 ovs_vhost3                                            33 
 211534 openvsw+  10 -10  268.5g 285464  32244 S   0.0   0.4   0:00.00 ct_clean4                                             29 
 211535 openvsw+  10 -10  268.5g 285464  32244 S   0.0   0.4   0:00.00 ipf_clean5                                             7 
 211635 openvsw+  10 -10  268.5g 285464  32244 S   0.0   0.4   0:00.00 pmd-c52/id:64                                         52 
 211638 openvsw+  10 -10  268.5g 285464  32244 S   0.0   0.4   0:00.00 pmd-c54/id:65                                         54 
 211639 openvsw+  10 -10  268.5g 285464  32244 S   0.0   0.4   0:00.00 vhost_reconn                                          11 
 211641 openvsw+  10 -10  268.5g 285464  32244 S   0.0   0.4   0:00.04 vhost-events                                          51 
 656270 openvsw+  10 -10  268.5g 285464  32244 S   0.0   0.4   0:00.00 handler67                                              2 
 656271 openvsw+  10 -10  268.5g 285464  32244 S   0.0   0.4   0:00.00 revalidator66                                          2