Bug 2031085

Summary: ovs-vswitchd running more threads than expected
Product: OpenShift Container Platform Reporter: Ian Miller <imiller>
Component: Telco EdgeAssignee: Dahir Osman <dosman>
Telco Edge sub component: RAN QA Contact: yliu1
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: unspecified CC: dosman, jramsay
Version: 4.10   
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-10 16:33:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ian Miller 2021-12-10 13:30:28 UTC 
Description of problem:
The ovs-vswitchd process on DU-configured SNO runs as part of the management overhead set of processes which are constrained to the management cores. When ovs-vswitchd starts up it samples the system for number of cores and the cpu-affinity mask to determine how many worker threads should be created. The number of revalidator threads should be (cpus/4 + 1). On our 80 core system with 4 cores reserved for management there are 21 revalidator threads rather than the expected 2.

Version-Release number of selected component (if applicable): 4.10


How reproducible: ~always


Steps to Reproduce:
1. Install SNO with networking OVNKubernetes and full DU configuration (as deployed via gitops ZTP)
2. Wait for cluster to stabilize post-reboot
3. Verify number of worker threads on ovs-vswitchd using: ps -Teo pid,tid,comm,cmd | egrep ovs | cut -b -100


Actual results:
   3161    3161 ovsdb-server    ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -v
   4329    4329 ovs-vswitchd    ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4461 handler1        ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4464 handler2        ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4467 handler3        ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4468 handler4        ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4471 urcu5           ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4473 handler6        ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4478 handler7        ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4482 handler8        ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4483 handler9        ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4489 handler10       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4491 handler11       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4493 handler12       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4496 handler13       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4500 handler14       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4504 handler15       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4505 handler16       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4508 handler17       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4509 handler18       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4512 handler19       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4516 handler20       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4518 handler21       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4519 handler22       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4523 handler23       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4526 handler24       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4528 handler25       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4531 handler26       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4533 handler27       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4535 handler28       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4542 handler29       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4547 handler30       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4549 handler31       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4555 handler32       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4556 handler33       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4558 handler34       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4562 handler35       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4568 handler36       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4570 handler37       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4574 handler38       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4575 handler39       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4580 handler40       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4585 handler41       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4588 handler42       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4594 handler43       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4596 handler44       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4597 handler45       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4599 handler46       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4604 handler47       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4608 handler48       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4611 handler49       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4615 handler50       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4618 handler51       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4622 handler52       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4624 handler53       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4626 handler54       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4629 handler55       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4633 handler56       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4638 handler57       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4639 handler58       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4641 handler59       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4645 handler60       ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4648 revalidator61   ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4650 revalidator62   ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4651 revalidator63   ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4653 revalidator64   ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4656 revalidator65   ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4660 revalidator66   ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4661 revalidator67   ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4663 revalidator68   ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4665 revalidator69   ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4669 revalidator70   ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4672 revalidator71   ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4677 revalidator72   ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4680 revalidator73   ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4684 revalidator74   ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4688 revalidator75   ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4690 revalidator76   ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4691 revalidator77   ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4696 revalidator78   ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4700 revalidator79   ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4703 revalidator80   ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   4329    4707 revalidator81   ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl


Expected results:
2 revalidator threads and similarly reduced number of handler threads.

Additional info:
Comment 5 yliu1 2022-02-04 17:50:12 UTC 
Verified on 4.10.0-rc.1 with latest 4.10 ztp container image.

[root@worker-2 core]# ps -Teo pid,tid,comm,cmd | egrep ovs | cut -b -100
   3209    3209 ovsdb-server    ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -v
   3833    3833 ovs-vswitchd    ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   3833    3843 urcu5           ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   3833   91149 handler325      ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   3833   91150 handler324      ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   3833   91151 revalidator322  ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
   3833   91152 revalidator323  ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsysl
  35311   35311 bash            /bin/bash -c set -euo pipefail  # Rotate audit log files when then g
  39182   39182 bash            /bin/bash -c set -xem if [[ -f /env/_master ]]; then   set -o allexp
  47316   47316 ovn-ctl         /bin/sh /usr/share/ovn/scripts/ovn-ctl --db-nb-cluster-local-port=96
  47570   47570 ovsdb-server    ovsdb-server -vconsole:info -vfile:off -vPATTERN:console:%D{%Y-%m-%d
  47570   47624 log_fsync0      ovsdb-server -vconsole:info -vfile:off -vPATTERN:console:%D{%Y-%m-%d
  47570   47625 urcu1           ovsdb-server -vconsole:info -vfile:off -vPATTERN:console:%D{%Y-%m-%d
  53597   53597 bash            /bin/bash -c set -xm if [[ -f /env/_master ]]; then   set -o allexpo
  56036   56036 ovn-ctl         /bin/sh /usr/share/ovn/scripts/ovn-ctl --db-sb-cluster-local-port=96
  56160   56160 ovsdb-server    ovsdb-server -vconsole:info -vfile:off -vPATTERN:console:%D{%Y-%m-%d
  56160   56274 log_fsync0      ovsdb-server -vconsole:info -vfile:off -vPATTERN:console:%D{%Y-%m-%d
  56160   56275 urcu1           ovsdb-server -vconsole:info -vfile:off -vPATTERN:console:%D{%Y-%m-%d
Comment 8 errata-xmlrpc 2022-03-10 16:33:22 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056