Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 1692812

Summary: [OvS 2.11] High CPU in ovs-vswitchd on qrouter migration or service restart
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Flavio Leitner <fleitner>
Component: openvswitch2.11Assignee: Flavio Leitner <fleitner>
Status: CLOSED ERRATA QA Contact: haidong li <haili>
Severity: high Docs Contact:
Priority: unspecified    
Version: FDP 19.ACC: apevec, chrisw, cswanson, ctrautma, fhallal, fleitner, haili, jhsiao, moddi, njohnston, qding, ragiman, ralongi, rhos-maint, rkhan, sputhenp, tredaelli
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openvswitch2.11-2.11.0-5.el7fdn Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1686003
: 1692813 (view as bug list) Environment:
Last Closed: 2019-04-23 08:26:35 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1692813    

Comment 4 haidong li 2019-04-09 10:59:29 UTC 
Hi Flavio,
   Can this bug be reproduced without setting up a openstack environment? If we can reproduce it directly on ovs or ovn,we can test it in a short time. Thanks.
Comment 5 Flavio Leitner 2019-04-09 17:18:01 UTC 
(In reply to haidong li from comment #4)

I used the env. provided by Marco, but based on what we know now,
I think if you have a internal port moved to another netns might
reproduce the issue.

fbl
Comment 7 haidong li 2019-04-10 07:44:32 UTC 
Thanks a lot,Flavio and Mauro.I can reproduce this bug with steps you provide on old version 2.11.0-3.el7fdp:

[root@hp-dl380pg8-04 openvswitch]# top

top - 03:42:12 up 1 day,  4:23,  3 users,  load average: 0.78, 0.30, 0.16
Tasks: 306 total,   2 running, 304 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.6 sy,  0.0 ni, 99.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 32802316 total, 24258624 free,  2195568 used,  6348124 buff/cache
KiB Swap: 16515068 total, 16515068 free,        0 used. 29825416 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                
56138 openvsw+  10 -10 1934652 229148  17392 R 105.0  0.7   0:35.29 ovs-vswitchd                                           
   10 root      20   0       0      0      0 S   0.3  0.0   0:27.31 rcu_sched                                              
 3825 root      20   0  150732  85452  85020 S   0.3  0.3   0:50.02 systemd-journal                                        
15880 root      20   0  115704   2384   1688 S   0.3  0.0   0:03.20 bash                                                   
18630 root      20   0  381264  49524  45748 S   0.3  0.2   0:27.64 rsyslogd                                               
    1 root      20   0  195252   8424   4220 S   0.0  0.0   0:22.49 systemd                                                
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.04 kthreadd                                               
    3 root      20   0       0      0      0 S   0.0  0.0   0:00.27 ksoftirqd/0                                            
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H                                           
    8 root      rt   0       0      0      0 S   0.0  0.0   0:00.89 migration/0                                            
    9 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh    
[root@hp-dl380pg8-04 openvswitch]# cat /var/log/openvswitch/ovs-vswitchd.log | grep CPU
2019-04-10T06:48:36.422Z|00480|poll_loop|INFO|wakeup due to 0-ms timeout at ../ofproto/ofproto-dpif.c:1757 (72% CPU usage)
2019-04-10T06:48:36.422Z|00481|poll_loop|INFO|wakeup due to [POLLIN] on fd 15 (<->/var/run/openvswitch/db.sock) at ../lib/stream-fd.c:157 (72% CPU usage)
2019-04-10T06:48:36.432Z|00482|poll_loop|INFO|wakeup due to [POLLIN] on fd 16 (NETLINK_ROUTE<->NETLINK_ROUTE) at ../lib/netlink-socket.c:1401 (72% CPU usage)
2019-04-10T06:48:36.434Z|00483|poll_loop|INFO|wakeup due to [POLLIN] on fd 22 (FIFO pipe:[4174135]) at ../vswitchd/bridge.c:384 (72% CPU usage)
2019-04-10T06:48:36.611Z|00484|poll_loop|INFO|wakeup due to 0-ms timeout at ../ofproto/ofproto-dpif.c:1757 (72% CPU usage)
2019-04-10T06:48:36.628Z|00485|poll_loop|INFO|wakeup due to [POLLIN] on fd 16 (NETLINK_ROUTE<->NETLINK_ROUTE) at ../lib/netlink-socket.c:1401 (72% CPU usage)
2019-04-10T06:48:36.628Z|00486|poll_loop|INFO|wakeup due to [POLLIN] on fd 16 (NETLINK_ROUTE<->NETLINK_ROUTE) at ../lib/netlink-socket.c:1401 (72% CPU usage)
2019-04-10T06:48:42.275Z|00498|poll_loop|INFO|wakeup due to 0-ms timeout at ../ofproto/ofproto-dpif.c:1757 (94% CPU usage)
2019-04-10T06:48:49.323Z|00553|poll_loop|INFO|wakeup due to [POLLIN] on fd 22 (FIFO pipe:[4174135]) at ../vswitchd/bridge.c:384 (100% CPU usage)
2019-04-10T06:48:54.770Z|00558|poll_loop|INFO|wakeup due to [POLLIN] on fd 22 (FIFO pipe:[4174135]) at ../vswitchd/bridge.c:384 (100% CPU usage)
2019-04-10T06:49:00.172Z|00567|poll_loop|INFO|wakeup due to 0-ms timeout at ../ofproto/ofproto-dpif.c:1757 (98% CPU usage)
2019-04-10T06:49:06.184Z|00578|poll_loop|INFO|wakeup due to 0-ms timeout at ../ofproto/ofproto-dpif.c:1757 (94% CPU usage)

[root@hp-dl380pg8-04 log]# cat messages  | grep SELinux
[99047.727742] SELinux: unrecognized netlink message: protocol=0 nlmsg_type=1 sclass=31
[99047.762646] SELinux: unrecognized netlink message: protocol=0 nlmsg_type=1 sclass=31
[99047.798805] SELinux: unrecognized netlink message: protocol=0 nlmsg_type=1 sclass=31
[99047.834110] SELinux: unrecognized netlink message: protocol=0 nlmsg_type=1 sclass=31
[99047.871381] SELinux: unrecognized netlink message: protocol=0 nlmsg_type=1 sclass=31
[99047.911956] SELinux: unrecognized netlink message: protocol=0 nlmsg_type=1 sclass=31
[99047.950286] SELinux: unrecognized netlink message: protocol=0 nlmsg_type=1 sclass=31
[99047.990548] SELinux: unrecognized netlink message: protocol=0 nlmsg_type=1 sclass=31
[99048.038732] SELinux: unrecognized netlink message: protocol=0 nlmsg_type=1 sclass=31
[99048.076732] SELinux: unrecognized netlink message: protocol=0 nlmsg_type=1 sclass=31
[99048.116670] SELinux: unrecognized netlink message: protocol=0 nlmsg_type=0 sclass=31
[99048.154461] SELinux: unrecognized netlink message: protocol=0 nlmsg_type=0 sclass=31
[99048.190848] SELinux: unrecognized netlink message: protocol=0 nlmsg_type=0 sclass=31
[99048.227719] SELinux: unrecognized netlink message: protocol=0 nlmsg_type=0 sclass=31
[99048.271691] SELinux: unrecognized netlink message: protocol=0 nlmsg_type=0 sclass=31
[99048.310117] SELinux: unrecognized netlink message: protocol=0 nlmsg_type=1 sclass=31
[99048.345362] SELinux: unrecognized netlink message: protocol=0 nlmsg_type=1 sclass=31
[99048.381785] SELinux: unrecognized netlink message: protocol=0 nlmsg_type=1 sclass=31
[99048.419684] SELinux: unrecognized netlink message: protocol=0 nlmsg_type=1 sclass=31
[99048.459109] SELinux: unrecognized netlink message: protocol=0 nlmsg_type=1 sclass=31



============================================================================================
verified on the latest version:
[root@hp-dl380pg8-04 openvswitch]# rpm -qa | grep openvswitch
openvswitch-selinux-extra-policy-1.0-11.el7fdp.noarch
openvswitch2.11-2.11.0-5.el7fdp.x86_64
kernel-kernel-networking-openvswitch-ovn-1.0-121.noarch
[root@hp-dl380pg8-04 openvswitch]# 

[root@hp-dl380pg8-04 openvswitch]# cat /var/log/openvswitch/ovs-vswitchd.log | grep CPU
2019-04-10T07:17:22.292Z|00002|ovs_numa|INFO|Discovered 12 CPU cores on NUMA node 0
2019-04-10T07:17:22.292Z|00003|ovs_numa|INFO|Discovered 12 CPU cores on NUMA node 1
2019-04-10T07:17:22.292Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 24 CPU cores

[root@hp-dl380pg8-04 openvswitch]# top

top - 03:38:12 up 1 day,  4:19,  3 users,  load average: 0.38, 0.10, 0.07
Tasks: 302 total,   1 running, 301 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.4 us,  1.0 sy,  0.0 ni, 98.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 32802316 total, 24254212 free,  2200572 used,  6347532 buff/cache
KiB Swap: 16515068 total, 16515068 free,        0 used. 29820632 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                
47269 openvsw+  10 -10 1934656 229164  17392 S  24.3  0.7   0:09.72 ovs-vswitchd                                           
15880 root      20   0  115704   2332   1688 S   1.3  0.0   0:02.63 bash                                                   
47211 openvsw+  10 -10   69492   3240   1604 S   1.0  0.0   0:00.24 ovsdb-server                                           
   10 root      20   0       0      0      0 S   0.3  0.0   0:27.01 rcu_sched                                              
   19 root      rt   0       0      0      0 S   0.3  0.0   0:01.31 migration/2                                            
   24 root      rt   0       0      0      0 S   0.3  0.0   0:01.07 migration/3                                            
   39 root      rt   0       0      0      0 S   0.3  0.0   0:02.08 migration/6                                            
   50 root      rt   0       0      0      0 S   0.3  0.0   0:01.56 migration/8                                            
   60 root      rt   0       0      0      0 S   0.3  0.0   0:01.34 migration/10                                           
   90 root      rt   0       0      0      0 S   0.3  0.0   0:01.21 migration/16                                           
  100 root      rt   0       0      0      0 S   0.3  0.0   0:00.84 migration/18                                           
  105 root      rt   0       0      0      0 S   0.3  0.0   0:01.29 migration/19                                           
  115 root      rt   0       0      0      0 S   0.3  0.0   0:00.94 migration/21                                           
29108 qemu      20   0 2921544 725264  10232 S   0.3  2.2   3:34.56 qemu
Comment 9 errata-xmlrpc 2019-04-23 08:26:35 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0799