Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 1761411

Summary: [OVN] [RHEL8] ovn-controller is using 100% cpu when receive lots of incorrect arp packets
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Numan Siddique <nusiddiq>
Component: ovn2.11Assignee: Dumitru Ceara <dceara>
Status: CLOSED CURRENTRELEASE QA Contact: haidong li <haili>
Severity: unspecified Docs Contact:
Priority: high    
Version: FDP 19.GCC: ctrautma, dceara, haili, jishi, mmichels, nusiddiq, ovs-qe, qding
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1729846 Environment:
Last Closed: 2020-04-21 08:03:55 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1729846    
Bug Blocks:    

Description Numan Siddique 2019-10-14 11:03:39 UTC 
+++ This bug was initially created as a clone of Bug #1729846 +++

Description of problem:
ovn-controller is using 100% cpu when receive lots of incorrect arp packets

Version-Release number of selected component (if applicable):
[root@dell-per730-19 ovn]# uname -a
Linux dell-per730-19.rhts.eng.pek2.redhat.com 3.10.0-1060.el7.x86_64 #1 SMP Mon Jul 1 18:28:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@dell-per730-19 ovn]# rpm -qa | grep openvswitch
openvswitch-selinux-extra-policy-1.0-11.el7fdp.noarch
openvswitch2.11-2.11.0-14.el7fdp.x86_64
[root@dell-per730-19 ovn]# rpm -qa | grep ovn
ovn2.11-host-2.11.0-19.el7fdp.x86_64
ovn2.11-central-2.11.0-19.el7fdp.x86_64
ovn2.11-2.11.0-19.el7fdp.x86_64
[root@dell-per730-19 ovn]#

How reproducible:
everytime

Steps to Reproduce:
1.setup ovn environment and add a lot of logical switch port
2.connect a guest to logical switch
3.send a lot of incorrect arp packets with 0.0.0.0 as the "Target IP address"

Actual results:
the cpu usage is 101% by the ovn-controller

scripts in guest:
from scapy.all import *
for x in range(1000): 
  sendp(Ether(src="00:de:ad:01:00:01", dst="ff:ff:ff:ff:ff:ff")/ARP(op=1,hwsrc='00:de:ad:01:00:01',hwdst='00:00:00:00:00:00',psrc='172.16.102.11',pdst='0.0.0.0'),iface="eth1")

[root@dell-per730-19 ~]# top

top - 02:11:07 up 3 days,  4:07,  2 users,  load average: 0.18, 0.06, 0.06
Tasks: 465 total,   2 running, 463 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.8 us,  0.3 sy,  0.0 ni, 95.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 65708808 total, 56629192 free,  3094836 used,  5984780 buff/cache
KiB Swap: 29241340 total, 29241340 free,        0 used. 62099024 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                           
11493 root      10 -10  285408   9848   1716 R 100.7  0.0   1:58.82 ovn-controller                                    
11650 qemu      20   0 2974064 929068  11900 S  43.4  1.4   2:51.37 qemu-kvm                                          
11419 openvsw+  10 -10 2323772  97776  17916 S  10.9  0.1   1:43.32 ovs-vswitchd                                      
10527 root      20   0 1555548  26604  14540 S   1.0  0.0   0:03.44 libvirtd                                          
11819 qemu      20   0 2950492 614672  11904 S   1.0  0.9   1:37.65 qemu-kvm                                          
 5834 root      20   0  152752   5740   4408 S   0.7  0.0   0:02.07 sshd                                              
 6064 root      20   0  162312   2684   1600 R   0.7  0.0   1:16.23 top                                               
12325 root      20   0       0      0      0 S   0.7  0.0   0:00.42 vhost-11650                                       
13975 root      20   0  347304   6876   5160 S   0.7  0.0   0:00.17 virsh                                             
    9 root      20   0       0      0      0 S   0.3  0.0   0:52.33 rcu_sched                                         
 1443 root      20   0   22296   1960    996 S   0.3  0.0   3:28.58 irqbalance                                        
 3592 root      20   0  548544   9536   6784 S   0.3  0.0   0:24.74 NetworkManager                                    
12048 root      20   0       0      0      0 S   0.3  0.0   0:00.19 kworker/8:1                                       
12355 root      20   0       0      0      0 S   0.3  0.0   0:00.07 vhost-1181

[root@dell-per730-19 ovn]# cat /var/log/openvswitch/ovn-controller.log | grep CPU | tail -4
2019-07-15T06:11:07.320Z|00056|poll_loop|INFO|wakeup due to 0-ms timeout at ovn/controller/pinctrl.c:2489 (100% CPU usage)
2019-07-15T06:11:08.244Z|00058|poll_loop|INFO|wakeup due to 0-ms timeout at ovn/controller/pinctrl.c:2489 (100% CPU usage)
2019-07-15T06:11:14.249Z|00060|poll_loop|INFO|wakeup due to 0-ms timeout at ovn/controller/pinctrl.c:2489 (100% CPU usage)
2019-07-15T06:11:21.322Z|00062|poll_loop|INFO|wakeup due to [POLLIN] on fd 21 (20.0.0.25:48756<->20.0.0.25:6642) at lib/stream-fd.c:157 (99% CPU usage)
[root@dell-per730-19 ovn]# 

Expected results:
cpu usage is not high

Additional info:

--- Additional comment from Dumitru Ceara on 2019-07-23 11:11:28 UTC ---

Fix sent upstream for review: https://patchwork.ozlabs.org/patch/1135580/

--- Additional comment from Mark Michelson on 2019-10-11 15:40:09 UTC ---

Moving back to "MODIFIED" so the errata tool won't be angry.

--- Additional comment from errata-xmlrpc on 2019-10-11 15:41:22 UTC ---

Bug report changed to ON_QA status by Errata System.
A QE request has been submitted for advisory RHBA-2019:47208-01
https://errata.devel.redhat.com/advisory/47208
Comment 1 haidong li 2019-10-16 10:09:20 UTC 
[root@dell-per730-42 ovn]# rpm -qa | grep ovn
kernel-kernel-networking-openvswitch-ovn-1.0-148.noarch
ovn2.11-central-2.11.1-7.el8fdp.x86_64
ovn2.11-2.11.1-7.el8fdp.x86_64
ovn2.11-host-2.11.1-7.el8fdp.x86_64
[root@dell-per730-42 ovn]# 
from scapy.all import *
for x in range(1000): 
  sendp(Ether(src="00:de:ad:01:00:01", dst="ff:ff:ff:ff:ff:ff")/ARP(op=1,hwsrc='00:de:ad:01:00:01',hwdst='00:00:00:00:00:00',psrc='172.16.102.11',pdst='0.0.0.0'),iface="eth1")
[root@dell-per730-42 ~]# top

top - 05:23:48 up  1:43,  2 users,  load average: 0.13, 0.05, 0.09
Tasks: 548 total,   1 running, 547 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.8 us,  0.2 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  31967.2 total,  24027.0 free,   1900.5 used,   6039.7 buff/cache
MiB Swap:  16128.0 total,  16128.0 free,      0.0 used.  29592.3 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                            
22753 qemu      20   0 7517516 604136  21112 S  20.9   1.8   0:55.73 qemu-kvm                                           
21738 root      20   0 1972900  44148  31436 S   2.3   0.1   0:03.05 libvirtd                                           
27509 root      20   0  246740  13688  11768 S   1.0   0.0   0:01.07 virsh                                              
 3717 root      20   0  151196   5028   3780 S   0.7   0.0   0:01.71 sshd                                               
22520 openvsw+  10 -10 3407944 154012  33040 S   0.7   0.5   0:11.05 ovs-vswitchd                                       
17340 root      20   0       0      0      0 I   0.3   0.0   0:00.08 kworker/u97:2-events_unbound                       
22586 root      10 -10  263724   4904   3004 S   0.3   0.0   0:01.91 ovn-controller                                     
22913 qemu      20   0 6092188 570660  20876 S   0.3   1.7   0:41.01 qemu-kvm                                           
23165 root      20   0       0      0      0 S   0.3   0.0   0:00.09 vhost-22753                                        
23238 root      20   0       0      0      0 S   0.3   0.0   0:00.05 vhost-22913                                        
27652 root      20   0       0      0      0 I   0.3   0.0   0:00.05 kworker/u98:3-events_unbound                       
27654 root      20   0   64384   5336   4036 R   0.3   0.0   0:00.02 top                                                
    1 root      20   0  244620  11568   8276 S   0.0   0.0   0:04.97 systemd                                            
    2 root      20   0       0      0      0 S   0.0   0.0   0:00.01 kthreadd                                           
    3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp