Bug 1695722
| Summary: | ovn-controller is using 100% cpu on OVN deployment, delay on security groups being applied | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux Fast Datapath | Reporter: | Timothy Redaelli <tredaelli> |
| Component: | ovn2.11 | Assignee: | Timothy Redaelli <tredaelli> |
| Status: | CLOSED ERRATA | QA Contact: | haidong li <haili> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | FDP 19.B | CC: | ctrautma, pvauter, yinxu |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | ovn2.11-2.11.0-7.el7fdn | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-04-30 09:23:26 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Timothy Redaelli
2019-04-03 16:55:38 UTC
reproduced on the old version:
[root@hp-dl380pg8-04 opt]# uname -a
Linux hp-dl380pg8-04.rhts.eng.pek2.redhat.com 3.10.0-957.el7.x86_64 #1 SMP Thu Oct 4 20:48:51 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@hp-dl380pg8-04 opt]# rpm -qa | grep openvswitch
openvswitch-selinux-extra-policy-1.0-11.el7fdp.noarch
openvswitch2.11-2.11.0-4.el7fdp.x86_64
kernel-kernel-networking-openvswitch-ovn-1.0-121.noarch
[root@hp-dl380pg8-04 opt]# rpm -qa | grep ovn
ovn2.11-host-2.11.0-4.el7fdp.x86_64
ovn2.11-common-2.11.0-4.el7fdp.x86_64
ovn2.11-central-2.11.0-4.el7fdp.x86_64
ovn2.11-2.11.0-4.el7fdp.x86_64
kernel-kernel-networking-openvswitch-ovn-1.0-121.noarch
[root@hp-dl380pg8-04 opt]#
add a lot of logical ports and excute the script:
from scapy.all import *
for x in range(1000):
sendp(Ether(src="00:de:ad:01:00:01", dst="ff:ff:ff:ff:ff:ff")/IP(src="0.0.0.0", dst="255.255.255.255")/UDP(sport=68,dport=67)/BOOTP(chaddr="00:de:ad:01:00:01",ciaddr="0.0.0.0")/DHCP(options=[("message-type","discover")])/DHCP(options=[("subnet_mask","255.255.255.0")])/DHCP(options=[("router","0.0.0.0")])/DHCP(options=[("name_server", "0.0.0.0"), "end"]), iface="eth1")
[root@hp-dl380pg8-04 ovn]# top
top - 02:22:28 up 2 days, 3:03, 4 users, load average: 0.80, 0.27, 0.23
Tasks: 314 total, 2 running, 312 sleeping, 0 stopped, 0 zombie
%Cpu(s): 6.1 us, 0.3 sy, 0.0 ni, 93.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 32802316 total, 25893460 free, 1955284 used, 4953572 buff/cache
KiB Swap: 16515068 total, 16515068 free, 0 used. 30065744 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
59606 root 10 -10 223824 19980 1676 R 100.3 0.1 82:17.21 ovn-controller
60028 qemu 20 0 2932040 1.1g 10236 S 45.8 3.6 21:10.92 qemu-kvm
59533 openvsw+ 10 -10 1938872 241900 17392 S 6.0 0.7 14:44.17 ovs-vswitchd
56162 root 20 0 347080 7036 5188 S 1.0 0.0 0:06.11 virsh
59079 root 20 0 1538776 26328 14312 S 1.0 0.1 0:08.69 libvirtd
59943 root 20 0 150432 5684 4340 S 1.0 0.0 0:11.24 sshd
60230 root 20 0 0 0 0 S 0.7 0.0 0:10.58 vhost-60028
10 root 20 0 0 0 0 S 0.3 0.0 2:24.81 rcu_sched
[root@hp-dl380pg8-04 opt]# cat /var/log/openvswitch/ovn-controller.log | grep CPU
2019-04-11T06:22:14.381Z|01075|poll_loop|INFO|wakeup due to [POLLIN] on fd 22 (<->/var/run/openvswitch/br-int.mgmt) at lib/stream-fd.c:157 (101% CPU usage)
2019-04-11T06:22:14.564Z|01076|poll_loop|INFO|wakeup due to [POLLIN] on fd 22 (<->/var/run/openvswitch/br-int.mgmt) at lib/stream-fd.c:157 (101% CPU usage)
2019-04-11T06:22:14.745Z|01077|poll_loop|INFO|wakeup due to [POLLIN] on fd 22 (<->/var/run/openvswitch/br-int.mgmt) at lib/stream-fd.c:157 (101% CPU usage)
2019-04-11T06:22:14.926Z|01078|poll_loop|INFO|wakeup due to [POLLIN] on fd 22 (<->/var/run/openvswitch/br-int.mgmt) at lib/stream-fd.c:157 (101% CPU usage)
2019-04-11T06:22:15.108Z|01079|poll_loop|INFO|wakeup due to [POLLIN] on fd 22 (<->/var/run/openvswitch/br-int.mgmt) at lib/stream-fd.c:157 (101% CPU usage)
2019-04-11T06:22:15.288Z|01080|poll_loop|INFO|wakeup due to [POLLIN] on fd 22 (<->/var/run/openvswitch/br-int.mgmt) at lib/stream-fd.c:157 (101% CPU usage)
2019-04-11T06:22:15.469Z|01081|poll_loop|INFO|wakeup due to [POLLIN] on fd 22 (<->/var/run/openvswitch/br-int.mgmt) at lib/stream-fd.c:157 (101% CPU usage)
2019-04-11T06:22:15.651Z|01082|poll_loop|INFO|wakeup due to [POLLIN] on fd 22 (<->/var/run/openvswitch/br-int.mgmt) at lib/stream-fd.c:157 (101% CPU usage)
2019-04-11T06:22:15.831Z|01085|poll_loop|INFO|wakeup due to [POLLIN] on fd 22 (<->/var/run/openvswitch/br-int.mgmt) at lib/stream-fd.c:157 (101% CPU usage)
2019-04-11T06:22:16.011Z|01086|poll_loop|INFO|wakeup due to [POLLIN] on fd 22 (<->/var/run/openvswitch/br-int.mgmt) at lib/stream-fd.c:157 (101% CPU usage)
2019-04-11T06:22:20.528Z|01090|poll_loop|INFO|wakeup due to [POLLIN] on fd 22 (<->/var/run/openvswitch/br-int.mgmt) at lib/stream-fd.c:157 (102% CPU usage)
2019-04-11T06:22:26.505Z|01096|poll_loop|INFO|wakeup due to [POLLIN] on fd 22 (<->/var/run/openvswitch/br-int.mgmt) at lib/stream-fd.c:157 (102% CPU usage)
===================================================================================================================================
verified on the latest version:
[root@hp-dl380pg8-04 scapy]# rpm -qa | grep openvswitch
openvswitch-selinux-extra-policy-1.0-11.el7fdp.noarch
openvswitch2.11-2.11.0-5.el7fdp.x86_64
kernel-kernel-networking-openvswitch-ovn-1.0-121.noarch
[root@hp-dl380pg8-04 scapy]# rpm -qa | grep ovn
ovn2.11-central-2.11.0-7.el7fdp.x86_64
ovn2.11-2.11.0-7.el7fdp.x86_64
ovn2.11-host-2.11.0-7.el7fdp.x86_64
kernel-kernel-networking-openvswitch-ovn-1.0-121.noarch
[root@hp-dl380pg8-04 scapy]#
[root@hp-dl380pg8-04 scapy]#
top - 06:15:23 up 2 days, 6:56, 4 users, load average: 0.09, 0.38, 0.52
Tasks: 314 total, 1 running, 313 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.5 us, 0.4 sy, 0.0 ni, 98.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 32802316 total, 26416436 free, 1588120 used, 4797760 buff/cache
KiB Swap: 16515068 total, 16515068 free, 0 used. 30434164 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
60896 qemu 20 0 2929992 810684 10236 S 43.9 2.5 1:30.35 qemu-kvm
60409 openvsw+ 10 -10 1939560 234360 17392 S 7.0 0.7 0:25.01 ovs-vswitc+
59570 root 20 0 1547032 24220 14524 S 1.0 0.1 0:01.90 libvirtd
56012 root 20 0 347080 6876 5164 S 0.7 0.0 0:00.54 virsh
59943 root 20 0 150432 5684 4340 S 0.7 0.0 0:11.81 sshd
61121 root 20 0 0 0 0 S 0.7 0.0 0:00.35 vhost-60896
10 root 20 0 0 0 0 S 0.3 0.0 2:32.25 rcu_sched
115 root rt 0 0 0 0 S 0.3 0.0 0:01.14 migration/+
145 root 20 0 0 0 0 S 0.3 0.0 0:01.99 kworker/11+
3751 root 20 0 0 0 0 S 0.3 0.0 0:15.79 xfsaild/dm+
6782 root 20 0 0 0 0 S 0.3 0.0 0:30.08 xfsaild/dm+
50665 root 20 0 162144 2456 1592 R 0.3 0.0 0:00.10 top
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0907 |