Bug 2011525

Summary: Rate-limit incoming BFD to prevent ovn-controller DoS
Product: OpenShift Container Platform Reporter: Sai Sindhur Malleni <smalleni>
Component: NetworkingAssignee: Surya Seetharaman <surya>
Networking sub component: ovn-kubernetes QA Contact: Anurag saxena <anusaxen>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: ctrautma, dblack, dcbw, dceara, ffernand, jiji, jlema, murali, nusiddiq, surya, trozet
Version: 4.10   
Target Milestone: ---   
Target Release: 4.11.0   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-08-10 10:38:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Description Flags
DBs and conf.db of worker node none

Description Sai Sindhur Malleni 2021-10-06 18:11:28 UTC
Description of problem:

On 4.10 OpenShift cluster with ~250 worker nodes, we launched around 35-60 pods per node. All the ovn-controller instances across the cluster seems be utilizing a reasonable amount of memory except two nodes. Once the pods are launched, they are not deleted and the cluster is in a steady state.

They are:

Over time their memory keeps increasing without bounds and without any activity happening in the cluster. I am suspecting a memory leak here.


Looking at memory stats on one of the problem nodes we see

[root@worker000-fc640 ~]# ovn-appctl -t ovn-controller memory/show
lflow-cache-entries-cache-expr:16030 lflow-cache-entries-cache-matches:9621 lflow-cache-size-KB:65688 ofctrl_desired_flow_usage-KB:37379 ofctrl_installed_flow_usage-KB:28470 ofctrl_sb_flow_ref_usage-KB:9361

Comparing it a node which is NOT exhibiting this leak, I don't spot many differences

Node without leak:
sh-4.4#  ovn-appctl -t ovn-controller memory/show
lflow-cache-entries-cache-expr:15804 lflow-cache-entries-cache-matches:9547 lflow-cache-size-KB:64684 ofctrl_desired_flow_usage-KB:37304 ofctrl_installed_flow_usage-KB:28425 ofctrl_sb_flow_ref_usage-KB:9326

Version-Release number of selected component (if applicable):

[kni@e16-h12-b02-fc640 web-burner]$ oc rsh -c ovn-controller ovnkube-node-qj8dq                                                        
sh-4.4# rpm -qa | grep ovn

How reproducible:
Only reproducible on some nodes

Steps to Reproduce:
1. Deploy a large cluster
2. Launch a few pods
3. remain at steady state and watch ovn-controller memory grow boundlessly on some nodes

Actual results:
ovn-controller memory on some nodes keeps growing without bounds indicating a memory leak

Expected results:
Memory should be within reasonable bounds and not grow at steady state

Additional info:

Comment 1 Sai Sindhur Malleni 2021-10-06 18:18:57 UTC
Placed DBS and provided Numan access to those.

Comment 2 Sai Sindhur Malleni 2021-10-06 19:24:09 UTC
Created attachment 1830071 [details]
DBs and conf.db of worker node

Comment 3 Sai Sindhur Malleni 2021-10-07 02:06:18 UTC
perf record output shows pinctrl0 thread is hot on CPu

 Event count (approx.): 6723064170
# Overhead  Command         Shared Object        Symbol                                                                                
# ........  ..............  ...................  ...........................................                                           
     1.97%  ovn_pinctrl0    libpthread-2.28.so   [.] __pthread_rwlock_wrlock                                                           
     1.84%  ovn_pinctrl0    libpthread-2.28.so   [.] __pthread_rwlock_rdlock                                                           
     1.77%  ovn_pinctrl0    libpthread-2.28.so   [.] __pthread_rwlock_unlock                                                           
     1.75%  ovn_pinctrl0    [kernel.kallsyms]    [k] copy_user_enhanced_fast_string                                                    
     1.62%  ovn_pinctrl0    libc-2.28.so         [.] _int_malloc
     1.53%  ovn_pinctrl0    [kernel.kallsyms]    [k] avc_has_perm
     1.16%  ovn_pinctrl0    [kernel.kallsyms]    [k] _raw_spin_lock
     1.09%  ovn_pinctrl0    libc-2.28.so         [.] malloc
     0.96%  ovn_pinctrl0    libc-2.28.so         [.] __memmove_avx_unaligned_erms                                                      
     0.96%  ovn_pinctrl0    libc-2.28.so         [.] _int_free
     0.80%  ovn_pinctrl0    ovn-controller       [.] 0x00000000000b87c1                                                                
     0.75%  ovn_pinctrl0    [kernel.kallsyms]    [k] copy_user_generic_unrolled                                                        
     0.71%  ovn_pinctrl0    libc-2.28.so         [.] __memset_avx2_unaligned_erms                                                      
     0.69%  ovn_pinctrl0    libpthread-2.28.so   [.] __pthread_enable_asynccancel                                                      
     0.69%  ovn_pinctrl0    libc-2.28.so         [.] __memcmp_avx2_movbe                                                               
     0.67%  ovn_pinctrl0    ovn-controller       [.] 0x000000000011c8fd                                                                
     0.65%  ovn_pinctrl0    [kernel.kallsyms]    [k] find_vma
     0.61%  ovn_pinctrl0    ovn-controller       [.] 0x00000000000469bb                                                                
     0.61%  ovn_pinctrl0    [kernel.kallsyms]    [k] skb_set_owner_w                                                                   
     0.60%  ovn_pinctrl0    ovn-controller       [.] 0x00000000000bdfbb                                                                
     0.58%  ovn_pinctrl0    ovn-controller       [.] 0x00000000000b87a6

Comment 6 Dan Williams 2021-11-18 16:46:30 UTC
As discussed above the solution is to use OVN's meter functionality to rate-limit packet-in to ovn-controller, which would be configured by ovn-kubernetes. This should be done for BFD and chk-pkt-len at least.

Comment 8 Surya Seetharaman 2022-01-14 20:28:50 UTC
Opened Upstream PR: https://github.com/ovn-org/ovn-kubernetes/pull/2752

Comment 14 errata-xmlrpc 2022-08-10 10:38:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.