Bug 2011525
Summary: | Rate-limit incoming BFD to prevent ovn-controller DoS | ||||||
---|---|---|---|---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Sai Sindhur Malleni <smalleni> | ||||
Component: | Networking | Assignee: | Surya Seetharaman <surya> | ||||
Networking sub component: | ovn-kubernetes | QA Contact: | Anurag saxena <anusaxen> | ||||
Status: | CLOSED ERRATA | Docs Contact: | |||||
Severity: | high | ||||||
Priority: | high | CC: | ctrautma, dblack, dcbw, dceara, ffernand, jiji, jlema, murali, nusiddiq, surya, trozet | ||||
Version: | 4.10 | ||||||
Target Milestone: | --- | ||||||
Target Release: | 4.11.0 | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | No Doc Update | |||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2022-08-10 10:38:04 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Attachments: |
|
Description
Sai Sindhur Malleni
2021-10-06 18:11:28 UTC
Placed DBS and provided Numan access to those. Created attachment 1830071 [details]
DBs and conf.db of worker node
perf record output shows pinctrl0 thread is hot on CPu Event count (approx.): 6723064170 # # Overhead Command Shared Object Symbol # ........ .............. ................... ........................................... # 1.97% ovn_pinctrl0 libpthread-2.28.so [.] __pthread_rwlock_wrlock 1.84% ovn_pinctrl0 libpthread-2.28.so [.] __pthread_rwlock_rdlock 1.77% ovn_pinctrl0 libpthread-2.28.so [.] __pthread_rwlock_unlock 1.75% ovn_pinctrl0 [kernel.kallsyms] [k] copy_user_enhanced_fast_string 1.62% ovn_pinctrl0 libc-2.28.so [.] _int_malloc 1.53% ovn_pinctrl0 [kernel.kallsyms] [k] avc_has_perm 1.16% ovn_pinctrl0 [kernel.kallsyms] [k] _raw_spin_lock 1.09% ovn_pinctrl0 libc-2.28.so [.] malloc 0.96% ovn_pinctrl0 libc-2.28.so [.] __memmove_avx_unaligned_erms 0.96% ovn_pinctrl0 libc-2.28.so [.] _int_free 0.80% ovn_pinctrl0 ovn-controller [.] 0x00000000000b87c1 0.75% ovn_pinctrl0 [kernel.kallsyms] [k] copy_user_generic_unrolled 0.71% ovn_pinctrl0 libc-2.28.so [.] __memset_avx2_unaligned_erms 0.69% ovn_pinctrl0 libpthread-2.28.so [.] __pthread_enable_asynccancel 0.69% ovn_pinctrl0 libc-2.28.so [.] __memcmp_avx2_movbe 0.67% ovn_pinctrl0 ovn-controller [.] 0x000000000011c8fd 0.65% ovn_pinctrl0 [kernel.kallsyms] [k] find_vma 0.61% ovn_pinctrl0 ovn-controller [.] 0x00000000000469bb 0.61% ovn_pinctrl0 [kernel.kallsyms] [k] skb_set_owner_w 0.60% ovn_pinctrl0 ovn-controller [.] 0x00000000000bdfbb 0.58% ovn_pinctrl0 ovn-controller [.] 0x00000000000b87a6 As discussed above the solution is to use OVN's meter functionality to rate-limit packet-in to ovn-controller, which would be configured by ovn-kubernetes. This should be done for BFD and chk-pkt-len at least. Opened Upstream PR: https://github.com/ovn-org/ovn-kubernetes/pull/2752 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069 |