Bug 2011525
| Summary: | Rate-limit incoming BFD to prevent ovn-controller DoS | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Sai Sindhur Malleni <smalleni> | ||||
| Component: | Networking | Assignee: | Surya Seetharaman <surya> | ||||
| Networking sub component: | ovn-kubernetes | QA Contact: | Anurag saxena <anusaxen> | ||||
| Status: | CLOSED ERRATA | Docs Contact: | |||||
| Severity: | high | ||||||
| Priority: | high | CC: | ctrautma, dblack, dcbw, dceara, ffernand, jiji, jlema, murali, nusiddiq, surya, trozet | ||||
| Version: | 4.10 | ||||||
| Target Milestone: | --- | ||||||
| Target Release: | 4.11.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | No Doc Update | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2022-08-10 10:38:04 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Sai Sindhur Malleni
2021-10-06 18:11:28 UTC
Placed DBS and provided Numan access to those. Created attachment 1830071 [details]
DBs and conf.db of worker node
perf record output shows pinctrl0 thread is hot on CPu
Event count (approx.): 6723064170
#
# Overhead Command Shared Object Symbol
# ........ .............. ................... ...........................................
#
1.97% ovn_pinctrl0 libpthread-2.28.so [.] __pthread_rwlock_wrlock
1.84% ovn_pinctrl0 libpthread-2.28.so [.] __pthread_rwlock_rdlock
1.77% ovn_pinctrl0 libpthread-2.28.so [.] __pthread_rwlock_unlock
1.75% ovn_pinctrl0 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
1.62% ovn_pinctrl0 libc-2.28.so [.] _int_malloc
1.53% ovn_pinctrl0 [kernel.kallsyms] [k] avc_has_perm
1.16% ovn_pinctrl0 [kernel.kallsyms] [k] _raw_spin_lock
1.09% ovn_pinctrl0 libc-2.28.so [.] malloc
0.96% ovn_pinctrl0 libc-2.28.so [.] __memmove_avx_unaligned_erms
0.96% ovn_pinctrl0 libc-2.28.so [.] _int_free
0.80% ovn_pinctrl0 ovn-controller [.] 0x00000000000b87c1
0.75% ovn_pinctrl0 [kernel.kallsyms] [k] copy_user_generic_unrolled
0.71% ovn_pinctrl0 libc-2.28.so [.] __memset_avx2_unaligned_erms
0.69% ovn_pinctrl0 libpthread-2.28.so [.] __pthread_enable_asynccancel
0.69% ovn_pinctrl0 libc-2.28.so [.] __memcmp_avx2_movbe
0.67% ovn_pinctrl0 ovn-controller [.] 0x000000000011c8fd
0.65% ovn_pinctrl0 [kernel.kallsyms] [k] find_vma
0.61% ovn_pinctrl0 ovn-controller [.] 0x00000000000469bb
0.61% ovn_pinctrl0 [kernel.kallsyms] [k] skb_set_owner_w
0.60% ovn_pinctrl0 ovn-controller [.] 0x00000000000bdfbb
0.58% ovn_pinctrl0 ovn-controller [.] 0x00000000000b87a6
As discussed above the solution is to use OVN's meter functionality to rate-limit packet-in to ovn-controller, which would be configured by ovn-kubernetes. This should be done for BFD and chk-pkt-len at least. Opened Upstream PR: https://github.com/ovn-org/ovn-kubernetes/pull/2752 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069 |