Bug 2052975 - High cpu load on Juniper Qfx5120 Network switches after upgrade to Openshift 4.8.26
Summary: High cpu load on Juniper Qfx5120 Network switches after upgrade to Openshift ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.8
Hardware: x86_64
OS: Linux
high
urgent
Target Milestone: ---
: 4.11.0
Assignee: Surya Seetharaman
QA Contact: huirwang
URL:
Whiteboard:
Depends On: 2054394
Blocks: 2063834
TreeView+ depends on / blocked
 
Reported: 2022-02-10 11:34 UTC by Jonas Nordell
Modified: 2022-11-01 05:55 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 2063834 (view as bug list)
Environment:
Last Closed: 2022-08-10 10:49:18 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift ovn-kubernetes pull 982 0 None open Bug 2052975: Bump OVN to ovn-2021-21.12.0-30.el8fdp 2022-03-04 16:53:34 UTC
Github openshift ovn-kubernetes pull 988 0 None open Bug 2052975: Downstream merge 07-03-2022 2022-03-07 17:33:12 UTC
Github ovn-org ovn-kubernetes pull 2835 0 None open Stop sending GARPs for LB VIPs on GR 2022-03-04 16:53:36 UTC
Red Hat Product Errata RHSA-2022:5069 0 None None None 2022-08-10 10:49:37 UTC

Description Jonas Nordell 2022-02-10 11:34:22 UTC
Description of problem:

After a cluster upgrade from OCP 4.7.33 to OCP 4.8.26 a big increase in CPU load on the network switches has started to affect all connected entities. The reason seams to be related to the fact that the switches is discovering Service IP addresses on the node NIC. 


Version-Release number of selected component (if applicable):

OCP 3.8.26


How reproducible:
In the customer environment on several clusters. 

Steps to Reproduce:
1. Upgrade cluster from 4.7.33 to 4.8.26
2. Look at assigned IPs on the network
3.

Actual results:
The switch is detecting many service IPs 

Expected results:
The switch should not detect service IPs

Additional info:

Comment 18 Surya Seetharaman 2022-02-14 21:13:31 UTC
Marking BZ with a depends on https://bugzilla.redhat.com/show_bug.cgi?id=2054394 since the fix really needs to go into OVN before we can consume it. This bug will continue to be open to track the OVN bump needed to consume this feature and to disable the GARPs for LB IPs on OVN-K side for egressIPs once the OVN fix lands. Note that this will take a few weeks.

Comment 22 Surya Seetharaman 2022-02-28 17:11:10 UTC
Upstream fix: https://github.com/ovn-org/ovn-kubernetes/pull/2835

Comment 24 Surya Seetharaman 2022-03-07 08:46:41 UTC
moving it back to post to pick up the second commit: https://github.com/ovn-org/ovn-kubernetes/pull/2835

Comment 30 errata-xmlrpc 2022-08-10 10:49:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069


Note You need to log in before you can comment on or make changes to this bug.