Bug 1956955 - Services sync causes too many ovn load balancer deletes
Summary: Services sync causes too many ovn load balancer deletes
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.8
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.9.0
Assignee: Tim Rozet
QA Contact: Kedar Kulkarni
URL:
Whiteboard: perfscale-ovn
Depends On:
Blocks: 1986573
TreeView+ depends on / blocked
 
Reported: 2021-05-04 18:17 UTC by Mohit Sheth
Modified: 2021-10-18 17:30 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1986573 (view as bug list)
Environment:
Last Closed: 2021-10-18 17:30:51 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift ovn-kubernetes pull 545 0 None open Bug 1956955: Reduces number of OVN operations in services #2201 2021-05-19 18:57:44 UTC
Github openshift ovn-kubernetes pull 567 0 None open Bug 1956955: Batching: Fixes finding maximum bash arguments 2021-06-09 17:53:39 UTC
Github openshift ovn-kubernetes pull 582 0 None closed Bug 1973813: 6-21-2021 merge 2021-06-22 18:37:27 UTC
Github ovn-org ovn-kubernetes pull 2201 0 None open Reduces number of remove VIP operations in svcs 2021-05-05 04:14:58 UTC
Github ovn-org ovn-kubernetes pull 2221 0 None closed Split large nbctl transactions 2021-05-19 18:38:15 UTC
Github ovn-org ovn-kubernetes pull 2266 0 None open Declare a maximum line length for batching 2021-06-18 17:31:03 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:30:54 UTC

Description Mohit Sheth 2021-05-04 18:17:37 UTC 
Description of problem:
While running cluster-density at 250 node scale with 2000 projects, we see that there are 46000 nbctl commands and out of those 29000 are remove load_balancer.*vips

Version-Release number of selected component (if applicable):
ovs-vswitchd (Open vSwitch) 2.15.0

How reproducible:


Steps to Reproduce:
1. Run cluster-density 2k at 250 node scale

Actual results:
Lots of remove load_balancer deletes

Expected results:
Lesser deletes

Additional info:
ovnkube-master logs http://dell-r510-01.perf.lab.eng.rdu2.redhat.com/msheth/ovnkube-master-04-may/ovnkube-master-mkzfk.log
Comment 16 Kedar Kulkarni 2021-07-14 20:26:16 UTC 
Hi,

I ran Cluster Density 250 workers for 2000 iterations and grepped through the log, the remove load_balancer call count was 15322, vs 29000 as reported originally. There were no errors there about bash too many args. I used the following build "4.9.0-0.nightly-2021-07-12-203753" . Based on this info, I am closing this bz as verified. 

Thanks,
KK.
Comment 19 errata-xmlrpc 2021-10-18 17:30:51 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759
Note You need to log in before you can comment on or make changes to this bug.