Bug 2002425

Summary: [SCALE] ovn-controller takes too long to set ports as ovn-installed
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Tim Rozet <trozet>
Component: OVNAssignee: Dumitru Ceara <dceara>
Status: CLOSED NOTABUG QA Contact: Jianlin Shi <jishi>
Severity: urgent Docs Contact:
Priority: high    
Version: RHEL 8.0CC: ctrautma, dcbw, dceara, jiji, mmichels
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: perfscale-ovn
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-09-10 18:13:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1986216    

Description Tim Rozet 2021-09-08 19:14:37 UTC 
Description of problem:
In a 20 node cluster, running node-density lite with 249 pods per node, ovn-controller takes too long to set ports up via ovn-installed.

Example: https://gist.githubusercontent.com/trozet/f08d3fd4c6b991c40cc56122ee9d34b2/raw/22c14543822d76b21b4734f69857854606f7aa71/gistfile1.txt

The result is that some pods are taking over 300 seconds to be considered as up. This can be mitigated by slowing the rate at which ports are created in OVN. There are no long poll intervals in ovn-controller during this time, and only 35k openflow flows.

Version-Release number of selected component (if applicable):
ovn21.09-21.09.0-18.el8fdp.x86_64
Comment 1 Dumitru Ceara 2021-09-10 16:20:22 UTC 
@trozet Given that the symptoms were root caused to bug 2003193 and ovn-controller was legitimately waiting for OVS flows to report that flows were installed, I think we can safely close the OVN bug, do you agree?
Comment 2 Dan Williams 2021-09-10 18:13:20 UTC 
Yes, that's correct.
Comment 3 Red Hat Bugzilla 2023-09-15 01:14:52 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days