Bug 1962338

+++ This bug was initially created as a clone of Bug #1960042 +++

Description of problem:
OCP4.8 w/ OVNKubernetes as the SDN. Scaled to 300 nodes, we are seeing ovn-northd consume an entire core:

      1 root      20   0 1521272   1.4g   7420 R  98.7   1.1 370:31.04 ovn-northd                                                                                                                                                                             

TimR also noted that it is taking 30+ seconds for northd to process changes.


Version-Release number of selected component (if applicable):
4.8

How reproducible:
100%

Steps to Reproduce:
1. Deploy OCP4.8
2. Scale to 300 nodes
3. Run clusterdensity 2k

Actual results:

--- Additional comment from Joe Talerico on 2021-05-12 21:12:57 UTC ---

full must-gather : http://dell-r510-01.perf.lab.eng.rdu2.redhat.com/rook/must-gather-bz1960042.tar.gz

--- Additional comment from Dumitru Ceara on 2021-05-18 21:50:54 UTC ---

(In reply to Joe Talerico from comment #1)
> full must-gather :
> http://dell-r510-01.perf.lab.eng.rdu2.redhat.com/rook/must-gather-bz1960042.
> tar.gz

I might have missed it but it seems to me that the must-gather doesn't
include the OVN NB database file, it only includes container logs.
Would it be possible to attach a NB DB from a scaled clusters from the
time when the issue is hit?  That would make investigation easier as
we don't need a complete cluster to profile northd.

Thanks,
Dumitru

--- Additional comment from Tim Rozet on 2021-05-19 13:25:28 UTC ---



--- Additional comment from Dumitru Ceara on 2021-05-19 15:33:03 UTC ---

After initial analysis, a decent time of the poll interval is spent in
building router load balancer logical flows.

1. get_router_load_balancer_ips() is called way too often, we can just
precompute those IPs once per iteration, instead of calling it for every
logical router port.  An initial test shows that this change reduces
the loop iteration time from ~22s to ~19s.

2. ovn_lflow_add_at() always builds a logical flow record even though
this will be discarded if the logical flow is aggregated on a datapath
group.  We can instead try to delay the creation of new flow records
until really necessary.  This saves a decent amount of allocations and
memory copying.  An initial test shows that this change reduces the
loop iteration time further to ~15s.

3. We can try to change the way load balancer flows are built.
Currently for X routers (or switches) with Y load balancers applied to
them we do:
- for every router:
  - for every load balancer:
    - parse and generate lots of common lb stuff (e.g., VIPs, backends)
    - generate one logical flow per VIP.

I think we can save quite a lot of CPU by changing this to:
- for every load balancer:
  - parse and generate lots of common lb stuff (e.g., VIPs, backends)
  - for every router:
    - generate one logical flow per VIP.

4. I also enabled northd parallelization and this further reduced the
loop iteration times to ~8s with the cost of northd consuming up to
900% CPU.  However, northd parallelization is a new feature and needs
further testing and needs to be enabled in CI.

I'll open OVN BZs for all items above so we can track the work
independently.