Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2279159

Summary: OVN revision numbers bumped on Noop port updates
Product: Red Hat OpenStack Reporter: Flavio Piccioni <fpiccion>
Component: openstack-neutronAssignee: Jakub Libosvar <jlibosva>
Status: CLOSED ERRATA QA Contact: Eran Kuris <ekuris>
Severity: high Docs Contact:
Priority: urgent    
Version: 17.1 (Wallaby)CC: astupnik, bcafarel, beagles, chrisw, dsedgmen, froyo, gthiemon, jcrumple, jlibosva, ldavidde, mariel, mburns, michjohn, njohnston, scohen, tweining
Target Milestone: z4Keywords: Triaged
Target Release: 17.1   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-neutron-18.6.1-17.1.20240822200817.85ff760.el9ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-11-21 09:40:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Flavio Piccioni 2024-05-05 15:02:05 UTC 
Description of problem:
After 16.2 to 17.1.2 upgrade, a platform running a huge amount (1300) of (ovn provider) loadbalancers (OCP/kuryr) starts facing memory saturation problems

8<--------------------------------------8<--------------------------------------8<--------------------------------------8<--------------------------------------
11:20:01 PM 439243504 439289468  84511512     16.02      2104   2697344 131671248     24.96    641108  82966240       228
11:30:01 PM 415935828 416097212 107729232     20.42      2104   2778304 152151584     28.85    647676 106172292      1212
11:40:00 PM 378074792 378675192 145151780     27.52      2104   3197932 191180400     36.24    661504 143943200      1524
11:50:01 PM 323490780 332469352 190942196     36.20      3540  11236780 226774032     42.99   3678064 194585840      2784
12:00:01 AM 313016400 322273884 201144180     38.13      3540  11517316 232714004     44.12   3684468 204962728      6800
12:10:02 AM 311540316 320924424 202521796     38.39      3540  11633156 233079340     44.19   3668132 206505232      2892
12:20:01 AM 303514140 312847828 210590724     39.92      3540  11566816 239634176     45.43   3543708 214586952     10148
12:30:00 AM 276519224 287163776 236297916     44.80      3540  12796844 249643008     47.33   3619152 241335120      1704
12:40:00 AM 247699864 258306092 265170324     50.27      3540  12735944 274905296     52.12   3574716 270187264      3540
12:50:00 AM 224926460 235624468 287851356     54.57      3540  12822012 295148172     55.96   3580844 292924628      3940
01:00:01 AM 204274472 215107832 308381568     58.46      3540  12940108 313340296     59.40   3583844 313517256     63236
01:10:01 AM 188949632 199709068 323911788     61.41      4524  12727016 327153968     62.02   3549408 328828448      5384
01:20:00 AM 166752812 177590968 346023140     65.60      4524  12803148 350177320     66.39   3557056 350932036      5408
01:30:00 AM 155222132 166151136 357469708     67.77      4524  12888320 358115448     67.89   3560196 362497116      1624
01:40:01 AM 131953824 142899712 380714120     72.18      4524  12898348 377099276     71.49   3565088 385666096      1976
01:50:01 AM 102762116 113840768 409768104     77.69      4524  13029368 403790280     76.55   3566900 414818428      5896
02:00:01 AM  89457712 100208752 423161616     80.22      4524  12994188 415097032     78.70   3587412 428120092      1496
02:10:01 AM  67541820  78221272 445140620     84.39      4524  12919044 435059988     82.48   3610472 449942076      9528
02:20:01 AM  41957100  51998844 471588468     89.41      2996  12467652 457720744     86.78   3439036 476012552     10612
02:30:00 AM  27488700  36481344 487107888     92.35      2376  11421464 471481476     89.38   3199712 490746820      5984
02:40:00 AM  17865024  26726556 496860876     94.20      2376  11290616 481416508     91.27   3128324 500310444      3748
02:50:01 AM   2164876   9378736 514239868     97.49      2372   9694496 495622648     93.96   2802748 516364544      6328

03:02:48 AM  LINUX RESTART	(96 CPU)
8<--------------------------------------8<--------------------------------------8<--------------------------------------8<--------------------------------------


From podman stats we are able to find out that memory-leaking process seems to be octavia_api:

8<--------------------------------------8<--------------------------------------8<--------------------------------------8<--------------------------------------
ID            NAME                                                                                  CPU %       MEM USAGE / LIMIT  MEM %       NET IO      BLOCK IO           PIDS        CPU TIME          AVG CPU %
[..]
212ae0c320fb  octavia_api                                                                           453.58%     322.3GB / 540.1GB  59.67%      0B / 0B     4.276GB / 30.21MB  2
[..]
8<--------------------------------------8<--------------------------------------8<--------------------------------------8<--------------------------------------


After tripleo_octavia_api restart:
8<--------------------------------------8<--------------------------------------8<--------------------------------------8<--------------------------------------
25e00d7be61b  octavia_api                                                                           22.46%      7.257GB / 540.1GB  1.34%       0B / 0B     16.38kB / 1.233MB  59          5m28.47749s       22.46%
8<--------------------------------------8<--------------------------------------8<--------------------------------------8<--------------------------------------
  

Version-Release number of selected component (if applicable):
RHOSP 17.1.2


How reproducible:
create a huge (> 1000k) amount of loadbalancers (ovn provider)

Actual results:
octavia_api containers quickly saturate host's memory
Comment 31 errata-xmlrpc 2024-11-21 09:40:18 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (RHOSP 17.1.4 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:9974