Bug 1724543

Summary:	Memory leak in neutron-server due to neutron-lbaas haproxy driver
Product:	Red Hat OpenStack	Reporter:	Carlos Goncalves <cgoncalves>
Component:	openstack-neutron-lbaas	Assignee:	Carlos Goncalves <cgoncalves>
Status:	CLOSED ERRATA	QA Contact:	Bruna Bonguardo <bbonguar>
Severity:	low	Docs Contact:
Priority:	low
Version:	10.0 (Newton)	CC:	amuller, apevec, bperkins, lhh, scohen
Target Milestone:	zstream	Keywords:	Triaged, ZStream
Target Release:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-neutron-lbaas-9.2.2-10.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-10-16 09:41:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Carlos Goncalves 2019-06-27 09:04:45 UTC

LBaaS uses a dict for caching load balancer resources (including children resources). That alone already increases memory consumption. A neutron-server restart solves this.

But the real issue is that LBaaS leaves some data behind when a LB is deleted, increasing memory without freeing it (unless neutron-server is restart) hence the leak.

In my test output below you can see memory not being freed when LBs are deleted. #B is size in bytes of the dict (cache). I tried to fix the leak -- we can see that memory is now being freed and when there are no LBs the size of the cache is the size of an empty dictionary (280 bytes).


$ python
Python 2.7.5 (default, Jul 13 2018, 13:06:57) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.getsizeof(dict())
280
>>> 

PRE-PATCH
=========

Start
#LB=0   #L=0    #B=280  <--- empty dict

Create 1 LB with 10 listeners
#LB=1   #L=1    #B=30341
#LB=1   #L=2    #B=47653
#LB=1   #L=3    #B=69385
#LB=1   #L=4    #B=95721
#LB=1   #L=5    #B=126865
#LB=1   #L=6    #B=162493
#LB=1   #L=7    #B=202745
#LB=1   #L=8    #B=247621
#LB=1   #L=9    #B=297761
#LB=1   #L=10   #B=351905
#LB=1   #L=11   #B=410713
#LB=1   #L=12   #B=474093
#LB=1   #L=13   #B=542145
#LB=1   #L=14   #B=614761
#LB=1   #L=15   #B=691993
#LB=1   #L=16   #B=773909
#LB=1   #L=17   #B=861745
#LB=1   #L=18   #B=952905
#LB=1   #L=19   #B=1048761
#LB=1   #L=20   #B=1149157

Create 1 more LB with 10 listeners
#LB=2   #L=21   #B=1176493
#LB=2   #L=22   #B=1193829
#LB=2   #L=23   #B=1215593
#LB=2   #L=24   #B=1241989
#LB=2   #L=25   #B=1273185
#LB=2   #L=26   #B=1308873
#LB=2   #L=27   #B=1349161
#LB=2   #L=28   #B=1394109
#LB=2   #L=29   #B=1444289
#LB=2   #L=30   #B=1498517
#LB=2   #L=31   #B=1557417
#LB=2   #L=32   #B=1620897
#LB=2   #L=33   #B=1689001
#LB=2   #L=34   #B=1761729
#LB=2   #L=35   #B=1839145
#LB=2   #L=36   #B=1921193
#LB=2   #L=37   #B=2009097
#LB=2   #L=38   #B=2100401
#LB=2   #L=39   #B=2196409
#LB=2   #L=40   #B=2297049

Delete 1/2 LB
#LB=2   #L=39   #B=2195129
#LB=2   #L=38   #B=2099185
#LB=2   #L=37   #B=2007945
#LB=2   #L=36   #B=1920037
#LB=2   #L=35   #B=1838121
#LB=2   #L=34   #B=1760829
#LB=2   #L=33   #B=1688161
#LB=2   #L=32   #B=1620065
#LB=2   #L=31   #B=1556601
#LB=2   #L=30   #B=1497769
#LB=2   #L=29   #B=1443609
#LB=2   #L=28   #B=1393461
#LB=2   #L=27   #B=1348585
#LB=2   #L=26   #B=1308341
#LB=2   #L=25   #B=1272753
#LB=2   #L=24   #B=1241629
#LB=2   #L=23   #B=1215321
#LB=2   #L=22   #B=1193625
#LB=2   #L=21   #B=1176569
#LB=2   #L=21   #B=1176569

Delete 1/1 LB
#LB=2   #L=20   #B=1074813
#LB=2   #L=19   #B=979101
#LB=2   #L=18   #B=888005
#LB=2   #L=17   #B=800301
#LB=2   #L=16   #B=718509
#LB=2   #L=15   #B=641273
#LB=2   #L=14   #B=568661
#LB=2   #L=13   #B=500673
#LB=2   #L=12   #B=437309
#LB=2   #L=11   #B=378569
#LB=2   #L=10   #B=324493
#LB=2   #L=9    #B=274421
#LB=2   #L=8    #B=229645
#LB=2   #L=7    #B=189457
#LB=2   #L=6    #B=153893
#LB=2   #L=5    #B=122793
#LB=2   #L=4    #B=96525
#LB=2   #L=3    #B=74873
#LB=2   #L=2    #B=57829
#LB=2   #L=2    #B=57829



POST PATCH
=============


Start
#LB=0   #L=0    #B=280  <--- empty dict

Create 1 LB with 10 listeners
#LB=1   #L=1    #B=30333
#LB=1   #L=2    #B=47641
#LB=1   #L=1    #B=30409
#LB=0   #L=0    #B=280
#LB=1   #L=1    #B=30341
#LB=1   #L=2    #B=47653
#LB=1   #L=3    #B=69369
#LB=1   #L=4    #B=95701
#LB=1   #L=5    #B=126865
#LB=1   #L=6    #B=162465
#LB=1   #L=7    #B=202713
#LB=1   #L=8    #B=247549
#LB=1   #L=9    #B=297681
#LB=1   #L=10   #B=351817
#LB=1   #L=11   #B=410617
#LB=1   #L=12   #B=474041
#LB=1   #L=13   #B=542089
#LB=1   #L=14   #B=614761
#LB=1   #L=15   #B=692057
#LB=1   #L=16   #B=773909
#LB=1   #L=17   #B=861673
#LB=1   #L=18   #B=952829
#LB=1   #L=19   #B=1048681
#LB=1   #L=20   #B=1149157

Create 1 more LB with 10 listeners
#LB=2   #L=21   #B=1176485
#LB=2   #L=22   #B=1193817
#LB=2   #L=23   #B=1215577
#LB=2   #L=24   #B=1241969
#LB=2   #L=25   #B=1273161
#LB=2   #L=26   #B=1308845
#LB=2   #L=27   #B=1349129
#LB=2   #L=28   #B=1394073
#LB=2   #L=29   #B=1444289
#LB=2   #L=30   #B=1498561
#LB=2   #L=31   #B=1557417
#LB=2   #L=32   #B=1620897
#LB=2   #L=33   #B=1689057
#LB=2   #L=34   #B=1761849
#LB=2   #L=35   #B=1839209
#LB=2   #L=36   #B=1921193
#LB=2   #L=37   #B=2009169
#LB=2   #L=38   #B=2100477
#LB=2   #L=39   #B=2196409
#LB=2   #L=40   #B=2297049

Delete 1/2 LB
#LB=2   #L=39   #B=2195049
#LB=2   #L=38   #B=2099185
#LB=2   #L=37   #B=2007873
#LB=2   #L=36   #B=1920037
#LB=2   #L=35   #B=1838121
#LB=2   #L=34   #B=1760769
#LB=2   #L=33   #B=1688105
#LB=2   #L=32   #B=1620013
#LB=2   #L=31   #B=1556553
#LB=2   #L=30   #B=1497725
#LB=2   #L=29   #B=1443529
#LB=2   #L=28   #B=1393389
#LB=2   #L=27   #B=1348553
#LB=2   #L=26   #B=1308341
#LB=2   #L=25   #B=1272753
#LB=2   #L=24   #B=1241629
#LB=2   #L=23   #B=1215321
#LB=2   #L=22   #B=1193625
#LB=2   #L=21   #B=1176569
#LB=1   #L=20   #B=1149157

Delete 1/1 LB
#LB=1   #L=20   #B=1149157
#LB=1   #L=19   #B=1047321
#LB=1   #L=16   #B=772753
#LB=1   #L=18   #B=952221
#LB=1   #L=16   #B=772753
#LB=1   #L=14   #B=613801
#LB=1   #L=15   #B=690969
#LB=1   #L=13   #B=541193
#LB=1   #L=12   #B=473209
#LB=1   #L=11   #B=409897
#LB=1   #L=10   #B=351201
#LB=1   #L=9    #B=297081
#LB=1   #L=8    #B=247009
#LB=1   #L=7    #B=202201
#LB=1   #L=6    #B=162017
#LB=1   #L=5    #B=126481
#LB=1   #L=4    #B=95381
#LB=1   #L=3    #B=69097
#LB=1   #L=2    #B=47449
#LB=1   #L=1    #B=30417
#LB=0   #L=0    #B=280  <---- empty dict

Comment 8 errata-xmlrpc 2019-10-16 09:41:17 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3115