Bug 1769418

Summary:	Several memory leaks reported by Valgrind for 389-ds 1.3.9.1-10.
Product:	Red Hat Enterprise Linux 7	Reporter:	Têko Mihinto <tmihinto>
Component:	389-ds-base	Assignee:	thierry bordaz <tbordaz>
Status:	CLOSED ERRATA	QA Contact:	RHDS QE <ds-qe-bugs>
Severity:	medium	Docs Contact:	Marc Muehlfeld <mmuehlfe>
Priority:	high
Version:	7.7	CC:	horst.thaller, jreznik, ldelouw, mhonek, mreynolds, msauton, nkinder, pasik, rrelyea, spichugi, tbordaz, tmihinto, vashirov
Target Milestone:	rc	Keywords:	ZStream
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	389-ds-base-1.3.10.2-1.el7	Doc Type:	Bug Fix
Doc Text:	.Directory Server no longer leaks memory when using ACIs with an `ip` bind rule When a Directory Server Access Control Instruction (ACI) contains an `ip` bind rule, the server stores the value of the `ip` keyword as a reference while evaluating the ACI. Previously, when the evaluations were completed Directory Server did not free the `ip` value. As a consequence, the server leaked around 100 bytes of memory each time the server evaluated an ACI with an `ip` bind rule. With this update, Directory Server keeps track of the `ip` value in the per-connection structure and frees the structure when the connection is closed. As a consequence, Directory Server no longer leaks memory in the mentioned scenario.	Story Points:	---
Clone Of:
Clones:	1790975 1803023 (view as bug list)		Environment:
Last Closed:	2020-09-29 19:46:50 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1788833, 1790975, 1803023

Description Têko Mihinto 2019-11-06 15:33:17 UTC

Description of problem:

A customer is experiencing a high memory usage with RHDS.
The memory consumption starts from a few GB and can go up to 100 GB after a week.

Version-Release number of selected component (if applicable):

$ cat etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)
$
$ grep 389-ds-base installed-rpms
389-ds-base-1.3.9.1-10.el7.x86_64                           Mon Aug 26 14:13:40 2019
389-ds-base-debuginfo-1.3.9.1-10.el7.x86_64                 Tue Nov  5 10:20:03 2019
389-ds-base-libs-1.3.9.1-10.el7.x86_64                      Mon Aug 26 14:13:26 2019
$

How reproducible:

Always at customer site.

Steps to Reproduce:

1. Restart the RHDS instance.
The sum of the configured caches is about 1.3 GB:
$ grep total errors | tail -1
[05/Nov/2019:11:32:52.600935718 +0000] - NOTICE - ldbm_back_start - total cache size: 1316466524 B;
$

2. The memory usage will grow significantly:
Eg:
$ head -1 ./ps
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
$
$ grep ns-slapd ./ps
dirsrv    3696 79.9 93.6 64044164 61602916 ?   Ssl  Oct15 18237:22 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-XXX -i /var/run/dirsrv/slapd-XXX.pid
$


Actual results:

High memory usage indicating memory leaks.

Expected results:

The memory usage should not go that high.
Need to fix the memory leaks.

Additional info:

From the Valgrind output:

* The leak summary:

==127975== LEAK SUMMARY:
==127975==    definitely lost: 29,132 bytes in 67 blocks
==127975==    indirectly lost: 136,872 bytes in 2,478 blocks
==127975==      possibly lost: 1,432 bytes in 5 blocks
==127975==    still reachable: 4,524,109 bytes in 26,967 blocks
==127975==                       of which reachable via heuristic:
==127975==                         stdstring          : 30 bytes in 1 blocks
==127975==         suppressed: 0 bytes in 0 blocks

* A couple of leak signatures:

==127975== 277,032 bytes in 51 blocks are still reachable in loss record 1,251 of 1,253
==127975==    at 0x4C2BF79: calloc (vg_replace_malloc.c:762)
==127975==    by 0x9D382A1: PORT_ZAlloc_Util (in /usr/lib64/libnssutil3.so)
==127975==    by 0x169F1BBE: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x169F1D0F: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x169D7E0D: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x169F0FFA: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x6C2EBBE: PK11_DeriveWithTemplate (in /usr/lib64/libnss3.so)
==127975==    by 0x6C2EC50: PK11_Derive (in /usr/lib64/libnss3.so)
==127975==    by 0x6992A82: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69B164A: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699AD58: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699C7C2: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699CD5C: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699E31E: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699F388: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69A5711: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69A63A4: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69AA460: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x122160: connection_read_ldap_data (connection.c:1097)
==127975==    by 0x122160: connection_read_operation (connection.c:1181)
==127975==    by 0x122996: connection_threadmain (connection.c:1628)
==127975==    by 0x7338BFA: ??? (in /usr/lib64/libnspr4.so)
==127975==    by 0x7978EA4: start_thread (pthread_create.c:307)
==127975==    by 0x83088CC: clone (clone.S:111)

...

==127975== 120,950 (80 direct, 120,870 indirect) bytes in 1 blocks are definitely lost in loss record 1,244 of 1,253
==127975==    at 0x4C2BF79: calloc (vg_replace_malloc.c:762)
==127975==    by 0x50944D3: slapi_ch_calloc (ch_malloc.c:175)
==127975==    by 0x1343D722: vlvSearch_new (vlv_srch.c:46)
==127975==    by 0x134398F7: vlv_init_search_entry (vlv.c:320)
==127975==    by 0x509E308: dse_call_callback.isra.1 (dse.c:2553)
==127975==    by 0x509FFBC: do_dse_search (dse.c:1567)
==127975==    by 0x509FFBC: dse_search (dse.c:1669)
==127975==    by 0x50DA461: op_shared_search (opshared.c:766)
==127975==    by 0x50EC036: search_internal_callback_pb (plugin_internal_op.c:727)
==127975==    by 0x50EC287: search_internal_pb (plugin_internal_op.c:592)
==127975==    by 0x50EC49C: slapi_search_internal (plugin_internal_op.c:439)
==127975==    by 0x1343A9D1: vlv_init (vlv.c:441)
==127975==    by 0x13404BF2: ldbm_instance_startall (instance.c:312)
==127975==    by 0x13438D99: ldbm_back_start (start.c:383)
==127975==    by 0x50E7096: plugin_call_func (plugin.c:2028)
==127975==    by 0x50E770A: plugin_call_one (plugin.c:1978)
==127975==    by 0x50E770A: plugin_dependency_startall (plugin.c:1737)
==127975==    by 0x50E770A: plugin_startall (plugin.c:1950)
==127975==    by 0x119A17: main (main.c:1141)

Comment 2 mreynolds 2019-11-06 16:03:16 UTC

Initial observations:


==127975== 277,032 bytes in 51 blocks are still reachable in loss record 1,251 of 1,253
==127975==    at 0x4C2BF79: calloc (vg_replace_malloc.c:762)
==127975==    by 0x9D382A1: PORT_ZAlloc_Util (in /usr/lib64/libnssutil3.so)
==127975==    by 0x169F1BBE: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x169F1D0F: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x169D7E0D: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x169F0FFA: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x6C2EBBE: PK11_DeriveWithTemplate (in /usr/lib64/libnss3.so)
==127975==    by 0x6C2EC50: PK11_Derive (in /usr/lib64/libnss3.so)
==127975==    by 0x6992A82: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69B164A: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699AD58: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699C7C2: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699CD5C: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699E31E: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699F388: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69A5711: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69A63A4: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69AA460: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x122160: connection_read_ldap_data (connection.c:1097)
==127975==    by 0x122160: connection_read_operation (connection.c:1181)
==127975==    by 0x122996: connection_threadmain (connection.c:1628)
==127975==    by 0x7338BFA: ??? (in /usr/lib64/libnspr4.so)
==127975==    by 0x7978EA4: start_thread (pthread_create.c:307)
==127975==    by 0x83088CC: clone (clone.S:111)


If this is a recurring leak it could explain some of this.  But it's a leak in NSS, not DS itself.  Did you try reproducing this with ldclt over ldaps?  If not, could you try it?



==127975== 120,950 (80 direct, 120,870 indirect) bytes in 1 blocks are definitely lost in loss record 1,244 of 1,253
==127975==    at 0x4C2BF79: calloc (vg_replace_malloc.c:762)
==127975==    by 0x50944D3: slapi_ch_calloc (ch_malloc.c:175)
==127975==    by 0x1343D722: vlvSearch_new (vlv_srch.c:46)
==127975==    by 0x134398F7: vlv_init_search_entry (vlv.c:320)
==127975==    by 0x509E308: dse_call_callback.isra.1 (dse.c:2553)
==127975==    by 0x509FFBC: do_dse_search (dse.c:1567)
==127975==    by 0x509FFBC: dse_search (dse.c:1669)
==127975==    by 0x50DA461: op_shared_search (opshared.c:766)
==127975==    by 0x50EC036: search_internal_callback_pb (plugin_internal_op.c:727)
==127975==    by 0x50EC287: search_internal_pb (plugin_internal_op.c:592)
==127975==    by 0x50EC49C: slapi_search_internal (plugin_internal_op.c:439)
==127975==    by 0x1343A9D1: vlv_init (vlv.c:441)
==127975==    by 0x13404BF2: ldbm_instance_startall (instance.c:312)
==127975==    by 0x13438D99: ldbm_back_start (start.c:383)
==127975==    by 0x50E7096: plugin_call_func (plugin.c:2028)
==127975==    by 0x50E770A: plugin_call_one (plugin.c:1978)
==127975==    by 0x50E770A: plugin_dependency_startall (plugin.c:1737)
==127975==    by 0x50E770A: plugin_startall (plugin.c:1950)
==127975==    by 0x119A17: main (main.c:1141)

This is a red herring.  This is a one time startup leak, they are a lot of these "leaks" unfortunately.  Any time you see "plugin_startall" in the stack it should be ignored.


I'm more worried that there is a leak in some hash or array that incorrectly keeps growing, but then it actually gets freed but only at server shutdown - so valgrind or asan would not detect it.  This would be more difficult to investigate and we would really need a reproducer.

Comment 4 mreynolds 2019-11-06 20:15:29 UTC

I looked over the valgrind output, and there are lot of stacks with very large leaks like this (which you already mentioned):

==127975== 277,032 bytes in 51 blocks are still reachable in loss record 1,251 of 1,253
==127975==    at 0x4C2BF79: calloc (vg_replace_malloc.c:762)
==127975==    by 0x9D382A1: PORT_ZAlloc_Util (in /usr/lib64/libnssutil3.so)
==127975==    by 0x169F1BBE: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x169F1D0F: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x169D7E0D: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x169F0FFA: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x6C2EBBE: PK11_DeriveWithTemplate (in /usr/lib64/libnss3.so)
==127975==    by 0x6C2EC50: PK11_Derive (in /usr/lib64/libnss3.so)
==127975==    by 0x6992A82: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69B164A: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699AD58: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699C7C2: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699CD5C: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699E31E: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699F388: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69A5711: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69A63A4: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69AA460: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x122160: connection_read_ldap_data (connection.c:1097)
==127975==    by 0x122160: connection_read_operation (connection.c:1181)
==127975==    by 0x122996: connection_threadmain (connection.c:1628)
==127975==    by 0x7338BFA: ??? (in /usr/lib64/libnspr4.so)
==127975==    by 0x7978EA4: start_thread (pthread_create.c:307)
==127975==    by 0x83088CC: clone (clone.S:111)

Everything else in the valgrind report looks okay.

What version of NSS is on the customers system?

Comment 5 Têko Mihinto 2019-11-06 21:12:33 UTC

Hi Mark,

$ grep ^nss installed-rpms 
nss-3.44.0-4.el7.i686                                       Mon Aug 26 14:14:27 2019
nss-3.44.0-4.el7.x86_64                                     Mon Aug 26 14:13:04 2019
nss-pem-1.0.3-7.el7.i686                                    Mon Aug 26 14:14:27 2019
nss-pem-1.0.3-7.el7.x86_64                                  Mon Aug 26 14:13:04 2019
nss-softokn-3.44.0-5.el7.i686                               Mon Aug 26 14:14:27 2019
nss-softokn-3.44.0-5.el7.x86_64                             Mon Aug 26 14:13:04 2019
nss-softokn-freebl-3.44.0-5.el7.i686                        Mon Aug 26 14:14:24 2019
nss-softokn-freebl-3.44.0-5.el7.x86_64                      Mon Aug 26 14:12:56 2019
nss-sysinit-3.44.0-4.el7.x86_64                             Mon Aug 26 14:13:04 2019
nss-tools-3.44.0-4.el7.x86_64                               Mon Aug 26 14:13:11 2019
nss-util-3.44.0-3.el7.i686                                  Mon Aug 26 14:14:25 2019
nss-util-3.44.0-3.el7.x86_64                                Mon Aug 26 14:12:57 2019
$

Please let me know if you need further details.

Regards,
Têko.

Comment 6 mreynolds 2019-11-07 14:01:28 UTC

What would really be useful now is to get a clean valgrind report.  So the customer needs to install the devel and debuginfo packages for all the nss packages.  Then when they run the valgrind test again we should hopefully see the full stack and symbols.  Then we can share that with the NSS team and move forward on the investigation.

Comment 8 Têko Mihinto 2019-11-08 09:00:28 UTC

Hi Mark,

I have attached the Valgrind output after the NSS devel and debug packages have been installed.

We have the NSS calls in the new output.
Eg:

==83837== 298,760 bytes in 55 blocks are still reachable in loss record 1,263 of 1,265
==83837==    at 0x4C2BF79: calloc (vg_replace_malloc.c:762)
==83837==    by 0x9D382A1: PORT_ZAlloc_Util (secport.c:116)
==83837==    by 0x169F1BBE: sftk_GetObjectFromList (pkcs11u.c:921)
==83837==    by 0x169F1D0F: sftk_NewObject (pkcs11u.c:1028)
==83837==    by 0x169D7E0D: sftk_buildSSLKey (pkcs11c.c:6151)
==83837==    by 0x169F0FFA: NSC_DeriveKey (pkcs11c.c:7154)
==83837==    by 0x6C2EBBE: PK11_DeriveWithTemplate (pk11skey.c:1589)
==83837==    by 0x6C2EC50: PK11_Derive (pk11skey.c:1445)
==83837==    by 0x6992A82: ssl3_DeriveConnectionKeys (ssl3con.c:3575)
==83837==    by 0x6992A82: ssl3_InitPendingCipherSpecs (ssl3con.c:1793)
==83837==    by 0x69B164A: ssl3_HandleECDHClientKeyExchange (ssl3ecc.c:316)
==83837==    by 0x699AD58: ssl3_HandleClientKeyExchange (ssl3con.c:10172)
==83837==    by 0x699AD58: ssl3_HandlePostHelloHandshakeMessage (ssl3con.c:11922)
==83837==    by 0x699AD58: ssl3_HandleHandshakeMessage (ssl3con.c:11817)
==83837==    by 0x699C7C2: ssl3_HandleHandshake (ssl3con.c:11991)
==83837==    by 0x699C7C2: ssl3_HandleNonApplicationData (ssl3con.c:12510)
==83837==    by 0x699CD5C: ssl3_HandleRecord (ssl3con.c:12791)
==83837==    by 0x699E31E: ssl3_GatherCompleteHandshake (ssl3gthr.c:512)
==83837==    by 0x699F388: ssl_GatherRecord1stHandshake (sslcon.c:74)
==83837==    by 0x69A5711: ssl_Do1stHandshake (sslsecur.c:41)
==83837==    by 0x69A63A4: ssl_SecureRecv (sslsecur.c:808)
==83837==    by 0x69AA460: ssl_Recv (sslsock.c:2943)
==83837==    by 0x122160: connection_read_ldap_data (connection.c:1097)
==83837==    by 0x122160: connection_read_operation (connection.c:1181)
==83837==    by 0x122996: connection_threadmain (connection.c:1628)
==83837==    by 0x7338BFA: _pt_root (ptthread.c:201)
==83837==    by 0x7978EA4: start_thread (pthread_create.c:307)
==83837==    by 0x83088CC: clone (clone.S:111)
==83837==


Regards,
Têko.

Comment 10 thierry bordaz 2019-11-08 09:55:59 UTC

Hi Teko,

I think there is a leak in acl with IP relate ACL while manipulating the client network address.
The fix looks simple.

==83837== 60,144 bytes in 537 blocks are definitely lost in loss record 1,253 of 1,265
==83837==    at 0x4C29E63: malloc (vg_replace_malloc.c:309)
==83837==    by 0x5094272: slapi_ch_malloc (ch_malloc.c:95)
==83837==    by 0x12111247: DS_LASIpGetter (acllas.c:265)
==83837==    by 0x123418BA: ACL_GetAttribute (method.cpp:130)
==83837==    by 0x123404BC: LASIpEval (lasip.cpp:496)
==83837==    by 0x123422B4: ACLEvalAce(NSErr_s*, ACLEvalHandle*, ACLExprHandle*, unsigned long*, PListStruct_s**, PListStruct_s*) (oneeval.cpp:225)
==83837==    by 0x12342D70: ACL_INTEvalTestRights(NSErr_s*, ACLEvalHandle*, char const**, char const**, char**, char**, char**, int*, unsigned long*) (oneeval.cpp:753)
==83837==    by 0x123432A5: ACL_EvalTestRights (oneeval.cpp:963)
==83837==    by 0x1210438E: acl__TestRights.constprop.4 (acl.c:3294)
==83837==    by 0x121077AF: acl_access_allowed (acl.c:591)
==83837==    by 0x1211A976: acl_access_allowed_main (aclplugin.c:371)
==83837==    by 0x50EA9AB: plugin_call_acl_plugin (plugin_acl.c:62)
==83837==    by 0x50B020C: test_filter_access (filterentry.c:956)
==83837==    by 0x50B0E34: slapi_vattr_filter_test_ext_internal (filterentry.c:867)
==83837==    by 0x50B1A25: slapi_vattr_filter_test_ext (filterentry.c:771)
==83837==    by 0x13429762: ldbm_back_next_search_entry_ext (ldbm_search.c:1650)
==83837==    by 0x50D8A95: iterate (opshared.c:1260)
==83837==    by 0x50D8A95: send_results_ext.constprop.4 (opshared.c:1613)
==83837==    by 0x50DABE0: op_shared_search (opshared.c:822)
==83837==    by 0x135B5D: do_search (search.c:351)
==83837==    by 0x123949: connection_dispatch_operation (connection.c:649)
==83837==    by 0x123949: connection_threadmain (connection.c:1788)
==83837==    by 0x7338BFA: _pt_root (ptthread.c:201)
==83837==    by 0x7978EA4: start_thread (pthread_create.c:307)
==83837==    by 0x83088CC: clone (clone.S:111)
==83837== 
==83837== 60,144 bytes in 537 blocks are definitely lost in loss record 1,254 of 1,265
==83837==    at 0x4C29E63: malloc (vg_replace_malloc.c:309)
==83837==    by 0x5094272: slapi_ch_malloc (ch_malloc.c:95)
==83837==    by 0x12111247: DS_LASIpGetter (acllas.c:265)
==83837==    by 0x123418BA: ACL_GetAttribute (method.cpp:130)
==83837==    by 0x123404BC: LASIpEval (lasip.cpp:496)
==83837==    by 0x123422B4: ACLEvalAce(NSErr_s*, ACLEvalHandle*, ACLExprHandle*, unsigned long*, PListStruct_s**, PListStruct_s*) (oneeval.cpp:225)
==83837==    by 0x12342D70: ACL_INTEvalTestRights(NSErr_s*, ACLEvalHandle*, char const**, char const**, char**, char**, char**, int*, unsigned long*) (oneeval.cpp:753)
==83837==    by 0x123432A5: ACL_EvalTestRights (oneeval.cpp:963)
==83837==    by 0x1210438E: acl__TestRights.constprop.4 (acl.c:3294)
==83837==    by 0x121077AF: acl_access_allowed (acl.c:591)
==83837==    by 0x1211A976: acl_access_allowed_main (aclplugin.c:371)
==83837==    by 0x50EA9AB: plugin_call_acl_plugin (plugin_acl.c:62)
==83837==    by 0x50B020C: test_filter_access (filterentry.c:956)
==83837==    by 0x50B0F01: slapi_vattr_filter_test_ext_internal (filterentry.c:855)
==83837==    by 0x50B1A25: slapi_vattr_filter_test_ext (filterentry.c:771)
==83837==    by 0x509FE56: dse_search (dse.c:1651)
==83837==    by 0x50DA461: op_shared_search (opshared.c:766)
==83837==    by 0x135B5D: do_search (search.c:351)
==83837==    by 0x123949: connection_dispatch_operation (connection.c:649)
==83837==    by 0x123949: connection_threadmain (connection.c:1788)
==83837==    by 0x7338BFA: _pt_root (ptthread.c:201)
==83837==    by 0x7978EA4: start_thread (pthread_create.c:307)
==83837==    by 0x83088CC: clone (clone.S:111)

Comment 11 Têko Mihinto 2019-11-08 10:04:32 UTC

Hi Thierry,

Thanks for the update!

Would you need the ACIs from the customer?

Regards,
Têko.

Comment 12 thierry bordaz 2019-11-08 13:29:09 UTC

Hi Teko,

Indeed it would help as I have a doubt about the fix.
So far I managed to reproduce it but the fix is not that immediate. Still investigating before opening a ticket.

Comment 13 mreynolds 2019-11-08 14:08:28 UTC

Teko, can you get an access log so we can see the exact details on the SSL connections being made to the server? Thanks!

Comment 14 mreynolds 2019-11-08 14:58:52 UTC

(In reply to mreynolds from comment #13)
> Teko, can you get an access log so we can see the exact details on the SSL
> connections being made to the server? Thanks!

I found the access log in the SOS report.  They are using StartTLS (TLS1.2) for all their secure client connections.

Comment 19 thierry bordaz 2019-11-12 13:40:44 UTC

Upstream ticket:
https://pagure.io/389-ds-base/issue/50709

Comment 33 thierry bordaz 2019-12-09 14:41:56 UTC

Fix pushed upstream -> POST

Comment 34 mreynolds 2020-01-15 21:43:12 UTC

There is a regression with the original fix that can cause a crash

Comment 38 thierry bordaz 2020-01-21 10:13:39 UTC

Regression was fixed (https://pagure.io/389-ds-base/pull-request/50833) and is pushed master, 1.4.2, 1.4.1, 1.4.0
In addition the BZ was also backported to 1.3.10

Comment 40 thierry bordaz 2020-01-24 13:31:10 UTC

The backport in 1.3.10 is incomplete and can trigger a hang --> ASSIGNED

Comment 42 thierry bordaz 2020-02-07 10:24:05 UTC

Second leak in ACI IP is track with a different BZ/ticket: https://bugzilla.redhat.com/show_bug.cgi?id=1796558 / https://pagure.io/389-ds-base/issue/50857

Comment 48 Viktor Ashirov 2020-03-17 12:52:25 UTC

============================================================== test session starts ===============================================================
platform linux -- Python 3.6.8, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.6.8', 'Platform': 'Linux-3.10.0-1127.el7.x86_64-x86_64-with-redhat-7.8-Maipo', 'Packages': {'pytest': '5.3.5', 'py': '1.8.1', 'pluggy': '0.13.1'}, 'Plugins': {'metadata': '1.8.0', 'html': '2.1.0'}}
389-ds-base: 1.3.10.2-1.1asan.el7
nss: 3.44.0-7.el7_7
nspr: 4.21.0-1.el7
openldap: 2.4.44-21.el7_6
cyrus-sasl: 2.1.26-23.el7
FIPS: disabled
rootdir: /mnt/tests/rhds/tests/upstream/ds/dirsrvtests, inifile: pytest.ini
plugins: metadata-1.8.0, html-2.1.0
collected 57 items

dirsrvtests/tests/suites/acl/keywords_part2_test.py::test_access_from_certain_network_only_ip PASSED                                       [  1%]
dirsrvtests/tests/suites/acl/keywords_part2_test.py::test_connectin_from_an_unauthorized_network PASSED                                    [  3%]
dirsrvtests/tests/suites/acl/keywords_part2_test.py::test_ip_keyword_test_noip_cannot PASSED                                               [  5%]
dirsrvtests/tests/suites/acl/keywords_part2_test.py::test_user_can_access_the_data_at_any_time PASSED                                      [  7%]
dirsrvtests/tests/suites/acl/keywords_part2_test.py::test_user_can_access_the_data_only_in_the_morning PASSED                              [  8%]
dirsrvtests/tests/suites/acl/keywords_part2_test.py::test_user_can_access_the_data_only_in_the_afternoon PASSED                            [ 10%]
dirsrvtests/tests/suites/acl/keywords_part2_test.py::test_timeofday_keyword PASSED                                                         [ 12%]
dirsrvtests/tests/suites/acl/keywords_part2_test.py::test_dayofweek_keyword_test_everyday_can_access PASSED                                [ 14%]
dirsrvtests/tests/suites/acl/keywords_part2_test.py::test_dayofweek_keyword_today_can_access PASSED                                        [ 15%]
dirsrvtests/tests/suites/acl/keywords_part2_test.py::test_user_cannot_access_the_data_at_all PASSED                                        [ 17%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_user_binds_with_a_password_and_can_access_the_data PASSED                              [ 19%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_user_binds_with_a_bad_password_and_cannot_access_the_data PASSED                       [ 21%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_anonymous_user_cannot_access_the_data PASSED                                           [ 22%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_authenticated_but_has_no_rigth_on_the_data PASSED                                      [ 24%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_the_bind_client_is_accessing_the_directory PASSED                                      [ 26%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_users_binds_with_a_password_and_can_access_the_data PASSED                             [ 28%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_user_binds_without_any_password_and_cannot_access_the_data PASSED                      [ 29%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_user_can_access_the_data_when_connecting_from_any_machine PASSED                       [ 31%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_user_can_access_the_data_when_connecting_from_internal_ds_network_only PASSED          [ 33%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_user_can_access_the_data_when_connecting_from_some_network_only PASSED                 [ 35%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_from_an_unauthorized_network PASSED                                                    [ 36%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_user_cannot_access_the_data_when_connecting_from_an_unauthorized_network_2 PASSED      [ 38%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_user_cannot_access_the_data_if_not_from_a_certain_domain PASSED                        [ 40%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_dnsalias_keyword_test_nodns_cannot PASSED                                              [ 42%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_user_can_access_from_ipv4_or_ipv6_address[127.0.0.1] PASSED                            [ 43%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_user_can_access_from_ipv4_or_ipv6_address[[::1]] PASSED                                [ 45%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_success[6-5] PASSED                                              [ 47%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_success[5-5] PASSED                                              [ 49%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_success[5-25] PASSED                                             [ 50%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_limits_fail[50-200-cn=config,cn=ldbm database,cn=plugins,cn=config-nsslapd-idlistscanlimit-100-UNWILLING_TO_PERFORM] PASSED [ 52%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_limits_fail[5-15-cn=config-nsslapd-timelimit-20-UNAVAILABLE_CRITICAL_EXTENSION] PASSED [ 54%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_limits_fail[21-50-cn=config-nsslapd-sizelimit-20-SIZELIMIT_EXCEEDED] PASSED [ 56%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_limits_fail[21-50-cn=config-nsslapd-pagedsizelimit-5-SIZELIMIT_EXCEEDED] PASSED [ 57%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_limits_fail[5-50-cn=config,cn=ldbm database,cn=plugins,cn=config-nsslapd-lookthroughlimit-20-ADMINLIMIT_EXCEEDED] PASSED [ 59%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_sort_success PASSED                                              [ 61%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_abandon PASSED                                                   [ 63%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_with_timelimit PASSED                                            [ 64%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_dns_ip_aci[dns = "localhost.localdomain"] PASSED                 [ 66%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_dns_ip_aci[ip = "127.0.0.1"] PASSED                              [ 68%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_multiple_paging PASSED                                           [ 70%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_invalid_cookie[1000] PASSED                                      [ 71%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_invalid_cookie[-1] PASSED                                        [ 73%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_abandon_with_zero_size PASSED                                    [ 75%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_pagedsizelimit_success PASSED                                    [ 77%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_nspagedsizelimit[5-15-PASS] PASSED                               [ 78%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_nspagedsizelimit[15-5-SIZELIMIT_EXCEEDED] PASSED                 [ 80%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_paged_limits[conf_attr_values0-ADMINLIMIT_EXCEEDED] PASSED       [ 82%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_paged_limits[conf_attr_values1-PASS] PASSED                      [ 84%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_paged_user_limits[conf_attr_values0-ADMINLIMIT_EXCEEDED] PASSED  [ 85%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_paged_user_limits[conf_attr_values1-PASS] PASSED                 [ 87%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_ger_basic PASSED                                                        [ 89%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_multi_suffix_search PASSED                                              [ 91%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_maxsimplepaged_per_conn_success[None] PASSED                            [ 92%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_maxsimplepaged_per_conn_success[-1] PASSED                              [ 94%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_maxsimplepaged_per_conn_success[1000] PASSED                            [ 96%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_maxsimplepaged_per_conn_failure[0] PASSED                               [ 98%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_maxsimplepaged_per_conn_failure[1] PASSED                               [100%]

================================================== 57 passed, 13 warnings in 185.37s (0:03:05) ===================================================


Automated tests pass, no memory leak (described in this bugzilla) found.
Marking as VERIFIED.

Comment 53 Marc Muehlfeld 2020-07-06 10:31:20 UTC

Thierry, can you please review the release note (see Doc Text field) for this bug fix? Thanks.

Comment 58 errata-xmlrpc 2020-09-29 19:46:50 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (389-ds-base bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3894