RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1769418 - Several memory leaks reported by Valgrind for 389-ds 1.3.9.1-10.
Summary: Several memory leaks reported by Valgrind for 389-ds 1.3.9.1-10.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: 389-ds-base
Version: 7.7
Hardware: x86_64
OS: Linux
high
medium
Target Milestone: rc
: ---
Assignee: thierry bordaz
QA Contact: RHDS QE
Marc Muehlfeld
URL:
Whiteboard:
Depends On:
Blocks: 1788833 1790975 1803023
TreeView+ depends on / blocked
 
Reported: 2019-11-06 15:33 UTC by Têko Mihinto
Modified: 2023-09-21 09:11 UTC (History)
13 users (show)

Fixed In Version: 389-ds-base-1.3.10.2-1.el7
Doc Type: Bug Fix
Doc Text:
.Directory Server no longer leaks memory when using ACIs with an `ip` bind rule When a Directory Server Access Control Instruction (ACI) contains an `ip` bind rule, the server stores the value of the `ip` keyword as a reference while evaluating the ACI. Previously, when the evaluations were completed Directory Server did not free the `ip` value. As a consequence, the server leaked around 100 bytes of memory each time the server evaluated an ACI with an `ip` bind rule. With this update, Directory Server keeps track of the `ip` value in the per-connection structure and frees the structure when the connection is closed. As a consequence, Directory Server no longer leaks memory in the mentioned scenario.
Clone Of:
: 1790975 1803023 (view as bug list)
Environment:
Last Closed: 2020-09-29 19:46:50 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github 389ds 389-ds-base issues 3764 0 None closed Several memory leaks reported by Valgrind for 389-ds 1.3.9.1-10. 2021-01-27 12:13:39 UTC
Red Hat Product Errata RHBA-2020:3894 0 None None None 2020-09-29 19:47:59 UTC

Description Têko Mihinto 2019-11-06 15:33:17 UTC 
Description of problem:

A customer is experiencing a high memory usage with RHDS.
The memory consumption starts from a few GB and can go up to 100 GB after a week.

Version-Release number of selected component (if applicable):

$ cat etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)
$
$ grep 389-ds-base installed-rpms
389-ds-base-1.3.9.1-10.el7.x86_64                           Mon Aug 26 14:13:40 2019
389-ds-base-debuginfo-1.3.9.1-10.el7.x86_64                 Tue Nov  5 10:20:03 2019
389-ds-base-libs-1.3.9.1-10.el7.x86_64                      Mon Aug 26 14:13:26 2019
$

How reproducible:

Always at customer site.

Steps to Reproduce:

1. Restart the RHDS instance.
The sum of the configured caches is about 1.3 GB:
$ grep total errors | tail -1
[05/Nov/2019:11:32:52.600935718 +0000] - NOTICE - ldbm_back_start - total cache size: 1316466524 B;
$

2. The memory usage will grow significantly:
Eg:
$ head -1 ./ps
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
$
$ grep ns-slapd ./ps
dirsrv    3696 79.9 93.6 64044164 61602916 ?   Ssl  Oct15 18237:22 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-XXX -i /var/run/dirsrv/slapd-XXX.pid
$


Actual results:

High memory usage indicating memory leaks.

Expected results:

The memory usage should not go that high.
Need to fix the memory leaks.

Additional info:

From the Valgrind output:

* The leak summary:

==127975== LEAK SUMMARY:
==127975==    definitely lost: 29,132 bytes in 67 blocks
==127975==    indirectly lost: 136,872 bytes in 2,478 blocks
==127975==      possibly lost: 1,432 bytes in 5 blocks
==127975==    still reachable: 4,524,109 bytes in 26,967 blocks
==127975==                       of which reachable via heuristic:
==127975==                         stdstring          : 30 bytes in 1 blocks
==127975==         suppressed: 0 bytes in 0 blocks

* A couple of leak signatures:

==127975== 277,032 bytes in 51 blocks are still reachable in loss record 1,251 of 1,253
==127975==    at 0x4C2BF79: calloc (vg_replace_malloc.c:762)
==127975==    by 0x9D382A1: PORT_ZAlloc_Util (in /usr/lib64/libnssutil3.so)
==127975==    by 0x169F1BBE: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x169F1D0F: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x169D7E0D: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x169F0FFA: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x6C2EBBE: PK11_DeriveWithTemplate (in /usr/lib64/libnss3.so)
==127975==    by 0x6C2EC50: PK11_Derive (in /usr/lib64/libnss3.so)
==127975==    by 0x6992A82: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69B164A: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699AD58: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699C7C2: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699CD5C: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699E31E: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699F388: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69A5711: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69A63A4: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69AA460: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x122160: connection_read_ldap_data (connection.c:1097)
==127975==    by 0x122160: connection_read_operation (connection.c:1181)
==127975==    by 0x122996: connection_threadmain (connection.c:1628)
==127975==    by 0x7338BFA: ??? (in /usr/lib64/libnspr4.so)
==127975==    by 0x7978EA4: start_thread (pthread_create.c:307)
==127975==    by 0x83088CC: clone (clone.S:111)

...

==127975== 120,950 (80 direct, 120,870 indirect) bytes in 1 blocks are definitely lost in loss record 1,244 of 1,253
==127975==    at 0x4C2BF79: calloc (vg_replace_malloc.c:762)
==127975==    by 0x50944D3: slapi_ch_calloc (ch_malloc.c:175)
==127975==    by 0x1343D722: vlvSearch_new (vlv_srch.c:46)
==127975==    by 0x134398F7: vlv_init_search_entry (vlv.c:320)
==127975==    by 0x509E308: dse_call_callback.isra.1 (dse.c:2553)
==127975==    by 0x509FFBC: do_dse_search (dse.c:1567)
==127975==    by 0x509FFBC: dse_search (dse.c:1669)
==127975==    by 0x50DA461: op_shared_search (opshared.c:766)
==127975==    by 0x50EC036: search_internal_callback_pb (plugin_internal_op.c:727)
==127975==    by 0x50EC287: search_internal_pb (plugin_internal_op.c:592)
==127975==    by 0x50EC49C: slapi_search_internal (plugin_internal_op.c:439)
==127975==    by 0x1343A9D1: vlv_init (vlv.c:441)
==127975==    by 0x13404BF2: ldbm_instance_startall (instance.c:312)
==127975==    by 0x13438D99: ldbm_back_start (start.c:383)
==127975==    by 0x50E7096: plugin_call_func (plugin.c:2028)
==127975==    by 0x50E770A: plugin_call_one (plugin.c:1978)
==127975==    by 0x50E770A: plugin_dependency_startall (plugin.c:1737)
==127975==    by 0x50E770A: plugin_startall (plugin.c:1950)
==127975==    by 0x119A17: main (main.c:1141)
Comment 2 mreynolds 2019-11-06 16:03:16 UTC 
Initial observations:


==127975== 277,032 bytes in 51 blocks are still reachable in loss record 1,251 of 1,253
==127975==    at 0x4C2BF79: calloc (vg_replace_malloc.c:762)
==127975==    by 0x9D382A1: PORT_ZAlloc_Util (in /usr/lib64/libnssutil3.so)
==127975==    by 0x169F1BBE: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x169F1D0F: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x169D7E0D: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x169F0FFA: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x6C2EBBE: PK11_DeriveWithTemplate (in /usr/lib64/libnss3.so)
==127975==    by 0x6C2EC50: PK11_Derive (in /usr/lib64/libnss3.so)
==127975==    by 0x6992A82: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69B164A: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699AD58: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699C7C2: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699CD5C: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699E31E: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699F388: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69A5711: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69A63A4: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69AA460: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x122160: connection_read_ldap_data (connection.c:1097)
==127975==    by 0x122160: connection_read_operation (connection.c:1181)
==127975==    by 0x122996: connection_threadmain (connection.c:1628)
==127975==    by 0x7338BFA: ??? (in /usr/lib64/libnspr4.so)
==127975==    by 0x7978EA4: start_thread (pthread_create.c:307)
==127975==    by 0x83088CC: clone (clone.S:111)


If this is a recurring leak it could explain some of this.  But it's a leak in NSS, not DS itself.  Did you try reproducing this with ldclt over ldaps?  If not, could you try it?



==127975== 120,950 (80 direct, 120,870 indirect) bytes in 1 blocks are definitely lost in loss record 1,244 of 1,253
==127975==    at 0x4C2BF79: calloc (vg_replace_malloc.c:762)
==127975==    by 0x50944D3: slapi_ch_calloc (ch_malloc.c:175)
==127975==    by 0x1343D722: vlvSearch_new (vlv_srch.c:46)
==127975==    by 0x134398F7: vlv_init_search_entry (vlv.c:320)
==127975==    by 0x509E308: dse_call_callback.isra.1 (dse.c:2553)
==127975==    by 0x509FFBC: do_dse_search (dse.c:1567)
==127975==    by 0x509FFBC: dse_search (dse.c:1669)
==127975==    by 0x50DA461: op_shared_search (opshared.c:766)
==127975==    by 0x50EC036: search_internal_callback_pb (plugin_internal_op.c:727)
==127975==    by 0x50EC287: search_internal_pb (plugin_internal_op.c:592)
==127975==    by 0x50EC49C: slapi_search_internal (plugin_internal_op.c:439)
==127975==    by 0x1343A9D1: vlv_init (vlv.c:441)
==127975==    by 0x13404BF2: ldbm_instance_startall (instance.c:312)
==127975==    by 0x13438D99: ldbm_back_start (start.c:383)
==127975==    by 0x50E7096: plugin_call_func (plugin.c:2028)
==127975==    by 0x50E770A: plugin_call_one (plugin.c:1978)
==127975==    by 0x50E770A: plugin_dependency_startall (plugin.c:1737)
==127975==    by 0x50E770A: plugin_startall (plugin.c:1950)
==127975==    by 0x119A17: main (main.c:1141)

This is a red herring.  This is a one time startup leak, they are a lot of these "leaks" unfortunately.  Any time you see "plugin_startall" in the stack it should be ignored.


I'm more worried that there is a leak in some hash or array that incorrectly keeps growing, but then it actually gets freed but only at server shutdown - so valgrind or asan would not detect it.  This would be more difficult to investigate and we would really need a reproducer.
Comment 4 mreynolds 2019-11-06 20:15:29 UTC 
I looked over the valgrind output, and there are lot of stacks with very large leaks like this (which you already mentioned):

==127975== 277,032 bytes in 51 blocks are still reachable in loss record 1,251 of 1,253
==127975==    at 0x4C2BF79: calloc (vg_replace_malloc.c:762)
==127975==    by 0x9D382A1: PORT_ZAlloc_Util (in /usr/lib64/libnssutil3.so)
==127975==    by 0x169F1BBE: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x169F1D0F: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x169D7E0D: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x169F0FFA: ??? (in /usr/lib64/libsoftokn3.so)
==127975==    by 0x6C2EBBE: PK11_DeriveWithTemplate (in /usr/lib64/libnss3.so)
==127975==    by 0x6C2EC50: PK11_Derive (in /usr/lib64/libnss3.so)
==127975==    by 0x6992A82: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69B164A: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699AD58: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699C7C2: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699CD5C: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699E31E: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x699F388: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69A5711: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69A63A4: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x69AA460: ??? (in /usr/lib64/libssl3.so)
==127975==    by 0x122160: connection_read_ldap_data (connection.c:1097)
==127975==    by 0x122160: connection_read_operation (connection.c:1181)
==127975==    by 0x122996: connection_threadmain (connection.c:1628)
==127975==    by 0x7338BFA: ??? (in /usr/lib64/libnspr4.so)
==127975==    by 0x7978EA4: start_thread (pthread_create.c:307)
==127975==    by 0x83088CC: clone (clone.S:111)

Everything else in the valgrind report looks okay.

What version of NSS is on the customers system?
Comment 5 Têko Mihinto 2019-11-06 21:12:33 UTC 
Hi Mark,

$ grep ^nss installed-rpms 
nss-3.44.0-4.el7.i686                                       Mon Aug 26 14:14:27 2019
nss-3.44.0-4.el7.x86_64                                     Mon Aug 26 14:13:04 2019
nss-pem-1.0.3-7.el7.i686                                    Mon Aug 26 14:14:27 2019
nss-pem-1.0.3-7.el7.x86_64                                  Mon Aug 26 14:13:04 2019
nss-softokn-3.44.0-5.el7.i686                               Mon Aug 26 14:14:27 2019
nss-softokn-3.44.0-5.el7.x86_64                             Mon Aug 26 14:13:04 2019
nss-softokn-freebl-3.44.0-5.el7.i686                        Mon Aug 26 14:14:24 2019
nss-softokn-freebl-3.44.0-5.el7.x86_64                      Mon Aug 26 14:12:56 2019
nss-sysinit-3.44.0-4.el7.x86_64                             Mon Aug 26 14:13:04 2019
nss-tools-3.44.0-4.el7.x86_64                               Mon Aug 26 14:13:11 2019
nss-util-3.44.0-3.el7.i686                                  Mon Aug 26 14:14:25 2019
nss-util-3.44.0-3.el7.x86_64                                Mon Aug 26 14:12:57 2019
$

Please let me know if you need further details.

Regards,
Têko.
Comment 6 mreynolds 2019-11-07 14:01:28 UTC 
What would really be useful now is to get a clean valgrind report.  So the customer needs to install the devel and debuginfo packages for all the nss packages.  Then when they run the valgrind test again we should hopefully see the full stack and symbols.  Then we can share that with the NSS team and move forward on the investigation.
Comment 8 Têko Mihinto 2019-11-08 09:00:28 UTC 
Hi Mark,

I have attached the Valgrind output after the NSS devel and debug packages have been installed.

We have the NSS calls in the new output.
Eg:

==83837== 298,760 bytes in 55 blocks are still reachable in loss record 1,263 of 1,265
==83837==    at 0x4C2BF79: calloc (vg_replace_malloc.c:762)
==83837==    by 0x9D382A1: PORT_ZAlloc_Util (secport.c:116)
==83837==    by 0x169F1BBE: sftk_GetObjectFromList (pkcs11u.c:921)
==83837==    by 0x169F1D0F: sftk_NewObject (pkcs11u.c:1028)
==83837==    by 0x169D7E0D: sftk_buildSSLKey (pkcs11c.c:6151)
==83837==    by 0x169F0FFA: NSC_DeriveKey (pkcs11c.c:7154)
==83837==    by 0x6C2EBBE: PK11_DeriveWithTemplate (pk11skey.c:1589)
==83837==    by 0x6C2EC50: PK11_Derive (pk11skey.c:1445)
==83837==    by 0x6992A82: ssl3_DeriveConnectionKeys (ssl3con.c:3575)
==83837==    by 0x6992A82: ssl3_InitPendingCipherSpecs (ssl3con.c:1793)
==83837==    by 0x69B164A: ssl3_HandleECDHClientKeyExchange (ssl3ecc.c:316)
==83837==    by 0x699AD58: ssl3_HandleClientKeyExchange (ssl3con.c:10172)
==83837==    by 0x699AD58: ssl3_HandlePostHelloHandshakeMessage (ssl3con.c:11922)
==83837==    by 0x699AD58: ssl3_HandleHandshakeMessage (ssl3con.c:11817)
==83837==    by 0x699C7C2: ssl3_HandleHandshake (ssl3con.c:11991)
==83837==    by 0x699C7C2: ssl3_HandleNonApplicationData (ssl3con.c:12510)
==83837==    by 0x699CD5C: ssl3_HandleRecord (ssl3con.c:12791)
==83837==    by 0x699E31E: ssl3_GatherCompleteHandshake (ssl3gthr.c:512)
==83837==    by 0x699F388: ssl_GatherRecord1stHandshake (sslcon.c:74)
==83837==    by 0x69A5711: ssl_Do1stHandshake (sslsecur.c:41)
==83837==    by 0x69A63A4: ssl_SecureRecv (sslsecur.c:808)
==83837==    by 0x69AA460: ssl_Recv (sslsock.c:2943)
==83837==    by 0x122160: connection_read_ldap_data (connection.c:1097)
==83837==    by 0x122160: connection_read_operation (connection.c:1181)
==83837==    by 0x122996: connection_threadmain (connection.c:1628)
==83837==    by 0x7338BFA: _pt_root (ptthread.c:201)
==83837==    by 0x7978EA4: start_thread (pthread_create.c:307)
==83837==    by 0x83088CC: clone (clone.S:111)
==83837==


Regards,
Têko.
Comment 10 thierry bordaz 2019-11-08 09:55:59 UTC 
Hi Teko,

I think there is a leak in acl with IP relate ACL while manipulating the client network address.
The fix looks simple.

==83837== 60,144 bytes in 537 blocks are definitely lost in loss record 1,253 of 1,265
==83837==    at 0x4C29E63: malloc (vg_replace_malloc.c:309)
==83837==    by 0x5094272: slapi_ch_malloc (ch_malloc.c:95)
==83837==    by 0x12111247: DS_LASIpGetter (acllas.c:265)
==83837==    by 0x123418BA: ACL_GetAttribute (method.cpp:130)
==83837==    by 0x123404BC: LASIpEval (lasip.cpp:496)
==83837==    by 0x123422B4: ACLEvalAce(NSErr_s*, ACLEvalHandle*, ACLExprHandle*, unsigned long*, PListStruct_s**, PListStruct_s*) (oneeval.cpp:225)
==83837==    by 0x12342D70: ACL_INTEvalTestRights(NSErr_s*, ACLEvalHandle*, char const**, char const**, char**, char**, char**, int*, unsigned long*) (oneeval.cpp:753)
==83837==    by 0x123432A5: ACL_EvalTestRights (oneeval.cpp:963)
==83837==    by 0x1210438E: acl__TestRights.constprop.4 (acl.c:3294)
==83837==    by 0x121077AF: acl_access_allowed (acl.c:591)
==83837==    by 0x1211A976: acl_access_allowed_main (aclplugin.c:371)
==83837==    by 0x50EA9AB: plugin_call_acl_plugin (plugin_acl.c:62)
==83837==    by 0x50B020C: test_filter_access (filterentry.c:956)
==83837==    by 0x50B0E34: slapi_vattr_filter_test_ext_internal (filterentry.c:867)
==83837==    by 0x50B1A25: slapi_vattr_filter_test_ext (filterentry.c:771)
==83837==    by 0x13429762: ldbm_back_next_search_entry_ext (ldbm_search.c:1650)
==83837==    by 0x50D8A95: iterate (opshared.c:1260)
==83837==    by 0x50D8A95: send_results_ext.constprop.4 (opshared.c:1613)
==83837==    by 0x50DABE0: op_shared_search (opshared.c:822)
==83837==    by 0x135B5D: do_search (search.c:351)
==83837==    by 0x123949: connection_dispatch_operation (connection.c:649)
==83837==    by 0x123949: connection_threadmain (connection.c:1788)
==83837==    by 0x7338BFA: _pt_root (ptthread.c:201)
==83837==    by 0x7978EA4: start_thread (pthread_create.c:307)
==83837==    by 0x83088CC: clone (clone.S:111)
==83837== 
==83837== 60,144 bytes in 537 blocks are definitely lost in loss record 1,254 of 1,265
==83837==    at 0x4C29E63: malloc (vg_replace_malloc.c:309)
==83837==    by 0x5094272: slapi_ch_malloc (ch_malloc.c:95)
==83837==    by 0x12111247: DS_LASIpGetter (acllas.c:265)
==83837==    by 0x123418BA: ACL_GetAttribute (method.cpp:130)
==83837==    by 0x123404BC: LASIpEval (lasip.cpp:496)
==83837==    by 0x123422B4: ACLEvalAce(NSErr_s*, ACLEvalHandle*, ACLExprHandle*, unsigned long*, PListStruct_s**, PListStruct_s*) (oneeval.cpp:225)
==83837==    by 0x12342D70: ACL_INTEvalTestRights(NSErr_s*, ACLEvalHandle*, char const**, char const**, char**, char**, char**, int*, unsigned long*) (oneeval.cpp:753)
==83837==    by 0x123432A5: ACL_EvalTestRights (oneeval.cpp:963)
==83837==    by 0x1210438E: acl__TestRights.constprop.4 (acl.c:3294)
==83837==    by 0x121077AF: acl_access_allowed (acl.c:591)
==83837==    by 0x1211A976: acl_access_allowed_main (aclplugin.c:371)
==83837==    by 0x50EA9AB: plugin_call_acl_plugin (plugin_acl.c:62)
==83837==    by 0x50B020C: test_filter_access (filterentry.c:956)
==83837==    by 0x50B0F01: slapi_vattr_filter_test_ext_internal (filterentry.c:855)
==83837==    by 0x50B1A25: slapi_vattr_filter_test_ext (filterentry.c:771)
==83837==    by 0x509FE56: dse_search (dse.c:1651)
==83837==    by 0x50DA461: op_shared_search (opshared.c:766)
==83837==    by 0x135B5D: do_search (search.c:351)
==83837==    by 0x123949: connection_dispatch_operation (connection.c:649)
==83837==    by 0x123949: connection_threadmain (connection.c:1788)
==83837==    by 0x7338BFA: _pt_root (ptthread.c:201)
==83837==    by 0x7978EA4: start_thread (pthread_create.c:307)
==83837==    by 0x83088CC: clone (clone.S:111)
Comment 11 Têko Mihinto 2019-11-08 10:04:32 UTC 
Hi Thierry,

Thanks for the update!

Would you need the ACIs from the customer?

Regards,
Têko.
Comment 12 thierry bordaz 2019-11-08 13:29:09 UTC 
Hi Teko,

Indeed it would help as I have a doubt about the fix.
So far I managed to reproduce it but the fix is not that immediate. Still investigating before opening a ticket.
Comment 13 mreynolds 2019-11-08 14:08:28 UTC 
Teko, can you get an access log so we can see the exact details on the SSL connections being made to the server? Thanks!
Comment 14 mreynolds 2019-11-08 14:58:52 UTC 
(In reply to mreynolds from comment #13)
> Teko, can you get an access log so we can see the exact details on the SSL
> connections being made to the server? Thanks!

I found the access log in the SOS report.  They are using StartTLS (TLS1.2) for all their secure client connections.
Comment 19 thierry bordaz 2019-11-12 13:40:44 UTC 
Upstream ticket:
https://pagure.io/389-ds-base/issue/50709
Comment 33 thierry bordaz 2019-12-09 14:41:56 UTC 
Fix pushed upstream -> POST
Comment 34 mreynolds 2020-01-15 21:43:12 UTC 
There is a regression with the original fix that can cause a crash
Comment 38 thierry bordaz 2020-01-21 10:13:39 UTC 
Regression was fixed (https://pagure.io/389-ds-base/pull-request/50833) and is pushed master, 1.4.2, 1.4.1, 1.4.0
In addition the BZ was also backported to 1.3.10
Comment 40 thierry bordaz 2020-01-24 13:31:10 UTC 
The backport in 1.3.10 is incomplete and can trigger a hang --> ASSIGNED
Comment 42 thierry bordaz 2020-02-07 10:24:05 UTC 
Second leak in ACI IP is track with a different BZ/ticket: https://bugzilla.redhat.com/show_bug.cgi?id=1796558 / https://pagure.io/389-ds-base/issue/50857
Comment 48 Viktor Ashirov 2020-03-17 12:52:25 UTC 
============================================================== test session starts ===============================================================
platform linux -- Python 3.6.8, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.6.8', 'Platform': 'Linux-3.10.0-1127.el7.x86_64-x86_64-with-redhat-7.8-Maipo', 'Packages': {'pytest': '5.3.5', 'py': '1.8.1', 'pluggy': '0.13.1'}, 'Plugins': {'metadata': '1.8.0', 'html': '2.1.0'}}
389-ds-base: 1.3.10.2-1.1asan.el7
nss: 3.44.0-7.el7_7
nspr: 4.21.0-1.el7
openldap: 2.4.44-21.el7_6
cyrus-sasl: 2.1.26-23.el7
FIPS: disabled
rootdir: /mnt/tests/rhds/tests/upstream/ds/dirsrvtests, inifile: pytest.ini
plugins: metadata-1.8.0, html-2.1.0
collected 57 items

dirsrvtests/tests/suites/acl/keywords_part2_test.py::test_access_from_certain_network_only_ip PASSED                                       [  1%]
dirsrvtests/tests/suites/acl/keywords_part2_test.py::test_connectin_from_an_unauthorized_network PASSED                                    [  3%]
dirsrvtests/tests/suites/acl/keywords_part2_test.py::test_ip_keyword_test_noip_cannot PASSED                                               [  5%]
dirsrvtests/tests/suites/acl/keywords_part2_test.py::test_user_can_access_the_data_at_any_time PASSED                                      [  7%]
dirsrvtests/tests/suites/acl/keywords_part2_test.py::test_user_can_access_the_data_only_in_the_morning PASSED                              [  8%]
dirsrvtests/tests/suites/acl/keywords_part2_test.py::test_user_can_access_the_data_only_in_the_afternoon PASSED                            [ 10%]
dirsrvtests/tests/suites/acl/keywords_part2_test.py::test_timeofday_keyword PASSED                                                         [ 12%]
dirsrvtests/tests/suites/acl/keywords_part2_test.py::test_dayofweek_keyword_test_everyday_can_access PASSED                                [ 14%]
dirsrvtests/tests/suites/acl/keywords_part2_test.py::test_dayofweek_keyword_today_can_access PASSED                                        [ 15%]
dirsrvtests/tests/suites/acl/keywords_part2_test.py::test_user_cannot_access_the_data_at_all PASSED                                        [ 17%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_user_binds_with_a_password_and_can_access_the_data PASSED                              [ 19%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_user_binds_with_a_bad_password_and_cannot_access_the_data PASSED                       [ 21%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_anonymous_user_cannot_access_the_data PASSED                                           [ 22%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_authenticated_but_has_no_rigth_on_the_data PASSED                                      [ 24%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_the_bind_client_is_accessing_the_directory PASSED                                      [ 26%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_users_binds_with_a_password_and_can_access_the_data PASSED                             [ 28%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_user_binds_without_any_password_and_cannot_access_the_data PASSED                      [ 29%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_user_can_access_the_data_when_connecting_from_any_machine PASSED                       [ 31%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_user_can_access_the_data_when_connecting_from_internal_ds_network_only PASSED          [ 33%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_user_can_access_the_data_when_connecting_from_some_network_only PASSED                 [ 35%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_from_an_unauthorized_network PASSED                                                    [ 36%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_user_cannot_access_the_data_when_connecting_from_an_unauthorized_network_2 PASSED      [ 38%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_user_cannot_access_the_data_if_not_from_a_certain_domain PASSED                        [ 40%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_dnsalias_keyword_test_nodns_cannot PASSED                                              [ 42%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_user_can_access_from_ipv4_or_ipv6_address[127.0.0.1] PASSED                            [ 43%]
dirsrvtests/tests/suites/acl/keywords_test.py::test_user_can_access_from_ipv4_or_ipv6_address[[::1]] PASSED                                [ 45%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_success[6-5] PASSED                                              [ 47%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_success[5-5] PASSED                                              [ 49%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_success[5-25] PASSED                                             [ 50%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_limits_fail[50-200-cn=config,cn=ldbm database,cn=plugins,cn=config-nsslapd-idlistscanlimit-100-UNWILLING_TO_PERFORM] PASSED [ 52%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_limits_fail[5-15-cn=config-nsslapd-timelimit-20-UNAVAILABLE_CRITICAL_EXTENSION] PASSED [ 54%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_limits_fail[21-50-cn=config-nsslapd-sizelimit-20-SIZELIMIT_EXCEEDED] PASSED [ 56%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_limits_fail[21-50-cn=config-nsslapd-pagedsizelimit-5-SIZELIMIT_EXCEEDED] PASSED [ 57%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_limits_fail[5-50-cn=config,cn=ldbm database,cn=plugins,cn=config-nsslapd-lookthroughlimit-20-ADMINLIMIT_EXCEEDED] PASSED [ 59%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_sort_success PASSED                                              [ 61%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_abandon PASSED                                                   [ 63%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_with_timelimit PASSED                                            [ 64%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_dns_ip_aci[dns = "localhost.localdomain"] PASSED                 [ 66%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_dns_ip_aci[ip = "127.0.0.1"] PASSED                              [ 68%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_multiple_paging PASSED                                           [ 70%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_invalid_cookie[1000] PASSED                                      [ 71%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_invalid_cookie[-1] PASSED                                        [ 73%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_abandon_with_zero_size PASSED                                    [ 75%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_pagedsizelimit_success PASSED                                    [ 77%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_nspagedsizelimit[5-15-PASS] PASSED                               [ 78%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_nspagedsizelimit[15-5-SIZELIMIT_EXCEEDED] PASSED                 [ 80%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_paged_limits[conf_attr_values0-ADMINLIMIT_EXCEEDED] PASSED       [ 82%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_paged_limits[conf_attr_values1-PASS] PASSED                      [ 84%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_paged_user_limits[conf_attr_values0-ADMINLIMIT_EXCEEDED] PASSED  [ 85%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_search_paged_user_limits[conf_attr_values1-PASS] PASSED                 [ 87%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_ger_basic PASSED                                                        [ 89%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_multi_suffix_search PASSED                                              [ 91%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_maxsimplepaged_per_conn_success[None] PASSED                            [ 92%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_maxsimplepaged_per_conn_success[-1] PASSED                              [ 94%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_maxsimplepaged_per_conn_success[1000] PASSED                            [ 96%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_maxsimplepaged_per_conn_failure[0] PASSED                               [ 98%]
dirsrvtests/tests/suites/paged_results/paged_results_test.py::test_maxsimplepaged_per_conn_failure[1] PASSED                               [100%]

================================================== 57 passed, 13 warnings in 185.37s (0:03:05) ===================================================


Automated tests pass, no memory leak (described in this bugzilla) found.
Marking as VERIFIED.
Comment 53 Marc Muehlfeld 2020-07-06 10:31:20 UTC 
Thierry, can you please review the release note (see Doc Text field) for this bug fix? Thanks.
Comment 58 errata-xmlrpc 2020-09-29 19:46:50 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (389-ds-base bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3894
Note You need to log in before you can comment on or make changes to this bug.