Bug 784119

+++ This bug was initially created as a clone of Bug #771493 +++

Description of problem:
When processing a great deal of individual changes triggered from the memberof plugin in realtime, schema compat is capable of a sort of storm situation which can consume all available memory to the point of exhaustion.

Version-Release number of selected component (if applicable):
slapi-nis-0.28-1.fc15.x86_64

How reproducible:
100%

Steps to Reproduce:
0. Start with a vanilla FreeIPA Server Install
1. For performance reasons, I find its best to disable schema compat for the import.
2. Import the attached ldif to populate the Directory or run the attached scripted commands to populate the Directory. (This will take 'a long time')
3. Delete the sudorule named admins: ipa sudorule-del admins
4. Run various ldap / freeipa commands: ipa host-find, ldapsearch etc, Notice the huge performance hit.
  
Actual results:
Substantial Memory usage and intense usage of slapd process causes serious performance hit. 

Expected results:
Release previously stored values when computing new schema compat modifications and don't consume all available memory.
Optimized performance during memberof updates.

Additional info:

--- Additional comment from jr.aquino on 2012-01-03 16:45:50 EST ---

Created attachment 550537 [details]
Ldif for complete test directory

--- Additional comment from jr.aquino on 2012-01-03 16:47:21 EST ---

Created attachment 550538 [details]
Scripts to populate Test FreeIPA Directory

As an alternative to importing the previously attached ldif, you should be able to use the included batch scripts to fully populate the test FreeIPA database.

--- Additional comment from nalin on 2012-01-19 14:44:10 EST ---

0.31 and later fix some memory leaks, one of which is quite severe.  Please retest with the candidate update, once it's built, and let me know if the server's memory usage continues to grow without apparent bound -- on my test systems, it's affected (sometimes contrary to my expectations) somewhat by the cache sizes set in the server's cn=config entry.

--- Additional comment from rmeggins on 2012-01-19 15:01:35 EST ---

(In reply to comment #3)
> 0.31 and later fix some memory leaks, one of which is quite severe.  Please
> retest with the candidate update, once it's built, and let me know if the
> server's memory usage continues to grow without apparent bound -- on my test
> systems, it's affected (sometimes contrary to my expectations) somewhat by the
> cache sizes set in the server's cn=config entry.

Yeah - there appears to be a memory leak if the cache is too small to hold all of the entries.  What happens then is that the entries are churned - repeatedly added and deleted - there must be either a memory leak during deletion, or just the accumulated memory fragmentation over time that causes the memory usage to go up.  This may be https://fedorahosted.org/389/ticket/51 - the solution for that is to make sure the nsslapd-cachememsize is large enough to contain all of the data.

--- Additional comment from updates on 2012-01-19 15:02:50 EST ---

slapi-nis-0.34-1.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/slapi-nis-0.34-1.fc15

--- Additional comment from updates on 2012-01-19 15:03:11 EST ---

slapi-nis-0.34-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/slapi-nis-0.34-1.fc16

--- Additional comment from updates on 2012-01-21 16:49:29 EST ---

Package slapi-nis-0.34-1.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing slapi-nis-0.34-1.fc16'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-0722/slapi-nis-0.34-1.fc16
then log in and leave karma (feedback).