Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1095847

Summary:	CoS cache re-scanning severely impacts performance
Product:	Red Hat Enterprise Linux 6	Reporter:	mreynolds
Component:	389-ds-base	Assignee:	mreynolds
Status:	CLOSED ERRATA	QA Contact:	Sankar Ramalingam <sramling>
Severity:	high	Docs Contact:
Priority:	high
Version:	6.6	CC:	amsharma, jgalipea, mreynolds, nhosoi, nkinder, rmeggins
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	389-ds-base-1.2.11.15-34.el6	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-10-14 07:55:02 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description mreynolds 2014-05-08 16:11:50 UTC

This bug is created as a clone of upstream ticket:
https://fedorahosted.org/389/ticket/47649

on 1.2.11
https://fedorahosted.org/389/ticket/47762

A newly-installed directory server with 15 classic CoS rules and >200,000 entries suffers poor write performance.

After a small number (typically 6) of consecutive write requests (basic attribute changes to a single entry using ldapmodify) the ''ns-slapd'' process hits >100% CPU and stays there for ''at least'' 10 seconds per update, and blocks the client process attempting the update.

Enabling the Plug-ins logging shows that the CoS cache is being rebuilt after every attribute change - even though the attribute which is being changed does not have CoS rules.

----

The RHEL6.5 server is a VMware ESXi VM with 8GB RAM and 2x CPUs, running the latest EPEL package for RHEL6 (v1.2.11.15-32).

All attributes referred to by the CoS rules (mostly custom attributes) have been indexed. Replication has __not__ been configured so far.

This was initially described in [[https://lists.fedoraproject.org/pipermail/389-users/2014-March/016910.html|this 389-users thread]] which [[https://lists.fedoraproject.org/pipermail/389-users/2014-April/016922.html|continued here]].

I will continue to add more details...

Thanks for looking into this,
Steve

Comment 1 mreynolds 2014-05-09 21:20:34 UTC

Fixed upstream.

Comment 2 Sankar Ramalingam 2014-05-12 10:12:16 UTC

Requesting you to provide steps to reproduce the problem.

Comment 3 mreynolds 2014-05-12 14:07:04 UTC

(In reply to Sankar Ramalingam from comment #2)
> Requesting you to provide steps to reproduce the problem.

I was only able to reproduce this with the customer's private data.  I will start working on a new "generic" testcase to reproduce the issue.

Comment 6 mreynolds 2014-05-14 16:01:00 UTC

Verification Steps

[1]  Create a single instance of 389, using "dc=example,dc=com"
[2]  Import the cos-db.ldif file
[3]  restart the server - just to be sure the cos cache gets created.
[4]  Run this ldapmodify:

    ldapmodify -D "cn=directory manager" -w password -f mod-entry.ldif

If the ldapmodify completes in under 5 seconds, the bug is verified.  Previously it would take 5 - 10 minutes to complete.

Comment 10 Amita Sharma 2014-07-17 07:19:08 UTC

HI Mark,

Modify operation does not take much time ::
[root@dhcp201-155 ~]# tail -f /var/log/dirsrv/slapd-dhcp201-155/errors
[17/Jul/2014:02:01:56 -0400] - import userRoot: Indexing complete.  Post-processing...
[17/Jul/2014:02:01:56 -0400] - import userRoot: Generating numSubordinates complete.
[17/Jul/2014:02:01:58 -0400] - import userRoot: Flushing caches...
[17/Jul/2014:02:01:58 -0400] - import userRoot: Closing files...
[17/Jul/2014:02:02:03 -0400] - All database threads now stopped
[17/Jul/2014:02:02:03 -0400] - import userRoot: Import complete.  Processed 216303 entries in 98 seconds. (2207.17 entries/sec)
[17/Jul/2014:02:46:03 -0400] - 389-Directory/1.2.11.15 B2014.183.1827 starting up
[17/Jul/2014:02:46:03 -0400] - WARNING: userRoot: entry cache size 10485760B is less than db size 185778176B; We recommend to increase the entry cache size nsslapd-cachememsize.
[17/Jul/2014:02:46:04 -0400] - I'm resizing my cache now...cache was 417939456 and is now 8000000
[17/Jul/2014:02:46:59 -0400] - slapd started.  Listening on All Interfaces port 389 for LDAP requests

[root@dhcp201-155 ~]# service dirsrv start
Starting dirsrv: 
    dhcp201-155...                                         [  OK  ]

[root@dhcp201-155 ~]# ldapmodify -x -h localhost -p 389 -D "cn=Directory Manager" -w Secret123 -f /export/mod-entry.ldif 
modifying entry "uid=st8, ou=people, dc=example,dc=com"

modifying entry "uid=st8, ou=people, dc=example,dc=com"

modifying entry "uid=st8, ou=people, dc=example,dc=com"

modifying entry "uid=st8, ou=people, dc=example,dc=com"

modifying entry "uid=st8, ou=people, dc=example,dc=com"

modifying entry "uid=st8, ou=people, dc=example,dc=com"

modifying entry "uid=st8, ou=people, dc=example,dc=com"

modifying entry "uid=st8, ou=people, dc=example,dc=com"

modifying entry "uid=st8, ou=people, dc=example,dc=com"

modifying entry "uid=st8, ou=people, dc=example,dc=com"

modifying entry "uid=st8, ou=people, dc=example,dc=com"

modifying entry "uid=st8, ou=people, dc=example,dc=com"

But restarting dirsrv taking more time than usual approx. 1 min..
Is it due to large db?
Is it OK and as expected.
Please confirm.

[root@dhcp201-155 ~]# date
Thu Jul 17 02:51:20 EDT 2014
[root@dhcp201-155 ~]# service dirsrv restart
Shutting down dirsrv: 
    dhcp201-155...                                         [  OK  ]
Starting dirsrv: 
    dhcp201-155...                                         [  OK  ]
[root@dhcp201-155 ~]# date
Thu Jul 17 02:52:21 EDT 2014

Comment 11 mreynolds 2014-07-18 19:49:59 UTC

Yes, this is expected.  At start up the cos plugin builds its cache (which is quite large in this particular testcase).  This looks verified to me.

Comment 12 Amita Sharma 2014-07-21 11:26:49 UTC

Marking bug as VERIFIED.

Comment 13 errata-xmlrpc 2014-10-14 07:55:02 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1385.html