Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
Prasad has done a great analysis of the issue I am describing here.
Customer has a group of 1400 members more or less but plenty of nested groups. This group is member of 13 other groups.
Adding a new member to this group takes 53 seconds. We have asked for pstacks and we see this:
=====================================
Thread 21 (Thread 0x7fbad9ffb700 (LWP 23197)):
#0 0x00007fbb16f081a7 in slapi_log_access () from /usr/lib64/dirsrv/libslapd.so.0
#1 0x00007fbb16f16bea in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0
#2 0x00007fbb16f278de in search_internal_callback_pb () from /usr/lib64/dirsrv/libslapd.so.0
#3 0x00007fbb07dd27bd in wrap_search_internal_get_entry () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#4 0x00007fbb07dc7287 in format_maybe_add_sdn_list () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#5 0x00007fbb07dc9875 in format_deref_rx.isra.6 () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#6 0x00007fbb07dc9e79 in format_deref_r () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#7 0x00007fbb07dc7e19 in format_expand () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#8 0x00007fbb07dc8f66 in format_format () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#9 0x00007fbb07dcab79 in format_get_data_set () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#10 0x00007fbb07dccd43 in format_collect () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#11 0x00007fbb07dc7e19 in format_expand () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#12 0x00007fbb07dcd3b1 in format_ifeq () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#13 0x00007fbb07dc7e19 in format_expand () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#14 0x00007fbb07dc8f66 in format_format () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#15 0x00007fbb07dcab79 in format_get_data_set () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#16 0x00007fbb07dcc39b in format_link () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#17 0x00007fbb07dc7e19 in format_expand () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#18 0x00007fbb07dc8f66 in format_format () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#19 0x00007fbb07dcab79 in format_get_data_set () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#20 0x00007fbb07dbd459 in backend_set_entry_from () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#21 0x00007fbb07dc0b16 in backend_shr_set_entry_cb () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#22 0x00007fbb16f3858d in send_ldap_search_entry_ext () from /usr/lib64/dirsrv/libslapd.so.0
#23 0x00007fbb16f38dcc in send_ldap_search_entry () from /usr/lib64/dirsrv/libslapd.so.0
#24 0x00007fbb16f15ad3 in iterate.isra.0.constprop.3 () from /usr/lib64/dirsrv/libslapd.so.0
#25 0x00007fbb16f15c7a in send_results_ext.constprop.2 () from /usr/lib64/dirsrv/libslapd.so.0
#26 0x00007fbb16f1765e in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0
#27 0x00007fbb16f278de in search_internal_callback_pb () from /usr/lib64/dirsrv/libslapd.so.0
#28 0x00007fbb07dc29e6 in backend_shr_update_references_cb () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#29 0x00007fbb07dd069f in map_data_foreach_map () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#30 0x00007fbb07dc0dfb in backend_shr_update_references () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#31 0x00007fbb07dc43f1 in backend_shr_modify_cb.part.20 () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#32 0x00007fbb07dc47d1 in backend_shr_betxn_post_modify_cb () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#33 0x00007fbb16f227d8 in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0
#34 0x00007fbb16f22aa3 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0
#35 0x00007fbb0a1d43c9 in ldbm_back_modify () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
#36 0x00007fbb16f104e1 in op_shared_modify () from /usr/lib64/dirsrv/libslapd.so.0
#37 0x00007fbb16f1182f in do_modify () from /usr/lib64/dirsrv/libslapd.so.0
#38 0x00007fbb173f5320 in connection_threadmain ()
#39 0x00007fbb1510496b in _pt_root () from /lib64/libnspr4.so
#40 0x00007fbb14aa5dc5 in start_thread () from /lib64/libpthread.so.0
#41 0x00007fbb147d2ced in clone () from /lib64/libc.so.6
=====================================
The quantity of total internal operations is more or less:
=====================================
[20/Jul/2016:15:41:58 -0500] conn=180 op=5 MOD dn="cn=role_il_users,cn=groups,cn=accounts,dc=dev,dc=fpd,dc=cat,dc=com"
[20/Jul/2016:15:42:54 -0500] conn=180 op=5 RESULT err=0 tag=103 nentries=0 etime=56 csn=578fe293000300040000
in between we have 408302 operations !
most of them taking less than a second and searching only the user entry. Except one taking 34 secs. but we cannot match internal op and its result (a RFE was already logged)
======================================
So, we decide to disable compat plugin:
# ipa-compat-manage disable
and the command is completed in only 4 seconds from cli side but from ldap side:
[03/Aug/2016:12:05:27 -0500] conn=14 op=5 MOD dn="cn=role_il_users,cn=groups,cn=accounts,dc=dev,dc=fpd,dc=cat,dc=com"
[03/Aug/2016:12:05:27 -0500] conn=14 op=5 RESULT err=0 tag=103 nentries=0 etime=0 csn=57a224de001200040000
The quantity of internal ops is 57.
Version-Release number of selected component (if applicable):
slapi-nis-0.54-9.el7_2.x86_64
How reproducible: always
Steps to Reproduce: we could re-generated customer scenario if needed.
We could give more information on the operations by uploading customer access logs if needed.
Regards,
Prasad and German.
Some comments:
In fact one MOD triggers 400K internal searches.
* The vast majority of them are base search of user/host entries.
* 60% of those entries are searched > 100 times.
* for each entry, at least one search retrieves all attributes.
* searches are done under the same txn, so those entries do not change during the MOD
Hello,
Here is the brief description customer has mentioned about the issue :
In IPA master-replica environment, create a group which is member of 13-15 other groups each containing about 500-1400 users.
Now, try addition/removal of users to that group, which takes about a minute to finish and will make both master and replica unresponsive to any ldap queries or login attempts during the time.
Make sure compat plugin is enabled on both the systems.
Hi German,
Thanks for reporting this new related case.
This bug is high priority to 7.4 and will be fixed in that release.
Can you confirm that during this long MOD, DS has a similar pstack https://bugzilla.redhat.com/show_bug.cgi?id=1364144#c0.
Did you reproduce it in-house or give some detail how to reproduce it.
thanks
Comment 20Florence Blanc-Renaud
2020-02-14 15:42:12 UTC
Thank you taking your time and submitting this request for Red Hat Enterprise Linux 7. Unfortunately, this bug cannot be kept even as a stretch goal and was postponed to RHEL8.
Thank you taking your time and submitting this request for Red Hat Enterprise Linux. It was unfortunately not given priority Red Hat Enterprise Linux.
Given that this request is not planned for a close release, it is highly unlikely it will be fixed in this major version of Red Hat Enterprise Linux. We are therefore closing the request as WONTFIX.
To request that Red Hat reconsiders the decision, please reopen the Bugzilla with the help of Red Hat Customer Service and provide additional business and/or technical details about it's importance to you.