Bug 1627846

Summary: Contention on virtual attribute lookup
Product: Red Hat Enterprise Linux 7 Reporter: German Parente <gparente>
Component: 389-ds-baseAssignee: thierry bordaz <tbordaz>
Status: CLOSED ERRATA QA Contact: RHDS QE <ds-qe-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 7.7-AltCC: anazmy, cpelland, mreynolds, nkinder, pasik, rmeggins, spichugi, tbordaz, vashirov
Target Milestone: rc   
Target Release: 7.7   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 389-ds-base-1.3.9.1-4.el7 Doc Type: If docs needed, set a value
Doc Text:
Cause: When evaluating a filter component, it is necessary to check if the attribute is real or virtual Consequence: Contention on a RW lock protecting the list of virtual attributes even if the lock is most often free or acquired in read. Fix: The search thread acquires the lock in read and set a flag, into a private thread variable, so that further access to the list will just check the flag. Result: no more contention on virtual attribute lock
 Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-06 12:58:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description German Parente 2018-09-11 16:58:54 UTC 
Description of problem:

this is a copy of upstream bug:

https://pagure.io/389-ds-base/issue/49873

The tests locally have been done by Ahmed Nazmy.

When doing "id <user>" in a high number of clients at the same time, we can see stack traces like these ones:

Thread 4 (Thread 0x7f23067d9700 (LWP 64916)):
#0  0x00007f234235f1e5 in vattr_hash_fn () at /usr/lib64/dirsrv/libslapd.so.0
#1  0x00007f23406eea52 in PL_HashTableLookupConst () at /lib64/libplds4.so
#2  0x00007f234235f2e4 in vattr_map_lookup () at /usr/lib64/dirsrv/libslapd.so.0
#3  0x00007f23423602f8 in vattr_map_namespace_sp_getlist () at /usr/lib64/dirsrv/libslapd.so.0
#4  0x00007f23423603be in vattr_test_filter () at /usr/lib64/dirsrv/libslapd.so.0
#5  0x00007f23422eea54 in slapi_vattr_filter_test_ext_internal () at /usr/lib64/dirsrv/libslapd.so.0
#6  0x00007f23422ef736 in slapi_vattr_filter_test_ext () at /usr/lib64/dirsrv/libslapd.so.0
#7  0x00007f23376f1df2 in acl__resource_match_aci () at /usr/lib64/dirsrv/plugins/libacl-plugin.so
#8  0x00007f23376f3270 in acl_access_allowed () at /usr/lib64/dirsrv/plugins/libacl-plugin.so
#9  0x00007f2337706987 in acl_access_allowed_main () at /usr/lib64/dirsrv/plugins/libacl-plugin.so
#10 0x00007f2342327ecc in plugin_call_acl_plugin () at /usr/lib64/dirsrv/libslapd.so.0
#11 0x00007f2342328397 in slapi_access_allowed () at /usr/lib64/dirsrv/libslapd.so.0
#12 0x00007f2336a6d17b in deref_check_access.constprop.2 () at /usr/lib64/dirsrv/plugins/libderef-plugin.so
#13 0x00007f2336a6d7d3 in deref_pre_entry () at /usr/lib64/dirsrv/plugins/libderef-plugin.so
#14 0x00007f2342324548 in plugin_call_func () at /usr/lib64/dirsrv/libslapd.so.0
#15 0x00007f2342324803 in plugin_call_plugins () at /usr/lib64/dirsrv/libslapd.so.0
#16 0x00007f234233a46f in send_ldap_search_entry_ext () at /usr/lib64/dirsrv/libslapd.so.0
#17 0x00007f234233ae5c in send_ldap_search_entry () at /usr/lib64/dirsrv/libslapd.so.0
#18 0x00007f2342315cf4 in send_entry.isra.0 () at /usr/lib64/dirsrv/libslapd.so.0
#19 0x00007f2342316258 in send_results_ext.constprop.4 () at /usr/lib64/dirsrv/libslapd.so.0
#20 0x00007f2342318121 in op_shared_search () at /usr/lib64/dirsrv/libslapd.so.0
#21 0x000055c4cf71873e in do_search ()
#22 0x000055c4cf7065fa in connection_threadmain ()
#23 0x00007f23404d7bab in _pt_root () at /lib64/libnspr4.so
#24 0x00007f233fe77dd5 in start_thread () at /lib64/libpthread.so.0
#25 0x00007f233f524b3d in clone () at /lib64/libc.so.6


Thread 19 (Thread 0x7f230dfe8700 (LWP 64901)):
#0  0x00007f233fe7e4cd in __lll_lock_wait () at /lib64/libpthread.so.0
#1  0x00007f233fe7b610 in pthread_rwlock_unlock () at /lib64/libpthread.so.0
#2  0x00007f234235f2f7 in vattr_map_lookup () at /usr/lib64/dirsrv/libslapd.so.0
#3  0x00007f2342360294 in vattr_map_namespace_sp_getlist () at /usr/lib64/dirsrv/libslapd.so.0
#4  0x00007f23423603be in vattr_test_filter () at /usr/lib64/dirsrv/libslapd.so.0
#5  0x00007f23422eea54 in slapi_vattr_filter_test_ext_internal () at /usr/lib64/dirsrv/libslapd.so.0
#6  0x00007f23422ee8c1 in slapi_vattr_filter_test_ext_internal () at /usr/lib64/dirsrv/libslapd.so.0
#7  0x00007f23422ef736 in slapi_vattr_filter_test_ext () at /usr/lib64/dirsrv/libslapd.so.0
#8  0x00007f23376f1df2 in acl__resource_match_aci () at /usr/lib64/dirsrv/plugins/libacl-plugin.so
#9  0x00007f23376f3270 in acl_access_allowed () at /usr/lib64/dirsrv/plugins/libacl-plugin.so
#10 0x00007f2337706987 in acl_access_allowed_main () at /usr/lib64/dirsrv/plugins/libacl-plugin.so
#11 0x00007f2342327ecc in plugin_call_acl_plugin () at /usr/lib64/dirsrv/libslapd.so.0
#12 0x00007f2342328397 in slapi_access_allowed () at /usr/lib64/dirsrv/libslapd.so.0
#13 0x00007f2336a6d17b in deref_check_access.constprop.2 () at /usr/lib64/dirsrv/plugins/libderef-plugin.so
#14 0x00007f2336a6d7d3 in deref_pre_entry () at /usr/lib64/dirsrv/plugins/libderef-plugin.so
#15 0x00007f2342324548 in plugin_call_func () at /usr/lib64/dirsrv/libslapd.so.0
#16 0x00007f2342324803 in plugin_call_plugins () at /usr/lib64/dirsrv/libslapd.so.0
#17 0x00007f234233a46f in send_ldap_search_entry_ext () at /usr/lib64/dirsrv/libslapd.so.0
#18 0x00007f234233ae5c in send_ldap_search_entry () at /usr/lib64/dirsrv/libslapd.so.0
#19 0x00007f2342315cf4 in send_entry.isra.0 () at /usr/lib64/dirsrv/libslapd.so.0
#20 0x00007f2342316258 in send_results_ext.constprop.4 () at /usr/lib64/dirsrv/libslapd.so.0
#21 0x00007f2342318121 in op_shared_search () at /usr/lib64/dirsrv/libslapd.so.0
#22 0x000055c4cf71873e in do_search ()
#23 0x000055c4cf7065fa in connection_threadmain ()
#24 0x00007f23404d7bab in _pt_root () at /lib64/libnspr4.so
#25 0x00007f233fe77dd5 in start_thread () at /lib64/libpthread.so.0
#26 0x00007f233f524b3d in clone () at /lib64/libc.so.6

Thread 18 (Thread 0x7f230d7e7700 (LWP 64902)):
#0  0x00007f233f59500a in __strlen_sse2_pminub () at /lib64/libc.so.6
#1  0x00007f23422d79f5 in slapi_dn_normalize_ext () at /usr/lib64/dirsrv/libslapd.so.0
#2  0x00007f23422d99ff in slapi_dn_normalize_case_ext () at /usr/lib64/dirsrv/libslapd.so.0
#3  0x00007f233b68d9eb in value_normalize_ext () at /usr/lib64/dirsrv/plugins/libsyntax-plugin.so
#4  0x00007f233b68dd9d in value_cmp () at /usr/lib64/dirsrv/plugins/libsyntax-plugin.so
#5  0x00007f233b689f5e in string_filter_ava () at /usr/lib64/dirsrv/plugins/libsyntax-plugin.so
#6  0x00007f233b686bb3 in dn_filter_ava () at /usr/lib64/dirsrv/plugins/libsyntax-plugin.so
#7  0x00007f234232b456 in plugin_call_syntax_filter_ava_sv () at /usr/lib64/dirsrv/libslapd.so.0
#8  0x00007f23422ee2db in test_ava_filter () at /usr/lib64/dirsrv/libslapd.so.0
#9  0x00007f2342360899 in vattr_test_filter () at /usr/lib64/dirsrv/libslapd.so.0
#10 0x00007f23422eea54 in slapi_vattr_filter_test_ext_internal () at /usr/lib64/dirsrv/libslapd.so.0
#11 0x00007f23422ef736 in slapi_vattr_filter_test_ext () at /usr/lib64/dirsrv/libslapd.so.0
#12 0x00007f23376f1df2 in acl__resource_match_aci () at /usr/lib64/dirsrv/plugins/libacl-plugin.so
#13 0x00007f23376f3270 in acl_access_allowed () at /usr/lib64/dirsrv/plugins/libacl-plugin.so
#14 0x00007f2337706987 in acl_access_allowed_main () at /usr/lib64/dirsrv/plugins/libacl-plugin.so
#15 0x00007f2342327ecc in plugin_call_acl_plugin () at /usr/lib64/dirsrv/libslapd.so.0
#16 0x00007f2342328397 in slapi_access_allowed () at /usr/lib64/dirsrv/libslapd.so.0
#17 0x00007f2336a6d17b in deref_check_access.constprop.2 () at /usr/lib64/dirsrv/plugins/libderef-plugin.so
#18 0x00007f2336a6d7d3 in deref_pre_entry () at /usr/lib64/dirsrv/plugins/libderef-plugin.so
#19 0x00007f2342324548 in plugin_call_func () at /usr/lib64/dirsrv/libslapd.so.0
#20 0x00007f2342324803 in plugin_call_plugins () at /usr/lib64/dirsrv/libslapd.so.0
#21 0x00007f234233a46f in send_ldap_search_entry_ext () at /usr/lib64/dirsrv/libslapd.so.0
#22 0x00007f234233ae5c in send_ldap_search_entry () at /usr/lib64/dirsrv/libslapd.so.0
#23 0x00007f2342315cf4 in send_entry.isra.0 () at /usr/lib64/dirsrv/libslapd.so.0
#24 0x00007f2342316258 in send_results_ext.constprop.4 () at /usr/lib64/dirsrv/libslapd.so.0
#25 0x00007f2342318121 in op_shared_search () at /usr/lib64/dirsrv/libslapd.so.0
#26 0x000055c4cf71873e in do_search ()
#27 0x000055c4cf7065fa in connection_threadmain ()
#28 0x00007f23404d7bab in _pt_root () at /lib64/libnspr4.so
#29 0x00007f233fe77dd5 in start_thread () at /lib64/libpthread.so.0
#30 0x00007f233f524b3d in clone () at /lib64/libc.so.6

High contention in virtual attributes, dn normalization, aci evaluation after deref plugin operations.

 

Version-Release number of selected component (if applicable): 389-ds-base-1.3.7.5-25.el7_5.x86_64
Comment 5 thierry bordaz 2019-02-06 12:56:57 UTC 
Fix pushed upstream -> POST
Comment 7 Viktor Ashirov 2019-06-21 14:17:51 UTC 
Build tested: 389-ds-base-1.3.9.1-10.el7.x86_64

Acceptance tests pass, I don't see the stack traces from the description anymore.
Marking as VERIFIED, SanityOnly.
Comment 9 errata-xmlrpc 2019-08-06 12:58:51 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2152