Doing a simple paged results search against a subtree that uses ip or dns based ACIs will hang the server. For example: dn: dc=example,dc=com aci: (targetattr != "userPassword") (version 3.0;acl "Anonymous access within domain";allow (read,compare,search)(userdn = "ldap:///anyone") and (dns="localhost" or dns="localdomain" or dns="*.localdomain");) If you repeatedly do simple paged result searches against dc=example,dc=com, the server will hang with a stack trace like this: Thread 10 (Thread 0x7f44fa9f0700 (LWP 7422)): #0 0x000000305280dfe4 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x000000305280934e in _L_lock_995 () from /lib64/libpthread.so.0 #2 0x00000030528092b6 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x0000003061023bd9 in PR_Lock (lock=0x19890f0) at ../../../mozilla/nsprpub/pr/src/pthreads/ptsynch.c:206 #4 0x00007f4521f17112 in slapi_pblock_get (pblock=0x193ee50, arg=850, value=0x7f44fa9dfdc0) at ../ds.git/ldap/servers/slapd/pblock.c:254 #5 0x00007f451eee241f in DS_LASDnsGetter (errp=0x0, subject=0x1822d08, resource=0x0, auth_info=0x0, global_auth=0x0, arg=0x0) at ../ds.git/ldap/servers/plugins/acl/acllas.c:376 #6 0x00007f451ec9bf3e in ACL_GetAttribute (errp=0x0, attr=0x7f451ecb1093 "dns", val=0x7f44fa9dff68, subject=0x1822d08, resource=0x0, auth_info=0x0, global_auth=0x0) at ../ds.git/lib/libaccess/method.cpp:159 #7 0x00007f451ec9996f in LASDnsEval (errp=0x0, attr_name=0x7f44e0001268 "dns", comparator=CMP_OP_EQ, attr_pattern=0x7f44e00011d8 "localhost", cachable=0x7f44fa9e0020, LAS_cookie=0x7f44e0002ec8, subject=0x1822d08, resource=0x0, auth_info=0x0, global_auth=0x0) at ../ds.git/lib/libaccess/lasdns.cpp:404 #8 0x00007f451ec9c999 in ACLEvalAce (errp=0x0, acleval=0x1822f28, ace=0x7f44e000db08, cachable=0x7f44fa9e2bc8, autharray=0x0, global_auth=0x0) at ../ds.git/lib/libaccess/oneeval.cpp:254 #9 0x00007f451ec9d990 in ACL_INTEvalTestRights (errp=0x0, acleval=0x1822f28, rights=0x7f44fa9e2d38, map_generic=0x7f451f0fb5c0, deny_type=0x7f44fa9e2d60, deny_response=0x7f44fa9e2d58, acl_tag=0x7f44fa9e2d50, expr_num=0x7f44fa9e2d80, cachable=0x7f44fa9e2cc8) at ../ds.git/lib/libaccess/oneeval.cpp:785 #10 0x00007f451ec9e012 in ACL_EvalTestRights (errp=0x0, acleval=0x1822f28, rights=0x7f44fa9e2d30, map_generic=0x7f451f0fb5c0, deny_type=0x7f44fa9e2d60, deny_response=0x7f44fa9e2d58, acl_tag=0x7f44fa9e2d50, expr_num=0x7f44fa9e2d80) at ../ds.git/lib/libaccess/oneeval.cpp:995 #11 0x00007f451eed9653 in acl__TestRights (aclpb=0x184b3c0, access=2, right=0x7f44fa9e2e60, map_generic=0x7f451f0fb5c0, result_reason=0x7f44fa9e2e00) at ../ds.git/ldap/servers/plugins/acl/acl.c:3068 #12 0x00007f451eed5113 in acl_access_allowed (pb=0x193ee50, e=0x7f44e0000a50, attr=0x7f4488007850 "objectClass", val=0x0, access=2) at ../ds.git/ldap/servers/plugins/acl/acl.c:590 #13 0x00007f451eef01dc in acl_access_allowed_main (pb=0x193ee50, e=0x7f44e0000a50, attrs=0x7f44fa9e4fe0, val=0x0, access=2, flags=0, errbuf=0x0) at ../ds.git/ldap/servers/plugins/acl/aclplugin.c:381 #14 0x00007f4521f2281e in plugin_call_acl_plugin (pb=0x193ee50, e=0x7f44e0000a50, attrs=0x7f44fa9e4fe0, val=0x0, access=2, flags=0, errbuf=0x0) at ../ds.git/ldap/servers/slapd/plugin_acl.c:90 #15 0x00007f4521eeabb3 in test_filter_access (pb=0x193ee50, e=0x7f44e0000a50, attr_type=0x7f4488007850 "objectClass", attr_val=0x0) at ../ds.git/ldap/servers/slapd/filterentry.c:1011 #16 0x00007f4521eea793 in slapi_vattr_filter_test_ext_internal (pb=0x193ee50, e=0x7f44e0000a50, f=0x7f4488007390, verify_access=1, only_check_access=0, access_check_done=0x7f44fa9e50b8) at ../ds.git/ldap/servers/slapd/filterentry.c:926 #17 0x00007f4521eea38d in slapi_vattr_filter_test_ext (pb=0x193ee50, e=0x7f44e0000a50, f=0x7f4488007390, verify_access=1, only_check_access=0) at ../ds.git/ldap/servers/slapd/filterentry.c:839 #18 0x00007f4521eea2b5 in slapi_vattr_filter_test (pb=0x193ee50, e=0x7f44e0000a50, f=0x7f4488007390, verify_access=1) at ../ds.git/ldap/servers/slapd/filterentry.c:787 #19 0x00007f451cdd847a in ldbm_back_next_search_entry_ext (pb=0x193ee50, use_extension=0) at ../ds.git/ldap/servers/slapd/back-ldbm/ldbm_search.c:1412 #20 0x00007f451cdd78dd in ldbm_back_next_search_entry (pb=0x193ee50) at ../ds.git/ldap/servers/slapd/back-ldbm/ldbm_search.c:1141 #21 0x00007f4521f15344 in iterate (pb=0x193ee50, be=0x1774ea0, send_result=1, pnentries=0x7f44fa9eb358, pagesize=2, pr_statp=0x7f44fa9eb334) at ../ds.git/ldap/servers/slapd/opshared.c:1162 #22 0x00007f4521f15c2b in send_results_ext (pb=0x193ee50, send_result=1, nentries=0x7f44fa9eb358, pagesize=2, pr_stat=0x7f44fa9eb334) at ../ds.git/ldap/servers/slapd/opshared.c:1545 #23 0x00007f4521f144d1 in op_shared_search (pb=0x193ee50, send_result=1) at ../ds.git/ldap/servers/slapd/opshared.c:599 #24 0x000000000042caf7 in do_search (pb=0x193ee50) at ../ds.git/ldap/servers/slapd/search.c:393 #25 0x0000000000413fde in connection_dispatch_operation (conn=0x7f45181f5560, op=0x1990ef0, pb=0x193ee50) at ../ds.git/ldap/servers/slapd/connection.c:611 #26 0x00000000004158e4 in connection_threadmain () at ../ds.git/ldap/servers/slapd/connection.c:2328 #27 0x0000003061029633 in _pt_root (arg=0x197c0f0) at ../../../mozilla/nsprpub/pr/src/pthreads/ptthread.c:187 #28 0x00000030528077e1 in start_thread () from /lib64/libpthread.so.0 #29 0x0000003051ce577d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f4521c577c0 (LWP 7366)): #0 0x000000305280dfe4 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x000000305280934e in _L_lock_995 () from /lib64/libpthread.so.0 #2 0x00000030528092b6 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x0000003061023bd9 in PR_Lock (lock=0x19890f0) at ../../../mozilla/nsprpub/pr/src/pthreads/ptsynch.c:206 #4 0x0000000000418eda in setup_pr_read_pds (ct=0x1996fb0, n_tcps=0x171d4d0, s_tcps=0x0, i_unix=0x0, num_to_read=0x7fff551bb248) at ../ds.git/ldap/servers/slapd/daemon.c:1193 #5 0x0000000000417f7c in slapd_daemon (ports=0x7fff551bb390) at ../ds.git/ldap/servers/slapd/daemon.c:663 #6 0x000000000041fc23 in main (argc=7, argv=0x7fff551bb508) at ../ds.git/ldap/servers/slapd/main.c:1239 The connection mutex is acquired here: opshared.c:1160 if ( is_paged && pb->pb_conn && pb->pb_conn->c_mutex ) { PR_Lock( pb->pb_conn->c_mutex ); }
Created attachment 521072 [details] 0001-Bug-735121-simple-paged-search-ip-dns-based-ACI-hang.patch
To ssh://git.fedorahosted.org/git/389/ds.git 576d90b..b5f77c6 master -> master commit b5f77c693d50ca3acf58eb7a41d9eded59328632 Author: Rich Megginson <rmeggins> Date: Thu Sep 1 10:30:06 2011 -0600 Reviewed by: nhosoi (Thanks!) Branch: master Fix Description: The pb_conn->c_mutex lock around be->be_next_search_entry() was too big. Other code inside be->be_next_search_entry needs to lock the conn. In this particular case, the aci code needed to access the client ip address from the conn object. The fix is to remove the mutex around be->be_next_search_entry() in iterate() and instead make sure code that accesses pb_conn locks first. Platforms tested: RHEL6 x86_64 Flag Day: no Doc impact: no
Rich, this appears to have fixed: http://lists.fedoraproject.org/pipermail/389-users/2011-August/013516.html. My issues must have been related to dns or ip based ACIs. THANK YOU!!!
My problem (http://lists.fedoraproject.org/pipermail/389-users/2011-September/013574.html) is also fixed. We have indeed several IP-based ACIs. Re-deploying 1.2.9.9 now :) Thanks, Rich! 389DS developers (Noriko, you and Nathan) reactivity still continues to amaze me after all these years :)
Latest Result from tet: SIMPLEPAGED startup 100% (1/1) SIMPLEPAGED run 100% (10/10) SIMPLEPAGED cleanup 100% (1/1) as per comment#3, Rich added test for this bug fix. Hence marking as VERIFIED.