Description of problem: Crash Version-Release number of selected component (if applicable): 1.2.6 x86_64 on RHEL 5.5 How reproducible: Steps to Reproduce: 1. we imported out old LDIF file and left server running for a bit Actual results: Program terminated with signal 11, Segmentation fault. #0 0x00002ac6b1aa2a1f in ?? () from /usr/lib64/dirsrv/libslapd.so.0 (gdb) bt #0 0x00002ac6b1aa2a1f in ?? () from /usr/lib64/dirsrv/libslapd.so.0 #1 0x00002ac6b1aa396c in slapi_attr_type_cmp () from /usr/lib64/dirsrv/libslapd.so.0 #2 0x00002ac6b1abf5fa in test_ava_filter () from /usr/lib64/dirsrv/libslapd.so.0 #3 0x00002ac6b1b12bda in vattr_test_filter () from /usr/lib64/dirsrv/libslapd.so.0 #4 0x00002ac6b1abfc08 in ?? () from /usr/lib64/dirsrv/libslapd.so.0 #5 0x00002ac6b1abff92 in ?? () from /usr/lib64/dirsrv/libslapd.so.0 #6 0x00002ac6b1abfd8c in ?? () from /usr/lib64/dirsrv/libslapd.so.0 #7 0x00002ac6b1ac00c2 in slapi_vattr_filter_test_ext () from /usr/lib64/dirsrv/libslapd.so.0 #8 0x00002ac6b63b0ba6 in ldbm_back_next_search_entry_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so #9 0x00002ac6b1adeec1 in ?? () from /usr/lib64/dirsrv/libslapd.so.0 #10 0x00002ac6b1adf3f2 in ?? () from /usr/lib64/dirsrv/libslapd.so.0 #11 0x00002ac6b1adfdb5 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0 #12 0x00000000004251ef in ?? () #13 0x00000000004134be in ber_sockbuf_free () #14 0x0000003667e284ad in ?? () from /usr/lib64/libnspr4.so #15 0x0000003665e0673d in start_thread () from /lib64/libpthread.so.0 #16 0x00000036656d3d1d in clone () from /lib64/libc.so.6 Not sure if this is helpful but just in case: (gdb) x /6xw 0x00002ac6b1aa2a1f 0x2ac6b1aa2a1f: 0x01be0f41 0x8dd1460f 0x708dbf48 0x19f98320 0x2ac6b1aa2a2f: 0x39c6460f 0x441275c2 (gdb) x/6xw 0x00002ac6b1aa396c 0x2ac6b1aa396c <slapi_attr_type_cmp+108>: 0x0874c085 0x01b85d5b 0xc3000000 0xe8ef8948 0x2ac6b1aa397c <slapi_attr_type_cmp+124>: 0xfffff050 0x48c08548 Expected results: Additional info: This is on a production box that was okay on 1.2.5, then started returning bad results intermittently when it was upgraded to 1.2.6 (indexing issue? I've tried a reindex). I've built a brand new 1.2.6 and imported the old LDIF. Things looked good but it's crashed after a few hours with the backtrace above. Nothing in the error logs :( Access logs (company name changed) [23/Sep/2010:00:00:42 +0100] conn=7732 op=1215343 SRCH base="dc=ds,dc=example,dc=com" scope=2 filter="(&(objectClass=groupOfUniqueNames)(cn=Example Software Limited))" attrs=ALL [23/Sep/2010:00:00:42 +0100] conn=7732 op=1215343 RESULT err=0 tag=101 nentries=1 etime=0 389-Directory/1.2.6 B2010.238.2134 ds2.man.alfresco.com:636 (/etc/dirsrv/slapd-ds2)
Can you install the 389-ds-base-debuginfo package so that we can get more data from the core dump? Can you provide your LDIF data so that we can see if there is something about the data that is causing the problem? I mean, the server shouldn't crash anyway, but so far we have not been able to figure out how to reproduce the problem.
Thanks for the fast response. I tried to find that but it only found the i386 one, I've looked a bit harder and found it this time! I can't provide the LDIF data I'm afraid :-( The original import failed and we had to cleanup the LDIF quite a bit ("c"(country) field had full names in not "DE", some strange uids etc. We never forced anything in so the final import should be clean I believe. Program terminated with signal 11, Segmentation fault. #0 comp_cmp (s1=0x2aaab0d2dfb0 "memberOf", s2=0x0) at ldap/servers/slapd/attr.c:98 98 while ( *s1 && *s1 != ';' && tolower( *s1 ) == tolower( *s2 ) ) { (gdb) bt full #0 comp_cmp (s1=0x2aaab0d2dfb0 "memberOf", s2=0x0) at ldap/servers/slapd/attr.c:98 No locals. #1 0x00002ac6b1aa396c in slapi_attr_type_cmp (a1=0x2aaab0d2dfb0 "memberOf", a2=0x0, opt=109) at ldap/servers/slapd/attr.c:131 rc = <value optimized out> #2 0x00002ac6b1abf5fa in test_ava_filter (pb=0x0, e=0x2aaab02b5050, a=0x2aaab1c64770, ava=0x2aaab10d3690, ftype=163, verify_access=0, only_check_access=0, access_check_done=0x51747bd8) at ldap/servers/slapd/filterentry.c:342 rc = -1 #3 0x00002ac6b1b12bda in vattr_test_filter (pb=0x2aaab22e8960, e=0x2aaab02b5050, f=0x2aaab10d3670, filter_type=FILTER_TYPE_AVA, type=0x2aaab0d2dfb0 "memberOf") at ldap/servers/slapd/vattr.c:605 acl_test_done = 0 rc = -1 list = 0x0 sdn = <value optimized out> be = <value optimized out> namespace_dn = <value optimized out> #4 0x00002ac6b1abfc08 in slapi_vattr_filter_test_ext_internal ( pb=0x2aaab22e8960, e=0x2aaab02b5050, f=0x2aaab10d3670, verify_access=0, only_check_access=<value optimized out>, access_check_done=0x51747d14) ---Type <return> to continue, or q <return> to quit--- at ldap/servers/slapd/filterentry.c:894 rc = 0 #5 0x00002ac6b1abff92 in vattr_test_filter_list (pb=0x2aaab22e8960, e=0x2aaab02b5050, flist=0x2aaab10d3670, ftype=160, verify_access=0, only_check_access=0, access_check_done=0x51747d14) at ldap/servers/slapd/filterentry.c:1035 nomatch = 0 f = 0x2aaab10d3670 #6 0x00002ac6b1abfd8c in slapi_vattr_filter_test_ext_internal ( pb=0x2aaab22e8960, e=0x2aaab02b5050, f=0x2aaab2044b50, verify_access=109, only_check_access=0, access_check_done=0x51747d14) at ldap/servers/slapd/filterentry.c:962 rc = 0 #7 0x00002ac6b1ac00c2 in slapi_vattr_filter_test_ext (pb=0x2aaab22e8960, e=0x2aaab02b5050, f=0x2aaab2044b50, verify_access=1, only_check_access=<value optimized out>) at ldap/servers/slapd/filterentry.c:824 rc = 44 access_check_done = 0 #8 0x00002ac6b63b0ba6 in ldbm_back_next_search_entry_ext (pb=0x2aaab22e8960, use_extension=0) at ldap/servers/slapd/back-ldbm/ldbm_search.c:1356 filter_test = 1 be = 0x72c0e90 ---Type <return> to continue, or q <return> to quit--- inst = 0x73848c0 li = 0x7280460 scope = 2 managedsait = 1 attr = 0x0 filter = 0x2aaab2044b50 base = 0x2aaab005b070 "dc=people,dc=ds,dc=example,dc=com" sr = 0x2aaab20d0490 id = 22383 e = 0x2aaab18a4020 nentries = <value optimized out> curtime = <value optimized out> stoptime = 1285200040 optime = 1285196440 tlimit = 3600 llimit = -1 slimit = 999917 isroot = 0 urls = 0x0 err = 0 basesdn = {flag = 0 '\000', dn = 0x2aaab005b070 "dc=people,dc=ds,dc=example,dc=com", ndn = 0x2aaab005b070 "dc=people,dc=ds,dc=example,dc=com", ---Type <return> to continue, or q <return> to quit--- ndn_len = 34} target_uniqueid = 0x0 rc = <value optimized out> #9 0x00002ac6b1adeec1 in iterate (pb=0x2aaab22e8960, be=0x72c0e90, send_result=1, pnentries=0x5174a6a4, pagesize=-1, pr_statp=0x5174a698) at ldap/servers/slapd/opshared.c:1088 gerentry = 0x2aaab15159c0 operation = 0x2aaab25c3d50 rc = 0 rval = 1 attrsonly = 0 done = 0 e = 0x2aaaac5575d0 attrs = 0x2aaab187f890 pr_stat = 0 #10 0x00002ac6b1adf3f2 in send_results_ext (pb=0x2aaab22e8960, send_result=1, nentries=0x5174a6a4, pagesize=-1, pr_stat=0x5174a698) at ldap/servers/slapd/opshared.c:1461 be = 0x72c0e90 rc = <value optimized out> #11 0x00002ac6b1adfdb5 in op_shared_search (pb=0x2aaab22e8960, send_result=1) at ldap/servers/slapd/opshared.c:688 be_suffix = 0x7232d30 ---Type <return> to continue, or q <return> to quit--- err = 0 next_be = 0x0 base = 0x2aaab236bf20 "dc=people,dc=ds,dc=example,dc=com" fstr = 0x2aaab15159c0 "(&(objectClass=inetOrgPerson)(memberOf=cn=app_partners_access,dc=groups,dc=ds,dc=example,dc=com)(!(modifyTimestamp<=20100920230533Z)))" scope = 2 be = 0x72c0e90 be_single = 0x0 be_list = {0x72c0e90, 0x0, 0x0, 0x2aaab098ba80, 0x2aaab254af90, 0x2aaab254af80, 0x30, 0x2aaab098ba50, 0x50, 0x3665674632, 0x0, 0x2aaab0000020, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2aaab0000020, 0x2aaab098ba60, 0x42, 0x2, 0x2aaab098ba50, 0x2aaab1c2fde0, 0x3665675292, 0x20, 0x30, 0x2aaab10d3698, 0x5174a440, 0x2ac6b1b1458b, 0x366b229702, 0x0, 0x2aaab098ba60, 0x42, 0x2aaab1c2fde0, 0x2, 0x42, 0x2aaab1c2fde0, 0x3667e153ce, 0x20000, 0xa00000000, 0x3900000000000000, 0x2aaab0000020, 0x5174a700, 0x20, 0x0, 0x7c816000004b3b00, 0x2ac6b1b1c2d2, 0x22, 0x5174a700, 0x2aaab1c2fde0, 0x2, 0x3667e122d3, 0x0, 0x2ac6b1b1c2d6, 0xffffffde, 0x0, 0x5174a700, 0x3667e11344, 0x0, 0x2aaab0000020, 0x2aaab195b100, 0x5174a740, 0x0, 0x2aaab195b0f0, 0x0, 0x22b25c41a4, 0x20, 0x30, 0x0, 0x0, 0x0, 0x0, 0x2aaab0000088, 0x2aaab195b100, 0x40, 0x2ac6b1b1c196, 0x3a, 0x40, 0x2ac6b1b1c197, 0x3667e153ce, 0x0, 0x0, ---Type <return> to continue, or q <return> to quit--- 0x3b00000000000000, 0x2aaab0000020, 0x2aaab25c3ed0, 0x2aaab25c3ed0, 0x5174a8bc, 0x5174a620, 0x28, 0x366b229702, 0x2aaab25c3ed0, 0xa2002aaab25c4121, 0x5174a8bc, 0x5174a620, 0x5174a8bc, 0x5174a620, 0xa2, 0x366b2297fd, 0x4, 0x366b22c613} referral_list = {0x0 <repeats 62 times>, 0x2f7065532f32325b, 0x3a32323a30313032, 0x302b2031303a3534, 0x205d303031, 0x0 <repeats 15 times>, 0xa0, 0x2aaab0000088, 0x5174a2c0, 0x0, 0x85, 0x41, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0xa00000000, 0x0, 0x0} ebuf = "\033\000\000\000\000\000\000\000\020�tQ\000\000\000\000�\000\000\000\000\000\000\000�\227\"k6", '\000' <repeats 26 times>, "$\033\000\000\000\000\000\000\000`�\230��*\000\000p6\r��*\000\000\060�tQ\000\000\000\000P\235\"k6\000\000\000(\000\000\000\000\000\000\000\002\227\"k6\000\000\000 �o��*\000\000 �o��*\000�<�tQ\000\000\000\000��tQ\000\000\000\000\201\000\000\000\000\000\000\000��tQ\000\000\000\000�\000\000\000\000\000\000\000O\230\"k6\000\000\000\004\000\000\000\000\000\000\000\023�\"k6\000\000\000\000\000\000\213�*\000\201<�tQ\000\000\000\000\001\000\000\000\000\000\000\000"... attrlistbuf = "\"1.1\"\000\000\000P�tQ\000\000\000\000H�D\000\000\000\000\000\002\227\"k6\000\000\000[22/Sep/2010:23\004\001\000\000\000\000\000\000\000P�tQ\000\000\000\000\001\000\000\000\000\000\000\000P�tQ\000\000\000\000\004\000\000\000\000\000\000\000�\227\"k6", '\000' <repeats 11 times>, "x\000\000��*\000\000\000\000\000\000\000\000\000\003\001\000\000\000\000\000\000\000G�D\000\000\000\000\000H�D", '\000' <repeats 13 times>, "P\235\"k6\000\000\000@Y\001\000\000---Type <return> to continue, or q <return> to quit--- \000\000\000 t챪*\000\000X$re6\000\000\000\060X\001\000\000\000\000\000 \000\000��*\000\000L\"ge"... attrliststr = 0x5174e6b0 "\"1.1\"" attrs = 0x2aaab187f890 rc = 0 internal_op = <value optimized out> sdn = {flag = 4 '\004', dn = 0x2aaab236bf20 "dc=people,dc=ds,dc=example,dc=com", ndn = 0x2aaab005b070 "dc=people,dc=ds,dc=example,dc=com", ndn_len = 34} operation = 0x2aaab25c3d50 referral = <value optimized out> errorbuf = "\000A\\��*\000\000\000;K\000\000`\201|", '\000' <repeats 16 times>, "��tQ\000\000\000\000(\000\000\000\000\000\000\000\020\024���*\000\000��tQ\000\000\000\000�>\\��*\000\000�!�g6\000\000\000`\"�g6\000\000\000\220�T��*\000\000��T��*\000\000B\000\000\000�*\000\000\000\000\000\000\000\000\000\000 \000\000��*\000\000p�\036\a\000\000\000\000\064\201���*\000\000\020\000\000\000\060\000\000\000 �tQ", '\000' <repeats 12 times>, "�Lge6\000\000\000\t\000\000\000\000\000\000\000��±�*", '\000' <repeats 18 times>, "\003\000\000\000\000\000\000\000"... nentries = 0 pnentries = 83 flag_search_base_found = 0 ---Type <return> to continue, or q <return> to quit--- flag_no_such_object = 0 flag_referral = <value optimized out> flag_psearch = <value optimized out> err_code = 0 ctrlp = 0x2aaab1d4cbd0 ctl_value = 0x0 iscritical = 0 be_name = 0x0 index = -1 sent_result = 0 pr_stat = 0 pagesize = -1 curr_search_count = 0 pr_be = <value optimized out> pr_search_result = 0x0 #12 0x00000000004251ef in do_search (pb=0x2aaab22e8960) at ldap/servers/slapd/search.c:381 operation = 0x2aaab25c3d50 ber = 0x2aaab25c3ed0 err = <value optimized out> attrsonly = 0 scope = 2 deref = 3 ---Type <return> to continue, or q <return> to quit--- sizelimit = 0 timelimit = 0 rawbase = 0x2aaab236bf20 "dc=people,dc=ds,dc=example,dc=com" base = 0x2aaab236bf20 "dc=people,dc=ds,dc=example,dc=com" fstr = 0x2aaab15159c0 "(&(objectClass=inetOrgPerson)(memberOf=cn=app_partners_access,dc=groups,dc=ds,dc=example,dc=com)(!(modifyTimestamp<=20100920230533Z)))" filter = 0x2aaab2044b50 attrs = 0x2aaab187f890 gerattrs = 0x0 psearch = 0 psbvp = 0x0 changetypes = -1408393860 send_entchg_controls = 10922 changesonly = 0 rc = 0 original_base = 0x2aaab236bf20 "dc=people,dc=ds,dc=example,dc=com" new_base = 0x0 baselen = 34 strict = <value optimized out> #13 0x00000000004134be in connection_threadmain () at ldap/servers/slapd/connection.c:606 is_timedout = <value optimized out> ---Type <return> to continue, or q <return> to quit--- curtime = 46912609867056 pb = 0x2aaab22e8960 interval = 10000 conn = 0x2aaaaaab1410 op = 0x2aaab25c3d50 tag = 99 thread_turbo_flag = 0 ret = <value optimized out> more_data = 0 replication_connection = 0 #14 0x0000003667e284ad in ?? () from /usr/lib64/libnspr4.so No symbol table info available. #15 0x0000003665e0673d in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #16 0x00000036656d3d1d in clone () from /lib64/libc.so.6 No symbol table info available. It's dumped again this afternoon - looks similar to me: (gdb) bt full #0 comp_cmp (s1=0x1c72f6c0 "memberOf", s2=0x0) at ldap/servers/slapd/attr.c:98 No locals. #1 0x00002b77a3d2f96c in slapi_attr_type_cmp (a1=0x1c72f6c0 "memberOf", a2=0x0, opt=109) at ldap/servers/slapd/attr.c:131 rc = <value optimized out> #2 0x00002b77a3d4b5fa in test_ava_filter (pb=0x0, e=0x2aaaacfd7a40, a=0x2aaaae0cefc0, ava=0x1cdf6f50, ftype=163, verify_access=0, only_check_access=0, access_check_done=0x4a7b8b48) at ldap/servers/slapd/filterentry.c:342 rc = -1 #3 0x00002b77a3d9ebda in vattr_test_filter (pb=0x2aaaac527580, e=0x2aaaacfd7a40, f=0x1cdf6f30, filter_type=FILTER_TYPE_AVA, type=0x1c72f6c0 "memberOf") at ldap/servers/slapd/vattr.c:605 acl_test_done = 0 rc = -1 list = 0x0 sdn = <value optimized out> be = <value optimized out> namespace_dn = <value optimized out> #4 0x00002b77a3d4bc08 in slapi_vattr_filter_test_ext_internal ( pb=0x2aaaac527580, e=0x2aaaacfd7a40, f=0x1cdf6f30, verify_access=0, only_check_access=<value optimized out>, access_check_done=0x4a7b8d14) at ldap/servers/slapd/filterentry.c:894 ---Type <return> to continue, or q <return> to quit--- rc = 0 #5 0x00002b77a3d4bf92 in vattr_test_filter_list (pb=0x2aaaac527580, e=0x2aaaacfd7a40, flist=0x1cdf6f30, ftype=161, verify_access=0, only_check_access=0, access_check_done=0x4a7b8d14) at ldap/servers/slapd/filterentry.c:1035 nomatch = 1 f = 0x1cdf6f30 #6 0x00002b77a3d4bd8c in slapi_vattr_filter_test_ext_internal ( pb=0x2aaaac527580, e=0x2aaaacfd7a40, f=0x1cdf6ea0, verify_access=109, only_check_access=0, access_check_done=0x4a7b8d14) at ldap/servers/slapd/filterentry.c:962 rc = 0 #7 0x00002b77a3d4bf92 in vattr_test_filter_list (pb=0x2aaaac527580, e=0x2aaaacfd7a40, flist=0x1cdf6ea0, ftype=160, verify_access=0, only_check_access=0, access_check_done=0x4a7b8d14) at ldap/servers/slapd/filterentry.c:1035 nomatch = 1 f = 0x1cdf6ea0 #8 0x00002b77a3d4bd8c in slapi_vattr_filter_test_ext_internal ( pb=0x2aaaac527580, e=0x2aaaacfd7a40, f=0x1c72f870, verify_access=109, only_check_access=0, access_check_done=0x4a7b8d14) at ldap/servers/slapd/filterentry.c:962 rc = 0 ---Type <return> to continue, or q <return> to quit--- #9 0x00002b77a3d4c0c2 in slapi_vattr_filter_test_ext (pb=0x2aaaac527580, e=0x2aaaacfd7a40, f=0x1c72f870, verify_access=1, only_check_access=<value optimized out>) at ldap/servers/slapd/filterentry.c:824 rc = 44 access_check_done = 0 #10 0x00002b77a863cba6 in ldbm_back_next_search_entry_ext (pb=0x2aaaac527580, use_extension=0) at ldap/servers/slapd/back-ldbm/ldbm_search.c:1356 filter_test = -1 be = 0x1a4cd830 inst = 0x1a4cd2f0 li = 0x1a48b690 scope = 2 managedsait = 1 attr = 0x0 filter = 0x1c72f870 base = 0x1cafd260 "dc=people,dc=ds,dc=example,dc=com" sr = 0x1c72f830 id = 83901 e = 0x2aaaae0ced60 nentries = <value optimized out> curtime = <value optimized out> stoptime = 1285251598 ---Type <return> to continue, or q <return> to quit--- optime = 1285247998 tlimit = 3600 llimit = -1 slimit = 997801 isroot = 0 urls = 0x0 err = 0 basesdn = {flag = 0 '\000', dn = 0x1cafd260 "dc=people,dc=ds,dc=example,dc=com", ndn = 0x1cafd260 "dc=people,dc=ds,dc=example,dc=com", ndn_len = 34} target_uniqueid = 0x0 rc = <value optimized out> #11 0x00002b77a3d6aec1 in iterate (pb=0x2aaaac527580, be=0x1a4cd830, send_result=1, pnentries=0x4a7bb6a4, pagesize=-1, pr_statp=0x4a7bb698) at ldap/servers/slapd/opshared.c:1088 gerentry = 0x1c72f370 operation = 0x2aaaac04afd0 rc = 0 rval = 1 attrsonly = 0 done = 0 e = 0x2aaaacfd72e0 attrs = 0x0 ---Type <return> to continue, or q <return> to quit--- pr_stat = 0 #12 0x00002b77a3d6b3f2 in send_results_ext (pb=0x2aaaac527580, send_result=1, nentries=0x4a7bb6a4, pagesize=-1, pr_stat=0x4a7bb698) at ldap/servers/slapd/opshared.c:1461 be = 0x1a4cd830 rc = <value optimized out> #13 0x00002b77a3d6bdb5 in op_shared_search (pb=0x2aaaac527580, send_result=1) at ldap/servers/slapd/opshared.c:688 be_suffix = 0x1a4cd940 err = 0 next_be = 0x0 base = 0x1c732fd0 "dc=people,dc=ds,dc=example,dc=com" fstr = 0x1c72f370 "(&(|(memberOf=cn=Staff,dc=network,dc=example,dc=application,dc=ds,dc=example,dc=com)(memberOf=cn=Enterprise,dc=network,dc=example,dc=application,dc=ds,dc=example,dc=com))(objectClass=inetOrgPerson"... scope = 2 be = 0x1a4cd830 be_single = 0x0 be_list = {0x1a4cd830, 0x0, 0x0, 0x1cafd460, 0x1c72f290, 0x1c72f280, 0x30, 0x1cafd430, 0xe0, 0x3665674632, 0x0, 0x36659529e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x36659529e0, 0x1cafd440, 0xc9, 0x2, 0x1cafd430, 0x1cafd520, 0x3665675292, 0x20, 0x30, 0x1cafd548, 0x4a7bb440, 0x2b77a3da058b, 0x366b229702, 0x0, 0x40000001cafd440, 0x1cafd548, ---Type <return> to continue, or q <return> to quit--- 0x4a7bb440, 0x1cafd548, 0x4a7bb440, 0x4, 0x366b2297fd, 0x0, 0xa00000000, 0xd00000000000000, 0x1cafd548, 0x2b77a3da0589, 0x2b77a3da058b, 0x4, 0x366b229d50, 0x2b77a3da82cc, 0xa9, 0x4a7bb700, 0x1cafd520, 0x2, 0x3667e122d3, 0x0, 0x2b77a3da82d0, 0xffffff57, 0x0, 0x4a7bb700, 0x3667e11344, 0x0, 0x36659529e0, 0x1bfab860, 0x4a7bb740, 0x0, 0x1bfab850, 0x2aaaac04b43b, 0x2aaaac04b467, 0x20, 0x30, 0x0, 0x4a7bb590, 0x1cdf6f30, 0x366b229702, 0x3665952a48, 0xa30000001bfab860, 0x0, 0x4a7bb590, 0xe, 0x4a7bb590, 0xa3, 0x366b2297fd, 0x0, 0x0, 0x5800000000000000, 0x0, 0x2aaaac04b150, 0x2aaaac04b150, 0x1cafd550, 0x366b229702, 0x28, 0x40000366b229702, 0x2aaaac04b150, 0x2aaaac04b150, 0x2aaaac04b150, 0x36659529e0, 0x4, 0xd, 0x2aaaac04b150, 0xd, 0x4, 0x366b22c613} referral_list = {0x0 <repeats 62 times>, 0x2f7065532f33325b, 0x3a34313a30313032, 0x302b2032303a3430, 0x205d303031, 0x0 <repeats 15 times>, 0x100, 0x3665952a48, 0x4a7bb2c0, 0x0, 0xe1, 0xd1, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, 0x2f7065532f33325b, 0x1c30313032, 0x302b2035353a3331, 0x205d303031} ebuf = "\000\000\000\000\000\000\000\000\020�{J\000\000\000\000�\000\000\000\000\000\000\000�\227\"k6\000\000\000\060�{J\000\000\000\000\016", '\000' <repeats 11 times>, "w+\000\034", '\000' <repeats 16 times>, "�n�\034\000\000\000\000\060�{J\000\000\000\000P\235\"k6\000\000\000(\000\000\000\000\000\000\000\002\227\"k6", '\000' <repeats 18 times>, "�<�{J\000\000\000\000��{J\000\000\000\000\001\000\000\000\000\000\000\000��{J\000\000\000\000�\000\000\000\000\000\000\00---Type <return> to continue, or q <return> to quit--- 0O\230\"k6\000\000\000��{J\000\000\000\000�Lge6\000\000\000\000\000\000�\000\000\000\201<�{J\000\000\000\000��{J\000\000\000\000"... attrlistbuf = "\"uid\"\000\000\000P�{J\000\000\000\000H�D\000\000\000\000\000\002\227\"k6\000\000\000[23/Sep/2010:14\004\001\000\000\000\000\000\000\000P�{J\000\000\000\000\001\000\000\000\000\000\000\000P�{J\000\000\000\000\004\000\000\000\000\000\000\000�\227\"k6\000\000\000\020\000\000\000\000\000\000\000\020", '\000' <repeats 14 times>, "\003\001\000\000\000\000\000\000\000G�D\000\000\000\000\000H�D", '\000' <repeats 13 times>, "P\235\"k6\000\000\000�\000\000\000\000\000\000\000\004", '\000' <repeats 47 times>, "^\020�g6\000\000\000\000\000\000\000\001\000\000\000\020�{J\000\000\000\000��{J"... attrliststr = 0x2b77a3dadc54 "ALL" attrs = 0x0 rc = 0 internal_op = <value optimized out> sdn = {flag = 4 '\004', dn = 0x1c732fd0 "dc=people,dc=ds,dc=example,dc=com", ndn = 0x1cafd260 "dc=people,dc=ds,dc=example,dc=com", ndn_len = 34} operation = 0x2aaaac04afd0 referral = <value optimized out> errorbuf = "\000�\004��*\000\000g�\004��*\000\000\000\000\000\000\001\000\000\000�\000\000\000\000\000\000\000H*\225e6\000\000\000P�{J\000\000\000\000P`���*\000\000\221\000\000\000\000\000\000\000\200\000\000\000\000\000\000\000\000\217�\000\000��Z", '\000' <repeats 17 times>, "�{J\000\000\000\000Hկ\034", '\00---Type <return> to continue, or q <return> to quit--- 0' <repeats 12 times>, "�)\225e6\000\000\000PL7\032\000\000\000\000\f\000\000\000\000\000\000\000P`���*\000\000��{J\000\000\000\000P�\004��*\000\000�Lge6\000\000\000\f\000\000\000\000\000\000\000\000�{J\000\000\000\000\000�{J\000\000\000\000(\000\000\000\000\000\000\000P"... nentries = 0 pnentries = 2199 flag_search_base_found = 0 flag_no_such_object = 0 flag_referral = <value optimized out> flag_psearch = <value optimized out> err_code = 0 ctrlp = 0x1cdf9540 ctl_value = 0x0 iscritical = 0 be_name = 0x0 index = -1 sent_result = 0 pr_stat = 0 pagesize = -1 curr_search_count = 0 pr_be = <value optimized out> pr_search_result = 0x0 #14 0x00000000004251ef in do_search (pb=0x2aaaac527580) ---Type <return> to continue, or q <return> to quit--- at ldap/servers/slapd/search.c:381 operation = 0x2aaaac04afd0 ber = 0x2aaaac04b150 err = <value optimized out> attrsonly = 0 scope = 2 deref = 3 sizelimit = 2147483647 timelimit = 0 rawbase = 0x1c732fd0 "dc=people,dc=ds,dc=example,dc=com" base = 0x1c732fd0 "dc=people,dc=ds,dc=example,dc=com" fstr = 0x1c72f370 "(&(|(memberOf=cn=Staff,dc=network,dc=example,dc=application,dc=ds,dc=example,dc=com)(memberOf=cn=Enterprise,dc=network,dc=example,dc=application,dc=ds,dc=example,dc=com))(objectClass=inetOrgPerson"... filter = 0x1c72f870 attrs = 0x0 gerattrs = 0x0 psearch = 0 psbvp = 0x0 changetypes = -1392387300 send_entchg_controls = 10922 changesonly = 0 rc = 0 ---Type <return> to continue, or q <return> to quit--- original_base = 0x1c732fd0 "dc=people,dc=ds,dc=example,dc=com" new_base = 0x0 baselen = 34 strict = <value optimized out> #15 0x00000000004134be in connection_threadmain () at ldap/servers/slapd/connection.c:606 is_timedout = <value optimized out> curtime = 477311024 pb = 0x2aaaac527580 interval = 10000 conn = 0x2aaaaaab6050 op = 0x2aaaac04afd0 tag = 99 thread_turbo_flag = 0 ret = <value optimized out> more_data = 0 replication_connection = 0 #16 0x0000003667e284ad in ?? () from /usr/lib64/libnspr4.so No symbol table info available. #17 0x0000003665e0673d in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #18 0x00000036656d3d1d in clone () from /lib64/libc.so.6 No symbol table info available.
Here are the instructions I wrote for when we setup the new server in case it's of any use or shows an error on my part. yum install 389-ds setup-ds-admin.pl Custom setup type Accept default computer name (ds2.man.example.com) Accept default system user, system group Accept default of not registering with an existing config server admin ID=admin admin password .... administration domain = ds.example.com Accept default port of 389 Accept default Directory server identifier (ds2) Set Suffix = "dc=ds, dc=example, dc=com" Accept default Directory Manager DN (cn=Directory Manager) Set Directory Manager password .... Accept default to not install example entries Do not attempt to import the LDIF file - (or if you do, check the log) Accept default admin port (9830) Accept default bind address (0.0.0.0) Accept default to run admin server as nobody run "389-consolei -x nologo" (java app) from the CLI admin / ... / localhost:9830 To import LDIF into LDAP see: http://www.redhat.com/docs/manuals/dir-server/8.1/admin/Populating_Directory_Databases.html#Populating_Directory_Databases-Importing_Data If you get errors about "violates attribute syntax" then check the schema and Adrian's ldif_fix script. We used the first option for troubleshooting: /etc/init.d/dirsrv stop /usr/lib64/dirsrv/slapd-ds2/ldif2db -n userRoot -i /tmp/exportfixed5.ldif Run 389-console, open up the server, Configuration Tab, Data->(dc=example),userRoot, Add attribute: Add organisation (equality,presence,substring) Add uidNumber (equality,presence) Add gidNumber (equality,presence) Now we need to make some config changes: dn: cn=config,cn=ldbm database,cn=plugins,cn=config nsslapd-lookthroughlimit: -1 (default is 5000 was 5000000) dn: cn=config nsslapd-maxdescriptors: 4096 (default is 1024) nsslapd-sizelimit: 1000000 Run 389-console, open up the server, Configuration Tab, Plug-ins: attribute uniqueness (enable, arguments are defaults) arguments 1: uid arguments 2: dc=ds,dc=example,dc=com referential integrity postoperation (enable, add argument #8) arguments 1: 0 arguments 2: /var/log/dirsrv/slapd-ds2/referint arguments 3: 0 arguments 4: member arguments 5: uniquemember arguments 6: owner arguments 7: seeAlso arguments 8: manager Space Insensitive String Syntax (enable) MemberOf Plugin (enable + change below) Advanced -> memberofgroupattr: uniqueMember (default is member) You must "Save" each plugin /etc/init.d/dirsrv restart echo "* - nofile 8192" > /etc/security/limits.d/ldap.conf su - /etc/init.d/dirsrv restart
Thanks for the detailed instructions and stack traces. Are there any other operations reported in the access log from around the time of the crash? Is this server a consumer for replication? Do you ever use the fixup-memberof.pl script to add memberOf attributes to entries that don't yet have them but should? The server runs fine for a while, then crashes? Does it seem to be a search related to member, uniquemember, or memberOf that causes it to crash? The stack trace is odd - the value should never be NULL at this point. Seems like it could be a race condition with another thread. If you can get another crash, try this in gdb: thread apply all bt full and attach the output of that as an attachment to this bug. I've tried various combinations of searches with memberOf enabled, but I still cannot reproduce the crash.
Created attachment 915155 [details] Comment (This comment was longer than 65,535 characters and has been moved to an attachment by Red Hat Bugzilla).
Thanks for the very detailed stack trace. This proves that this is a duplicate of 631862. I'm going to target this bug for 1.2.6.1 *** This bug has been marked as a duplicate of bug 631862 ***
fabulous :-) Bit annoyed I didn't suggest that bug - I saw it when looking at the old server and was going to try turning off the referential checking. I wasn't aware of any deletes that occurred but it's certainly possible.
(In reply to comment #7) > fabulous :-) Bit annoyed I didn't suggest that bug - I saw it when looking at > the old server and was going to try turning off the referential checking. I > wasn't aware of any deletes that occurred but it's certainly possible. I think you have shown that the problem can occur without deletes - it's really a problem of loading an entry into the cache and "fixing it up" in a non-thread safe manner.