Hi, I am running dirsrv 1.2.6-1 on rhel 5.5 and experiencing random crashes. I can't seem to pin it down to a set a reproducible steps. I have managed to get a core file. See below #0 0x0000003938230265 in raise () from /lib64/libc.so.6 #1 0x0000003938231d10 in abort () from /lib64/libc.so.6 #2 0x000000393826a84b in __libc_message () from /lib64/libc.so.6 #3 0x0000003938272886 in free () from /lib64/libc.so.6 #4 0x00002b3410ebe1c6 in slapi_ch_free (ptr=0x2aab0ce4b460) at ldap/servers/slapd/ch_malloc.c:363 #5 0x00002b3410eb95b3 in attr_done (a=0x5ee7) at ldap/servers/slapd/attr.c:353 #6 0x00002b3410eb962b in slapi_attr_free (ppa=<value optimized out>) at ldap/servers/slapd/attr.c:341 #7 0x00002b3410eb9de4 in attrlist_free (alist=<value optimized out>) at ldap/servers/slapd/attrlist.c:53 #8 0x00002b3410ecbd10 in slapi_entry_free (e=0x2aaab3e2cb30) at ldap/servers/slapd/entry.c:1918 #9 0x00002b3410eb67fd in op_shared_add (pb=0x2aab057aed00) at ldap/servers/slapd/add.c:718 #10 0x00002b3410eb7ef8 in do_add (pb=0x2aab057aed00) at ldap/servers/slapd/add.c:259 #11 0x00000000004135a3 in connection_threadmain () at ldap/servers/slapd/connection.c:563 #12 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #13 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #14 0x00000039382d3f6d in clone () from /lib64/libc.so.6 (gdb) where #0 0x0000003938230265 in raise () from /lib64/libc.so.6 #1 0x0000003938231d10 in abort () from /lib64/libc.so.6 #2 0x000000393826a84b in __libc_message () from /lib64/libc.so.6 #3 0x0000003938272886 in free () from /lib64/libc.so.6 #4 0x00002b3410ebe1c6 in slapi_ch_free (ptr=0x2aab0ce4b460) at ldap/servers/slapd/ch_malloc.c:363 #5 0x00002b3410eb95b3 in attr_done (a=0x5ee7) at ldap/servers/slapd/attr.c:353 #6 0x00002b3410eb962b in slapi_attr_free (ppa=<value optimized out>) at ldap/servers/slapd/attr.c:341 #7 0x00002b3410eb9de4 in attrlist_free (alist=<value optimized out>) at ldap/servers/slapd/attrlist.c:53 #8 0x00002b3410ecbd10 in slapi_entry_free (e=0x2aaab3e2cb30) at ldap/servers/slapd/entry.c:1918 #9 0x00002b3410eb67fd in op_shared_add (pb=0x2aab057aed00) at ldap/servers/slapd/add.c:718 #10 0x00002b3410eb7ef8 in do_add (pb=0x2aab057aed00) at ldap/servers/slapd/add.c:259 #11 0x00000000004135a3 in connection_threadmain () at ldap/servers/slapd/connection.c:563 #12 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #13 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #14 0x00000039382d3f6d in clone () from /lib64/libc.so.6 (gdb) list 358 if(recording) 359 { 360 remove_memory_record(*ptr); 361 } 362 #endif 363 free (*ptr); 364 *ptr = NULL; 365 if(!counters_created) 366 { 367 create_counters(); (gdb) up #5 0x00002b3410eb95b3 in attr_done (a=0x5ee7) at ldap/servers/slapd/attr.c:353 353 slapi_ch_free((void**)&a->a_type); (gdb) list 348 void 349 attr_done(Slapi_Attr *a) 350 { 351 if(a!=NULL) 352 { 353 slapi_ch_free((void**)&a->a_type); 354 csn_free(&a->a_deletioncsn); 355 slapi_valueset_done(&a->a_present_values); 356 slapi_valueset_done(&a->a_deleted_values); 357 { If you need anymore information from the core file let me know. It's 2GB in size so it's not easy for me to upload it. Regards, Daniel
Can you confirm what version of 389-ds-base you are using? This bug should have been fixed in 389-ds-base-1.2.6.1-2 if you are not sure, just do rpm -qi 389-ds-base and paste the output in this bug
I am running 389-ds-base-1.2.6.1-2: rpm -qi 389-ds-base Name : 389-ds-base Relocations: (not relocatable) Version : 1.2.6.1 Vendor: Fedora Project Release : 2.el5 Build Date: Thu 30 Sep 2010 09:15:13 AM EST Install Date: Mon 18 Oct 2010 03:45:22 PM EST Build Host: x86-02.phx2.fedoraproject.org Group : System Environment/Daemons Source RPM: 389-ds-base-1.2.6.1-2.el5.src.rpm Size : 5855143 License: GPLv2 with exceptions Signature : DSA/SHA1, Fri 01 Oct 2010 01:56:46 AM EST, Key ID 119cc036217521f6 Packager : Fedora Project URL : http://port389.org/ Summary : 389 Directory Server (base) Description : 389 Directory Server is an LDAPv3 compliant server. The base package includes the LDAP server and command line utilities for server administration. $ rpm -qa | grep 389 389-console-1.1.4-1.el5 389-dsgw-1.1.5-1.el5 389-admin-1.1.11-1.el5 389-ds-console-doc-1.2.3-1.el5 389-ds-base-1.2.6.1-2.el5 389-ds-console-1.2.3-1.el5 389-ds-1.2.1-1.el5 389-admin-console-1.1.5-1.el5 389-ds-base-debuginfo-1.2.6.1-2.el5 389-admin-console-doc-1.1.5-1.el5
ok - do this - when examining the core file in gdb, type the following: thread apply all bt and attach the output to this bug as an attachment
I also note that you do not have 389-adminutil installed - do you have adminutil installed? Is this a machine that did/does have redhat-ds installed on it?
The machine doesn't have redhat-ds. 389-adminutil is not installed either. (gdb) thread apply all bt Thread 52 (Thread 24295): #0 0x00000039382cb186 in poll () from /lib64/libc.so.6 #1 0x000000393c6246e4 in PR_Poll () from /usr/lib64/libnspr4.so #2 0x0000000000416009 in slapd_daemon (ports=0x7fff08505270) at ldap/servers/slapd/daemon.c:664 #3 0x000000000041d4c4 in main (argc=7, argv=0x7fff08505608) at ldap/servers/slapd/main.c:1243 Thread 51 (Thread 24298): #0 0x00000039382cd212 in select () from /lib64/libc.so.6 #1 0x00002b3410f21fc9 in DS_Sleep (ticks=<value optimized out>) at ldap/servers/slapd/util.c:788 #2 0x00002b3414742c08 in deadlock_threadmain (param=<value optimized out>) at ldap/servers/slapd/back-ldbm/dblayer.c:3589 #3 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #4 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 50 (Thread 24299): #0 0x00000039382cd212 in select () from /lib64/libc.so.6 #1 0x00002b3410f21fc9 in DS_Sleep (ticks=<value optimized out>) at ldap/servers/slapd/util.c:788 #2 0x00002b3414742e39 in checkpoint_threadmain (param=0xf19dfe0) at ldap/servers/slapd/back-ldbm/dblayer.c:3736 #3 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #4 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 49 (Thread 24300): #0 0x00000039382cd212 in select () from /lib64/libc.so.6 #1 0x00002b3410f21fc9 in DS_Sleep (ticks=<value optimized out>) at ldap/servers/slapd/util.c:788 #2 0x00002b3414742985 in trickle_threadmain (param=<value optimized out>) at ldap/servers/slapd/back-ldbm/dblayer.c:3887 #3 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #4 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 48 (Thread 24301): #0 0x00000039382cd212 in select () from /lib64/libc.so.6 #1 0x00002b3410f21fc9 in DS_Sleep (ticks=<value optimized out>) at ldap/servers/slapd/util.c:788 #2 0x00002b3414743267 in perf_threadmain (param=<value optimized out>) at ldap/servers/slapd/back-ldbm/dblayer.c:3524 #3 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #4 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 47 (Thread 24302): #0 0x0000003938a0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622e2d in PR_WaitCondVar () from /usr/lib64/libnspr4.so #2 0x00002b3410f17799 in slapi_wait_condvar (cvar=0xf5ce750, timeout=0x0) at ldap/servers/slapd/slapi2nspr.c:179 #3 0x00002b3413f0955e in cos_cache_wait_on_change (arg=<value optimized out>) at ldap/servers/plugins/cos/cos_cache.c:478 #4 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #5 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #6 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 46 (Thread 24390): #0 0x00000039382cd212 in select () from /lib64/libc.so.6 #1 0x00002b3410f21fc9 in DS_Sleep (ticks=<value optimized out>) at ldap/servers/slapd/util.c:788 #2 0x00002b3414cd0716 in _cl5DeadlockMain (param=<value optimized out>) at ldap/servers/plugins/replication/cl5_api.c:3223 ---Type <return> to continue, or q <return> to quit--- #3 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #4 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 45 (Thread 24391): #0 0x00000039382cd212 in select () from /lib64/libc.so.6 #1 0x00002b3410f21fc9 in DS_Sleep (ticks=<value optimized out>) at ldap/servers/slapd/util.c:788 #2 0x00002b3414cd0530 in _cl5CheckpointMain (param=<value optimized out>) at ldap/servers/plugins/replication/cl5_api.c:3307 #3 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #4 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 44 (Thread 24392): #0 0x00000039382cd212 in select () from /lib64/libc.so.6 #1 0x00002b3410f21fc9 in DS_Sleep (ticks=<value optimized out>) at ldap/servers/slapd/util.c:788 #2 0x00002b3414cd0447 in _cl5TrickleMain (param=<value optimized out>) at ldap/servers/plugins/replication/cl5_api.c:3335 #3 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #4 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 43 (Thread 24393): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00002b3414cd6c88 in _cl5TrimMain (param=<value optimized out>) at ldap/servers/plugins/replication/cl5_api.c:4239 #4 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #5 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #6 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 42 (Thread 24394): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00002b3414ce7085 in protocol_sleep (prp=0xf5e5e40, duration=300000) at ldap/servers/plugins/replication/repl5_inc_protocol.c:1325 #4 0x00002b3414ce8e86 in repl5_inc_run (prp=0xf5e5e40) at ldap/servers/plugins/replication/repl5_inc_protocol.c:810 #5 0x00002b3414cec374 in prot_thread_main (arg=<value optimized out>) at ldap/servers/plugins/replication/repl5_protocol.c:311 #6 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #7 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #8 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 41 (Thread 24395): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00002b3414ce7085 in protocol_sleep (prp=0xf5f2e50, duration=300000) at ldap/servers/plugins/replication/repl5_inc_protocol.c:1325 #4 0x00002b3414ce8e86 in repl5_inc_run (prp=0xf5f2e50) at ldap/servers/plugins/replication/repl5_inc_protocol.c:810 #5 0x00002b3414cec374 in prot_thread_main (arg=<value optimized out>) at ldap/servers/plugins/replication/repl5_protocol.c:311 #6 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #7 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #8 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 40 (Thread 24396): ---Type <return> to continue, or q <return> to quit--- #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00002b3414ce7085 in protocol_sleep (prp=0xf5f5a70, duration=300000) at ldap/servers/plugins/replication/repl5_inc_protocol.c:1325 #4 0x00002b3414ce8e86 in repl5_inc_run (prp=0xf5f5a70) at ldap/servers/plugins/replication/repl5_inc_protocol.c:810 #5 0x00002b3414cec374 in prot_thread_main (arg=<value optimized out>) at ldap/servers/plugins/replication/repl5_protocol.c:311 #6 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #7 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #8 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 39 (Thread 24397): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00002b3414ce7085 in protocol_sleep (prp=0xf5de690, duration=300000) at ldap/servers/plugins/replication/repl5_inc_protocol.c:1325 #4 0x00002b3414ce8e86 in repl5_inc_run (prp=0xf5de690) at ldap/servers/plugins/replication/repl5_inc_protocol.c:810 #5 0x00002b3414cec374 in prot_thread_main (arg=<value optimized out>) at ldap/servers/plugins/replication/repl5_protocol.c:311 #6 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #7 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #8 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 38 (Thread 24398): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00002b3414ce7085 in protocol_sleep (prp=0x2aaaac002250, duration=300000) at ldap/servers/plugins/replication/repl5_inc_protocol.c:1325 #4 0x00002b3414ce8e86 in repl5_inc_run (prp=0x2aaaac002250) at ldap/servers/plugins/replication/repl5_inc_protocol.c:810 #5 0x00002b3414cec374 in prot_thread_main (arg=<value optimized out>) at ldap/servers/plugins/replication/repl5_protocol.c:311 #6 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #7 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #8 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 37 (Thread 24399): #0 0x0000003938a0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622e2d in PR_WaitCondVar () from /usr/lib64/libnspr4.so #2 0x00002b3410f17799 in slapi_wait_condvar (cvar=0x2aaab0001d40, timeout=0x0) at ldap/servers/slapd/slapi2nspr.c:179 #3 0x00002b341619486f in roles_cache_wait_on_change (arg=<value optimized out>) at ldap/servers/plugins/roles/roles_cache.c:432 #4 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #5 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #6 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 36 (Thread 24400): #0 0x0000003938a0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622e2d in PR_WaitCondVar () from /usr/lib64/libnspr4.so #2 0x00002b3410f17799 in slapi_wait_condvar (cvar=0x2aaab00035a0, timeout=0x0) at ldap/servers/slapd/slapi2nspr.c:179 #3 0x00002b341619486f in roles_cache_wait_on_change (arg=<value optimized out>) at ldap/servers/plugins/roles/roles_cache.c:432 #4 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #5 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #6 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 35 (Thread 24401): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x0000000000419673 in housecleaning (cur_time=<value optimized out>) at ldap/servers/slapd/house.c:77 #4 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #5 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #6 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 34 (Thread 24402): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00002b3410ed12ff in eq_loop (arg=<value optimized out>) at ldap/servers/slapd/eventq.c:355 #4 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #5 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #6 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 33 (Thread 24403): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x4b4c5098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 32 (Thread 24404): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x4bec6098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 31 (Thread 24406): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x4d2c8098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 30 (Thread 24407): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x4dcc9098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 ---Type <return> to continue, or q <return> to quit--- #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 29 (Thread 24408): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x4e6ca098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 28 (Thread 24409): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x4f0cb098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 27 (Thread 24410): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x4facc098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 26 (Thread 24411): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x504cd098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 25 (Thread 24412): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x50ece098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 24 (Thread 24413): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x518cf098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 23 (Thread 24414): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x522d0098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 22 (Thread 24415): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x52cd1098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 21 (Thread 24416): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x536d2098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 20 (Thread 24417): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x540d3098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 ---Type <return> to continue, or q <return> to quit--- Thread 19 (Thread 24418): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x54ad4098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 18 (Thread 24419): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x554d5098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 17 (Thread 24420): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x55ed6098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 16 (Thread 24421): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x568d7098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 15 (Thread 24422): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x572d8098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 14 (Thread 24423): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x57cd9098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 13 (Thread 24424): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x586da098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 12 (Thread 24425): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x590db098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 11 (Thread 24426): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x59adc098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 10 (Thread 24427): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x5a4dd098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 9 (Thread 24428): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so ---Type <return> to continue, or q <return> to quit--- #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x5aede098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 8 (Thread 24429): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x5b8df098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 7 (Thread 24430): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x5c2e0098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 6 (Thread 24431): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x5cce1098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 5 (Thread 24432): #0 0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so #2 0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so #3 0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x5d6e2098, interval=10000) at ldap/servers/slapd/connection.c:1695 #4 0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173 #5 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #6 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #7 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 4 (Thread 24433): #0 0x00000039382cd212 in select () from /lib64/libc.so.6 #1 0x00002b3410f21fc9 in DS_Sleep (ticks=<value optimized out>) at ldap/servers/slapd/util.c:788 #2 0x0000000000416c0c in time_thread (nothing=<value optimized out>) at ldap/servers/slapd/daemon.c:466 #3 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #4 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- #5 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 3 (Thread 28531): #0 0x00000039382729ed in _int_malloc () from /lib64/libc.so.6 #1 0x0000003938274cde in malloc () from /lib64/libc.so.6 #2 0x0000003938279772 in strdup () from /lib64/libc.so.6 #3 0x00002b3410ebe382 in slapi_ch_strdup (s1=0xf0d6820 "sn") at ldap/servers/slapd/ch_malloc.c:277 #4 0x00002b3410eb981d in slapi_attr_init_locking_optional (a=0x11f68360, type=0x2aaaf80e6c56 "sn", use_lock=0) at ldap/servers/slapd/attr.c:274 #5 0x00002b3410eb9fbf in attrlist_find_or_create_locking_optional (alist=<value optimized out>, type=<value optimized out>, a=0x5eae1d20, use_lock=0) at ldap/servers/slapd/attrlist.c:85 #6 0x00002b3410ecebce in str2entry_fast (rawdn=0x0, s=0x0, flags=8, read_stateinfo=-1) at ldap/servers/slapd/entry.c:402 #7 0x00002b3410ecee4d in slapi_str2entry_ext (dn=0x10d1da70 "uid=slvi,ou=People,dc=deakin,dc=edu,dc=au", s=0x2aaaf80e6690 "rdn", flags=8) at ldap/servers/slapd/entry.c:1381 #8 0x00002b341477d737 in ldbm_back_ldbm2ldif (pb=<value optimized out>) at ldap/servers/slapd/back-ldbm/ldif2ldbm.c:1468 #9 0x00002b3410f1dca5 in task_export_thread (arg=<value optimized out>) at ldap/servers/slapd/task.c:1004 #10 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #11 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #12 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 2 (Thread 29560): #0 0x00000039382729fb in _int_malloc () from /lib64/libc.so.6 #1 0x0000003938274cde in malloc () from /lib64/libc.so.6 #2 0x0000003938279772 in strdup () from /lib64/libc.so.6 #3 0x00002b3410ebe382 in slapi_ch_strdup (s1=0x2aab013f3310 "cn=aaaexam,ou=auto.home,ou=Automount,dc=deakin,dc=edu,dc=au") at ldap/servers/slapd/ch_malloc.c:277 #4 0x00002b3410ec497e in slapi_sdn_get_ndn (sdn=0x5f4e2c80) at ldap/servers/slapd/dn.c:1937 #5 0x00002b3410eeadfe in sdn_is_nulldn (sdn=0x0) at ldap/servers/slapd/mapping_tree.c:1941 #6 0x00002b3410eeb348 in slapi_get_mapping_tree_node_by_dn (dn=0x5f4e2c80) at ldap/servers/slapd/mapping_tree.c:2746 #7 0x00002b3410eeb409 in slapi_get_suffix_by_dn (dn=0x0) at ldap/servers/slapd/mapping_tree.c:3050 #8 0x00002b3410f08f4c in _slapi_rdn_init_all_dn_ext (rdn=0x2aab016e2940, sdn=0x5f4e2c80) at ldap/servers/slapd/rdn.c:144 #9 0x00002b3410f09331 in slapi_rdn_init_all_dn (rdn=0x2aab016e2940, dn=0x2aab018dd750 "cn=aaaexam,ou=auto.home,ou=Automount,dc=deakin,dc=edu,dc=au") at ldap/servers/slapd/rdn.c:238 #10 0x00002b3410ece7bd in str2entry_fast (rawdn=0x155bfd90 "cn=aaaexam,ou=auto.home,ou=Automount,dc=deakin,dc=edu,dc=au", s=0x0, flags=8, read_stateinfo=-1) at ldap/servers/slapd/entry.c:297 #11 0x00002b3410ecee4d in slapi_str2entry_ext (dn=0x155bfd90 "cn=aaaexam,ou=auto.home,ou=Automount,dc=deakin,dc=edu,dc=au", s=0x2aaaac21f120 "rdn", flags=8) at ldap/servers/slapd/entry.c:1381 #12 0x00002b341477d737 in ldbm_back_ldbm2ldif (pb=<value optimized out>) at ldap/servers/slapd/back-ldbm/ldif2ldbm.c:1468 #13 0x00002b3410f1dca5 in task_export_thread (arg=<value optimized out>) at ldap/servers/slapd/task.c:1004 #14 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #15 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #16 0x00000039382d3f6d in clone () from /lib64/libc.so.6 Thread 1 (Thread 24405): #0 0x0000003938230265 in raise () from /lib64/libc.so.6 #1 0x0000003938231d10 in abort () from /lib64/libc.so.6 #2 0x000000393826a84b in __libc_message () from /lib64/libc.so.6 #3 0x0000003938272886 in free () from /lib64/libc.so.6 #4 0x00002b3410ebe1c6 in slapi_ch_free (ptr=0x2aab0ce4b460) at ldap/servers/slapd/ch_malloc.c:363 #5 0x00002b3410eb95b3 in attr_done (a=0x5ee7) at ldap/servers/slapd/attr.c:353 #6 0x00002b3410eb962b in slapi_attr_free (ppa=<value optimized out>) at ldap/servers/slapd/attr.c:341 #7 0x00002b3410eb9de4 in attrlist_free (alist=<value optimized out>) at ldap/servers/slapd/attrlist.c:53 #8 0x00002b3410ecbd10 in slapi_entry_free (e=0x2aaab3e2cb30) at ldap/servers/slapd/entry.c:1918 #9 0x00002b3410eb67fd in op_shared_add (pb=0x2aab057aed00) at ldap/servers/slapd/add.c:718 #10 0x00002b3410eb7ef8 in do_add (pb=0x2aab057aed00) at ldap/servers/slapd/add.c:259 ---Type <return> to continue, or q <return> to quit--- #11 0x00000000004135a3 in connection_threadmain () at ldap/servers/slapd/connection.c:563 #12 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so #13 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0 #14 0x00000039382d3f6d in clone () from /lib64/libc.so.6
Do you have ou=Automount,dc=deakin,dc=edu,dc=au and ou=People,dc=deakin,dc=edu,dc=au in separate databases? How did you invoke these export operations? Did you run db2ldif.pl from the command line? Did you use the console to export the database? Do you know what the add operation is for thread 1? In the debugger, you can do (gdb) thread 1 (gdb) up (gdb) up ... # keep going up until you get to slapi_entry_free (gdb) print *e I would like to find out what the DN is. Are you running RHEL5 or CentOS5? Looks like 64-bit, correct?
Yes People and Automount use separate databases. We use db2ldif.pl to export the databases from the command line. Thread 1: (gdb) up #8 0x00002b3410ecbd10 in slapi_entry_free (e=0x2aaab3e2cb30) at ldap/servers/slapd/entry.c:1918 1918 attrlist_free(e->e_attrs); (gdb) print *e $1 = {e_sdn = {flag = 16 '\020', dn = 0x0, ndn = 0x0, ndn_len = 0}, e_srdn = {flag = 0 '\000', rdn = 0x0, rdns = 0x0, butcheredupto = -1, nrdn = 0x0, all_rdns = 0x0, all_nrdns = 0x0}, e_uniqueid = 0x0, e_dncsnset = 0x0, e_maxcsn = 0x0, e_attrs = 0x2aab0ce4b460, e_deleted_attrs = 0x0, e_virtual_attrs = 0x0, e_virtual_watermark = 0, e_virtual_lock = 0x2aab0ce64470, e_extension = 0x0, e_flags = 0 '\000', e_aux_attrs = 0x0} (gdb) We are running RHEL 5 64bit.
Daniel, we ran a series of stress tests repeating 2 db2ldif.pl and adding (using a test tool "infadd"), but we have no luck to reproduce the crash so far. There could be some conditions to trigger the problem. It looks the crashed server is a master. Is it a single master or one of MMR servers? Are there any special configuration you could think of? For instance, password policy, CoS/Roles plugin, memberOf plugin, etc.? Any customer schema? Thanks for your help, in advance.
Hi Noriko, The server is a single master. The environment also has one consumer. The special config we have is: - We disable all password/account lockout policies and prevent users from changing their own password. - We have multiple custom schemas we apply. We also index attributes on the custom schemas - The file descriptors have been increased to 8000 - We increase the database cache size to: 52428800 The rest of the config is default from install. I am not sure if it's related but it seems hourly the task created by db2ldif.pl takes a long amount of time (about 30+ seconds) to complete. This is strange as it normally takes about 5 seconds to complete. Does db2ldif.pl use any sort of caching system? Regards, Daniel
> I am not sure if it's related but it seems hourly the task created by > db2ldif.pl takes a long amount of time (about 30+ seconds) to complete. This is > strange as it normally takes about 5 seconds to complete. Could you explain this a bit more? You create a task with db2ldif.pl hourly? Sometimes it takes 5 seconds and sometimes it takes 30+ seconds? Is there anything different about the times it takes 5 seconds and the times it takes 30+ seconds? Noriko, if this is cache related, you might try making the cache size the minimum size, to see if this is somehow related to swapping entries into and out of the cache during the export + add operations.
(In reply to comment #11) > Noriko, if this is cache related, you might try making the cache size the > minimum size, to see if this is somehow related to swapping entries into and > out of the cache during the export + add operations. All right. Let me try...
Our script runs an export every 5mins and diffs the ldif file with a generated one (from an external database which holds our user information). It's nice a quick during the hour but on the hour it takes a quite a long time. It's seems like the cache is expiring on the hour causing it to be much slower. Normally the export looks like this (doing ldapsearch in a loop on the task entry): dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 21000 entries (20%). dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 24000 entries (22%). dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 27000 entries (25%). dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 30000 entries (41%). dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 34000 entries (83%). dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 38000 entries (87%). dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 40000 entries (88%). dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 44000 entries (91%). dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 45000 entries (92%). dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 46000 entries (93%). dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 49000 entries (95%). dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 51000 entries (97%). dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 53000 entries (98%). nstaskexitcode: 0 nstaskstatus: userRoot: Processed 54855 entries (100%). However on a slow run it takes a lot longer: dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 12000 entries (13%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 15000 entries (15%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 19000 entries (18%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 21000 entries (20%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 23000 entries (22%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 26000 entries (24%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 27000 entries (25%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 30000 entries (41%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 34000 entries (83%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 34000 entries (83%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 35000 entries (84%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 35000 entries (84%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 36000 entries (85%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 37000 entries (86%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 38000 entries (87%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 39000 entries (88%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 40000 entries (88%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 41000 entries (89%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 42000 entries (90%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 43000 entries (91%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 44000 entries (91%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 45000 entries (92%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 46000 entries (93%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 47000 entries (94%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 48000 entries (94%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 49000 entries (95%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 50000 entries (96%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 51000 entries (97%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 52000 entries (97%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 53000 entries (98%). dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 54000 entries (99%). nstaskexitcode: 0 nstaskstatus: userRoot: Processed 54855 entries (100%).
Can you provide the arguments to the db2ldif.pl command that you use?
Also, would it be possible for you to try to reproduce the crash with 389-ds-base 1.2.7.5? If this is memory related, it may have been fixed when we did the Coverity fixes for 1.2.7
Hi, The command we use is db2ldif.pl -1 -u -N -D "cn=Directory Manager" -n userRoot -w "DIRMANAGERPWD" -a /tmp/ldapexport.ldif I'll upgrade one of our environments but I wasn't able to reproduce it on the spot last time. Regards, Daniel
(In reply to comment #17) > Hi, > > The command we use is > > db2ldif.pl -1 -u -N -D "cn=Directory Manager" -n userRoot -w "DIRMANAGERPWD" -a > /tmp/ldapexport.ldif > > I'll upgrade one of our environments but I wasn't able to reproduce it on the > spot last time. > > Regards, > Daniel In the stack trace in https://bugzilla.redhat.com/show_bug.cgi?id=656734#c5 there are two different export jobs running at the same time for different databases. cn=aaaexam,ou=auto.home,ou=Automount,dc=deakin,dc=edu,dc=au uid=slvi,ou=People,dc=deakin,dc=edu,dc=au In https://bugzilla.redhat.com/show_bug.cgi?id=656734#c7 you said these were in separate databases. I would assume that userRoot is the ou=People database. Is there a separate db2ldif.pl job for the ou=Automount database?
Yes, automountData is our database for ou=Automount. We also have netgroupData for ou=Netgroup groupData for ou=Groups. Regards, Daniel
(In reply to comment #19) > Yes, automountData is our database for ou=Automount. > > We also have > > netgroupData for ou=Netgroup > groupData for ou=Groups. > > Regards, > Daniel The core file showed two db2ldif jobs running - one for the ou=People database and one for the ou=Automount database - are these two separate cron jobs? What are the arguments for the ou=Automount db2ldif?
The arguments are the same for each job. There is a separate job for each database. We have spaced out the intervals the jobs have run but it looks like they may have overlapped (probably due to the slow run on the hour).
Ok. So it looks as though the following happens: 1) first userRoot db2ldif cron job runs and runs very, very slowly 2) second ou=Automount db2ldif cron job runs before userRoot one finishes 3) there is an ldap ADD operation I've tried tests using this sequence of operations (and several more) - no luck reproducing the problem.
Any luck reproducing with 1.2.7?
I have not been able to reproduce this with 1.2.7. Feel free to close this one. If it does occur again in the future i'll submit another bug. Thanks, Daniel
(In reply to comment #24) > I have not been able to reproduce this with 1.2.7. Feel free to close this one. > If it does occur again in the future i'll submit another bug. > > Thanks, > Daniel Excellent. Thanks Daniel!