Bug 656734 - dirsrv crash signal 4
Summary: dirsrv crash signal 4
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: 389
Classification: Retired
Component: Directory Server
Version: 1.2.6
Hardware: x86_64
OS: Linux
high
medium
Target Milestone: ---
Assignee: Rich Megginson
QA Contact: Chandrasekar Kannan
URL:
Whiteboard:
Depends On:
Blocks: 639035 389_1.2.8
TreeView+ depends on / blocked
 
Reported: 2010-11-24 00:28 UTC by Daniel
Modified: 2015-01-04 23:44 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-02-24 01:43:05 UTC


Attachments (Terms of Use)

Description Daniel 2010-11-24 00:28:01 UTC
Hi,

I am running dirsrv 1.2.6-1 on rhel 5.5 and experiencing random crashes. I can't seem to pin it down to a set a reproducible steps. I have managed to get a core file. See below


#0  0x0000003938230265 in raise () from /lib64/libc.so.6
#1  0x0000003938231d10 in abort () from /lib64/libc.so.6
#2  0x000000393826a84b in __libc_message () from /lib64/libc.so.6
#3  0x0000003938272886 in free () from /lib64/libc.so.6
#4  0x00002b3410ebe1c6 in slapi_ch_free (ptr=0x2aab0ce4b460) at ldap/servers/slapd/ch_malloc.c:363
#5  0x00002b3410eb95b3 in attr_done (a=0x5ee7) at ldap/servers/slapd/attr.c:353
#6  0x00002b3410eb962b in slapi_attr_free (ppa=<value optimized out>) at ldap/servers/slapd/attr.c:341
#7  0x00002b3410eb9de4 in attrlist_free (alist=<value optimized out>) at ldap/servers/slapd/attrlist.c:53
#8  0x00002b3410ecbd10 in slapi_entry_free (e=0x2aaab3e2cb30) at ldap/servers/slapd/entry.c:1918
#9  0x00002b3410eb67fd in op_shared_add (pb=0x2aab057aed00) at ldap/servers/slapd/add.c:718
#10 0x00002b3410eb7ef8 in do_add (pb=0x2aab057aed00) at ldap/servers/slapd/add.c:259
#11 0x00000000004135a3 in connection_threadmain () at ldap/servers/slapd/connection.c:563
#12 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so
#13 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0
#14 0x00000039382d3f6d in clone () from /lib64/libc.so.6

(gdb) where
#0  0x0000003938230265 in raise () from /lib64/libc.so.6
#1  0x0000003938231d10 in abort () from /lib64/libc.so.6
#2  0x000000393826a84b in __libc_message () from /lib64/libc.so.6
#3  0x0000003938272886 in free () from /lib64/libc.so.6
#4  0x00002b3410ebe1c6 in slapi_ch_free (ptr=0x2aab0ce4b460) at ldap/servers/slapd/ch_malloc.c:363
#5  0x00002b3410eb95b3 in attr_done (a=0x5ee7) at ldap/servers/slapd/attr.c:353
#6  0x00002b3410eb962b in slapi_attr_free (ppa=<value optimized out>) at ldap/servers/slapd/attr.c:341
#7  0x00002b3410eb9de4 in attrlist_free (alist=<value optimized out>) at ldap/servers/slapd/attrlist.c:53
#8  0x00002b3410ecbd10 in slapi_entry_free (e=0x2aaab3e2cb30) at ldap/servers/slapd/entry.c:1918
#9  0x00002b3410eb67fd in op_shared_add (pb=0x2aab057aed00) at ldap/servers/slapd/add.c:718
#10 0x00002b3410eb7ef8 in do_add (pb=0x2aab057aed00) at ldap/servers/slapd/add.c:259
#11 0x00000000004135a3 in connection_threadmain () at ldap/servers/slapd/connection.c:563
#12 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so
#13 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0
#14 0x00000039382d3f6d in clone () from /lib64/libc.so.6
(gdb) list
358             if(recording)
359             {
360                     remove_memory_record(*ptr);
361             }
362     #endif
363             free (*ptr);
364             *ptr = NULL;
365             if(!counters_created)
366             {
367                     create_counters();
(gdb) up
#5  0x00002b3410eb95b3 in attr_done (a=0x5ee7) at ldap/servers/slapd/attr.c:353
353                     slapi_ch_free((void**)&a->a_type);
(gdb) list
348     void
349     attr_done(Slapi_Attr *a)
350     {
351             if(a!=NULL)
352             {
353                     slapi_ch_free((void**)&a->a_type);
354                     csn_free(&a->a_deletioncsn);
355                     slapi_valueset_done(&a->a_present_values);
356                     slapi_valueset_done(&a->a_deleted_values);
357                     {

If you need anymore information from the core file let me know. It's 2GB in size so it's not easy for me to upload it.

Regards,
Daniel

Comment 1 Rich Megginson 2010-11-24 03:04:51 UTC
Can you confirm what version of 389-ds-base you are using?  This bug should have been fixed in 389-ds-base-1.2.6.1-2

if you are not sure, just do
rpm -qi 389-ds-base
and paste the output in this bug

Comment 2 Daniel 2010-11-24 03:09:32 UTC
I am running 389-ds-base-1.2.6.1-2:

rpm -qi 389-ds-base
Name        : 389-ds-base                  Relocations: (not relocatable)
Version     : 1.2.6.1                           Vendor: Fedora Project
Release     : 2.el5                         Build Date: Thu 30 Sep 2010 09:15:13 AM EST
Install Date: Mon 18 Oct 2010 03:45:22 PM EST      Build Host: x86-02.phx2.fedoraproject.org
Group       : System Environment/Daemons    Source RPM: 389-ds-base-1.2.6.1-2.el5.src.rpm
Size        : 5855143                          License: GPLv2 with exceptions
Signature   : DSA/SHA1, Fri 01 Oct 2010 01:56:46 AM EST, Key ID 119cc036217521f6
Packager    : Fedora Project
URL         : http://port389.org/
Summary     : 389 Directory Server (base)
Description :
389 Directory Server is an LDAPv3 compliant server.  The base package includes
the LDAP server and command line utilities for server administration.


$ rpm -qa | grep 389
389-console-1.1.4-1.el5
389-dsgw-1.1.5-1.el5
389-admin-1.1.11-1.el5
389-ds-console-doc-1.2.3-1.el5
389-ds-base-1.2.6.1-2.el5
389-ds-console-1.2.3-1.el5
389-ds-1.2.1-1.el5
389-admin-console-1.1.5-1.el5
389-ds-base-debuginfo-1.2.6.1-2.el5
389-admin-console-doc-1.1.5-1.el5

Comment 3 Rich Megginson 2010-11-29 17:30:18 UTC
ok - do this - when examining the core file in gdb, type the following:

thread apply all bt

and attach the output to this bug as an attachment

Comment 4 Rich Megginson 2010-11-30 23:02:00 UTC
I also note that you do not have 389-adminutil installed - do you have adminutil installed?  Is this a machine that did/does have redhat-ds installed on it?

Comment 5 Daniel 2010-11-30 23:59:09 UTC
The machine doesn't have redhat-ds. 389-adminutil is not installed either.


(gdb) thread apply all bt

 

Thread 52 (Thread 24295):

#0  0x00000039382cb186 in poll () from /lib64/libc.so.6

#1  0x000000393c6246e4 in PR_Poll () from /usr/lib64/libnspr4.so

#2  0x0000000000416009 in slapd_daemon (ports=0x7fff08505270) at ldap/servers/slapd/daemon.c:664

#3  0x000000000041d4c4 in main (argc=7, argv=0x7fff08505608) at ldap/servers/slapd/main.c:1243

 

Thread 51 (Thread 24298):

#0  0x00000039382cd212 in select () from /lib64/libc.so.6

#1  0x00002b3410f21fc9 in DS_Sleep (ticks=<value optimized out>) at ldap/servers/slapd/util.c:788

#2  0x00002b3414742c08 in deadlock_threadmain (param=<value optimized out>) at ldap/servers/slapd/back-ldbm/dblayer.c:3589

#3  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#4  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#5  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 50 (Thread 24299):

#0  0x00000039382cd212 in select () from /lib64/libc.so.6

#1  0x00002b3410f21fc9 in DS_Sleep (ticks=<value optimized out>) at ldap/servers/slapd/util.c:788

#2  0x00002b3414742e39 in checkpoint_threadmain (param=0xf19dfe0) at ldap/servers/slapd/back-ldbm/dblayer.c:3736

#3  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#4  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#5  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 49 (Thread 24300):

#0  0x00000039382cd212 in select () from /lib64/libc.so.6

#1  0x00002b3410f21fc9 in DS_Sleep (ticks=<value optimized out>) at ldap/servers/slapd/util.c:788

#2  0x00002b3414742985 in trickle_threadmain (param=<value optimized out>) at ldap/servers/slapd/back-ldbm/dblayer.c:3887

#3  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#4  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#5  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 48 (Thread 24301):

#0  0x00000039382cd212 in select () from /lib64/libc.so.6

#1  0x00002b3410f21fc9 in DS_Sleep (ticks=<value optimized out>) at ldap/servers/slapd/util.c:788

#2  0x00002b3414743267 in perf_threadmain (param=<value optimized out>) at ldap/servers/slapd/back-ldbm/dblayer.c:3524

#3  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#4  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#5  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 47 (Thread 24302):

#0  0x0000003938a0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622e2d in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#2  0x00002b3410f17799 in slapi_wait_condvar (cvar=0xf5ce750, timeout=0x0) at ldap/servers/slapd/slapi2nspr.c:179

#3  0x00002b3413f0955e in cos_cache_wait_on_change (arg=<value optimized out>) at ldap/servers/plugins/cos/cos_cache.c:478

#4  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#5  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#6  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 46 (Thread 24390):

#0  0x00000039382cd212 in select () from /lib64/libc.so.6

#1  0x00002b3410f21fc9 in DS_Sleep (ticks=<value optimized out>) at ldap/servers/slapd/util.c:788

#2  0x00002b3414cd0716 in _cl5DeadlockMain (param=<value optimized out>) at ldap/servers/plugins/replication/cl5_api.c:3223

---Type <return> to continue, or q <return> to quit---

#3  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#4  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#5  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 45 (Thread 24391):

#0  0x00000039382cd212 in select () from /lib64/libc.so.6

#1  0x00002b3410f21fc9 in DS_Sleep (ticks=<value optimized out>) at ldap/servers/slapd/util.c:788

#2  0x00002b3414cd0530 in _cl5CheckpointMain (param=<value optimized out>) at ldap/servers/plugins/replication/cl5_api.c:3307

#3  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#4  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#5  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 44 (Thread 24392):

#0  0x00000039382cd212 in select () from /lib64/libc.so.6

#1  0x00002b3410f21fc9 in DS_Sleep (ticks=<value optimized out>) at ldap/servers/slapd/util.c:788

#2  0x00002b3414cd0447 in _cl5TrickleMain (param=<value optimized out>) at ldap/servers/plugins/replication/cl5_api.c:3335

#3  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#4  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#5  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 43 (Thread 24393):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00002b3414cd6c88 in _cl5TrimMain (param=<value optimized out>) at ldap/servers/plugins/replication/cl5_api.c:4239

#4  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#5  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#6  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 42 (Thread 24394):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00002b3414ce7085 in protocol_sleep (prp=0xf5e5e40, duration=300000) at ldap/servers/plugins/replication/repl5_inc_protocol.c:1325

#4  0x00002b3414ce8e86 in repl5_inc_run (prp=0xf5e5e40) at ldap/servers/plugins/replication/repl5_inc_protocol.c:810

#5  0x00002b3414cec374 in prot_thread_main (arg=<value optimized out>) at ldap/servers/plugins/replication/repl5_protocol.c:311

#6  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#7  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#8  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 41 (Thread 24395):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00002b3414ce7085 in protocol_sleep (prp=0xf5f2e50, duration=300000) at ldap/servers/plugins/replication/repl5_inc_protocol.c:1325

#4  0x00002b3414ce8e86 in repl5_inc_run (prp=0xf5f2e50) at ldap/servers/plugins/replication/repl5_inc_protocol.c:810

#5  0x00002b3414cec374 in prot_thread_main (arg=<value optimized out>) at ldap/servers/plugins/replication/repl5_protocol.c:311

#6  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#7  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#8  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 40 (Thread 24396):

---Type <return> to continue, or q <return> to quit---

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00002b3414ce7085 in protocol_sleep (prp=0xf5f5a70, duration=300000) at ldap/servers/plugins/replication/repl5_inc_protocol.c:1325

#4  0x00002b3414ce8e86 in repl5_inc_run (prp=0xf5f5a70) at ldap/servers/plugins/replication/repl5_inc_protocol.c:810

#5  0x00002b3414cec374 in prot_thread_main (arg=<value optimized out>) at ldap/servers/plugins/replication/repl5_protocol.c:311

#6  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#7  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#8  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 39 (Thread 24397):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00002b3414ce7085 in protocol_sleep (prp=0xf5de690, duration=300000) at ldap/servers/plugins/replication/repl5_inc_protocol.c:1325

#4  0x00002b3414ce8e86 in repl5_inc_run (prp=0xf5de690) at ldap/servers/plugins/replication/repl5_inc_protocol.c:810

#5  0x00002b3414cec374 in prot_thread_main (arg=<value optimized out>) at ldap/servers/plugins/replication/repl5_protocol.c:311

#6  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#7  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#8  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 38 (Thread 24398):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00002b3414ce7085 in protocol_sleep (prp=0x2aaaac002250, duration=300000) at ldap/servers/plugins/replication/repl5_inc_protocol.c:1325

#4  0x00002b3414ce8e86 in repl5_inc_run (prp=0x2aaaac002250) at ldap/servers/plugins/replication/repl5_inc_protocol.c:810

#5  0x00002b3414cec374 in prot_thread_main (arg=<value optimized out>) at ldap/servers/plugins/replication/repl5_protocol.c:311

#6  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#7  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#8  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 37 (Thread 24399):

#0  0x0000003938a0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622e2d in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#2  0x00002b3410f17799 in slapi_wait_condvar (cvar=0x2aaab0001d40, timeout=0x0) at ldap/servers/slapd/slapi2nspr.c:179

#3  0x00002b341619486f in roles_cache_wait_on_change (arg=<value optimized out>) at ldap/servers/plugins/roles/roles_cache.c:432

#4  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#5  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#6  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 36 (Thread 24400):

#0  0x0000003938a0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622e2d in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#2  0x00002b3410f17799 in slapi_wait_condvar (cvar=0x2aaab00035a0, timeout=0x0) at ldap/servers/slapd/slapi2nspr.c:179

#3  0x00002b341619486f in roles_cache_wait_on_change (arg=<value optimized out>) at ldap/servers/plugins/roles/roles_cache.c:432

#4  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#5  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#6  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 35 (Thread 24401):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

---Type <return> to continue, or q <return> to quit---

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x0000000000419673 in housecleaning (cur_time=<value optimized out>) at ldap/servers/slapd/house.c:77

#4  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#5  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#6  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 34 (Thread 24402):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00002b3410ed12ff in eq_loop (arg=<value optimized out>) at ldap/servers/slapd/eventq.c:355

#4  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#5  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#6  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 33 (Thread 24403):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x4b4c5098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 32 (Thread 24404):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x4bec6098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 31 (Thread 24406):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x4d2c8098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 30 (Thread 24407):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x4dcc9098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

---Type <return> to continue, or q <return> to quit---

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 29 (Thread 24408):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x4e6ca098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 28 (Thread 24409):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x4f0cb098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 27 (Thread 24410):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x4facc098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 26 (Thread 24411):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x504cd098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 25 (Thread 24412):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x50ece098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

---Type <return> to continue, or q <return> to quit---

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 24 (Thread 24413):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x518cf098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 23 (Thread 24414):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x522d0098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 22 (Thread 24415):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x52cd1098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 21 (Thread 24416):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x536d2098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 20 (Thread 24417):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x540d3098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

---Type <return> to continue, or q <return> to quit---

Thread 19 (Thread 24418):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x54ad4098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 18 (Thread 24419):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x554d5098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 17 (Thread 24420):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x55ed6098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 16 (Thread 24421):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x568d7098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 15 (Thread 24422):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x572d8098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 14 (Thread 24423):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

---Type <return> to continue, or q <return> to quit---

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x57cd9098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 13 (Thread 24424):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x586da098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 12 (Thread 24425):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x590db098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 11 (Thread 24426):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x59adc098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 10 (Thread 24427):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x5a4dd098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 9 (Thread 24428):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

---Type <return> to continue, or q <return> to quit---

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x5aede098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 8 (Thread 24429):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x5b8df098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 7 (Thread 24430):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x5c2e0098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 6 (Thread 24431):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x5cce1098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 5 (Thread 24432):

#0  0x0000003938a0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000393c622159 in ?? () from /usr/lib64/libnspr4.so

#2  0x000000393c622d99 in PR_WaitCondVar () from /usr/lib64/libnspr4.so

#3  0x00000000004127c1 in connection_wait_for_new_pb (ppb=0x5d6e2098, interval=10000) at ldap/servers/slapd/connection.c:1695

#4  0x00000000004132c8 in connection_threadmain () at ldap/servers/slapd/connection.c:2173

#5  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#6  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#7  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 4 (Thread 24433):

#0  0x00000039382cd212 in select () from /lib64/libc.so.6

#1  0x00002b3410f21fc9 in DS_Sleep (ticks=<value optimized out>) at ldap/servers/slapd/util.c:788

#2  0x0000000000416c0c in time_thread (nothing=<value optimized out>) at ldap/servers/slapd/daemon.c:466

#3  0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#4  0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

---Type <return> to continue, or q <return> to quit---

#5  0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 3 (Thread 28531):

#0  0x00000039382729ed in _int_malloc () from /lib64/libc.so.6

#1  0x0000003938274cde in malloc () from /lib64/libc.so.6

#2  0x0000003938279772 in strdup () from /lib64/libc.so.6

#3  0x00002b3410ebe382 in slapi_ch_strdup (s1=0xf0d6820 "sn") at ldap/servers/slapd/ch_malloc.c:277

#4  0x00002b3410eb981d in slapi_attr_init_locking_optional (a=0x11f68360, type=0x2aaaf80e6c56 "sn", use_lock=0) at ldap/servers/slapd/attr.c:274

#5  0x00002b3410eb9fbf in attrlist_find_or_create_locking_optional (alist=<value optimized out>, type=<value optimized out>, a=0x5eae1d20, use_lock=0)

    at ldap/servers/slapd/attrlist.c:85

#6  0x00002b3410ecebce in str2entry_fast (rawdn=0x0, s=0x0, flags=8, read_stateinfo=-1) at ldap/servers/slapd/entry.c:402

#7  0x00002b3410ecee4d in slapi_str2entry_ext (dn=0x10d1da70 "uid=slvi,ou=People,dc=deakin,dc=edu,dc=au", s=0x2aaaf80e6690 "rdn", flags=8) at ldap/servers/slapd/entry.c:1381

#8  0x00002b341477d737 in ldbm_back_ldbm2ldif (pb=<value optimized out>) at ldap/servers/slapd/back-ldbm/ldif2ldbm.c:1468

#9  0x00002b3410f1dca5 in task_export_thread (arg=<value optimized out>) at ldap/servers/slapd/task.c:1004

#10 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#11 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#12 0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 2 (Thread 29560):

#0  0x00000039382729fb in _int_malloc () from /lib64/libc.so.6

#1  0x0000003938274cde in malloc () from /lib64/libc.so.6

#2  0x0000003938279772 in strdup () from /lib64/libc.so.6

#3  0x00002b3410ebe382 in slapi_ch_strdup (s1=0x2aab013f3310 "cn=aaaexam,ou=auto.home,ou=Automount,dc=deakin,dc=edu,dc=au") at ldap/servers/slapd/ch_malloc.c:277

#4  0x00002b3410ec497e in slapi_sdn_get_ndn (sdn=0x5f4e2c80) at ldap/servers/slapd/dn.c:1937

#5  0x00002b3410eeadfe in sdn_is_nulldn (sdn=0x0) at ldap/servers/slapd/mapping_tree.c:1941

#6  0x00002b3410eeb348 in slapi_get_mapping_tree_node_by_dn (dn=0x5f4e2c80) at ldap/servers/slapd/mapping_tree.c:2746

#7  0x00002b3410eeb409 in slapi_get_suffix_by_dn (dn=0x0) at ldap/servers/slapd/mapping_tree.c:3050

#8  0x00002b3410f08f4c in _slapi_rdn_init_all_dn_ext (rdn=0x2aab016e2940, sdn=0x5f4e2c80) at ldap/servers/slapd/rdn.c:144

#9  0x00002b3410f09331 in slapi_rdn_init_all_dn (rdn=0x2aab016e2940, dn=0x2aab018dd750 "cn=aaaexam,ou=auto.home,ou=Automount,dc=deakin,dc=edu,dc=au")

    at ldap/servers/slapd/rdn.c:238

#10 0x00002b3410ece7bd in str2entry_fast (rawdn=0x155bfd90 "cn=aaaexam,ou=auto.home,ou=Automount,dc=deakin,dc=edu,dc=au", s=0x0, flags=8, read_stateinfo=-1)

    at ldap/servers/slapd/entry.c:297

#11 0x00002b3410ecee4d in slapi_str2entry_ext (dn=0x155bfd90 "cn=aaaexam,ou=auto.home,ou=Automount,dc=deakin,dc=edu,dc=au", s=0x2aaaac21f120 "rdn", flags=8)

    at ldap/servers/slapd/entry.c:1381

#12 0x00002b341477d737 in ldbm_back_ldbm2ldif (pb=<value optimized out>) at ldap/servers/slapd/back-ldbm/ldif2ldbm.c:1468

#13 0x00002b3410f1dca5 in task_export_thread (arg=<value optimized out>) at ldap/servers/slapd/task.c:1004

#14 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#15 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#16 0x00000039382d3f6d in clone () from /lib64/libc.so.6

 

Thread 1 (Thread 24405):

#0  0x0000003938230265 in raise () from /lib64/libc.so.6

#1  0x0000003938231d10 in abort () from /lib64/libc.so.6

#2  0x000000393826a84b in __libc_message () from /lib64/libc.so.6

#3  0x0000003938272886 in free () from /lib64/libc.so.6

#4  0x00002b3410ebe1c6 in slapi_ch_free (ptr=0x2aab0ce4b460) at ldap/servers/slapd/ch_malloc.c:363

#5  0x00002b3410eb95b3 in attr_done (a=0x5ee7) at ldap/servers/slapd/attr.c:353

#6  0x00002b3410eb962b in slapi_attr_free (ppa=<value optimized out>) at ldap/servers/slapd/attr.c:341

#7  0x00002b3410eb9de4 in attrlist_free (alist=<value optimized out>) at ldap/servers/slapd/attrlist.c:53

#8  0x00002b3410ecbd10 in slapi_entry_free (e=0x2aaab3e2cb30) at ldap/servers/slapd/entry.c:1918

#9  0x00002b3410eb67fd in op_shared_add (pb=0x2aab057aed00) at ldap/servers/slapd/add.c:718

#10 0x00002b3410eb7ef8 in do_add (pb=0x2aab057aed00) at ldap/servers/slapd/add.c:259

---Type <return> to continue, or q <return> to quit---

#11 0x00000000004135a3 in connection_threadmain () at ldap/servers/slapd/connection.c:563

#12 0x000000393c6284ad in ?? () from /usr/lib64/libnspr4.so

#13 0x0000003938a0673d in start_thread () from /lib64/libpthread.so.0

#14 0x00000039382d3f6d in clone () from /lib64/libc.so.6

Comment 6 Rich Megginson 2010-12-02 20:32:14 UTC
Do you have ou=Automount,dc=deakin,dc=edu,dc=au and ou=People,dc=deakin,dc=edu,dc=au in separate databases?

How did you invoke these export operations?  Did you run db2ldif.pl from the command line?  Did you use the console to export the database?

Do you know what the add operation is for thread 1?  In the debugger, you can do

(gdb) thread 1
(gdb) up
(gdb) up
... # keep going up until you get to slapi_entry_free
(gdb) print *e

I would like to find out what the DN is.

Are you running RHEL5 or CentOS5?  Looks like 64-bit, correct?

Comment 7 Daniel 2010-12-05 22:29:42 UTC
Yes People and Automount use separate databases.

We use db2ldif.pl to export the databases from the command line.

Thread 1:

(gdb) up
#8  0x00002b3410ecbd10 in slapi_entry_free (e=0x2aaab3e2cb30) at ldap/servers/slapd/entry.c:1918
1918                    attrlist_free(e->e_attrs);
(gdb) print *e
$1 = {e_sdn = {flag = 16 '\020', dn = 0x0, ndn = 0x0, ndn_len = 0}, e_srdn = {flag = 0 '\000', rdn = 0x0, rdns = 0x0, butcheredupto = -1, nrdn = 0x0, all_rdns = 0x0,
    all_nrdns = 0x0}, e_uniqueid = 0x0, e_dncsnset = 0x0, e_maxcsn = 0x0, e_attrs = 0x2aab0ce4b460, e_deleted_attrs = 0x0, e_virtual_attrs = 0x0, e_virtual_watermark = 0,
  e_virtual_lock = 0x2aab0ce64470, e_extension = 0x0, e_flags = 0 '\000', e_aux_attrs = 0x0}
(gdb)

We are running RHEL 5 64bit.

Comment 9 Noriko Hosoi 2010-12-09 18:33:38 UTC
Daniel, we ran a series of stress tests repeating 2 db2ldif.pl and adding (using a test tool "infadd"), but we have no luck to reproduce the crash so far.

There could be some conditions to trigger the problem.  It looks the crashed server is a master.  Is it a single master or one of MMR servers?

Are there any special configuration you could think of?  For instance, password policy, CoS/Roles plugin, memberOf plugin, etc.?  Any customer schema?

Thanks for your help, in advance.

Comment 10 Daniel 2010-12-09 21:56:25 UTC
Hi Noriko,

The server is a single master. The environment also has one consumer. 

The special config we have is:

- We disable all password/account lockout policies and prevent users from changing their own password.
- We have multiple custom schemas we apply. We also index attributes on the custom schemas
- The file descriptors have been increased to 8000
- We increase the database cache size to: 52428800

The rest of the config is default from install.

I am not sure if it's related but it seems hourly the task created by db2ldif.pl takes a long amount of time (about 30+ seconds) to complete. This is strange as it normally takes about 5 seconds to complete. 

Does db2ldif.pl use any sort of caching system?

Regards,
Daniel

Comment 11 Rich Megginson 2010-12-09 22:08:54 UTC
> I am not sure if it's related but it seems hourly the task created by
> db2ldif.pl takes a long amount of time (about 30+ seconds) to complete. This is
> strange as it normally takes about 5 seconds to complete. 

Could you explain this a bit more?  You create a task with db2ldif.pl hourly?  Sometimes it takes 5 seconds and sometimes it takes 30+ seconds?  Is there anything different about the times it takes 5 seconds and the times it takes 30+ seconds?

Noriko, if this is cache related, you might try making the cache size the minimum size, to see if this is somehow related to swapping entries into and out of the cache during the export + add operations.

Comment 12 Noriko Hosoi 2010-12-09 22:15:36 UTC
(In reply to comment #11)
> Noriko, if this is cache related, you might try making the cache size the
> minimum size, to see if this is somehow related to swapping entries into and
> out of the cache during the export + add operations.

All right.  Let me try...

Comment 13 Daniel 2010-12-09 22:55:53 UTC
Our script runs an export every 5mins and diffs the ldif file with a generated one (from an external database which holds our user information). It's nice a quick during the hour but on the hour it takes a quite a long time. It's seems like the cache is expiring on the hour causing it to be much slower.

Normally the export looks like this (doing ldapsearch in a loop on the task entry):

dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 21000 entries (20%).
dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 24000 entries (22%).
dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 27000 entries (25%).
dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 30000 entries (41%).
dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 34000 entries (83%).
dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 38000 entries (87%).
dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 40000 entries (88%).
dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 44000 entries (91%).
dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 45000 entries (92%).
dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 46000 entries (93%).
dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 49000 entries (95%).
dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 51000 entries (97%).
dn: cn=export_2010_11_29_18_15_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 53000 entries (98%).
nstaskexitcode: 0 nstaskstatus: userRoot: Processed 54855 entries (100%).


However on a slow run it takes a lot longer:


dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 12000 entries (13%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 15000 entries (15%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 19000 entries (18%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 21000 entries (20%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 23000 entries (22%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 26000 entries (24%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 27000 entries (25%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 30000 entries (41%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 33000 entries (44%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 34000 entries (83%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 34000 entries (83%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 35000 entries (84%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 35000 entries (84%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 36000 entries (85%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 37000 entries (86%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 38000 entries (87%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 39000 entries (88%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 40000 entries (88%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 41000 entries (89%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 42000 entries (90%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 43000 entries (91%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 44000 entries (91%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 45000 entries (92%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 46000 entries (93%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 47000 entries (94%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 48000 entries (94%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 49000 entries (95%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 50000 entries (96%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 51000 entries (97%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 52000 entries (97%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 53000 entries (98%).
dn: cn=export_2010_11_29_18_00_2,cn=export,cn=tasks,cn=config nstaskstatus: userRoot: Processed 54000 entries (99%).
nstaskexitcode: 0 nstaskstatus: userRoot: Processed 54855 entries (100%).

Comment 15 Rich Megginson 2011-01-20 15:48:07 UTC
Can you provide the arguments to the db2ldif.pl command that you use?

Comment 16 Rich Megginson 2011-01-21 03:24:09 UTC
Also, would it be possible for you to try to reproduce the crash with 389-ds-base 1.2.7.5?  If this is memory related, it may have been fixed when we did the Coverity fixes for 1.2.7

Comment 17 Daniel 2011-01-23 22:17:23 UTC
Hi,

The command we use is

db2ldif.pl -1 -u -N -D "cn=Directory Manager" -n userRoot -w "DIRMANAGERPWD" -a /tmp/ldapexport.ldif

I'll upgrade one of our environments but I wasn't able to reproduce it on the spot last time.

Regards,
Daniel

Comment 18 Rich Megginson 2011-01-24 15:29:26 UTC
(In reply to comment #17)
> Hi,
> 
> The command we use is
> 
> db2ldif.pl -1 -u -N -D "cn=Directory Manager" -n userRoot -w "DIRMANAGERPWD" -a
> /tmp/ldapexport.ldif
> 
> I'll upgrade one of our environments but I wasn't able to reproduce it on the
> spot last time.
> 
> Regards,
> Daniel

In the stack trace in https://bugzilla.redhat.com/show_bug.cgi?id=656734#c5 there are two different export jobs running at the same time for different databases.

cn=aaaexam,ou=auto.home,ou=Automount,dc=deakin,dc=edu,dc=au

uid=slvi,ou=People,dc=deakin,dc=edu,dc=au

In https://bugzilla.redhat.com/show_bug.cgi?id=656734#c7 you said these were in separate databases.

I would assume that userRoot is the ou=People database.  Is there a separate db2ldif.pl job for the ou=Automount database?

Comment 19 Daniel 2011-01-24 22:10:37 UTC
Yes, automountData is our database for ou=Automount.

We also have

netgroupData for ou=Netgroup
groupData for ou=Groups.

Regards,
Daniel

Comment 20 Rich Megginson 2011-01-24 22:22:40 UTC
(In reply to comment #19)
> Yes, automountData is our database for ou=Automount.
> 
> We also have
> 
> netgroupData for ou=Netgroup
> groupData for ou=Groups.
> 
> Regards,
> Daniel

The core file showed two db2ldif jobs running - one for the ou=People database and one for the ou=Automount database - are these two separate cron jobs?  What are the arguments for the ou=Automount db2ldif?

Comment 21 Daniel 2011-01-24 22:31:21 UTC
The arguments are the same for each job. There is a separate job for each database. We have spaced out the intervals the jobs have run but it looks like they may have overlapped (probably due to the slow run on the hour).

Comment 22 Rich Megginson 2011-01-24 23:01:11 UTC
Ok.  So it looks as though the following happens:
1) first userRoot db2ldif cron job runs and runs very, very slowly
2) second ou=Automount db2ldif cron job runs before userRoot one finishes
3) there is an ldap ADD operation

I've tried tests using this sequence of operations (and several more) - no luck reproducing the problem.

Comment 23 Rich Megginson 2011-02-01 17:10:44 UTC
Any luck reproducing with 1.2.7?

Comment 24 Daniel 2011-02-23 22:11:28 UTC
I have not been able to reproduce this with 1.2.7. Feel free to close this one. If it does occur again in the future i'll submit another bug.

Thanks,
Daniel

Comment 25 Rich Megginson 2011-02-24 01:43:05 UTC
(In reply to comment #24)
> I have not been able to reproduce this with 1.2.7. Feel free to close this one.
> If it does occur again in the future i'll submit another bug.
> 
> Thanks,
> Daniel

Excellent.  Thanks Daniel!


Note You need to log in before you can comment on or make changes to this bug.