Bug 706225 - Stress tests - ldclt core dumps when opening multiple ldap connections
Stress tests - ldclt core dumps when opening multiple ldap connections
Product: Fedora
Classification: Fedora
Component: 389-ds-base (Show other bugs)
All Linux
unspecified Severity high
: ---
: ---
Assigned To: Rich Megginson
Fedora Extras Quality Assurance
: screened
Depends On:
Blocks: 690319
  Show dependency treegraph
Reported: 2011-05-19 15:38 EDT by Sankar Ramalingam
Modified: 2012-08-16 17:34 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2012-08-16 17:34:41 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Sankar Ramalingam 2011-05-19 15:38:46 EDT
Description of problem: Max connections stress test is dumping the core when ldap server is trying to open multiple connections using ldclt.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to reproduce:

1. Setup an directory server instance using rhel6.1 0day errata build of 389-ds-base.
2. Run the following ldclt command.
   threads=1; while [ $threads -lt 10 ]; do /usr/bin/ldclt -h $DS_HOST -p 389 -D "cn=directory manager" -w xxxxxxxxxx -s base -b "o=nosuchentry"  -I 32 -I 91 -f objectclass=* -q -v -i 60 -e esearch,random -r 0 -R 10 -n 250 -W 67 & threads=`expr $threads + 1`; done

3. It dumps the core.

Here is the stack trace and comments from Noriko....

    (gdb) bt
    #0  0x00b35424 in __kernel_vsyscall ()
    #1  0x0017ed71 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
    #2  0x0018064a in abort () at abort.c:92
    #3  0x001bcbfd in __libc_message (do_abort=2,
        fmt=0x2a76cc "*** glibc detected *** %s: %s: 0x%s ***\n")
        at ../sysdeps/unix/sysv/linux/libc_fatal.c:186
    #4  0x001c3041 in malloc_printerr (action=<value optimized out>,
        str=<value optimized out>, ptr=0xb2201358) at malloc.c:6283
    #5  0x00a47040 in ber_memfree_x (p=0xb2201358, ctx=0x0)
        at ../../../libraries/liblber/memory.c:152
    #6  0x0012dddc in ldap_int_initialize (gopts=0x152860, dbglvl=0x0)
        at init.c:631
    #7  0x0010f82a in ldap_create (ldp=0x374b521c) at open.c:106
    #8  0x0010ffaf in ldap_initialize (ldp=0x808d298,
        url=0x9956f08 "ldap://beast.idm.lab.bos.redhat.com:1389/") at open.c:223
    #9  0x0804d72a in connectToServer (tttctx=0x808d1e0)
        at ldap/servers/slapd/tools/ldclt/ldapfct.c:702
    #10 0x0804e595 in doExactSearch (tttctx=0x808d1e0)
        at ldap/servers/slapd/tools/ldclt/ldapfct.c:3730
    #11 0x08059650 in threadMain (arg=0x808d1e0)
        at ldap/servers/slapd/tools/ldclt/threadMain.c:1194
    #12 0x00c749e9 in start_thread (arg=0x374b5b70) at pthread_create.c:301
    #13 0x002320fe in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:133

Examining the value name and the value lda_int_hostname in ldap_int_initialize (the snippet of the code is found below).  Since they don't match, name is freed.

    #6  0x0012dddc in ldap_int_initialize (gopts=0x152860, dbglvl=0x0)
        at init.c:631
    631                LDAP_FREE( name );
    (gdb) p name
    $2 = 0xb2201358 "" <== can't trust the value since the process crashed in freeing this memory...
    (gdb) p ldap_int_hostname
    $3 = 0x6f0008c8 "beast.idm.lab.bos.redhat.com"

I'd think this fqdn checking is somehow not thread-safe.  The variable ldap_int_hostname is suspicious since it's global and replaced every time ldap_initialization is called.  The error message "double free" makes sense once ldap_int_hostname stores something wrong, multiple threads would try to fee the same memory almost at the same time...   But I could not figured how it causes this mismatch between the source ldap_int_hostname and returned ldap_int_hostname from ldap_pvt_get_fqdn.  It looks ldap_pvt_get_fqdn is implemented in the thread-safe manner (Note: openldap is compiled with LDAP_R_COMPILE).

    #if defined(HAVE_TLS) || defined(HAVE_CYRUS_SASL)
    char * ldap_int_hostname = NULL;

    void ldap_int_initialize( struct ldapoptions *gopts, int *dbglvl )
    #if defined(HAVE_TLS) || defined(HAVE_CYRUS_SASL)
            char    *name = ldap_int_hostname;

            ldap_int_hostname = ldap_pvt_get_fqdn( name );

            if ( name != NULL && name != ldap_int_hostname ) {
                LDAP_FREE( name );

The Max Connections test launches 250 ldclt threads.  Could it be possible to run the test with lower thread count such as 50?

Comment 3 Rich Megginson 2012-01-09 10:52:16 EST
Upstream ticket:
Comment 4 Fedora End Of Life 2012-08-16 17:34:44 EDT
This message is a notice that Fedora 14 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 14. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained.  At this time, all open bugs with a Fedora 'version'
of '14' have been closed as WONTFIX.

(Please note: Our normal process is to give advanced warning of this 
occurring, but we forgot to do that. A thousand apologies.)

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, feel free to reopen 
this bug and simply change the 'version' to a later Fedora version.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we were unable to fix it before Fedora 14 reached end of life. If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora, you are encouraged to click on 
"Clone This Bug" (top right of this page) and open it against that 
version of Fedora.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 

Note You need to log in before you can comment on or make changes to this bug.