Bug 755725

Summary: 389 programs linked against openldap crash during shutdown
Product: [Fedora] Fedora Reporter: Jr Aquino <jr.aquino>
Component: 389-ds-baseAssignee: Rich Megginson <rmeggins>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 15CC: edewata, nhosoi, nkinder, rmeggins
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: 389-dsgw-1.1.9-1.fc16 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-02-02 17:27:20 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 743970, 769107    
Attachments:
Description Flags
0001-Bug-755725-389-programs-linked-against-openldap-cras.patch nhosoi: review+

Description Jr Aquino 2011-11-21 21:18:43 UTC
Description of problem:
When restarting the dirsrv process, it receives a segfault

Version-Release number of selected component (if applicable):
389-ds-base-1.2.10-0.5.a5.fc15.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Install FreeIPA
2. Restart DIRSRV
3. check dmesg for segfaults
  
Actual results:
restart generates segfault

Expected results:
Restart without crashing

Additional info:


(gdb) thread apply all bt all

Thread 1 (Thread 0x7f4a76a387e0 (LWP 918)):
No symbol "all" in current context.
(gdb) thread apply all bt full

Thread 1 (Thread 0x7f4a76a387e0 (LWP 918)):
#0  __GI___libc_free (mem=0x185) at malloc.c:3709
        ar_ptr = <optimized out>
        p = <optimized out>
        hook = 0
#1  0x000000352be0e510 in ldap_int_destroy_global_options () at ../../../libraries/libldap/init.c:496
        gopts = 0x352c049360
#2  0x000000352660e896 in _dl_fini () at dl-fini.c:244
        array = 0x352c0475c8
        i = <optimized out>
        nmaps = 94
        nloaded = <optimized out>
        i = <optimized out>
        l = 0x7f4a76a459b0
        ns = 0
        maps = 0x7fff3ccac680
        maps_size = 752
        do_audit = 0
#3  0x0000003526a38991 in __run_exit_handlers (status=0, listp=0x3526d935a8, run_list_atexit=true) at exit.c:78
        atfct = <optimized out>
        onfct = <optimized out>
        cxafct = <optimized out>
        f = <optimized out>
#4  0x0000003526a38a15 in __GI_exit (status=<optimized out>) at exit.c:100
No locals.
#5  0x0000003526a213a4 in __libc_start_main (main=0x40c8d0 <main>, argc=7, ubp_av=0x7fff3ccacb08, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff3ccacaf8) at libc-start.c:258
        result = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, -6666163846686787399, 4255964, 140734213311232, 0, 0, 6666311345183307961, -6694800918663219015}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x7fff3ccacb48, 0x1}, data = {prev = 0x0, cleanup = 0x0, 
              canceltype = 1019923272}}}
        not_first_call = <optimized out>
#6  0x000000000040f105 in _start ()
No symbol table info available.

Comment 1 Jr Aquino 2011-11-21 21:52:17 UTC
Adjustment to Reproduce issue.  Standard install does NOT appear to generate the segfault.

(Corrected Method)
Steps to Reproduce:
1. Install FreeIPA
2. Install FreeIPA Replica Server
3. Restart FreeIPA Master
4. check dmesg for the segfault

Comment 2 Rich Megginson 2011-11-21 22:24:25 UTC
The crash is occurring here:
static void
ldap_int_destroy_global_options(void)
...
#ifdef HAVE_CYRUS_SASL
	if ( gopts->ldo_def_sasl_authcid ) {
		LDAP_FREE( gopts->ldo_def_sasl_authcid );
		gopts->ldo_def_sasl_authcid = NULL;
	}
#endif

Either gopts->ldo_def_sasl_authcid is corrupted, or someone else freed it and didn't set it to NULL.  I suspect that in order to trigger this bug you'd have to attempt a SASL/GSSAPI bind in replication or DNA.

Comment 3 Rich Megginson 2011-12-12 22:55:39 UTC
now happening in dsgw CGI programs as well:
Core was generated by `/usr/lib64/dirsrv/dsgw-cgi-bin/doauth'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000003eaca80c14 in ?? ()
Missing separate debuginfos, use: debuginfo-install glibc-2.14.90-19.x86_64
(gdb) up
#1 0x0000003ebda0eb90 in ldap_int_destroy_global_options () at ../../../libraries/libldap/init.c:496
496 LDAP_FREE( gopts->ldo_def_sasl_authcid );
(gdb) p *gopts
$1 = {ldo_valid = 0, ldo_debug = 32768, ldo_peer = 0x0, ldo_cldapdn = 0x0, ldo_is_udp = 0, ldo_tm_api = {tv_sec = 0, tv_usec = 0},
ldo_tm_net = {tv_sec = 0, tv_usec = 0}, ldo_version = 0, ldo_deref = 0, ldo_timelimit = 0, ldo_sizelimit = 0, ldo_tls_ctx = 0x0,
ldo_tls_connect_cb = 0, ldo_tls_connect_arg = 0x0, ldo_tls_info = {lt_certfile = 0x0, lt_keyfile = 0x0, lt_dhfile = 0x0,
lt_cacertfile = 0x0, lt_cacertdir = 0x0, lt_ciphersuite = 0x0, lt_crlfile = 0x0, lt_randfile = 0x0, lt_protocol_min = 0},
ldo_tls_mode = 0, ldo_tls_require_cert = 0, ldo_tls_impl = 0, ldo_defludp = 0x0, ldo_defport = 0, ldo_defbase = 0x0,
ldo_defbinddn = 0x0, ldo_def_sasl_mech = 0x0, ldo_def_sasl_realm = 0x0, ldo_def_sasl_authcid = 0x0, ldo_def_sasl_authzid = 0x0,
ldo_sasl_secprops = {min_ssf = 0, max_ssf = 0, maxbufsize = 0, security_flags = 0, property_names = 0x0, property_values = 0x0},
ldo_keepalive_idle = 0, ldo_keepalive_probes = 0, ldo_keepalive_interval = 0, ldo_refhoplimit = 0, ldo_sctrls = 0x0,
ldo_cctrls = 0x0, ldo_rebind_proc = 0, ldo_rebind_params = 0x0, ldo_nextref_proc = 0, ldo_nextref_params = 0x0,
ldo_urllist_proc = 0, ldo_urllist_params = 0x0, ldo_conn_cbs = 0x0, ldo_booleans = 0}
(gdb)

Comment 4 Rich Megginson 2011-12-14 23:53:23 UTC
I've been able to reproduce this using doauth from dsgw on F-16.  It looks as though the function ldap_int_destroy_global_options() is called twice during shutdown.  The first time it works fine.  The second time it fails.  In the debugger, I can see the value of ldo_def_sasl_authcid == 0x0, but
	if ( gopts->ldo_def_sasl_authcid ) {
		LDAP_FREE( gopts->ldo_def_sasl_authcid );
		gopts->ldo_def_sasl_authcid = NULL;
	}
this goes into LDAP_FREE anyway?????  And inside LDAP_FREE, the value is 0x185 (which strangely is 389 in decimal).  valgrind reports no errors except for the attempt to free 0x185 (which of course is not malloced).

Comment 5 Rich Megginson 2011-12-15 15:13:58 UTC
Created attachment 547304 [details]
0001-Bug-755725-389-programs-linked-against-openldap-cras.patch

Comment 6 Rich Megginson 2011-12-15 18:27:15 UTC
commit 91fa21f6af7871d73bb9d5e738ec4e31b5298463
Author: Rich Megginson <rmeggins>
Date:   Wed Dec 14 20:04:48 2011 -0700
    Reviewed by: nhosoi (Thanks!)
    Branch: master
    Fix Description: With recent versions of openldap, you cannot link with both
    ldap_r and ldap - when the shared object _fini is run, the _fini from the on
    will stomp on the _fini from the other, and the program will crash.  The fix
    is to link with ldap_r only in a threaded program, and ldap otherwise.
    Platforms tested: Fedora 16, RHEL6 x86_64
    Flag Day: no

Comment 7 Fedora Update System 2011-12-15 19:55:11 UTC
389-dsgw-1.1.8-1.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/389-dsgw-1.1.8-1.fc15

Comment 8 Fedora Update System 2011-12-15 19:56:36 UTC
389-dsgw-1.1.8-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/389-dsgw-1.1.8-1.fc16

Comment 9 Fedora Update System 2011-12-19 16:51:08 UTC
389-ds-base-1.2.10-0.6.a6.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/389-ds-base-1.2.10-0.6.a6.fc15

Comment 10 Fedora Update System 2011-12-19 16:52:31 UTC
389-ds-base-1.2.10-0.6.a6.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/389-ds-base-1.2.10-0.6.a6.el5

Comment 11 Fedora Update System 2011-12-19 16:53:25 UTC
389-ds-base-1.2.10-0.6.a6.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/389-ds-base-1.2.10-0.6.a6.fc16

Comment 12 Fedora Update System 2011-12-21 14:40:22 UTC
slapi-nis-0.28-1.fc16,freeipa-2.1.4-3.fc16,389-ds-base-1.2.10-0.6.a6.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/slapi-nis-0.28-1.fc16,freeipa-2.1.4-3.fc16,389-ds-base-1.2.10-0.6.a6.fc16

Comment 13 Fedora Update System 2011-12-22 14:28:47 UTC
389-ds-base-1.2.10-0.6.a6.fc15,slapi-nis-0.28-1.fc15,freeipa-2.1.4-2.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/389-ds-base-1.2.10-0.6.a6.fc15,slapi-nis-0.28-1.fc15,freeipa-2.1.4-2.fc15

Comment 14 Fedora Update System 2011-12-22 22:42:55 UTC
Package slapi-nis-0.28-1.fc16, freeipa-2.1.4-3.fc16, 389-ds-base-1.2.10-0.6.a6.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing slapi-nis-0.28-1.fc16 freeipa-2.1.4-3.fc16 389-ds-base-1.2.10-0.6.a6.fc16'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2011-17313/slapi-nis-0.28-1.fc16,freeipa-2.1.4-3.fc16,389-ds-base-1.2.10-0.6.a6.fc16
then log in and leave karma (feedback).

Comment 15 Rich Megginson 2012-01-10 20:36:59 UTC
Upstream ticket:
https://fedorahosted.org/389/ticket/252

Comment 16 Fedora Update System 2012-01-24 00:33:35 UTC
389-ds-base-1.2.10-0.8.a7.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/389-ds-base-1.2.10-0.8.a7.el5

Comment 17 Fedora Update System 2012-01-24 21:26:30 UTC
389-ds-base-1.2.10-0.9.a8.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/389-ds-base-1.2.10-0.9.a8.el5

Comment 18 Fedora Update System 2012-02-02 17:27:20 UTC
389-ds-base-1.2.10-0.6.a6.fc15, slapi-nis-0.28-1.fc15, freeipa-2.1.4-3.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 19 Fedora Update System 2012-02-02 17:32:12 UTC
slapi-nis-0.28-1.fc16, 389-ds-base-1.2.10-0.6.a6.fc16, freeipa-2.1.4-4.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 20 Fedora Update System 2012-02-04 00:46:17 UTC
389-dsgw-1.1.9-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/389-dsgw-1.1.9-1.fc16

Comment 21 Fedora Update System 2012-02-04 02:27:52 UTC
389-dsgw-1.1.9-1.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/389-dsgw-1.1.9-1.fc15

Comment 22 Fedora Update System 2012-02-04 03:38:02 UTC
389-ds-base-1.2.10-0.10.rc1.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/389-ds-base-1.2.10-0.10.rc1.el5

Comment 23 Fedora Update System 2012-02-13 20:13:59 UTC
389-ds-base-1.2.10.0-1.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/389-ds-base-1.2.10.0-1.el5

Comment 24 Fedora Update System 2012-02-23 15:50:51 UTC
389-ds-base-1.2.10.2-1.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/389-ds-base-1.2.10.2-1.el5

Comment 25 Fedora Update System 2012-03-05 23:13:34 UTC
389-ds-base-1.2.10.3-1.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/389-ds-base-1.2.10.3-1.el5

Comment 26 Fedora Update System 2012-03-27 14:04:43 UTC
389-ds-base-1.2.10.4-2.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/389-ds-base-1.2.10.4-2.el5

Comment 27 Fedora Update System 2012-03-29 19:21:19 UTC
389-ds-base-1.2.10.4-3.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/389-ds-base-1.2.10.4-3.el5

Comment 28 Fedora Update System 2012-03-31 03:26:01 UTC
389-dsgw-1.1.9-1.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 29 Fedora Update System 2012-03-31 03:26:51 UTC
389-dsgw-1.1.9-1.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.