Bug 1041732 - Segfault crash of 389 after ipa-adtrust-install
Summary: Segfault crash of 389 after ipa-adtrust-install
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: 389-ds-base
Version: 20
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Rich Megginson
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-12-12 18:59 UTC by Tomas Babej
Modified: 2014-01-06 15:42 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-01-06 15:42:31 UTC
Type: Bug


Attachments (Terms of Use)
Core dump captured by abrt (601.13 KB, application/gzip)
2013-12-12 18:59 UTC, Tomas Babej
no flags Details
First crash (2.75 MB, application/gzip)
2013-12-13 13:53 UTC, Tomas Babej
no flags Details
Second crash (2.21 MB, application/gzip)
2013-12-13 13:53 UTC, Tomas Babej
no flags Details
Log from the ipa-server-install and ipa-adtrust-install (3.61 MB, text/x-log)
2013-12-13 13:58 UTC, Tomas Babej
no flags Details
Journalctl output since 14:20 (277.69 KB, text/x-log)
2013-12-13 13:59 UTC, Tomas Babej
no flags Details
stacktrace 1 - slapi-nis (9.18 KB, text/plain)
2013-12-13 15:38 UTC, Rich Megginson
no flags Details
stacktrace 2 (163.46 KB, text/plain)
2013-12-13 17:26 UTC, Rich Megginson
no flags Details

Description Tomas Babej 2013-12-12 18:59:08 UTC
Created attachment 835995 [details]
Core dump captured by abrt

Description of problem:

After installing IPA AD trust support, Directory server crashes.


Version-Release number of selected component (if applicable):

389-ds-base-1.3.2.7-1.fc20.x86_64

How reproducible:

Always.

Steps to Reproduce:
1. Install IPA server on F20
2. Install IPA AD trust support

Actual results:

Directory server crashes, therefore Kerberos and DNS no longer work.

ns-slapd[4421]: GSSAPI server step 1
kernel: ns-slapd[4485]: segfault at 0 ip 00007f1ed8a596f0 sp 00007f1eb27dce88 error 4 in libback-ldbm.so[7f1ed8a24000+9e000]

Expected results:

Directory server does not crash.

Additional info:

Attaching core dump captured by abrt.

Comment 1 Rich Megginson 2013-12-12 19:05:19 UTC
Comment on attachment 835995 [details]
Core dump captured by abrt

According to the abrt dump, the crash is in winbindd:

executable: /usr/sbin/winbindd

package: samba-winbind-4.1.1-1.fc20

Comment 2 Rich Megginson 2013-12-12 19:06:27 UTC
perhaps there is another abrt file that contains the information for the 389-ds-base crash?  If not, please see http://port389.org/wiki/FAQ#Debugging_Crashes

Comment 3 Petr Spacek 2013-12-13 09:44:55 UTC
Tomas, please check if your backtrace is the same as in https://fedorahosted.org/389/ticket/47629 or not.

Comment 4 Martin Kosek 2013-12-13 10:15:31 UTC
Increasing severity, this instability affects FreeIPA function on F20.

Comment 5 Tomas Babej 2013-12-13 13:10:36 UTC
Rich is right, abrt indeed only captured the crash of the winbindd. I am reproducing according to the instructions on the 389's wiki, and will report update later today.

Comment 6 Tomas Babej 2013-12-13 13:49:28 UTC
I have a VM with snapshot that produces this error. I managed to capture the crash with abrt (probably the core dump size limit was the reason it wasn't captured in the first time).

There are actually *two* ns-slapd crashes. First some context from the jorunalctl:

[root@vm-227 ~]# journalctl -u dirsrv@DOM227-JENKINSAD-IDM-LAB-ENG-BRQ-REDHAT-COM.service

...
Dec 13 14:22:38 vm-227.dom227.jenkinsad.idm.lab.eng.brq.redhat.com ns-slapd[4291]: GSSAPI server step 1                                                                                           
Dec 13 14:22:38 vm-227.dom227.jenkinsad.idm.lab.eng.brq.redhat.com ns-slapd[4291]: GSSAPI server step 2                                                                                           
Dec 13 14:22:38 vm-227.dom227.jenkinsad.idm.lab.eng.brq.redhat.com ns-slapd[4291]: GSSAPI server step 3                                                                                           
Dec 13 14:22:41 vm-227.dom227.jenkinsad.idm.lab.eng.brq.redhat.com systemd[1]: Stopping 389 Directory Server DOM227-JENKINSAD-IDM-LAB-ENG-BRQ-REDHAT-COM....                                      
Dec 13 14:22:43 vm-227.dom227.jenkinsad.idm.lab.eng.brq.redhat.com systemd[1]: dirsrv@DOM227-JENKINSAD-IDM-LAB-ENG-BRQ-REDHAT-COM.service: main process exited, code=dumped, status=6/ABRT
Dec 13 14:22:43 vm-227.dom227.jenkinsad.idm.lab.eng.brq.redhat.com systemd[1]: Unit dirsrv@DOM227-JENKINSAD-IDM-LAB-ENG-BRQ-REDHAT-COM.service entered failed state.
Dec 13 14:22:43 vm-227.dom227.jenkinsad.idm.lab.eng.brq.redhat.com systemd[1]: Starting 389 Directory Server DOM227-JENKINSAD-IDM-LAB-ENG-BRQ-REDHAT-COM....
Dec 13 14:22:43 vm-227.dom227.jenkinsad.idm.lab.eng.brq.redhat.com systemd[1]: Started 389 Directory Server DOM227-JENKINSAD-IDM-LAB-ENG-BRQ-REDHAT-COM..
Dec 13 14:23:09 vm-227.dom227.jenkinsad.idm.lab.eng.brq.redhat.com ns-slapd[4683]: GSSAPI server step 1
Dec 13 14:23:09 vm-227.dom227.jenkinsad.idm.lab.eng.brq.redhat.com ns-slapd[4683]: GSSAPI server step 2
Dec 13 14:23:09 vm-227.dom227.jenkinsad.idm.lab.eng.brq.redhat.com ns-slapd[4683]: GSSAPI server step 3
Dec 13 14:23:10 vm-227.dom227.jenkinsad.idm.lab.eng.brq.redhat.com ns-slapd[4683]: GSSAPI server step 1
Dec 13 14:23:10 vm-227.dom227.jenkinsad.idm.lab.eng.brq.redhat.com ns-slapd[4683]: GSSAPI server step 2
Dec 13 14:23:10 vm-227.dom227.jenkinsad.idm.lab.eng.brq.redhat.com ns-slapd[4683]: GSSAPI server step 3
Dec 13 14:23:12 vm-227.dom227.jenkinsad.idm.lab.eng.brq.redhat.com systemd[1]: dirsrv@DOM227-JENKINSAD-IDM-LAB-ENG-BRQ-REDHAT-COM.service: main process exited, code=dumped, status=11/SEGV
Dec 13 14:23:12 vm-227.dom227.jenkinsad.idm.lab.eng.brq.redhat.com systemd[1]: Unit dirsrv@DOM227-JENKINSAD-IDM-LAB-ENG-BRQ-REDHAT-COM.service entered failed state.

First crash happened at *14:22:43* (while restarting directory server):

[2013-12-13T13:22:41Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>:   [15/21]: adding special DNS service records
[2013-12-13T13:22:42Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>:   [16/21]: enabling trusted domains support for older clients via Schema Compatibility plugin
[2013-12-13T13:22:42Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>:   [17/21]: restarting Directory Server to take MS PAC and LDAP plugins changes into account
[2013-12-13T13:22:47Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>:   [18/21]: adding fallback group

The second at *14:23:12* (ipa-adtrust-install finished and just after named restart)

[2013-12-13T13:22:48Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>:   [19/21]: setting SELinux booleans
[2013-12-13T13:23:09Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>:   [20/21]: starting CIFS services
[2013-12-13T13:23:10Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>:   [21/21]: adding SIDs to existing users and groups
[2013-12-13T13:23:10Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: Done configuring CIFS.
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: 
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: =============================================================================
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: Setup complete
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: 
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: You must make sure these network ports are open:
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: 	TCP Ports:
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: 	  * 138: netbios-dgm
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: 	  * 139: netbios-ssn
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: 	  * 445: microsoft-ds
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: 	UDP Ports:
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: 	  * 138: netbios-dgm
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: 	  * 139: netbios-ssn
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: 	  * 389: (C)LDAP
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: 	  * 445: microsoft-ds
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: 
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: Additionally you have to make sure the FreeIPA LDAP server is not reachable
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: by any domain controller in the Active Directory domain by closing down
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: the following ports for these servers:
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: 	TCP Ports:
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: 	  * 389, 636: LDAP/LDAPS
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: 
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: You may want to choose to REJECT the network packets instead of DROPing
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: them to avoid timeouts on the AD domain controllers.
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: 
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: =============================================================================
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10.out] <DEBUG>: 
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd10] <DEBUG>: Exit code: 0
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.ParamikoTransport] <INFO>: RUN ['systemctl', 'restart', 'named']
[2013-12-13T13:23:11Z ipa.ipatests.test_integration.host.Host.vm-227.cmd11] <DEBUG>: RUN ['systemctl', 'restart', 'named']

I'm attaching both crash dumps and a relevant parts of journalctl output and ipa-server-install.log

Comment 7 Tomas Babej 2013-12-13 13:53:04 UTC
Created attachment 836333 [details]
First crash

Comment 8 Tomas Babej 2013-12-13 13:53:41 UTC
Created attachment 836334 [details]
Second crash

Comment 9 Tomas Babej 2013-12-13 13:58:30 UTC
Created attachment 836335 [details]
Log from the ipa-server-install and ipa-adtrust-install

Comment 10 Tomas Babej 2013-12-13 13:59:16 UTC
Created attachment 836336 [details]
Journalctl output since 14:20

Comment 13 Rich Megginson 2013-12-13 15:38:19 UTC
Created attachment 836356 [details]
stacktrace 1 - slapi-nis

Comment 14 Rich Megginson 2013-12-13 15:38:55 UTC
Nalin, can you take a look at https://bugzilla.redhat.com/show_bug.cgi?id=1041732#c13 ?

Comment 15 Rich Megginson 2013-12-13 17:26:59 UTC
Created attachment 836423 [details]
stacktrace 2

The problem is at list_candidates():855:
        } else if ( ftype == LDAP_FILTER_AND ) {
            if (isnot && !idl_is_allids(tmp)) {
the search returned tmp == NULL (*err = -30988 DB_NOTFOUND) because objectclass=mepManagedEntry was not found.  This problem was introduced with this commit:

commit fae006821bd6e524c0f7f8d5f023f4fe5e160ef0
Author: Noriko Hosoi <nhosoi@redhat.com>
Date:   Wed Apr 17 14:55:56 2013 -0700

    Ticket #47313 - Indexed search with filter containing '&' and "!" with attribute subtypes gives wrong result

Comment 16 Tomas Babej 2013-12-16 15:23:50 UTC
Nalin, I created a separate tracking Bug 1043546 for the slapi-nis crash.

Comment 17 Noriko Hosoi 2013-12-16 17:18:12 UTC
(In reply to Rich Megginson from comment #15)
> Created attachment 836423 [details]
> stacktrace 2
> 
> The problem is at list_candidates():855:
>         } else if ( ftype == LDAP_FILTER_AND ) {
>             if (isnot && !idl_is_allids(tmp)) {
> the search returned tmp == NULL (*err = -30988 DB_NOTFOUND) because
> objectclass=mepManagedEntry was not found.  This problem was introduced with
> this commit:
> 
> commit fae006821bd6e524c0f7f8d5f023f4fe5e160ef0
> Author: Noriko Hosoi <nhosoi@redhat.com>
> Date:   Wed Apr 17 14:55:56 2013 -0700
> 
>     Ticket #47313 - Indexed search with filter containing '&' and "!" with
> attribute subtypes gives wrong result

The crash was fixed with this patch.
 0001-Ticket-47313-Indexed-search-with-filter-containing-a.2.patch​ (3.5 KB) - added by nhosoi 3 days ago.
    git patch file (master) -- Bug fix for bz 1041732 

If this bug 1041732 is about this stacktrace2, can we change the status to "POST"?

Comment 18 Rich Megginson 2013-12-16 17:35:08 UTC
(In reply to Noriko Hosoi from comment #17)
> (In reply to Rich Megginson from comment #15)
> > Created attachment 836423 [details]
> > stacktrace 2
> > 
> > The problem is at list_candidates():855:
> >         } else if ( ftype == LDAP_FILTER_AND ) {
> >             if (isnot && !idl_is_allids(tmp)) {
> > the search returned tmp == NULL (*err = -30988 DB_NOTFOUND) because
> > objectclass=mepManagedEntry was not found.  This problem was introduced with
> > this commit:
> > 
> > commit fae006821bd6e524c0f7f8d5f023f4fe5e160ef0
> > Author: Noriko Hosoi <nhosoi@redhat.com>
> > Date:   Wed Apr 17 14:55:56 2013 -0700
> > 
> >     Ticket #47313 - Indexed search with filter containing '&' and "!" with
> > attribute subtypes gives wrong result
> 
> The crash was fixed with this patch.
>  0001-Ticket-47313-Indexed-search-with-filter-containing-a.2.patch​ (3.5 KB)
> - added by nhosoi 3 days ago.
>     git patch file (master) -- Bug fix for bz 1041732 
> 
> If this bug 1041732 is about this stacktrace2, can we change the status to
> "POST"?

Yes.

Comment 19 Noriko Hosoi 2013-12-16 17:36:55 UTC
(In reply to Rich Megginson from comment #18)
> (In reply to Noriko Hosoi from comment #17)
> > If this bug 1041732 is about this stacktrace2, can we change the status to
> > "POST"?
> 
> Yes.

Thanks, Rich!

Comment 20 Tomas Babej 2013-12-16 23:44:42 UTC
The 389-ds-base-1.3.2.9-1 build fixed the issue for me. Thanks!

Comment 21 Adam Williamson 2013-12-28 02:46:10 UTC
The update has gone stable for F20 now; presumably this can be closed?


Note You need to log in before you can comment on or make changes to this bug.