Created attachment 592150 [details]
File: dso_list

Created attachment 592151 [details]
File: maps

Created attachment 592152 [details]
File: backtrace

This looks similar to https://fedorahosted.org/389/ticket/347 and https://fedorahosted.org/389/ticket/382 but not exactly - these issues are fixed in 1.2.10.9, available from updates-testing - can you try it?

yum update --enablerepo=updates-testing 389-ds-base 389-ds-base-libs 389-ds-base-debuginfo

Yeah, already tried, same crash.

Can we have your configuration/steps to reproduce the problem?

For instance, do you have multiple masters?  If yes, how many?

Could it be possible to share your error log and access log on the crashed server?
(/var/log/dirsrv/slapd-<ID>/{access,errors})

Yeah:

1. Setup FreeIPA (A) on F16 a long while ago, dealt with other crash issues.
2. Setup mirror in another datacenter (B), DNS + LDAP, no CA, no (local) NTP
3. Upgrade #1 to fIPA 2.1.4-5 + 389 1.2.10.6
4. Upgrade #2 to fIPA 2.1.4-7 + 389 1.2.10.6
5. Install new mirror in third datacenter (C), DNS + LDAP + CA, no (local) NTP
6. New mirror starts crashing after a few minutes.

7. Removed mirror C from replicas list using IPA tools (was previously 3-way master)
8. Removed mirror C from SRV records
9. Mirror C stops crashing.

last bit in the error log:

[15/Jun/2012:10:54:49 -0400] - 389-Directory/1.2.10.6 B2012.101.2023 starting up
[15/Jun/2012:10:54:49 -0400] - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[15/Jun/2012:10:54:50 -0400] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=eladian,dc=com--no CoS Templates found, which should be added before the CoS Definition.
[15/Jun/2012:10:54:50 -0400] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=eladian,dc=com--no CoS Templates found, which should be added before the CoS Definition.
[15/Jun/2012:10:54:50 -0400] set_krb5_creds - Could not get initial credentials for principal [ldap/ipa1-nyc4.eladian.com] in keytab [WRFILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see e-text))
[15/Jun/2012:10:54:50 -0400] - slapd started.  Listening on All Interfaces port 389 for LDAP requests
[15/Jun/2012:10:54:50 -0400] - Listening on All Interfaces port 636 for LDAPS requests
[15/Jun/2012:10:54:50 -0400] - Listening on /var/run/slapd-ELADIAN-COM.socket for LDAPI requests
[15/Jun/2012:10:54:50 -0400] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code 
may provide more information (Credentials cache file '/tmp/krb5cc_995' not found)) errno 0 (Success)
[15/Jun/2012:10:54:50 -0400] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
[15/Jun/2012:10:54:50 -0400] NSMMReplicationPlugin - agmt="cn=meToipa1-nyc1.eladian.com" (ipa1-nyc1:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Credentials cache file '/tmp/krb5cc_995' not found))
[15/Jun/2012:10:54:54 -0400] NSMMReplicationPlugin - agmt="cn=meToipa1-nyc1.eladian.com" (ipa1-nyc1:389): Replication bind with GSSAPI auth resumed

Last bit in the access log:q

[15/Jun/2012:10:54:54 -0400] conn=5 op=2 SRCH base="dc=eladian,dc=com" scope=2 filter="(&(|(objectClass=krbprincipalaux)(objectClass=krbprincipal))(krbPrincipalName=ldap/ipa1-nyc1.eladian.com))" attrs="krbPrincipalName krbCanonicalName objectClass krbPrincipalKey krbMaxRenewableAge krbMaxTicketLife krbTicketFlags krbPrincipalExpiration krbTicketPolicyReference krbUPEnabled krbPwdPolicyReference krbPasswordExpiration krbLastFailedAuth krbLoginFailedCount krbLastSuccessfulAuth nsAccountLock krbLastPwdChange krbLastAdminUnlock krbExtraData krbObjectReferences krballowedtodelegateto"
[15/Jun/2012:10:54:54 -0400] conn=5 op=2 RESULT err=0 tag=101 nentries=1 etime=0
[15/Jun/2012:10:54:54 -0400] conn=5 op=3 SRCH base="dc=eladian,dc=com" scope=2 filter="(&(|(objectClass=krbprincipalaux)(objectClass=krbprincipal))(krbPrincipalName=ldap/ipa1-nyc4.eladian.com))" attrs="krbPrincipalName krbCanonicalName objectClass krbPrincipalKey krbMaxRenewableAge krbMaxTicketLife krbTicketFlags krbPrincipalExpiration krbTicketPolicyReference krbUPEnabled krbPwdPolicyReference krbPasswordExpiration krbLastFailedAuth krbLoginFailedCount krbLastSuccessfulAuth nsAccountLock krbLastPwdChange krbLastAdminUnlock krbExtraData krbObjectReferences krballowedtodelegateto"
[15/Jun/2012:10:54:54 -0400] conn=5 op=3 RESULT err=0 tag=101 nentries=1 etime=0

Jun 15 10:55:22 ipa1-nyc4 kernel: [ 2720.321845] ns-slapd[2323]: segfault at 10 ip 00007f5df9aaeb5f sp 00007f5dd2ff66f0 error 4 in libslapd.so.0.0.0[7f5df9a6d000+ef000]

Probably also worth noting that the CA dirsrv was stable, it's the domain dirsrv that's crashing.

What are the operations coming in from bind?  If you turn off bind, and the crashes go away, it would be very useful to know what sorts of operations bind is doing.  The problem with the access log is that it is buffered, so the actual operation causing the crash could still be in memory in the buffer and not written to disk.

Something like this, every 30 seconds-or-so:

[15/Jun/2012:15:54:15 -0400] conn=3 op=1816 SRCH base="cn=dns,dc=eladian,dc=com" scope=2 filter="(objectClass=idnsConfigObject)" attrs="idnsforwardpolicy idnsforwarders idnsallowsyncptr idnszonerefresh idnspersistentsearch"
[15/Jun/2012:15:54:15 -0400] conn=3 op=1816 RESULT err=0 tag=101 nentries=0 etime=0
[15/Jun/2012:15:54:15 -0400] conn=3 op=1817 SRCH base="cn=dns,dc=eladian,dc=com" scope=2 filter="(&(objectClass=idnsZone)(idnsZoneActive=TRUE))" attrs="idnsName idnsUpdatePolicy idnsallowquery idnsallowtransfer idnsforwardpolicy idnsforwarders"
[15/Jun/2012:15:54:15 -0400] conn=3 op=1817 RESULT err=0 tag=101 nentries=7 etime=0
[15/Jun/2012:15:54:15 -0400] conn=4 op=2911 SRCH base="idnsname=31.172.in-addr.arpa.,cn=dns,dc=eladian,dc=com" scope=0 filter="(objectClass=idnsRecord)" attrs=ALL
[15/Jun/2012:15:54:15 -0400] conn=4 op=2911 RESULT err=0 tag=101 nentries=1 etime=0
[15/Jun/2012:15:54:15 -0400] conn=4 op=2912 SRCH base="idnsname=17.172.in-addr.arpa.,cn=dns,dc=eladian,dc=com" scope=0 filter="(objectClass=idnsRecord)" attrs=ALL
[15/Jun/2012:15:54:15 -0400] conn=4 op=2912 RESULT err=0 tag=101 nentries=1 etime=0
[15/Jun/2012:15:54:15 -0400] conn=4 op=2913 SRCH base="idnsname=168.192.in-addr.arpa.,cn=dns,dc=eladian,dc=com" scope=0 filter="(objectClass=idnsRecord)" attrs=ALL
[15/Jun/2012:15:54:15 -0400] conn=4 op=2913 RESULT err=0 tag=101 nentries=1 etime=0
[15/Jun/2012:15:54:15 -0400] conn=4 op=2914 SRCH base="idnsname=30.172.in-addr.arpa.,cn=dns,dc=eladian,dc=com" scope=0 filter="(objectClass=idnsRecord)" attrs=ALL
[15/Jun/2012:15:54:15 -0400] conn=4 op=2914 RESULT err=0 tag=101 nentries=1 etime=0
[15/Jun/2012:15:54:15 -0400] conn=4 op=2915 SRCH base="idnsname=239.in-addr.arpa.,cn=dns,dc=eladian,dc=com" scope=0 filter="(objectClass=idnsRecord)" attrs=ALL
[15/Jun/2012:15:54:15 -0400] conn=4 op=2915 RESULT err=0 tag=101 nentries=1 etime=0

Access log is buffered, so the last op which could the cause of the crash is not captured.

Could it be possible to use this named-pipe for the access log with no buffering?  Thanks!!

http://docs.redhat.com/docs/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/using-named-pipe.html
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
15.4.1. Using the Named Pipe for Logging
The Directory Server instance can use a named pipe for its logging simply by running the named pipe log script and giving the name of the pipe. (If the server is already running, then the log has to be reopened, but there is no configuration required otherwise.)

ds-logpipe.py /var/log/dirsrv/slapd-example/access

Running the ds-logpipe.py in this way has the advantage of being simple to implement and not requiring any Directory Server configuration changes. This is useful for fast debugging or monitoring, especially if you're looking for a specific type of event.
If the Directory Server instance will frequently or permanently use the named pipe rather than a real file for logging, then it is possible to reconfigure the instance to create the named pipe and use it for logging (as it does by default for the log files).
Three things need to be configured for the log configuration for the instance:

    The log file to use has to be changed to the pipe (nsslapd-*log, where the * can be access, error, or audit[5], depending on the log type being configured)
    Buffering should be disabled because the script already buffers the log entries (nsslapd-*log-logbuffering)
    Log rotation should be disabled so that the server doesn't attempt to rotate the named pipe (nsslapd-*log-maxlogsperdir, nsslapd-*log-logexpirationtime, and nsslapd-*log-logrotationtime) 

These configuration changes can be made in the Directory Server Console or using ldapmodify. 
For example, this switches the access log to access.pipe:

ldapmodify -D "cn=directory manager" -w secret -p 389 -h server.example.com -x

dn: cn=config
changetype: modify
replace: nsslapd-accesslog
nsslapd-accesslog: /var/log/dirsrv/slapd-instance_name/access.pipe
-
replace: nsslapd-accesslog-logbuffering
nsslapd-accesslog-logbuffering: off
-
replace: nsslapd-accesslog-maxlogsperdir
nsslapd-accesslog-maxlogsperdir: 1
- 
replace: nsslapd-accesslog-logexpirationtime
nsslapd-accesslog-logexpirationtime: -1
- 
replace: nsslapd-accesslog-logrotationtime
nsslapd-accesslog-logrotationtime: -1
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Hi James,

May I ask you one more thing?  I think I found the cause of the crash, but since cannot reproduce the problem in house, there's no way to verify it.  Could you please help us there?

I've put a tar ball containing my local build 389-ds-base-1.2.10.9.tar.gz at http://nhosoi.fedorapeople.org/.

Could you download and try it?
This is the contents of tar ball.
$ tar tvzf 389-ds-base-1.2.10.9.tar.gz 
drwxrwxr-x nhosoi/nhosoi     0 2012-06-15 14:14 RPMS/
drwxr-xr-x nhosoi/nhosoi     0 2012-06-15 14:14 RPMS/x86_64/
-rw-rw-r-- nhosoi/nhosoi 1371453 2012-06-15 14:14 RPMS/x86_64/389-ds-base-1.2.10.9-1.fc16.x86_64.rpm
-rw-rw-r-- nhosoi/nhosoi 3943441 2012-06-15 14:14 RPMS/x86_64/389-ds-base-debuginfo-1.2.10.9-1.fc16.x86_64.rpm
-rw-rw-r-- nhosoi/nhosoi  365753 2012-06-15 14:14 RPMS/x86_64/389-ds-base-libs-1.2.10.9-1.fc16.x86_64.rpm
-rw-rw-r-- nhosoi/nhosoi  108669 2012-06-15 14:14 RPMS/x86_64/389-ds-base-devel-1.2.10.9-1.fc16.x86_64.rpm

Please note that this is not a supported bit.  As you see above, it's built by me on my local host... :)  Thank you so much for your help!

I can certainly turn logging via a named pipe, but if the crashing member of a multi-master replica set (in IPA), will this also be replicated to the other members (i.e. am I going to add the overhead of full-bore logging to all of my machines, or just the one)?

(In reply to comment #13)
> I can certainly turn logging via a named pipe, but if the crashing member of
> a multi-master replica set (in IPA), will this also be replicated to the
> other members (i.e. am I going to add the overhead of full-bore logging to
> all of my machines, or just the one)?

We should have a fix out to updates-testing shortly, so just hold off on any additional debugging/logging for now.

Upstream ticket:
https://fedorahosted.org/389/ticket/390

389-ds-base-1.2.10.10-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/389-ds-base-1.2.10.10-1.fc16

389-ds-base-1.2.10.10-1.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/389-ds-base-1.2.10.10-1.fc15

Package 389-ds-base-1.2.10.10-1.fc15:
* should fix your issue,
* was pushed to the Fedora 15 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing 389-ds-base-1.2.10.10-1.fc15'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-9651/389-ds-base-1.2.10.10-1.fc15
then log in and leave karma (feedback).

389-ds-base-1.2.10.11-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/389-ds-base-1.2.10.11-1.fc16

389-ds-base-1.2.10.11-1.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/389-ds-base-1.2.10.11-1.fc15

389-ds-base-1.2.10.12-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/389-ds-base-1.2.10.12-1.fc16

389-ds-base-1.2.10.12-1.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/389-ds-base-1.2.10.12-1.fc15

389-ds-base-1.2.10.13-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/389-ds-base-1.2.10.13-1.fc16

389-ds-base-1.2.10.14-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/389-ds-base-1.2.10.14-1.fc16

389-ds-base-1.2.10.14-1.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.

This message is a reminder that Fedora 16 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 16. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '16'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 16's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 16 is end of life. If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora, you are encouraged to click on 
"Clone This Bug" and open it against that version of Fedora.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Fedora 16 changed to end-of-life (EOL) status on 2013-02-12. Fedora 16 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.