Bug 811291

Summary: [abrt] 389-ds-base-1.2.10.4-2.fc16: index_range_read_ext: Process /usr/sbin/ns-slapd was killed by signal 11 (SIGSEGV)
Product: Red Hat Enterprise Linux 6 Reporter: Rich Megginson <rmeggins>
Component: 389-ds-baseAssignee: Rich Megginson <rmeggins>
Status: CLOSED ERRATA QA Contact: IDM QE LIST <seceng-idm-qe-list>
Severity: high Docs Contact:
Priority: high    
Version: 6.3CC: jgalipea, mreynolds, nkinder, shaines, sramling
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 389-ds-base-1.2.10.2-6.el6 Doc Type: Bug Fix
Doc Text:
Cause: Performing delete and ranged search operations against the directory server under a high load. Consequence: Directory server crashes. Fix: Entries may be deleted out from under a ranged search request. Server should handle this case by not returning deleted entries and not crashing. Result: Server does not crash when performing range searches and deletions while under a high load.
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-06-20 07:15:05 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Rich Megginson 2012-04-10 16:02:16 UTC
This bug is created as a clone of upstream ticket:
https://fedorahosted.org/389/ticket/336

https://bugzilla.redhat.com/show_bug.cgi?id=808770 (''Fedora'')

{{{
libreport version: 2.0.8
abrt_version:   2.0.7
backtrace_rating: 4
cmdline:        /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-ELADIAN-COM -i
/var/run/dirsrv/slapd-ELADIAN-COM.pid -w
/var/run/dirsrv/slapd-ELADIAN-COM.startpid
comment:
crash_function: index_range_read_ext
executable:     /usr/sbin/ns-slapd
kernel:         3.1.6-1.fc16.x86_64
pid:            19109
pwd:            /var/log/dirsrv/slapd-ELADIAN-COM
reason:         Process /usr/sbin/ns-slapd was killed by signal 11 (SIGSEGV)
time:           Sat 31 Mar 2012 11:08:28 AM EDT
uid:            996
username:       dirsrv

backtrace:      Text file, 165873 bytes
dso_list:       Text file, 8455 bytes
maps:           Text file, 45972 bytes

build_ids:
:28579c89bda7f08dace218884d7a5561f3f5f89d
:ccd23fb93a7924ddd27ea9e7f054fa5309c92f32
:ee2f04900ae1f07517d91eba300ef385fccab1b8
:89017ab0b75e533e4294f74ee8f7db3daceea536
:bb5c9b5cca6d04252b061a2f840ab4ddfda9e733
:3ba5302e34546a45e9a85ac3429e7bc6ed624011
:638db099ff5fb986d1a092629d1bb1dca5be4904
:6a8868ece50dc746e4771ff0b5f6f314a45f4a77
:056358b51146e7dc68f8f712cf2b208bb4bd3654
:0f0c9eed63ed279ceb0aef109eff8bcbb351e4bb
:0bd515c1c778bb4b8f870a900d1543da39d6b662
:5083c24b4fd7dbf3cac9cf4c15876b0dfdceb4bb
:2639b5594fe4d7897db63ecd1ebb9681438317a8
:8d4c1cc04914e6f8dce0b485275a05d510543280
:48e9e9baf9f0cbd3653cb77ed45b83a7ee8aef2e
:2f71e81631ba518b573b7678465bcce24ae8530f
:1b2a3adcb5fa5b0594f5c9182e6a12e83d47b4b8
:4d7facfa8aead1e0060760f76c77f3ae0ed5becc
:ecd5ba5435e91ca72281c7c74cf39bf92e01b3a5
:5ac56dbb5866c475e1b0c3e4ceeb390b0f7508c0
:34e8b3cee73a6d0b8ac17c69ba4b1170724db64c
:aac33d0080dfd5b813520f4e8a5c36378e720b60
:a58aa84cd66c6863b87b0879885229c390c55162
:59292332bd88fce31f7557e60ff08d44ce254ec8
:cce23724534ba89f256874d714ee35672bc9f28b
:b01d2bc881174324d6dc5cbf0a3605239c70c72e
:ef95ed2b74ff19753827117d4e556f38b33e74b5
:9e12672793058b2f142b642851576ad036a2c4e6
:33324b7717adf03b8578f0b91fe2d53d765d3120
:0487340b0c803f5800eadd5238d3d9b2435ceeec
:c9f8613367b2e2e1d577edf20afa7ff17b3397ad
:0a49b6fd90960659888f0be3bb576495f9d41959
:ac2b4cc5fed8f857807a8830b4773f85be9642b2
:9f9aa00c80e740cbcf6628a52548bc352b90c898
:01175202f32ef3226de9abfe9384866107ba242c
:a0b2903547454e47fec356c1fa1a69a7593893c5
:ea44661a40777d6be2f79058c161d3bb4962891a
:a9791ba4315ed37156e8316c3e1380d8af2e33dd
:4b4285058f7f6b39b7e7c45df83fd36c6833bfe4
:e49b84b81d7329cc916ec84b2ec5bbf2f8852ff9
:6f130628ef6eb5e6c4203baef32d8382af19737a
:516780dcc6528878fd0301dbb03b30252111de56
:723cc9153ba157f7be2a431e089b875d9168e763
:8822ec50a7ca08984b34af0b3096794a5085c473
:1a212c7f1515542b310ba92f6109efc9b5bf2b6e
:7f1fbc967e3fba1e7008912447f54b7260204d1c
:919cd69551c389bdb0131a27a4d7ffc7c1ea6d2b
:57088c4741c6f62eb5d39889da7533bf7cc157f0
:64c606de82a17e176ea5c69c9571f4bcfd6c080d
:065f24e263b3a0d4316bfeedf855df8c2c35c03a
:c49811439aeff6168ae82e7cf808a8c03b1c9e48
:0ce9819eb7014de7e72eec40979dd5e4d6566dca
:bccc101a872bfd4670545e87b5d4004a2f890052
:c314e9cb57367719196b6f665fe6773a2d08add6
:54aeedc57e5f902a83f20308416ac2213627e5ca
:5ba371439acb075c48406bd7c66c80594d91c7e7
:e2e67b3ae2579e1667b73ad385f61552931024e9
:b4272a5a90d21f312b476d02b575b7222ff17e91
:6502dd4813f98137c70e6e05cc43828f5c2263c1
:dacf32a9a2f2b5077ba944e7b835cca6f637f78f
:fee9f980714278bbbbf7fa217189472cd6446f6d
:20efc30fde6abc23cfe693739cff23d67ee9d333
:75eaece6e0d1e075520cdfad5c398fdab4370846
:63365768c3b6cad3c5a303e46543c113a5326b80
:e814ef8432b7dec42a0b8ec12b9abd9fc7f57b40
:a28b15796672928bb44b262c88052ae601bf5955
:2982ba94b3d248b639aaa6218be955bb159bb2e3
:36fc045d11fc55f1b455247938e651666e2c127e
:f1e8182609d9eae40a8ddbbb72c9346166014f1a
:b010321c92dc1b5abf0007da423b778542817883
:efd7e116c58ad407b8936018a86a399993a11e79
:552354979eebf3d6c27346691a5b520ccf8f4e75
:aad91f4f4e3c8a5d8cf3be0deac2ac041b4e4e1c
:e5429e0905bedecc534b057783916ba43e06a66e
:788f31e354f4ad30dd74b958d821c980b37d04c2
:68c27733d3164d71c325fe621dd9bddf0f97ab47
:eac10d05c580f122193f7560fb72e743af88edb3
:a9a3ce4a014e11a12a90a0979d113504adada32d
:b9d3a4213a482d034bebc7dc1ed2901a734a894d
:789c958162e98ef211c3bebd162ab587eb9ece93
:b5918178168db4577a7744fef13d892cb5ac6d3f
:83cb4627bed5c756d4e6f23badf8a6b4b042f4e6
:040f78c63e49df93936f710287bdc74ae83111c0
:e567a4b498aa9dfc484575e666a17925f0944318
:2d4ac24a95409c92b794b521e85670af3d416590
:5734da0b584b126fcdfc17210761d923ff95cb26
:4613bd742f4dca43dc32603622cca5692dd60c24
:12f7326885e79c7a690d8b6a22109af42760f6af
:b3db147cfb8af552c8dd1b47c81dc66760eaea80

environ:
:BOOT_IMAGE=/vmlinuz-3.1.6-1.fc16.x86_64
:PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
:SYSFONT=latarcyrheb-sun16
:LANG=en_US.UTF-8
:PWD=/
:KEYTABLE=us
:'ulimit -n 8192'
:KRB5_KTNAME=/etc/dirsrv/ds.keytab
:'export KRB5_KTNAME=/etc/dirsrv/ds.keytab'
:SERVER_DIR=/usr/lib64/dirsrv
:SERVERBIN_DIR=/usr/sbin
:CONFIG_DIR=/etc/dirsrv/slapd-ELADIAN-COM
:INST_DIR=/var/lib/dirsrv/scripts-ELADIAN-COM
:RUN_DIR=/var/run/dirsrv
:PRODUCT_NAME=slapd

var_log_messages:
:Mar 25 03:25:11 ipa1-nyc1 ns-slapd: GSSAPI Error: An invalid name was supplied
(Hostname cannot be canonicalized)
:Mar 26 08:49:48 ipa1-nyc1 ns-slapd: GSSAPI Error: An invalid name was supplied
(Hostname cannot be canonicalized)
:Mar 26 14:29:19 ipa1-nyc1 ns-slapd: GSSAPI Error: An invalid name was supplied
(Hostname cannot be canonicalized)
:Mar 26 16:36:33 ipa1-nyc1 ns-slapd: GSSAPI Error: An invalid name was supplied
(Hostname cannot be canonicalized)
:Mar 27 15:30:23 ipa1-nyc1 kernel: [7150921.037468] ns-slapd[27549]: segfault
at 0 ip 00007f136092ac07 sp 00007f13497ee650 error 4 in
libback-ldbm.so[7f13608ec000+8d000]
:Mar 27 16:07:36 ipa1-nyc1 ns-slapd: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (Credentials cache file
'/tmp/krb5cc_996' not found)
:Mar 28 04:39:12 ipa1-nyc1 kernel: [7198275.020687] ns-slapd[15998]: segfault
at 0 ip 00007f388d8eec07 sp 00007f38757ee650 error 4 in
libback-ldbm.so[7f388d8b0000+8d000]
:Mar 28 07:01:26 ipa1-nyc1 ns-slapd: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (Credentials cache file
'/tmp/krb5cc_996' not found)
:Mar 28 09:44:27 ipa1-nyc1 kernel: [7216590.050214] ns-slapd[24987]: segfault
at 0 ip 00007f7c62c08c07 sp 00007f7c4b7f2650 error 4 in
libback-ldbm.so[7f7c62bca000+8d000]
:Mar 28 10:00:59 ipa1-nyc1 ns-slapd: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (Credentials cache file
'/tmp/krb5cc_996' not found)
:Mar 28 17:50:59 ipa1-nyc1 kernel: [7245793.824167] ns-slapd[27051]: segfault
at 17d ip 00000031d74810a4 sp 00007fff17327358 error 4 in libc-2.14.90.so
(deleted)[31d7400000+1ab000]
:Mar 28 17:51:43 ipa1-nyc1 ns-slapd: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (Credentials cache file
'/tmp/krb5cc_996' not found)
:Mar 28 17:53:33 ipa1-nyc1 ns-slapd: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (Credentials cache file
'/tmp/krb5cc_996' not found)
:Mar 28 17:53:59 ipa1-nyc1 ns-slapd[14914]: [28/Mar/2012:17:53:59 -0400] -
Information: Non-Secure Port Disabled
:Mar 28 17:56:06 ipa1-nyc1 ns-slapd[15846]: [28/Mar/2012:17:56:06 -0400] -
Information: Non-Secure Port Disabled
:Mar 28 17:56:08 ipa1-nyc1 ns-slapd: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (Credentials cache file
'/tmp/krb5cc_996' not found)
:Mar 28 18:09:51 ipa1-nyc1 ns-slapd: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (Credentials cache file
'/tmp/krb5cc_996' not found)
:Mar 29 20:33:09 ipa1-nyc1 kernel: [7341921.236497] ns-slapd[20295]: segfault
at 0 ip 00007f4e472cb0a7 sp 00007f4e267e8650 error 4 in
libback-ldbm.so[7f4e4728c000+8d000]
:Mar 29 21:41:19 ipa1-nyc1 ns-slapd: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (Credentials cache file
'/tmp/krb5cc_996' not found)
:Mar 29 22:16:29 ipa1-nyc1 ns-slapd: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (Credentials cache file
'/tmp/krb5cc_996' not found)
:Mar 31 11:08:28 ipa1-nyc1 kernel: [7480792.701830] ns-slapd[19141]: segfault
at 0 ip 00007f644bb230a7 sp 00007f6427fe3650 error 4 in
libback-ldbm.so[7f644bae4000+8d000]
:Mar 31 11:08:39 ipa1-nyc1 abrt[8052]: Saved core dump of pid 19109
(/usr/sbin/ns-slapd) to /var/spool/abrt/ccpp-2012-03-31-11:08:28-19109
(477249536 bytes)
}}}

Comment 1 Jenny Severance 2012-04-10 16:40:12 UTC
Please add steps to reproduce/verify

Comment 3 Rich Megginson 2012-04-10 19:11:28 UTC
Steps to reproduce/verify:

1) set up two masters for multi-master replication
2) enable the USN plugin on both servers
3) add indexes for uidNumber and gidNumber e.g.

dn: cn=uidnumber,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
objectclass: top
objectclass: nsIndex
cn: uidnumber
nssystemindex: false
nsindextype: eq
nsmatchingrule: integerOrderingMatch

dn: cn=gidnumber,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
objectclass: top
objectclass: nsIndex
cn: gidnumber
nssystemindex: false
nsindextype: eq
nsmatchingrule: integerOrderingMatch

in a separate shell/window, do the following:
usn=0 ; while [ 1 ] ; do ii=100 ; while [ $ii -gt 0 ] ; do ii=`expr $ii - 1` ; ldapsearch -xLLL -h localhost -p PORTNUMBEROFMASTER2 -D "cn=directory manager" -w password -b dc=example,dc=com -E pr=100/noprompt '(&(objectClass=posixAccount)(uid=*)(uidNumber=*)(gidNumber=*)(entryusn>='$usn')(!(entryusn='$usn')))' > /dev/null 2>&1 & done ; wait ; done

That is, do 100 ldapsearch processes all doing a simple paged range search, and wait for them all to complete, then do it again

in another shell/window, add 1000 posix users like this:
dn: uid=testuser$ii,ou=people,dc=example,dc=com
objectclass: inetOrgPerson
objectclass: posixAccount
cn: Test User$ii
sn: User$ii
uidNumber: 500$ii
gidNumber: 500$ii
homeDirectory: /home/testuser$ii

confirm that all users are replicated to both servers
next, delete all 1000 users - I do something like this:

ii=1000 ; while [ $ii -gt 0 ] ; do
  dn="uid=testuser$ii,ou=people,dc=example,dc=com"
  ldapdelete -x -h localhost -p PORTNUMBEROFMASTER1 -D "cn=directory manager" -w password "$dn"
  ii=`expr $ii - 1`
done

Usually you will see the ldapsearches complete with a message of "Done ..." - if you see them complete with an message like "Exit: 255 ..." or "Error: ..." that usually means the server has crashed.

Comment 4 Rich Megginson 2012-05-24 23:53:23 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause: Performing delete and ranged search operations against the directory server under a high load.
Consequence: Directory server crashes.
Fix: Entries may be deleted out from under a ranged search request.  Server should handle this case by not returning deleted entries and not crashing.
Result: Server does not crash when performing range searches and deletions while under a high load.

Comment 5 Sankar Ramalingam 2012-05-30 19:57:29 UTC
As given in comment #3, I ran simplepaged ldapsearch, ldapdelete in multiple threads. I didn't see any crash in the server. Hence, marking the bug as Verified.

Comment 6 errata-xmlrpc 2012-06-20 07:15:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-0813.html