Bug 988562 - deadlock after adding and deleting entries
Summary: deadlock after adding and deleting entries
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: 389-ds-base
Version: 6.5
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: rc
: ---
Assignee: Rich Megginson
QA Contact: Sankar Ramalingam
URL:
Whiteboard:
Depends On: 988559
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-07-25 20:14 UTC by Nathan Kinder
Modified: 2013-11-21 21:11 UTC (History)
4 users (show)

Fixed In Version: 389-ds-base-1.2.11.15-22.el6
Doc Type: Bug Fix
Doc Text:
Clone Of: 988559
Environment:
Last Closed: 2013-11-21 21:11:19 UTC
Target Upstream Version:


Attachments (Terms of Use)
LDIF file that adds and deletes 200 entries (31.87 KB, text/x-ldif)
2013-07-25 20:20 UTC, mreynolds
no flags Details
LDIF file that adds and deletes 200 entries (2) (32.46 KB, text/x-ldif)
2013-07-25 20:20 UTC, mreynolds
no flags Details
LDIF file that adds and deletes 200 entries (3) (31.87 KB, text/x-ldif)
2013-07-25 20:21 UTC, mreynolds
no flags Details
LDIF file that adds and deletes 200 entries (4) (32.46 KB, text/x-ldif)
2013-07-25 20:22 UTC, mreynolds
no flags Details
LDIF file that adds and deletes 200 entries (5) (32.46 KB, text/x-ldif)
2013-07-25 20:23 UTC, mreynolds
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2013:1653 normal SHIPPED_LIVE 389-ds-base bug fix update 2013-11-20 21:53:19 UTC

Description Nathan Kinder 2013-07-25 20:14:13 UTC
+++ This bug was initially created as a clone of Bug #988559 +++

This bug is created as a clone of upstream ticket:
https://fedorahosted.org/389/ticket/47449

If you have multiple clients, each adding and deleting users the server will deadlock.  I created 5 ldif files.  Each ldif file added and then deleted 200 entries.  Using 5 separate ldapmodify's the server will deadlock within a minute or so.

Appears to be an issue with an entry cache lock not being unlocked:

Thread 29 (Thread 0x7f7d16bfd700 (LWP 8337)):
#3  0x0000003b56223fe9 in PR_Lock () from /lib64/libnspr4.so
#4  0x0000003b5622410b in PR_EnterMonitor () from /lib64/libnspr4.so
#5  0x00007f7d29eb19bb in dblayer_lock_backend (be=0x2094160) at ../ds/ldap/servers/slapd/back-ldbm/dblayer.c:3942
#6  0x00007f7d29eb102f in dblayer_txn_begin (be=0x2094160, parent_txn=0x0, txn=0x7f7d16bfa860) at ../ds/ldap/servers/slapd/back-ldbm/dblayer.c:3664
#7  0x00007f7d29eeb814 in ldbm_back_delete (pb=0x7f7d16bfcaa0) at ../ds/ldap/servers/slapd/back-ldbm/ldbm_delete.c:257
#8  0x00007f7d2dc8def4 in op_shared_delete (pb=0x7f7d16bfcaa0) at ../ds/ldap/servers/slapd/delete.c:364
#9  0x00007f7d2dc8d6dd in do_delete (pb=0x7f7d16bfcaa0) at ../ds/ldap/servers/slapd/delete.c:128
#10 0x000000000041578e in connection_dispatch_operation (conn=0x7f7d2464f730, op=0x231ee80, pb=0x7f7d16bfcaa0) at ../ds/ldap/servers/slapd/connection.c:643
#11 0x0000000000417524 in connection_threadmain () at ../ds/ldap/servers/slapd/connection.c:2482

Thread 27 (Thread 0x7f7d157fb700 (LWP 8339)):
#3  0x0000003b56223fe9 in PR_Lock () from /lib64/libnspr4.so
#4  0x0000003b5622410b in PR_EnterMonitor () from /lib64/libnspr4.so
#5  0x00007f7d29eb19bb in dblayer_lock_backend (be=0x2094160) at ../ds/ldap/servers/slapd/back-ldbm/dblayer.c:3942
#6  0x00007f7d29eb102f in dblayer_txn_begin (be=0x2094160, parent_txn=0x0, txn=0x7f7d157f6790) at ../ds/ldap/servers/slapd/back-ldbm/dblayer.c:3664
#7  0x00007f7d29ede3f1 in ldbm_back_add (pb=0x7f7d157faaa0) at ../ds/ldap/servers/slapd/back-ldbm/ldbm_add.c:261
#8  0x00007f7d2dc7dc4b in op_shared_add (pb=0x7f7d157faaa0) at ../ds/ldap/servers/slapd/add.c:735
#9  0x00007f7d2dc7cb96 in do_add (pb=0x7f7d157faaa0) at ../ds/ldap/servers/slapd/add.c:258
#10 0x000000000041576c in connection_dispatch_operation (conn=0x7f7d2464f878, op=0x22ec000, pb=0x7f7d157faaa0) at ../ds/ldap/servers/slapd/connection.c:638
#11 0x0000000000417524 in connection_threadmain () at ../ds/ldap/servers/slapd/connection.c:2482

Thread 15 (Thread 0x7f7d09bf5700 (LWP 8351)):
#0  0x000000377560e054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000037756093be in _L_lock_995 () from /lib64/libpthread.so.0
#2  0x0000003775609326 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000003b56223fe9 in PR_Lock () from /lib64/libnspr4.so
#4  0x0000003b5622410b in PR_EnterMonitor () from /lib64/libnspr4.so
#5  0x00007f7d29eb19bb in dblayer_lock_backend (be=0x2094160) at ../ds/ldap/servers/slapd/back-ldbm/dblayer.c:3942
#6  0x00007f7d29eb102f in dblayer_txn_begin (be=0x2094160, parent_txn=0x0, txn=0x7f7d09bf0790) at ../ds/ldap/servers/slapd/back-ldbm/dblayer.c:3664
#7  0x00007f7d29ede3f1 in ldbm_back_add (pb=0x7f7d09bf4aa0) at ../ds/ldap/servers/slapd/back-ldbm/ldbm_add.c:261
#8  0x00007f7d2dc7dc4b in op_shared_add (pb=0x7f7d09bf4aa0) at ../ds/ldap/servers/slapd/add.c:735
#9  0x00007f7d2dc7cb96 in do_add (pb=0x7f7d09bf4aa0) at ../ds/ldap/servers/slapd/add.c:258
#10 0x000000000041576c in connection_dispatch_operation (conn=0x7f7d2464f4a0, op=0x232c850, pb=0x7f7d09bf4aa0) at ../ds/ldap/servers/slapd/connection.c:638
#11 0x0000000000417524 in connection_threadmain () at ../ds/ldap/servers/slapd/connection.c:2482


---> this thread is causing the deadlock

#3  0x0000003b56223fe9 in PR_Lock () from /lib64/libnspr4.so
#4  0x0000003b5622410b in PR_EnterMonitor () from /lib64/libnspr4.so
#5  0x00007f7d29ea7169 in cache_lock_entry (cache=0x21130b8, e=0x22f67a0) at ../ds/ldap/servers/slapd/back-ldbm/cache.c:1502
#6  0x00007f7d29ebee77 in find_entry_internal_dn (pb=0x7f7d073f0aa0, be=0x2094160, sdn=0x7f7ca400dec0, lock=1, txn=0x7f7d073ee860, flags=0)
    at ../ds/ldap/servers/slapd/back-ldbm/findentry.c:155
#7  0x00007f7d29ebf446 in find_entry_internal (pb=0x7f7d073f0aa0, be=0x2094160, addr=0x22f4b68, lock=1, txn=0x7f7d073ee860, flags=0)
    at ../ds/ldap/servers/slapd/back-ldbm/findentry.c:293
#8  0x00007f7d29ebf530 in find_entry2modify (pb=0x7f7d073f0aa0, be=0x2094160, addr=0x22f4b68, txn=0x7f7d073ee860)
    at ../ds/ldap/servers/slapd/back-ldbm/findentry.c:324
#9  0x00007f7d29eeb8b4 in ldbm_back_delete (pb=0x7f7d073f0aa0) at ../ds/ldap/servers/slapd/back-ldbm/ldbm_delete.c:273
#10 0x00007f7d2dc8def4 in op_shared_delete (pb=0x7f7d073f0aa0) at ../ds/ldap/servers/slapd/delete.c:364
#11 0x00007f7d2dc8d6dd in do_delete (pb=0x7f7d073f0aa0) at ../ds/ldap/servers/slapd/delete.c:128
#12 0x000000000041578e in connection_dispatch_operation (conn=0x7f7d2464f358, op=0x22f4a90, pb=0x7f7d073f0aa0) at ../ds/ldap/servers/slapd/connection.c:643
#13 0x0000000000417524 in connection_threadmain () at ../ds/ldap/servers/slapd/connection.c:2482

Comment 1 mreynolds 2013-07-25 20:20:12 UTC
Created attachment 778502 [details]
LDIF file that adds and deletes 200 entries

Comment 2 mreynolds 2013-07-25 20:20:54 UTC
Created attachment 778503 [details]
LDIF file that adds and deletes 200 entries (2)

Comment 3 mreynolds 2013-07-25 20:21:58 UTC
Created attachment 778504 [details]
LDIF file that adds and deletes 200 entries (3)

Comment 4 mreynolds 2013-07-25 20:22:30 UTC
Created attachment 778505 [details]
LDIF file that adds and deletes 200 entries (4)

Comment 5 mreynolds 2013-07-25 20:23:02 UTC
Created attachment 778506 [details]
LDIF file that adds and deletes 200 entries (5)

Comment 6 mreynolds 2013-07-25 20:27:39 UTC
Steps to reproduce:

[1]  Create an instance of 389 using "dc=example,dc=com" as the suffix.
[2]  Open 5 terminals, and run 5 ldapmodifys using the supplied ldifs at the same time. Example:

    ldapmodify -D "cn=directory manager" -w password -c -f add1.ldif

[3]  Keep running these ldapmodifys until the server hangs.  Usually happens in less than a minute.

Comment 8 Amita Sharma 2013-09-30 13:11:28 UTC
Followed steps in comment#6, Server did not hang so marking bug as VERIFIED.

Comment 9 errata-xmlrpc 2013-11-21 21:11:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1653.html


Note You need to log in before you can comment on or make changes to this bug.