Bug 816590 - Directory Server fails to add a record because of a deadlock
Summary: Directory Server fails to add a record because of a deadlock
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: 389-ds-base
Version: 17
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Rich Megginson
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-04-26 13:15 UTC by Martin Kosek
Modified: 2012-05-04 22:56 UTC (History)
6 users (show)

Fixed In Version: 389-ds-base-1.2.11.1-1.fc17
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-05-04 22:56:09 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Full access log (13.30 KB, text/plain)
2012-04-26 13:16 UTC, Martin Kosek
no flags Details
Full errors log (4.39 KB, text/plain)
2012-04-26 13:17 UTC, Martin Kosek
no flags Details
error log showing deadlock (4.47 KB, text/plain)
2012-04-26 14:58 UTC, John Dennis
no flags Details

Description Martin Kosek 2012-04-26 13:15:29 UTC
Description of problem:

When I run an installation of FreeIPA, Directory Server refuses to accept records added via ldapmodify:

# ipa-server-install -p secret123 -a secret123
...
  [16/18]: issuing RA agent certificate
  [17/18]: adding RA agent as a trusted user
  [18/18]: Configure HTTP to proxy connections
done configuring pki-cad.
Configuring directory server: Estimated time 1 minute
  [1/35]: creating directory server user
  [2/35]: creating directory server instance
  [3/35]: adding default schema
  [4/35]: enabling memberof plugin
  [5/35]: enabling referential integrity plugin
  [6/35]: enabling winsync plugin
  [7/35]: configuring replication version plugin
  [8/35]: enabling IPA enrollment plugin
  [9/35]: enabling ldapi
  [10/35]: configuring uniqueness plugin
  [11/35]: configuring uuid plugin
  [12/35]: configuring modrdn plugin
  [13/35]: enabling entryUSN plugin
  [14/35]: configuring lockout plugin
  [15/35]: creating indices
  [16/35]: configuring ssl for ds instance
  [17/35]: configuring certmap.conf
  [18/35]: configure autobind for root
  [19/35]: configure new location for managed entries
  [20/35]: restarting directory server
  [21/35]: adding default layout
  [22/35]: adding delegation layout
ipa         : CRITICAL Failed to load delegation.ldif: Command '/usr/bin/ldapmodify -h vm-109.idm.lab.bos.redhat.com -v -f /tmp/tmpVHWvQG -x -D cn=Directory Manager -y /tmp/tmptzYV3K' returned non-zero exit status 255
  [23/35]: adding replication acis
ipa         : CRITICAL Failed to load replica-acis.ldif: Command '/usr/bin/ldapmodify -h vm-109.idm.lab.bos.redhat.com -v -f /tmp/tmpCnPUew -x -D cn=Directory Manager -y /tmp/tmpzQm47t' returned non-zero exit status 255
  [24/35]: creating container for managed entries
ipa         : CRITICAL Failed to load managed-entries.ldif: Command '/usr/bin/ldapmodify -h vm-109.idm.lab.bos.redhat.com -v -f /tmp/tmpsjYjn4 -x -D cn=Directory Manager -y /tmp/tmpxjuwRA' returned non-zero exit status 255
...

This is what I found in errors log (full log attached):

/var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors:
...
[26/Apr/2012:08:00:13 -0400] - Listening on All Interfaces port 636 for LDAPS requests
[26/Apr/2012:08:00:13 -0400] - Listening on /var/run/slapd-IDM-LAB-BOS-REDHAT-COM.socket for LDAPI     requests
[26/Apr/2012:08:00:14 -0400] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=idm,dc=lab,   dc=bos,dc=redhat,dc=com--no CoS Templates found, which should be added before the CoS Definition.
[26/Apr/2012:08:00:14 -0400] entryrdn-index - _entryrdn_put_data: Adding the self link (62) failed:    BDB0068 DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30993)


Version-Release number of selected component (if applicable):
389-ds-base-1.2.11-0.1.a1.fc17.x86_64

How reproducible:


Steps to Reproduce:
1. Install freeipa-server RPM in F17 (I can provide the srpm/rpms)
2. Run ipa-server-install
3.
  
Actual results:
Installation fails because of DS error

Expected results:
Installation succeeds

Additional info:

Comment 1 Martin Kosek 2012-04-26 13:16:50 UTC
Created attachment 580472 [details]
Full access log

Comment 2 Martin Kosek 2012-04-26 13:17:14 UTC
Created attachment 580473 [details]
Full errors log

Comment 3 John Dennis 2012-04-26 14:58:58 UTC
Created attachment 580503 [details]
error log showing deadlock

I'm also seeing a deadlock that kills dirsrv. For me it occurred in a different place but appears to be the same issue.

entryrdn-index - _entryrdn_put_data: Adding the self link (61) failed: BDB0068 DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30993)

This is the installed version:

389-ds-base-1.2.11-0.1.a1.fc17.i686

Comment 4 Sumit Bose 2012-04-26 15:14:19 UTC
FYI, if I add a sleep for a few seconds after the ldapmodify operation related to "[21/35]: adding default layout", everything works fine. So maybe the deadlock is triggered by two or more parallel ldapmodify operations because the first one returns too early.

Comment 5 John Dennis 2012-04-26 15:23:36 UTC
re comment #4, interesting data point but I'm not sure how useful it is in practice. I'm hitting the deadlock in a different step

  [30/35]: initializing group membership
Unexpected error - see ipaserver-install.log for details:
 Can't contact LDAP server: 

seems like it might be a timing issue.

I'd have to add sleep to a whole lot of places to guard against a failure, or add the sleep to the run command so it sleeps after every call to a command line program, but even then I'm not sure you wouldn't hit the deadlock with normal IPA ldap operations.

Comment 6 Nathan Kinder 2012-04-26 15:42:32 UTC
I believe that this has already been fixed as a part of https://fedorahosted.org/389/ticket/335.  A new build of 1.2.11 on F17 is needed to get these latest fixes.

Comment 7 Nathan Kinder 2012-04-26 19:13:38 UTC
What version of freeipa-server is being used?  I'm getting other errors during installation when testing out a newer 389-ds-base build with freeipa-server-2.1.90.-.1.fc17.

Here is the error I get from running ipa-server-install:

  DESC 'an integer to order the sudoRole entries'    EQUALITY integerMatch    ORDERING integerOrderingMatch    SYNTAX 1.3.6.1.4.1.1466.115.121.1.27    X-ORIGIN 'SUDO' )
2012-04-26T18:42:33Z DEBUG 'set' object does not support item assignment
  File "/sbin/ipa-server-install", line 1092, in <module>
    rval = main()

  File "/sbin/ipa-server-install", line 1005, in main
    ds.apply_updates()

  File "/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py", line 425, in apply_updates
    ld.update(files)

  File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line 817, in update
    self.__run_updates(dn_list, all_updates)

  File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line 771, in __run_updates
    self.__update_record(all_updates[dn])

  File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line 657, in __update_record
    updated = self.is_schema_updated(entry.toDict())

  File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line 589, in is_schema_updated
    s = ldap.schema.SubSchema(s)

  File "/usr/lib64/python2.7/site-packages/ldap/schema/subentry.py", line 125, in __init__
    self.non_unique_names[se_class][se_id] = None

Comment 8 Fedora Update System 2012-05-02 18:02:27 UTC
389-ds-base-1.2.11.1-1.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/389-ds-base-1.2.11.1-1.fc17

Comment 9 Fedora Update System 2012-05-03 16:01:09 UTC
Package 389-ds-base-1.2.11.1-1.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing 389-ds-base-1.2.11.1-1.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-7214/389-ds-base-1.2.11.1-1.fc17
then log in and leave karma (feedback).

Comment 10 Fedora Update System 2012-05-04 22:56:09 UTC
389-ds-base-1.2.11.1-1.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.