Bug 833218

Summary: ldapmodify returns Operations error
Product: Red Hat Enterprise Linux 6 Reporter: Nathan Kinder <nkinder>
Component: 389-ds-baseAssignee: Rich Megginson <rmeggins>
Status: CLOSED ERRATA QA Contact: Sankar Ramalingam <sramling>
Severity: unspecified Docs Contact:
Priority: high    
Version: 6.4CC: jgalipea
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 389-ds-base-1.2.11.12-1.el6 Doc Type: Bug Fix
Doc Text:
Cause: Submitting an LDAP MODIFY operation when the directory server is heavily loaded. Consequence: The LDAP MODIFY operation returns an OPERATIONS_ERROR. Fix: Under a heavy load, the directory server will get deadlocks attempting to write to the database. The bug was caused by improper deadlock handling, which caused the database to report an error instead of retrying the transaction. The fix is to ensure that all deadlocks are handled correctly and their transactions are retried. Result: LDAP MODIFY operations in a heavily loaded directory server should not return OPERATIONS_ERROR.
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-21 08:18:11 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nathan Kinder 2012-06-18 22:00:07 UTC
This bug is created as a clone of upstream ticket:
https://fedorahosted.org/389/ticket/360

https://bugzilla.redhat.com/show_bug.cgi?id=819409 (''Fedora'')

{{{
Description of problem:
When installing FreeIPA, ldapmodify reports Operations Error and does not write
data to LDAP (privilege objects in this case). This causes subsequent issues in
FreeIPA install:

# ipa-server-install
...
Configuring directory server: Estimated time 1 minute
  [1/35]: creating directory server user
  [2/35]: creating directory server instance
  [3/35]: adding default schema
  [4/35]: enabling memberof plugin
  [5/35]: enabling referential integrity plugin
  [6/35]: enabling winsync plugin
  [7/35]: configuring replication version plugin
  [8/35]: enabling IPA enrollment plugin
  [9/35]: enabling ldapi
  [10/35]: configuring uniqueness plugin
  [11/35]: configuring uuid plugin
  [12/35]: configuring modrdn plugin
  [13/35]: enabling entryUSN plugin
  [14/35]: configuring lockout plugin
  [15/35]: creating indices
  [16/35]: configuring ssl for ds instance
  [17/35]: configuring certmap.conf
  [18/35]: configure autobind for root
  [19/35]: configure new location for managed entries
  [20/35]: restarting directory server
  [21/35]: adding default layout
  [22/35]: adding delegation layout
ipa         : CRITICAL Failed to load delegation.ldif: Command
'/usr/bin/ldapmodify -h vm-109.idm.lab.bos.redhat.com -v -f /tmp/tmpM7h8OS -x
-D cn=Directory Manager -y /tmp/tmpW0nOK4' returned non-zero exit status 1
  [23/35]: adding replication acis
  [24/35]: creating container for managed entries
  [25/35]: configuring user private groups
  [26/35]: configuring netgroups from hostgroups
...


ipaserver-install.log excerpt:
2012-05-07T06:45:15Z DEBUG   [22/35]: adding delegation layout
2012-05-07T06:45:16Z DEBUG args=/usr/bin/ldapmodify -h
vm-109.idm.lab.bos.redhat.com -v -f /tmp/       tmpM7h8OS -x -D cn=Directory
Manager -y /tmp/tmpW0nOK4
2012-05-07T06:45:16Z DEBUG stdout=add objectClass:
    top
    nsContainer
add cn:
    roles
adding new entry "cn=roles,cn=accounts,dc=idm,dc=lab,dc=bos,dc=redhat,dc=com"
modify complete

add objectClass:
    top
    nsContainer
add cn:
    pbac
adding new entry "cn=pbac,dc=idm,dc=lab,dc=bos,dc=redhat,dc=com"
modify complete
...
add objectClass:
    top
    groupofnames
    nestedgroup
add cn:
    Group Administrators
add description:
    Group Administrators
adding new entry "cn=Group
Administrators,cn=privileges,cn=pbac,dc=idm,dc=lab,dc=bos,dc=redhat,dc=com"


2012-05-07T06:45:16Z DEBUG stderr=ldap_initialize(
ldap://vm-109.idm.lab.bos.redhat.com )
ldap_add: Operations error (1)

2012-05-07T06:45:16Z CRITICAL Failed to load delegation.ldif: Command
'/usr/bin/ldapmodify -h vm-109.  idm.lab.bos.redhat.com -v -f /tmp/tmpM7h8OS -x
-D cn=Directory Manager -y /tmp/tmpW0nOK4' returned non-zero exit status 1


I found a strange error in dirsrv error log (full log attached) which may be
relevant:

[07/May/2012:02:45:13 -0400] - slapd stopped.
[07/May/2012:02:45:14 -0400] - 389-Directory/1.2.11.3 B2012.126.1429 starting
up
[07/May/2012:02:45:14 -0400] attrcrypt - No symmetric key found for cipher AES
in backend userRoot,    attempting to create one...
[07/May/2012:02:45:14 -0400] attrcrypt - Key for cipher AES successfully
generated and stored
[07/May/2012:02:45:14 -0400] attrcrypt - No symmetric key found for cipher 3DES
in backend userRoot,   attempting to create one...
[07/May/2012:02:45:14 -0400] attrcrypt - Key for cipher 3DES successfully
generated and stored
[07/May/2012:02:45:14 -0400] ipaenrollment_start - [file ipa_enrollment.c, line
390]: Failed to get    default realm?!
[07/May/2012:02:45:14 -0400] - slapd started.  Listening on All Interfaces port
389 for LDAP requests
[07/May/2012:02:45:14 -0400] - Listening on All Interfaces port 636 for LDAPS
requests
[07/May/2012:02:45:14 -0400] - Listening on
/var/run/slapd-IDM-LAB-BOS-REDHAT-COM.socket for LDAPI     requests
[07/May/2012:02:45:15 -0400] - Skipping CoS Definition cn=Password
Policy,cn=accounts,dc=idm,dc=lab,   dc=bos,dc=redhat,dc=com--no CoS Templates
found, which should be added before the CoS Definition.
[07/May/2012:02:45:15 -0400] - libdb: BDB0102 previous transaction deadlock
return not resolved
[07/May/2012:02:45:15 -0400] entryrdn-index - _entryrdn_put_data: Adding the
self link (61) failed:    Invalid argument (22)
[07/May/2012:02:45:15 -0400] - add: attempt to index 61 failed


Version-Release number of selected component (if applicable):
389-ds-base-1.2.11.3-1.fc17.x86_64

How reproducible:


Steps to Reproduce:
1. Install freeipa on F-17 and observe installation
2.
3.

Actual results:
Installation reports 389-ds errors

Expected results:
Installation succeeds without 389-ds errors

Additional info:
Target VM has 1G memory, there were several related warnings in the beginning
of the error log, not sure if it is connected with the error.
}}}

Comment 2 Rich Megginson 2012-07-07 14:32:30 UTC
Covered by the new test multi_plugin - the stress test
testcases/DS/6.0/multi_plugin
------------------------------------------------------------------------
r6706 | rmeggins | 2012-07-07 08:24:23 -0600 (Sat, 07 Jul 2012) | 16 lines

Add multi_plugin test suite
This test enables and configures multiple plugins in the style
of ipa.  The stress version of this test will enable the 
TXN_TESTING in the server which will exercise the txn deadlock
and retry code paths in the server.
Tickets covered by this test:
Ticket #345 - db deadlock return should not log error
Ticket #387 - managed entry sometimes doesn't delete the managed entry
Ticket #360 - ldapmodify returns Operations error
Ticket #335 - transaction retries need to be cache aware
Bugs covered by this test:
830336 db deadlock return should not log error
830343 managed entry sometimes doesn't delete the managed entry
833218 ldapmodify returns Operations error
833202 transaction retries need to be cache aware

Comment 4 Jenny Severance 2012-11-19 20:20:22 UTC
Verified

[2012-11-12 15:49:35] [multi_plugin][mp_01]
dn:
vendorversion: 389-Directory/1.2.11.15 B2012.317.946
[2012-11-12 15:49:35] [multi_plugin][mp_01] finished
TestCase [mp_01] result-> [PASS]
[2012-11-12 15:49:35] [multi_plugin][mp_add] - add 5 users
[2012-11-12 15:49:36] [multi_plugin][mp_add] no txn retries - increase TXN_TEST_HOLD_MSEC and/or decrease TXN_TEST_LOOP_MSEC
[2012-11-12 15:49:39] [multi_plugin][mp_add] group members in cn=mpusers,ou=Groups,dc=example,dc=com differ from current dn list - probable memberof failure due to busy/txn retries - allowing
1a2,5
> uid=testuser1,ou=People,dc=example,dc=com
> uid=testuser1,ou=People,dc=example,dc=com
> uid=testuser1,ou=People,dc=example,dc=com
> uid=testuser2,ou=People,dc=example,dc=com
2a7,10
> uid=testuser2,ou=People,dc=example,dc=com
> uid=testuser2,ou=People,dc=example,dc=com
> uid=testuser3,ou=People,dc=example,dc=com
> uid=testuser3,ou=People,dc=example,dc=com
3a12,15
> uid=testuser3,ou=People,dc=example,dc=com
> uid=testuser4,ou=People,dc=example,dc=com
> uid=testuser4,ou=People,dc=example,dc=com
> uid=testuser4,ou=People,dc=example,dc=com
5a18,20
> uid=testuser5,ou=People,dc=example,dc=com
> uid=testuser5,ou=People,dc=example,dc=com
> uid=testuser5,ou=People,dc=example,dc=com
[2012-11-12 15:49:39] [multi_plugin][mp_add] finished
TestCase [mp_add] result-> [PASS]
[2012-11-12 15:49:39] [multi_plugin][mp_mod] - modify users
[2012-11-12 15:49:39] [multi_plugin][mp_mod] no txn retries - increase TXN_TEST_HOLD_MSEC and/or decrease TXN_TEST_LOOP_MSEC
[2012-11-12 15:49:41] [multi_plugin][mp_mod] finished
TestCase [mp_mod] result-> [PASS]
[2012-11-12 15:49:41] [multi_plugin][mp_rename] - rename users
[2012-11-12 15:49:41] [multi_plugin][mp_rename] no txn retries - increase TXN_TEST_HOLD_MSEC and/or decrease TXN_TEST_LOOP_MSEC
[2012-11-12 15:49:43] [multi_plugin][mp_rename] finished
TestCase [mp_rename] result-> [PASS]
[2012-11-12 15:49:43] [multi_plugin][mp_subtreerename] - rename subtree
[2012-11-12 15:49:45] [multi_plugin][mp_subtreerename] finished
TestCase [mp_subtreerename] result-> [PASS]
[2012-11-12 15:49:45] [multi_plugin][mp_delete] delete users in /home/sramling/RHEL64/testcases/DS/6.0/tet_tmp_dir//multi_plugin/dnlist
[2012-11-12 15:49:46] [multi_plugin][mp_delete] no txn retries - increase TXN_TEST_HOLD_MSEC and/or decrease TXN_TEST_LOOP_MSEC
[2012-11-12 15:49:48] [multi_plugin][mp_delete] finished
TestCase [mp_delete] result-> [PASS]

version 

389-ds-base-1.2.11.15-3.el6

Comment 6 errata-xmlrpc 2013-02-21 08:18:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0503.html