Bug 1265851

Summary: Double free while adding entries
Product: Red Hat Enterprise Linux 6 Reporter: Noriko Hosoi <nhosoi>
Component: 389-ds-baseAssignee: Noriko Hosoi <nhosoi>
Status: CLOSED ERRATA QA Contact: Viktor Ashirov <vashirov>
Severity: urgent Docs Contact: Petr Bokoc <pbokoc>
Priority: urgent    
Version: 6.0CC: ekeck, jgalipea, nkinder, pbokoc, rmeggins, spichugi, tscherf
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: 389-ds-base-1.2.11.15-67.el6 Doc Type: Bug Fix
Doc Text:
Fixed a crash in the backend add function When a callback at `BE_TXN` in the backend add function failed on a cached entry, the function was attempting to free the entry twice instead of removing it from the cache and then freeing it. This update adds remove and free code to the backend add function and the function no longer attempts to free cached entries twice.
 Story Points: ---
Clone Of:
: 1314241 (view as bug list) Environment:
Last Closed: 2016-05-10 19:21:21 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1272422, 1314241    

Description Noriko Hosoi 2015-09-23 22:47:09 UTC 
This bug is created as a clone of upstream ticket:
https://fedorahosted.org/389/ticket/48287

This problem was found while investigating https://fedorahosted.org/freeipa/ticket/5235
It was reproduced on RHEL 6.7 (389-ds-base-1.2.11.15-60.el6.x86_64)

The test case to reproduce is:
 * install IPA master
 * install IPA replica
 * (enable core dump: yum install abrt, sysctl -w fs.suid_dumpable=1, 'ulimit -c unlimited' in /etc/sysconfig/dirsrv)
 * iterate the provided script 'steps.sh'
 * the master replica will crash

The crash is not systematic. On my VMs it happens once in ~[20,100] 'steps.sh'

Sometime the test case ends in breaking IPA topology (ipa user-add fails, or total init fails) without crash. It requires to reinstall the topology.

Each time (crash or IPA topology break) the following messages are logged:

[23/Sep/2015:15:31:25 +0200] NSMMReplicationPlugin - changelog program - _cl5WriteOperationTxn: retry (49) the transaction (csn=5602a9a9000000040000) failed (rc=-30994 (DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock))
[23/Sep/2015:15:31:25 +0200] NSMMReplicationPlugin - changelog program - _cl5WriteOperationTxn: failed to write entry with csn (5602a9a9000000040000); db error - -30994 DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock
[23/Sep/2015:15:31:25 +0200] NSMMReplicationPlugin - write_changelog_and_ruv: can't add a change for uid=user_1809_2,cn=users,cn=accounts,dc=abc,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com (uniqid: 51b06805-61f711e5-a12e9486-e75d4fed, optype: 16) to changelog csn 5602a9a9000000040000
Comment 1 Noriko Hosoi 2015-10-13 17:06:12 UTC 
Reproducer is in DS 48287.

It requires IPA master/client, but it is easy to reproduce (in a couple of hours).
Comment 3 Simon Pichugin 2016-01-20 15:48:39 UTC 
[root@ipaqa64vmf ~]# rpm -qa | grep 389-ds
389-ds-base-libs-1.2.11.15-71.el6.x86_64
389-ds-base-1.2.11.15-71.el6.x86_64

Verification steps:
1. Install IPA Master

2. Install IPA Replica

3. Enable core dump: 
 - yum install abrt
 - sysctl -w fs.suid_dumpable=1
 - In /etc/sysconfig/dirsrv, change 'ulimit -c' to unlimited

4. Iterate the provided script 'steps.sh' on the master:
[root@ipaqa64vmf ~]# cat steps.sh
#!/bin/sh

MASTER=ipamaster.redhat.com
REPLICA=ipareplica.redhat.com

CPT=0
EXT=${1:-}
if [ -z "${EXT}" ]
then
        EXT=$RANDOM
fi
kinit admin
ssh root@$REPLICA /usr/sbin/ipa-replica-manage re-initialize --from=$MASTER -p Secret123
while [ $CPT -lt 10 ]
do
        ipa user-add --first=fi --last=la user_${EXT}_${CPT}
        CPT=`expr $CPT + 1`
done

echo
echo
echo
sleep 10
CPT=0
while [ $CPT -lt 10 ]
do
        ldapsearch -D "cn=directory manager" -w Secret123 -h $REPLICA -p 389 -LLL -b"cn=users,cn=accounts,dc=redhat,dc=com" "uid=user_${EXT}_${CPT}" dn
        CPT=`expr $CPT + 1`
done

[root@ipaqa64vmf ~]# ./steps.sh
Password for admin.ENG.BOS.REDHAT.COM:
root.lab.eng.bos.redhat.com's password:

-------------------------
Added user "user_25018_0"
-------------------------
  User login: user_25018_0
  First name: fi
  Last name: la
  Full name: fi la
  Display name: fi la
  Initials: fl
  Home directory: /home/user_25018_0
  GECOS field: fi la
  Login shell: /bin/sh
  Kerberos principal: user_25018_0.ENG.BOS.REDHAT.COM
  Email address: user_25018_0.eng.bos.redhat.com
  UID: 983000022
  GID: 983000022
  Password: False
  Kerberos keys available: False
-------------------------
Added user "user_25018_1"
-------------------------
...
-------------------------
Added user "user_25018_9"
-------------------------

dn: uid=user_25018_0,cn=users,cn=accounts,dc=idmqe,dc=lab,dc=eng,dc=bos,dc=red
 hat,dc=com

dn: uid=user_25018_1,cn=users,cn=accounts,dc=idmqe,dc=lab,dc=eng,dc=bos,dc=red
 hat,dc=com
...
dn: uid=user_25018_9,cn=users,cn=accounts,dc=idmqe,dc=lab,dc=eng,dc=bos,dc=red
 hat,dc=com


5. Check for the error message:
[root@ipaqa64vmf log]# grep DB_LOCK_DEADLOCK dirsrv/slapd-PKI-IPA/errors
[root@ipaqa64vmf log]# echo $?
1
Comment 7 Viktor Ashirov 2016-03-08 17:21:50 UTC 
Hi Thorsten,

this fix is included in 389-ds-base-1.2.11.15-72.el6_7:
https://brewweb.devel.redhat.com/buildinfo?buildID=485184

It's targeted for BU#6.
Comment 10 errata-xmlrpc 2016-05-10 19:21:21 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0737.html