Bug 1265851 - Double free while adding entries
Double free while adding entries
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: 389-ds-base (Show other bugs)
6.0
All Linux
urgent Severity urgent
: rc
: ---
Assigned To: Noriko Hosoi
Viktor Ashirov
Petr Bokoc
: ZStream
Depends On:
Blocks: 1272422 1314241
  Show dependency treegraph
 
Reported: 2015-09-23 18:47 EDT by Noriko Hosoi
Modified: 2016-05-10 15:21 EDT (History)
7 users (show)

See Also:
Fixed In Version: 389-ds-base-1.2.11.15-67.el6
Doc Type: Bug Fix
Doc Text:
Fixed a crash in the backend add function When a callback at `BE_TXN` in the backend add function failed on a cached entry, the function was attempting to free the entry twice instead of removing it from the cache and then freeing it. This update adds remove and free code to the backend add function and the function no longer attempts to free cached entries twice.
Story Points: ---
Clone Of:
: 1314241 (view as bug list)
Environment:
Last Closed: 2016-05-10 15:21:21 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:0737 normal SHIPPED_LIVE 389-ds-base bug fix and enhancement update 2016-05-10 18:29:13 EDT

  None (edit)
Description Noriko Hosoi 2015-09-23 18:47:09 EDT
This bug is created as a clone of upstream ticket:
https://fedorahosted.org/389/ticket/48287

This problem was found while investigating https://fedorahosted.org/freeipa/ticket/5235
It was reproduced on RHEL 6.7 (389-ds-base-1.2.11.15-60.el6.x86_64)

The test case to reproduce is:
 * install IPA master
 * install IPA replica
 * (enable core dump: yum install abrt, sysctl -w fs.suid_dumpable=1, 'ulimit -c unlimited' in /etc/sysconfig/dirsrv)
 * iterate the provided script 'steps.sh'
 * the master replica will crash

The crash is not systematic. On my VMs it happens once in ~[20,100] 'steps.sh'

Sometime the test case ends in breaking IPA topology (ipa user-add fails, or total init fails) without crash. It requires to reinstall the topology.

Each time (crash or IPA topology break) the following messages are logged:

[23/Sep/2015:15:31:25 +0200] NSMMReplicationPlugin - changelog program - _cl5WriteOperationTxn: retry (49) the transaction (csn=5602a9a9000000040000) failed (rc=-30994 (DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock))
[23/Sep/2015:15:31:25 +0200] NSMMReplicationPlugin - changelog program - _cl5WriteOperationTxn: failed to write entry with csn (5602a9a9000000040000); db error - -30994 DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock
[23/Sep/2015:15:31:25 +0200] NSMMReplicationPlugin - write_changelog_and_ruv: can't add a change for uid=user_1809_2,cn=users,cn=accounts,dc=abc,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com (uniqid: 51b06805-61f711e5-a12e9486-e75d4fed, optype: 16) to changelog csn 5602a9a9000000040000
Comment 1 Noriko Hosoi 2015-10-13 13:06:12 EDT
Reproducer is in DS 48287.

It requires IPA master/client, but it is easy to reproduce (in a couple of hours).
Comment 3 Simon Pichugin 2016-01-20 10:48:39 EST
[root@ipaqa64vmf ~]# rpm -qa | grep 389-ds
389-ds-base-libs-1.2.11.15-71.el6.x86_64
389-ds-base-1.2.11.15-71.el6.x86_64

Verification steps:
1. Install IPA Master

2. Install IPA Replica

3. Enable core dump: 
 - yum install abrt
 - sysctl -w fs.suid_dumpable=1
 - In /etc/sysconfig/dirsrv, change 'ulimit -c' to unlimited

4. Iterate the provided script 'steps.sh' on the master:
[root@ipaqa64vmf ~]# cat steps.sh
#!/bin/sh

MASTER=ipamaster.redhat.com
REPLICA=ipareplica.redhat.com

CPT=0
EXT=${1:-}
if [ -z "${EXT}" ]
then
        EXT=$RANDOM
fi
kinit admin
ssh root@$REPLICA /usr/sbin/ipa-replica-manage re-initialize --from=$MASTER -p Secret123
while [ $CPT -lt 10 ]
do
        ipa user-add --first=fi --last=la user_${EXT}_${CPT}
        CPT=`expr $CPT + 1`
done

echo
echo
echo
sleep 10
CPT=0
while [ $CPT -lt 10 ]
do
        ldapsearch -D "cn=directory manager" -w Secret123 -h $REPLICA -p 389 -LLL -b"cn=users,cn=accounts,dc=redhat,dc=com" "uid=user_${EXT}_${CPT}" dn
        CPT=`expr $CPT + 1`
done

[root@ipaqa64vmf ~]# ./steps.sh
Password for admin@IDMQE.LAB.ENG.BOS.REDHAT.COM:
root@ipaqavmd.idmqe.lab.eng.bos.redhat.com's password:

-------------------------
Added user "user_25018_0"
-------------------------
  User login: user_25018_0
  First name: fi
  Last name: la
  Full name: fi la
  Display name: fi la
  Initials: fl
  Home directory: /home/user_25018_0
  GECOS field: fi la
  Login shell: /bin/sh
  Kerberos principal: user_25018_0@IDMQE.LAB.ENG.BOS.REDHAT.COM
  Email address: user_25018_0@idmqe.lab.eng.bos.redhat.com
  UID: 983000022
  GID: 983000022
  Password: False
  Kerberos keys available: False
-------------------------
Added user "user_25018_1"
-------------------------
...
-------------------------
Added user "user_25018_9"
-------------------------

dn: uid=user_25018_0,cn=users,cn=accounts,dc=idmqe,dc=lab,dc=eng,dc=bos,dc=red
 hat,dc=com

dn: uid=user_25018_1,cn=users,cn=accounts,dc=idmqe,dc=lab,dc=eng,dc=bos,dc=red
 hat,dc=com
...
dn: uid=user_25018_9,cn=users,cn=accounts,dc=idmqe,dc=lab,dc=eng,dc=bos,dc=red
 hat,dc=com


5. Check for the error message:
[root@ipaqa64vmf log]# grep DB_LOCK_DEADLOCK dirsrv/slapd-PKI-IPA/errors
[root@ipaqa64vmf log]# echo $?
1
Comment 7 Viktor Ashirov 2016-03-08 12:21:50 EST
Hi Thorsten,

this fix is included in 389-ds-base-1.2.11.15-72.el6_7:
https://brewweb.devel.redhat.com/buildinfo?buildID=485184

It's targeted for BU#6.
Comment 10 errata-xmlrpc 2016-05-10 15:21:21 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0737.html

Note You need to log in before you can comment on or make changes to this bug.