Bug 1316742

Summary: no plugin calls in tombstone purging
Product: Red Hat Enterprise Linux 7 Reporter: Noriko Hosoi <nhosoi>
Component: 389-ds-baseAssignee: Noriko Hosoi <nhosoi>
Status: CLOSED ERRATA QA Contact: Viktor Ashirov <vashirov>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.3CC: nkinder, rmeggins, spichugi
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 389-ds-base-1.3.5.2-1.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-03 20:40:34 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Noriko Hosoi 2016-03-11 00:39:04 UTC 
This bug is created as a clone of upstream ticket:
https://fedorahosted.org/389/ticket/48759

Another deadlock occurred involving memberof and retrocl, but what is specific to this case is that the sequence of calls memberof--> retrocl was triggered by the deletion of a tombstone.

tombstones are entries which are deleted, plugins have been called in the delete op, and purging is an internal db operation only.

For reference the two threads:


{{{
#0  0x00007f9832a7a705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f98330d02c3 in PR_EnterMonitor () from /lib64/libnspr4.so
#2  0x00007f98291f7132 in dblayer_lock_backend (be=<optimized out>) at ldap/servers/slapd/back-ldbm/dblayer.c:3946
#3  0x00007f98291fbf36 in dblayer_txn_begin (be=0x7f98367397e0, parent_txn=0x7f95cd065af0, txn=txn@entry=0x7f95dfff1370) at ldap/servers/slapd/back-ldbm/dblayer.c:3668
#4  0x00007f982921bb67 in ldbm_back_add (pb=0x7f95ccb1d430) at ldap/servers/slapd/back-ldbm/ldbm_add.c:272
#5  0x00007f9834c8bf1a in op_shared_add (pb=pb@entry=0x7f95ccb1d430) at ldap/servers/slapd/add.c:735
#6  0x00007f9834c8c773 in add_internal_pb (pb=pb@entry=0x7f95ccb1d430) at ldap/servers/slapd/add.c:434
#7  0x00007f9834c8d453 in slapi_add_internal_pb (pb=pb@entry=0x7f95ccb1d430) at ldap/servers/slapd/add.c:356
#8  0x00007f9827aa632d in write_replog_db (newsuperior=0x0, modrdn_mods=0x0, newrdn=0x0, log_e=0x0, curtime=1457532345, flag=0, log_m=0x7f95cd50deb0, dn=0x7f95cc7760a0 "uid=ccrgst74,cn=users,cn=accounts,dc=cbls,dc=ccr,dc=buffalo,dc=edu", optype=<optimized out>, pb=0x7f95cd45d8d0) at ldap/servers/plugins/retrocl/retrocl_po.c:377
#9  retrocl_postob (pb=0x7f95cd45d8d0, optype=<optimized out>) at ldap/servers/plugins/retrocl/retrocl_po.c:682
#10 0x00007f9834ce5280 in plugin_call_func (list=0x7f9836516b80, operation=operation@entry=561, pb=pb@entry=0x7f95cd45d8d0, call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:1952
#11 0x00007f9834ce54d8 in plugin_call_list (pb=0x7f95cd45d8d0, operation=561, list=<optimized out>) at ldap/servers/slapd/plugin.c:1886
#12 plugin_call_plugins (pb=pb@entry=0x7f95cd45d8d0, whichfunction=whichfunction@entry=561) at ldap/servers/slapd/plugin.c:459
#13 0x00007f9829236be9 in ldbm_back_modify (pb=<optimized out>) at ldap/servers/slapd/back-ldbm/ldbm_modify.c:847
#14 0x00007f9834cd20e1 in op_shared_modify (pb=pb@entry=0x7f95cd45d8d0, pw_change=pw_change@entry=0, old_pw=0x0) at ldap/servers/slapd/modify.c:1086
#15 0x00007f9834cd2ba4 in modify_internal_pb (pb=pb@entry=0x7f95cd45d8d0) at ldap/servers/slapd/modify.c:631
#16 0x00007f9834cd36a3 in slapi_modify_internal_pb (pb=pb@entry=0x7f95cd45d8d0) at ldap/servers/slapd/modify.c:486
#17 0x00007f98289116f1 in memberof_fix_memberof_callback (e=<optimized out>, callback_data=callback_data@entry=0x7f95dfff6030) at ldap/servers/plugins/memberof/memberof.c:2902
#18 0x00007f98289121d5 in memberof_modop_one_replace_r (pb=pb@entry=0x7f95cd5ee4b0, config=config@entry=0x7f95dfff6030, mod_op=mod_op@entry=1, group_sdn=group_sdn@entry=0x7f95cd35fad0, op_this_sdn=op_this_sdn@entry=0x7f95cd35fad0, replace_with_sdn=replace_with_sdn@entry=0x0, op_to_sdn=0x7f95cc862bc0, stack=stack@entry=0x0) at ldap/servers/plugins/memberof/memberof.c:1674
#19 0x00007f98289123c5 in memberof_modop_one_r (stack=0x0, op_to_sdn=<optimized out>, op_this_sdn=0x7f95cd35fad0, group_sdn=0x7f95cd35fad0, mod_op=1, config=0x7f95dfff6030, pb=0x7f95cd5ee4b0) at ldap/servers/plugins/memberof/memberof.c:1417
#20 memberof_mod_attr_list_r (pb=pb@entry=0x7f95cd5ee4b0, config=config@entry=0x7f95dfff6030, mod=mod@entry=1, group_sdn=group_sdn@entry=0x7f95cd35fad0, op_this_sdn=op_this_sdn@entry=0x7f95cd35fad0, attr=0x7f95cd28ec80, stack=stack@entry=0x0) at ldap/servers/plugins/memberof/memberof.c:1919
#21 0x00007f98289143e6 in memberof_mod_attr_list (attr=<optimized out>, group_sdn=0x7f95cd35fad0, mod=1, config=0x7f95dfff6030, pb=0x7f95cd5ee4b0) at ldap/servers/plugins/memberof/memberof.c:1858
#22 memberof_del_attr_list (attr=<optimized out>, group_sdn=0x7f95cd35fad0, config=0x7f95dfff6030, pb=0x7f95cd5ee4b0) at ldap/servers/plugins/memberof/memberof.c:1957
#23 memberof_postop_del (pb=0x7f95cd5ee4b0) at ldap/servers/plugins/memberof/memberof.c:579
#24 0x00007f9834ce5280 in plugin_call_func (list=0x7f98364f8f20, operation=operation@entry=563, pb=pb@entry=0x7f95cd5ee4b0, call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:1952
#25 0x00007f9834ce54d8 in plugin_call_list (pb=0x7f95cd5ee4b0, operation=563, list=<optimized out>) at ldap/servers/slapd/plugin.c:1886
#26 plugin_call_plugins (pb=pb@entry=0x7f95cd5ee4b0, whichfunction=whichfunction@entry=563) at ldap/servers/slapd/plugin.c:459
#27 0x00007f982922942e in ldbm_back_delete (pb=0x7f95cd5ee4b0) at ldap/servers/slapd/back-ldbm/ldbm_delete.c:1226
#28 0x00007f9834c99190 in op_shared_delete (pb=pb@entry=0x7f95cd5ee4b0) at ldap/servers/slapd/delete.c:364
#29 0x00007f9834c99342 in delete_internal_pb (pb=pb@entry=0x7f95cd5ee4b0) at ldap/servers/slapd/delete.c:242
#30 0x00007f9834c995f3 in slapi_delete_internal_pb (pb=pb@entry=0x7f95cd5ee4b0) at ldap/servers/slapd/delete.c:185
#31 0x00007f9828f76af8 in _delete_tombstone (tombstone_dn=0x7f944fde94b0 "nsuniqueid=d48e5121-cf5311e5-9a208f7e-e0b1a377,ipaUniqueID=ec5e3106-cf53-11e5-b8e8-a0369f577818,cn=hbac,dc=cbls,dc=ccr,dc=buffalo,dc=edu", uniqueid=0x7f95bee65750 "d48e5121-cf5311e5-9a208f7e-e0b1a377", ext_op_flags=0) at ldap/servers/plugins/replication/repl5_replica.c:2932
#32 0x00007f9828f79362 in process_reap_entry (entry=0x7f944f540430, cb_data=0x7f95dfffecc0) at ldap/servers/plugins/replication/repl5_replica.c:2993
#33 0x00007f9834cf9a7d in send_ldap_search_entry_ext (pb=pb@entry=0x7f95cd2ee540, e=<optimized out>, ectrls=ectrls@entry=0x0, attrs=0x7f95cc647490, attrsonly=0, send_result=send_result@entry=0, nentries=nentries@entry=0, urls=urls@entry=0x0) at ldap/servers/slapd/result.c:1544
#34 0x00007f9834cfa2bc in send_ldap_search_entry (pb=pb@entry=0x7f95cd2ee540, e=<optimized out>, ectrls=ectrls@entry=0x0, attrs=<optimized out>, attrsonly=<optimized out>) at ldap/servers/slapd/result.c:1084
#35 0x00007f9834cd76d3 in iterate (pb=pb@entry=0x7f95cd2ee540, pnentries=pnentries@entry=0x7f95dfff88c8, pagesize=pagesize@entry=-1, pr_statp=pr_statp@entry=0x7f95dfff8844, be=0x7f95cd2ee540, send_result=1) at ldap/servers/slapd/opshared.c:1485
#36 0x00007f9834cd787a in send_results_ext (pb=pb@entry=0x7f95cd2ee540, nentries=nentries@entry=0x7f95dfff88c8, pagesize=-1, pr_stat=pr_stat@entry=0x7f95dfff8844, send_result=1) at ldap/servers/slapd/opshared.c:1724
#37 0x00007f9834cd91c1 in op_shared_search (pb=pb@entry=0x7f95cd2ee540, send_result=send_result@entry=1) at ldap/servers/slapd/opshared.c:881
#38 0x00007f9834ce91de in search_internal_callback_pb (pb=pb@entry=0x7f95cd2ee540, callback_data=callback_data@entry=0x7f95dfffecc0, prc=prc@entry=0x7f9828f75be0 <get_reap_result>, psec=psec@entry=0x7f9828f79260 <process_reap_entry>, prec=prec@entry=0x0) at ldap/servers/slapd/plugin_internal_op.c:812
#39 0x00007f9834ce9759 in slapi_search_internal_callback_pb (pb=pb@entry=0x7f95cd2ee540, callback_data=callback_data@entry=0x7f95dfffecc0, prc=prc@entry=0x7f9828f75be0 <get_reap_result>, psec=psec@entry=0x7f9828f79260 <process_reap_entry>, prec=prec@entry=0x0) at ldap/servers/slapd/plugin_internal_op.c:593
#40 0x00007f9828f7b7b1 in _replica_reap_tombstones (arg=0x7f983bbdd820) at ldap/servers/plugins/replication/repl5_replica.c:3123
#41 0x00007f98330d57bb in _pt_root () from /lib64/libnspr4.so
#42 0x00007f9832a76df5 in start_thread () from /lib64/libpthread.so.0
#43 0x00007f98327a41ad in clone () from /lib64/libc.so.6

#0  0x00007f9832a7a705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f98330d02c3 in PR_EnterMonitor () from /lib64/libnspr4.so
#2  0x00007f982891431a in memberof_postop_del (pb=0x7f9505362290) at ldap/servers/plugins/memberof/memberof.c:554
#3  0x00007f9834ce5280 in plugin_call_func (list=0x7f98364f8f20, operation=operation@entry=563, pb=pb@entry=0x7f9505362290, call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:1952
#4  0x00007f9834ce54d8 in plugin_call_list (pb=0x7f9505362290, operation=563, list=<optimized out>) at ldap/servers/slapd/plugin.c:1886
#5  plugin_call_plugins (pb=pb@entry=0x7f9505362290, whichfunction=whichfunction@entry=563) at ldap/servers/slapd/plugin.c:459
#6  0x00007f982922942e in ldbm_back_delete (pb=0x7f9505362290) at ldap/servers/slapd/back-ldbm/ldbm_delete.c:1226
#7  0x00007f9834c99190 in op_shared_delete (pb=pb@entry=0x7f9505362290) at ldap/servers/slapd/delete.c:364
#8  0x00007f9834c99342 in delete_internal_pb (pb=pb@entry=0x7f9505362290) at ldap/servers/slapd/delete.c:242
#9  0x00007f9834c995f3 in slapi_delete_internal_pb (pb=pb@entry=0x7f9505362290) at ldap/servers/slapd/delete.c:185
#10 0x00007f9827aa6afe in delete_changerecord (cnum=cnum@entry=17901598) at ldap/servers/plugins/retrocl/retrocl_trim.c:117
#11 0x00007f9827aa6dd1 in trim_changelog () at ldap/servers/plugins/retrocl/retrocl_trim.c:316
#12 changelog_trim_thread_fn (arg=<optimized out>) at ldap/servers/plugins/retrocl/retrocl_trim.c:359
#13 0x00007f98330d57bb in _pt_root () from /lib64/libnspr4.so
#14 0x00007f9832a76df5 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f98327a41ad in clone () from /lib64/libc.so.6
}}}
Comment 3 Simon Pichugin 2016-07-13 07:55:13 UTC 
Build tested:
389-ds-base-1.3.5.10-3.el7.x86_64

========================== test session starts ==========================
platform linux2 -- Python 2.7.5, pytest-2.9.2, py-1.4.31, pluggy-0.3.1 -- /usr/bin/python
cachedir: dirsrvtests/tests/tickets/.cache
rootdir: /mnt/tests/rhds/tests/upstream/ds/dirsrvtests/tests/tickets, inifile:
plugins: html-1.9.0, cov-2.3.0
collected 2 items

dirsrvtests/tests/tickets/ticket48759_test.py::test_ticket48759 PASSED
dirsrvtests/tests/tickets/ticket48759_test.py::test_ticket48759_final PASSED

======================= 2 passed in 43.94 seconds =======================

Marking as verified.
Comment 5 errata-xmlrpc 2016-11-03 20:40:34 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2594.html