RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1186548 - ns-slapd crash in shutdown phase
Summary: ns-slapd crash in shutdown phase
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: 389-ds-base
Version: 7.1
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Noriko Hosoi
QA Contact: Viktor Ashirov
URL:
Whiteboard:
Depends On:
Blocks: 1195293
TreeView+ depends on / blocked
 
Reported: 2015-01-27 23:56 UTC by Scott Poore
Modified: 2020-09-13 21:19 UTC (History)
8 users (show)

Fixed In Version: 389-ds-base-1.3.4.0-1.el7
Doc Type: Bug Fix
Doc Text:
Cause: When shutting down the directory server, if some task threads were running, there was a small window that a task handle was freed by the shutting down main thread, although the task thread still referred the handle. Consequence: The server process crashed in the shutdown phase. Fix: The shutting down main thread wait for the task threads' finishing completely. Also task threads check more frequently on the server's shutdown. Result: There is no more chance for the server to crash in the shutdown phase.
Clone Of:
: 1195293 (view as bug list)
Environment:
Last Closed: 2015-11-19 11:43:27 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
abrt output email (174.77 KB, text/plain)
2015-01-27 23:58 UTC, Scott Poore 		no flags Details
pthread_mutex_lock lock abrt report (175.90 KB, text/plain)
2015-02-02 22:28 UTC, Scott Poore 		no flags Details
send_ldap_result_ext crash abrt report (215.84 KB, text/plain)
2015-02-02 22:29 UTC, Scott Poore 		no flags Details
slapi_get_object_extension crash abrt report (386.33 KB, text/plain)
2015-02-02 22:30 UTC, Scott Poore 		no flags Details
strlen crash abrt report (165.27 KB, text/plain)
2015-02-02 22:31 UTC, Scott Poore 		no flags Details
strlen_sse2_pminub crash abrt report (183.06 KB, text/plain)
2015-02-02 22:32 UTC, Scott Poore 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Github 389ds 389-ds-base issues 1336 0 None None None 2020-09-13 21:19:42 UTC
Red Hat Product Errata RHBA-2015:2351 0 normal SHIPPED_LIVE 389-ds-base bug fix and enhancement update 2015-11-19 10:28:44 UTC

Description Scott Poore 2015-01-27 23:56:54 UTC 
Description of problem:

abrt_version:   2.1.11
backtrace_rating: 4
cmdline:        /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-TESTRELM-TEST -i /var/run/dirsrv/slapd-TESTRELM-TEST.pid -w /var/run/dirsrv/slapd-TESTRELM-TEST.startpid
crash_function: __strlen_sse2_pminub
executable:     /usr/sbin/ns-slapd
hostname:       cloud-qe-15.testrelm.test
kernel:         3.10.0-123.el7.x86_64
last_occurrence: 1422390267
pid:            21902
pkg_arch:       x86_64
pkg_epoch:      0
pkg_name:       389-ds-base
pkg_release:    13.el7
pkg_version:    1.3.3.1
pwd:            /var/log/dirsrv/slapd-TESTRELM-TEST
runlevel:       N 3
time:           Tue 27 Jan 2015 03:24:27 PM EST
uid:            996
username:       dirsrv

And Noriko from DS Dev team helped by troubleshooting a similar core file to find the issue:

It looks the server crashed in the shutdown phase since memberof_fixup_task_thread was still running (about to finish, though).

Obviously, there is a bug in the task destruction handling -- not waiting enough to call task destructor.

(gdb) bt
#0  0x00007fa809846f02 in plugin_call_plugins (pb=pb@entry=0x7fa80aaa84c0, whichfunction=whichfunction@entry=503)
    at ldap/servers/slapd/plugin.c:365
#1  0x00007fa80983b87a in op_shared_search (pb=pb@entry=0x7fa80aaa84c0, send_result=send_result@entry=1)
    at ldap/servers/slapd/opshared.c:991
#2  0x00007fa80984ad0e in search_internal_callback_pb (pb=pb@entry=0x7fa80aaa84c0,
    callback_data=callback_data@entry=0x7fff1db11640, prc=prc@entry=0x7fa809849e60 <internal_plugin_result_callback>,
    psec=psec@entry=0x7fa809849eb0 <internal_plugin_search_entry_callback>,
    prec=prec@entry=0x7fa809849e70 <internal_plugin_search_referral_callback>) at ldap/servers/slapd/plugin_internal_op.c:812
#3  0x00007fa80984afa8 in search_internal_pb (pb=pb@entry=0x7fa80aaa84c0) at ldap/servers/slapd/plugin_internal_op.c:665
#4  0x00007fa80984b203 in slapi_search_internal_pb (pb=pb@entry=0x7fa80aaa84c0) at ldap/servers/slapd/plugin_internal_op.c:574
#5  0x00007fa7fcd8b87b in update_integrity (origSDN=0x7fa80a9bac80, newrDN=newrDN@entry=0x0, newsuperior=newsuperior@entry=0x0,
    logChanges=<optimized out>) at ldap/servers/plugins/referint/referint.c:1198
#6  0x00007fa7fcd8d142 in referint_postop_del (pb=0x7fa80aaa44f0) at ldap/servers/plugins/referint/referint.c:688
#7  0x00007fa809846db0 in plugin_call_func (list=0x7fa80a46d920, operation=operation@entry=563, pb=pb@entry=0x7fa80aaa44f0,
    call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:1952
#8  0x00007fa809847008 in plugin_call_list (pb=0x7fa80aaa44f0, operation=563, list=<optimized out>)
    at ldap/servers/slapd/plugin.c:1886
#9  plugin_call_plugins (pb=pb@entry=0x7fa80aaa44f0, whichfunction=whichfunction@entry=563) at ldap/servers/slapd/plugin.c:459
#10 0x00007fa809804a3d in dse_delete (pb=0x7fa80aaa44f0) at ldap/servers/slapd/dse.c:2547
#11 0x00007fa8097fb000 in op_shared_delete (pb=pb@entry=0x7fa80aaa44f0) at ldap/servers/slapd/delete.c:364
#12 0x00007fa8097fb1b2 in delete_internal_pb (pb=pb@entry=0x7fa80aaa44f0) at ldap/servers/slapd/delete.c:242
#13 0x00007fa8097fb463 in slapi_delete_internal_pb (pb=pb@entry=0x7fa80aaa44f0) at ldap/servers/slapd/delete.c:185
#14 0x00007fa80986e910 in destroy_task (when=when@entry=0, arg=<optimized out>) at ldap/servers/slapd/task.c:678
#15 0x00007fa809873987 in task_shutdown () at ldap/servers/slapd/task.c:2536 <== in the shutdown procedure
#16 0x00007fa809d18d99 in slapd_daemon (ports=ports@entry=0x7fff1db140b0) at ldap/servers/slapd/daemon.c:1334
#17 0x00007fa809d0b17c in main (argc=7, argv=0x7fff1db146d8) at ldap/servers/slapd/main.c:1279
Comment 1 Scott Poore 2015-01-27 23:58:37 UTC 
Created attachment 984921 [details]
abrt output email
Comment 4 Noriko Hosoi 2015-01-28 00:26:29 UTC 
Upstream ticket:
https://fedorahosted.org/389/ticket/48005
Comment 5 Scott Poore 2015-02-02 22:28:40 UTC 
Created attachment 987294 [details]
pthread_mutex_lock lock abrt report
Comment 7 Scott Poore 2015-02-02 22:29:44 UTC 
Created attachment 987296 [details]
send_ldap_result_ext crash abrt report
Comment 8 Scott Poore 2015-02-02 22:30:11 UTC 
Created attachment 987297 [details]
slapi_get_object_extension crash abrt report
Comment 10 Scott Poore 2015-02-02 22:31:11 UTC 
Created attachment 987299 [details]
strlen crash abrt report
Comment 12 Scott Poore 2015-02-02 22:32:15 UTC 
Created attachment 987301 [details]
strlen_sse2_pminub crash abrt report
Comment 23 Sankar Ramalingam 2015-02-20 13:32:14 UTC 
Hi Noriko, your previous comments states that its not possible to reproduce the crash from standalone 389-ds-base. Can you please share any pointers to verify this bug with 389-ds-base alone setup?
Comment 24 Noriko Hosoi 2015-02-20 17:07:05 UTC 
(In reply to Sankar Ramalingam from comment #23)
> Hi Noriko, your previous comments states that its not possible to reproduce
> the crash from standalone 389-ds-base. Can you please share any pointers to
> verify this bug with 389-ds-base alone setup?

If you are interested in, take a look at the CI test: dirsrvtests/tickets/ticket48005_test.py

But I haven't seen the crash even with the server without the patch.

Scott had to repeat his IPA upgrade test many times to reproduce it.  And to verify the fix, he repeated even more...  (Thanks, Scott!!)
Comment 29 Amita Sharma 2015-07-16 11:11:04 UTC 
Based on comment https://bugzilla.redhat.com/show_bug.cgi?id=1186548#c24
============================================= test session starts ==============================================
platform linux2 -- Python 2.7.5 -- py-1.4.30 -- pytest-2.7.2 -- /bin/python
rootdir: /export/ds/dirsrvtests/tickets, inifile: 
collected 7 items 

ticket48005_test.py::test_ticket48005_setup PASSED
ticket48005_test.py::test_ticket48005_memberof PASSED
ticket48005_test.py::test_ticket48005_automember PASSED
ticket48005_test.py::test_ticket48005_syntaxvalidate PASSED
ticket48005_test.py::test_ticket48005_usn PASSED
ticket48005_test.py::test_ticket48005_schemareload PASSED
ticket48005_test.py::test_ticket48005_final PASSED

========================================== 7 passed in 432.63 seconds ==========================================

Hence marking as VERIFIED.
Comment 30 errata-xmlrpc 2015-11-19 11:43:27 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2351.html
Note You need to log in before you can comment on or make changes to this bug.