Bug 2229150

Summary: bdb_start - Detected Disorderly Shutdown directory server is not starting
Product: Red Hat Enterprise Linux 8 Reporter: Rakesh Kumar <rakkumar>
Component: 389-ds-baseAssignee: thierry bordaz <tbordaz>
Status: CLOSED MIGRATED QA Contact: LDAP QA Team <idm-ds-qe-bugs>
Severity: medium Docs Contact:
Priority: high    
Version: 8.8CC: bsmejkal, fratto, idm-ds-dev-bugs, musoni, tbordaz, tmihinto, vashirov
Target Milestone: rcKeywords: MigratedToJIRA, Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2246307 (view as bug list) Environment:
Last Closed: 2023-09-19 14:28:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2246307, 2246362, 2246366, 2246367, 2265540    

Description Rakesh Kumar 2023-08-04 12:46:34 UTC
Description of problem:Detected Disorderly Shutdown last time Directory Server was running, recovering database


Version-Release number of selected component (if applicable):

389-ds-base-1.4.3.35-1.module+el8.8.0+19157+dcce0882.x86_64

Note: Though this bug had been fixed in 389-ds-base-1.4.2.4-7.module+el8.2.0+5670+0b8b1c2e [bug id: 1773115]

but we are getting the same error in latest version: 389-ds-base-1.4.3.35-1.module+el8.8.0+19157+dcce0882.x86_64


Steps to Reproduce:
[03/Aug/2023:08:40:04.805265238 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:08:58:58.899390868 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:00:26.775118199 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:03:55.313295154 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:05:17.474769486 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:07:45.603096969 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:08:21.253470562 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:13:14.074279152 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:18:58.793540788 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:23:40.063418220 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:29:48.668327261 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:30:59.598415633 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:35:32.229907856 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:36:49.130551958 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:40:19.555843712 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:44:13.122789859 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.

==============================

Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/ns-slapd...Reading symbols from /usr/lib/debug/usr/sbin/ns-slapd-1.4.3.35-1.module+el8.8.0+19157+dcce0882.x86_64.debug...done.
done.
(gdb) run -D /etc/dirsrv/slapd-dir -i /tmp/pid -d 0
Starting program: /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-dir -i /tmp/pid -d 0
Missing separate debuginfo for /usr/lib64/dirsrv/libslapd.so.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/94/91e474c34e1933e56b60364752d873a6cb10dd.debug
Missing separate debuginfo for /usr/lib64/dirsrv/libldaputil.so.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/42/6691649c58f00c606cef6241f0f84703056202.debug
Missing separate debuginfo for /usr/lib64/dirsrv/libsds.so.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/f3/63743501ba7272a90814a5545be9f2990974cd.debug
Missing separate debuginfo for /lib64/libsvrcore.so.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/d9/3ea59292568f9da19cd14909256b3940b21e39.debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Missing separate debuginfo for /usr/lib64/dirsrv/libns-dshttpd-1.4.3.35.so
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/4b/d8de45f89df20c1d1af7d6d12ea55bf1c6c7e0.debug
[New Thread 0x7fff9c505700 (LWP 245285)]
[New Thread 0x7fff9bd04700 (LWP 245286)]
[New Thread 0x7fff9b503700 (LWP 245287)]
[New Thread 0x7fff9ad02700 (LWP 245288)]
[New Thread 0x7fff9a501700 (LWP 245289)]

Thread 1 "ns-slapd" received signal SIGSEGV, Segmentation fault.
0x00007ffff7a5cbbc in slapi_exist_referral () from /usr/lib64/dirsrv/libslapd.so.0
Missing separate debuginfos, use: yum debuginfo-install audit-libs-3.0.7-4.el8.x86_64 cracklib-2.9.6-15.el8.x86_64 cyrus-sasl-gssapi-2.1.27-6.el8_5.x86_64 cyrus-sasl-lib-2.1.27-6.el8_5.x86_64 cyrus-sasl-md5-2.1.27-6.el8_5.x86_64 cyrus-sasl-plain-2.1.27-6.el8_5.x86_64 glibc-2.28-225.el8.x86_64 keyutils-libs-1.5.10-9.el8.x86_64 krb5-libs-1.18.2-25.el8_8.x86_64 libblkid-2.32.1-42.el8_8.x86_64 libcap-2.48-4.el8.x86_64 libcap-ng-0.7.11-1.el8.x86_64 libcom_err-1.45.6-5.el8.x86_64 libevent-2.1.8-5.el8.x86_64 libffi-3.1-24.el8.x86_64 libgcc-8.5.0-18.el8.x86_64 libgcrypt-1.8.5-7.el8_6.x86_64 libgpg-error-1.31-1.el8.x86_64 libicu-60.3-2.el8_1.x86_64 libmount-2.32.1-42.el8_8.x86_64 libnsl2-1.2.0-2.20180605git4a062cf.el8.x86_64 libselinux-2.9-8.el8.x86_64 libsss_nss_idmap-2.8.2-2.el8.x86_64 libstdc++-8.5.0-18.el8.x86_64 libtasn1-4.13-4.el8_7.x86_64 libtirpc-1.1.4-8.el8.x86_64 libxcrypt-4.1.1-6.el8.x86_64 lz4-libs-1.8.3-3.el8_4.x86_64 nspr-4.34.0-3.el8_6.x86_64 nss-3.79.0-11.el8_7.x86_64 nss-softokn-3.79.0-11.el8_7.x86_64 nss-softokn-freebl-3.79.0-11.el8_7.x86_64 nss-util-3.79.0-11.el8_7.x86_64 p11-kit-trust-0.23.22-1.el8.x86_64 pam-1.3.1-25.el8.x86_64 pcre-8.42-6.el8.x86_64 pcre2-10.32-3.el8_6.x86_64 slapi-nis-0.60.0-3.module+el8.8.0+18715+930e9ba0.x86_64 xz-libs-5.2.4-4.el8_6.x86_64 zlib-1.2.11-21.el8_7.x86_64
(gdb) bt full
#0  0x00007ffff7a5cbbc in slapi_exist_referral () at /usr/lib64/dirsrv/libslapd.so.0
#1  0x00007fffea096c8a in ldbm_instance_startall (li=0x5555559fbb30) at ldap/servers/slapd/back-ldbm/instance.c:320
        rc1 = <optimized out>
        inst_obj = 0x555555c9e860
        inst = 0x555555cb3cd0
        rc = 0
#2  0x00007fffea0c24eb in ldbm_back_start (pb=<optimized out>) at ldap/servers/slapd/back-ldbm/start.c:142
        li = 0x5555559fbb30
        retval = 0
        priv = 0x555555c8f040
#3  0x00007ffff7ab9d37 in plugin_call_func () at /usr/lib64/dirsrv/libslapd.so.0
#4  0x00007ffff7abcec9 in plugin_startall () at /usr/lib64/dirsrv/libslapd.so.0
#5  0x0000555555567d56 in main (argc=7, argv=0x7fffffffe3f8) at ldap/servers/slapd/main.c:1032
        rc = 0
        sdn = 0x0
        return_value = <optimized out>
        mcfg = 
          {extraname = 0x0, slapd_exemode = 1, n_port = 389, i_port = 1, s_port = 636, myname = 0x5555557c7d40 "ns-slapd", ldif_file = 0x0, ldif_files = 0, cmd_line_instance_name = 0x0, cmd_line_instance_names = 0x0, skip_db_protect_check = 0, db2ldif_include = 0x0, db2ldif_exclude = 0x0, ldif2db_removedupvals = 1, ldif2db_noattrindexes = 0, db2index_attrs = 0x0, ldif_printkey = 5, archive_name = 0x0, db2ldif_dump_replica = 0, db2ldif_dump_uniqueid = 1, ldif2db_generate_uniqueid = 1, ldif2db_namespaceid = 0x0, importexport_encrypt = 0, upgradedb_flags = 0, upgradednformat_dryrun = 0, is_quiet = 0, backuptools_verbose = 0, dbverify_verbose = 0, dbverify_dbdir = 0x0}
        slapdFrontendConfig = <optimized out>
        ports_info = 
          {n_port = 389, s_port = 636, n_listenaddr = 0x5555559797e0, s_listenaddr = 0x555555977b50, n_socket = 0x555555977cf0, i_listenaddr = 0x555555978020, i_port = 1, i_socket = 0x555555977de0, s_socket = 0x55555597b580}
        m = <optimized out>
        notify = <optimized out>
(gdb) thread apply all bt

Thread 6 (Thread 0x7fff9a501700 (LWP 245289)):
#0  0x00007ffff61fa67f in select () at /lib64/libc.so.6
#1  0x00007ffff7af56a4 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
#2  0x00007fffea0ccafc in locks_monitoring_threadmain (param=0x5555559fbb30)
    at ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:2991
#3  0x00007ffff44c8968 in _pt_root () at /lib64/libnspr4.so
#4  0x00007ffff3e631ca in start_thread () at /lib64/libpthread.so.0
#5  0x00007ffff610de73 in clone () at /lib64/libc.so.6

Thread 5 (Thread 0x7fff9ad02700 (LWP 245288)):
#0  0x00007ffff61fa67f in select () at /lib64/libc.so.6
#1  0x00007ffff7af56a4 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
#2  0x00007fffea0c0408 in perfctrs_wait (milliseconds=<optimized out>, priv=<optimized out>, db_env=<optimized out>)
    at ldap/servers/slapd/back-ldbm/perfctrs.c:80
#3  0x00007fffea0cc994 in perf_threadmain (param=0x5555559fbb30) at ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:2916
#4  0x00007ffff44c8968 in _pt_root () at /lib64/libnspr4.so
#5  0x00007ffff3e631ca in start_thread () at /lib64/libpthread.so.0
#6  0x00007ffff610de73 in clone () at /lib64/libc.so.6

Thread 4 (Thread 0x7fff9b503700 (LWP 245287)):
#0  0x00007ffff61fa67f in select () at /lib64/libc.so.6
#1  0x00007ffff7af56a4 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
#2  0x00007fffea0cc807 in trickle_threadmain (param=0x5555559fbb30) at ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:3976
#3  0x00007ffff44c8968 in _pt_root () at /lib64/libnspr4.so
#4  0x00007ffff3e631ca in start_thread () at /lib64/libpthread.so.0
#5  0x00007ffff610de73 in clone () at /lib64/libc.so.6

Thread 3 (Thread 0x7fff9bd04700 (LWP 245286)):
#0  0x00007ffff61fa67f in select () at /lib64/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--
#1  0x00007ffff7af56a4 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
#2  0x00007fffea0ceb43 in checkpoint_threadmain (param=0x5555559fbb30)
    at ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:3824
#3  0x00007ffff44c8968 in _pt_root () at /lib64/libnspr4.so
#4  0x00007ffff3e631ca in start_thread () at /lib64/libpthread.so.0
#5  0x00007ffff610de73 in clone () at /lib64/libc.so.6

Thread 2 (Thread 0x7fff9c505700 (LWP 245285)):
#0  0x00007ffff61fa67f in select () at /lib64/libc.so.6
#1  0x00007ffff7af56a4 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
#2  0x00007fffea0cc2e7 in deadlock_threadmain (param=0x5555559fbb30) at ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:3494
#3  0x00007ffff44c8968 in _pt_root () at /lib64/libnspr4.so
#4  0x00007ffff3e631ca in start_thread () at /lib64/libpthread.so.0
#5  0x00007ffff610de73 in clone () at /lib64/libc.so.6

Thread 1 (Thread 0x7ffff7fd8fc0 (LWP 245263)):
#0  0x00007ffff7a5cbbc in slapi_exist_referral () at /usr/lib64/dirsrv/libslapd.so.0
#1  0x00007fffea096c8a in ldbm_instance_startall (li=0x5555559fbb30) at ldap/servers/slapd/back-ldbm/instance.c:320
#2  0x00007fffea0c24eb in ldbm_back_start (pb=<optimized out>) at ldap/servers/slapd/back-ldbm/start.c:142
#3  0x00007ffff7ab9d37 in plugin_call_func () at /usr/lib64/dirsrv/libslapd.so.0
#4  0x00007ffff7abcec9 in plugin_startall () at /usr/lib64/dirsrv/libslapd.so.0
#5  0x0000555555567d56 in main (argc=7, argv=0x7fffffffe3f8) at ldap/servers/slapd/main.c:1032
(gdb)



Actual results:


Expected results:


Additional info:

Comment 1 Viktor Ashirov 2023-08-04 14:42:00 UTC
> 389-ds-base-1.4.3.35-1.module+el8.8.0+19157+dcce0882.x86_64

This is 389-ds module in RHEL, not RHDS. Moving to RHEL product.

Comment 2 thierry bordaz 2023-08-07 07:27:58 UTC
There is a good chance that this bug is already fixed upstream in https://github.com/389ds/389-ds-base/issues/5870

Comment 8 Francesco Ratto 2023-09-12 08:10:05 UTC
Hello 

 Can you please estimate a target date ?

 best

 Francesco

Comment 10 RHEL Program Management 2023-09-19 14:23:08 UTC
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

Comment 11 RHEL Program Management 2023-09-19 14:28:07 UTC
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.

Comment 12 Red Hat Bugzilla 2024-06-22 04:25:03 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days