Bug 2229150

Summary: bdb_start - Detected Disorderly Shutdown directory server is not starting
Product: Red Hat Enterprise Linux 8 Reporter: Rakesh Kumar <rakkumar>
Component: 389-ds-baseAssignee: thierry bordaz <tbordaz>
Status: POST --- QA Contact: LDAP QA Team <idm-ds-qe-bugs>
Severity: medium Docs Contact:
Priority: high    
Version: 8.8CC: bsmejkal, idm-ds-dev-bugs, musoni, tbordaz, vashirov
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Rakesh Kumar 2023-08-04 12:46:34 UTC
Description of problem:Detected Disorderly Shutdown last time Directory Server was running, recovering database


Version-Release number of selected component (if applicable):

389-ds-base-1.4.3.35-1.module+el8.8.0+19157+dcce0882.x86_64

Note: Though this bug had been fixed in 389-ds-base-1.4.2.4-7.module+el8.2.0+5670+0b8b1c2e [bug id: 1773115]

but we are getting the same error in latest version: 389-ds-base-1.4.3.35-1.module+el8.8.0+19157+dcce0882.x86_64


Steps to Reproduce:
[03/Aug/2023:08:40:04.805265238 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:08:58:58.899390868 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:00:26.775118199 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:03:55.313295154 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:05:17.474769486 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:07:45.603096969 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:08:21.253470562 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:13:14.074279152 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:18:58.793540788 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:23:40.063418220 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:29:48.668327261 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:30:59.598415633 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:35:32.229907856 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:36:49.130551958 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:40:19.555843712 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[03/Aug/2023:09:44:13.122789859 -0700] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.

==============================

Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/ns-slapd...Reading symbols from /usr/lib/debug/usr/sbin/ns-slapd-1.4.3.35-1.module+el8.8.0+19157+dcce0882.x86_64.debug...done.
done.
(gdb) run -D /etc/dirsrv/slapd-dir -i /tmp/pid -d 0
Starting program: /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-dir -i /tmp/pid -d 0
Missing separate debuginfo for /usr/lib64/dirsrv/libslapd.so.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/94/91e474c34e1933e56b60364752d873a6cb10dd.debug
Missing separate debuginfo for /usr/lib64/dirsrv/libldaputil.so.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/42/6691649c58f00c606cef6241f0f84703056202.debug
Missing separate debuginfo for /usr/lib64/dirsrv/libsds.so.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/f3/63743501ba7272a90814a5545be9f2990974cd.debug
Missing separate debuginfo for /lib64/libsvrcore.so.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/d9/3ea59292568f9da19cd14909256b3940b21e39.debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Missing separate debuginfo for /usr/lib64/dirsrv/libns-dshttpd-1.4.3.35.so
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/4b/d8de45f89df20c1d1af7d6d12ea55bf1c6c7e0.debug
[New Thread 0x7fff9c505700 (LWP 245285)]
[New Thread 0x7fff9bd04700 (LWP 245286)]
[New Thread 0x7fff9b503700 (LWP 245287)]
[New Thread 0x7fff9ad02700 (LWP 245288)]
[New Thread 0x7fff9a501700 (LWP 245289)]

Thread 1 "ns-slapd" received signal SIGSEGV, Segmentation fault.
0x00007ffff7a5cbbc in slapi_exist_referral () from /usr/lib64/dirsrv/libslapd.so.0
Missing separate debuginfos, use: yum debuginfo-install audit-libs-3.0.7-4.el8.x86_64 cracklib-2.9.6-15.el8.x86_64 cyrus-sasl-gssapi-2.1.27-6.el8_5.x86_64 cyrus-sasl-lib-2.1.27-6.el8_5.x86_64 cyrus-sasl-md5-2.1.27-6.el8_5.x86_64 cyrus-sasl-plain-2.1.27-6.el8_5.x86_64 glibc-2.28-225.el8.x86_64 keyutils-libs-1.5.10-9.el8.x86_64 krb5-libs-1.18.2-25.el8_8.x86_64 libblkid-2.32.1-42.el8_8.x86_64 libcap-2.48-4.el8.x86_64 libcap-ng-0.7.11-1.el8.x86_64 libcom_err-1.45.6-5.el8.x86_64 libevent-2.1.8-5.el8.x86_64 libffi-3.1-24.el8.x86_64 libgcc-8.5.0-18.el8.x86_64 libgcrypt-1.8.5-7.el8_6.x86_64 libgpg-error-1.31-1.el8.x86_64 libicu-60.3-2.el8_1.x86_64 libmount-2.32.1-42.el8_8.x86_64 libnsl2-1.2.0-2.20180605git4a062cf.el8.x86_64 libselinux-2.9-8.el8.x86_64 libsss_nss_idmap-2.8.2-2.el8.x86_64 libstdc++-8.5.0-18.el8.x86_64 libtasn1-4.13-4.el8_7.x86_64 libtirpc-1.1.4-8.el8.x86_64 libxcrypt-4.1.1-6.el8.x86_64 lz4-libs-1.8.3-3.el8_4.x86_64 nspr-4.34.0-3.el8_6.x86_64 nss-3.79.0-11.el8_7.x86_64 nss-softokn-3.79.0-11.el8_7.x86_64 nss-softokn-freebl-3.79.0-11.el8_7.x86_64 nss-util-3.79.0-11.el8_7.x86_64 p11-kit-trust-0.23.22-1.el8.x86_64 pam-1.3.1-25.el8.x86_64 pcre-8.42-6.el8.x86_64 pcre2-10.32-3.el8_6.x86_64 slapi-nis-0.60.0-3.module+el8.8.0+18715+930e9ba0.x86_64 xz-libs-5.2.4-4.el8_6.x86_64 zlib-1.2.11-21.el8_7.x86_64
(gdb) bt full
#0  0x00007ffff7a5cbbc in slapi_exist_referral () at /usr/lib64/dirsrv/libslapd.so.0
#1  0x00007fffea096c8a in ldbm_instance_startall (li=0x5555559fbb30) at ldap/servers/slapd/back-ldbm/instance.c:320
        rc1 = <optimized out>
        inst_obj = 0x555555c9e860
        inst = 0x555555cb3cd0
        rc = 0
#2  0x00007fffea0c24eb in ldbm_back_start (pb=<optimized out>) at ldap/servers/slapd/back-ldbm/start.c:142
        li = 0x5555559fbb30
        retval = 0
        priv = 0x555555c8f040
#3  0x00007ffff7ab9d37 in plugin_call_func () at /usr/lib64/dirsrv/libslapd.so.0
#4  0x00007ffff7abcec9 in plugin_startall () at /usr/lib64/dirsrv/libslapd.so.0
#5  0x0000555555567d56 in main (argc=7, argv=0x7fffffffe3f8) at ldap/servers/slapd/main.c:1032
        rc = 0
        sdn = 0x0
        return_value = <optimized out>
        mcfg = 
          {extraname = 0x0, slapd_exemode = 1, n_port = 389, i_port = 1, s_port = 636, myname = 0x5555557c7d40 "ns-slapd", ldif_file = 0x0, ldif_files = 0, cmd_line_instance_name = 0x0, cmd_line_instance_names = 0x0, skip_db_protect_check = 0, db2ldif_include = 0x0, db2ldif_exclude = 0x0, ldif2db_removedupvals = 1, ldif2db_noattrindexes = 0, db2index_attrs = 0x0, ldif_printkey = 5, archive_name = 0x0, db2ldif_dump_replica = 0, db2ldif_dump_uniqueid = 1, ldif2db_generate_uniqueid = 1, ldif2db_namespaceid = 0x0, importexport_encrypt = 0, upgradedb_flags = 0, upgradednformat_dryrun = 0, is_quiet = 0, backuptools_verbose = 0, dbverify_verbose = 0, dbverify_dbdir = 0x0}
        slapdFrontendConfig = <optimized out>
        ports_info = 
          {n_port = 389, s_port = 636, n_listenaddr = 0x5555559797e0, s_listenaddr = 0x555555977b50, n_socket = 0x555555977cf0, i_listenaddr = 0x555555978020, i_port = 1, i_socket = 0x555555977de0, s_socket = 0x55555597b580}
        m = <optimized out>
        notify = <optimized out>
(gdb) thread apply all bt

Thread 6 (Thread 0x7fff9a501700 (LWP 245289)):
#0  0x00007ffff61fa67f in select () at /lib64/libc.so.6
#1  0x00007ffff7af56a4 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
#2  0x00007fffea0ccafc in locks_monitoring_threadmain (param=0x5555559fbb30)
    at ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:2991
#3  0x00007ffff44c8968 in _pt_root () at /lib64/libnspr4.so
#4  0x00007ffff3e631ca in start_thread () at /lib64/libpthread.so.0
#5  0x00007ffff610de73 in clone () at /lib64/libc.so.6

Thread 5 (Thread 0x7fff9ad02700 (LWP 245288)):
#0  0x00007ffff61fa67f in select () at /lib64/libc.so.6
#1  0x00007ffff7af56a4 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
#2  0x00007fffea0c0408 in perfctrs_wait (milliseconds=<optimized out>, priv=<optimized out>, db_env=<optimized out>)
    at ldap/servers/slapd/back-ldbm/perfctrs.c:80
#3  0x00007fffea0cc994 in perf_threadmain (param=0x5555559fbb30) at ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:2916
#4  0x00007ffff44c8968 in _pt_root () at /lib64/libnspr4.so
#5  0x00007ffff3e631ca in start_thread () at /lib64/libpthread.so.0
#6  0x00007ffff610de73 in clone () at /lib64/libc.so.6

Thread 4 (Thread 0x7fff9b503700 (LWP 245287)):
#0  0x00007ffff61fa67f in select () at /lib64/libc.so.6
#1  0x00007ffff7af56a4 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
#2  0x00007fffea0cc807 in trickle_threadmain (param=0x5555559fbb30) at ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:3976
#3  0x00007ffff44c8968 in _pt_root () at /lib64/libnspr4.so
#4  0x00007ffff3e631ca in start_thread () at /lib64/libpthread.so.0
#5  0x00007ffff610de73 in clone () at /lib64/libc.so.6

Thread 3 (Thread 0x7fff9bd04700 (LWP 245286)):
#0  0x00007ffff61fa67f in select () at /lib64/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--
#1  0x00007ffff7af56a4 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
#2  0x00007fffea0ceb43 in checkpoint_threadmain (param=0x5555559fbb30)
    at ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:3824
#3  0x00007ffff44c8968 in _pt_root () at /lib64/libnspr4.so
#4  0x00007ffff3e631ca in start_thread () at /lib64/libpthread.so.0
#5  0x00007ffff610de73 in clone () at /lib64/libc.so.6

Thread 2 (Thread 0x7fff9c505700 (LWP 245285)):
#0  0x00007ffff61fa67f in select () at /lib64/libc.so.6
#1  0x00007ffff7af56a4 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
#2  0x00007fffea0cc2e7 in deadlock_threadmain (param=0x5555559fbb30) at ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:3494
#3  0x00007ffff44c8968 in _pt_root () at /lib64/libnspr4.so
#4  0x00007ffff3e631ca in start_thread () at /lib64/libpthread.so.0
#5  0x00007ffff610de73 in clone () at /lib64/libc.so.6

Thread 1 (Thread 0x7ffff7fd8fc0 (LWP 245263)):
#0  0x00007ffff7a5cbbc in slapi_exist_referral () at /usr/lib64/dirsrv/libslapd.so.0
#1  0x00007fffea096c8a in ldbm_instance_startall (li=0x5555559fbb30) at ldap/servers/slapd/back-ldbm/instance.c:320
#2  0x00007fffea0c24eb in ldbm_back_start (pb=<optimized out>) at ldap/servers/slapd/back-ldbm/start.c:142
#3  0x00007ffff7ab9d37 in plugin_call_func () at /usr/lib64/dirsrv/libslapd.so.0
#4  0x00007ffff7abcec9 in plugin_startall () at /usr/lib64/dirsrv/libslapd.so.0
#5  0x0000555555567d56 in main (argc=7, argv=0x7fffffffe3f8) at ldap/servers/slapd/main.c:1032
(gdb)



Actual results:


Expected results:


Additional info:

Comment 1 Viktor Ashirov 2023-08-04 14:42:00 UTC
> 389-ds-base-1.4.3.35-1.module+el8.8.0+19157+dcce0882.x86_64

This is 389-ds module in RHEL, not RHDS. Moving to RHEL product.

Comment 2 thierry bordaz 2023-08-07 07:27:58 UTC
There is a good chance that this bug is already fixed upstream in https://github.com/389ds/389-ds-base/issues/5870