1787921 – Crash on startup: Bus error in __env_faultmem.isra.1.part.2

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1787921 - Crash on startup: Bus error in __env_faultmem.isra.1.part.2

Summary: Crash on startup: Bus error in __env_faultmem.isra.1.part.2

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	389-ds-base
Sub Component:
Version:	7.7
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	urgent
Target Milestone:	rc
Target Release:	---
Assignee:	mreynolds
QA Contact:	RHDS QE
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	1610409 (view as bug list)
Depends On:
Blocks:	1788833 1790982 1790984
TreeView+	depends on / blocked

Reported:	2020-01-05 14:14 UTC by Graham Leggett
Modified:	2021-09-09 15:34 UTC (History)
CC List:	7 users (show)
Fixed In Version:	389-ds-base-1.3.10.2-1.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1790982 (view as bug list)
Environment:
Last Closed:	2020-09-29 19:46:56 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2020:3894	0	None	None	None	2020-09-29 19:48:24 UTC

Description Graham Leggett 2020-01-05 14:14:24 UTC

Description of problem:

389-ds-base crashes on startup after a previous disorderly shutdown.

Version-Release number of selected component (if applicable):

[root@phoebe ~]# rpm -q -a | grep 389-ds-base
389-ds-base-1.3.9.1-12.el7_7.x86_64
389-ds-base-libs-1.3.9.1-12.el7_7.x86_64
389-ds-base-debuginfo-1.3.9.1-12.el7_7.x86_64
[root@phoebe ~]# rpm -q -a | grep libdb
libdb-5.3.21-25.el7.x86_64
libdb-debuginfo-5.3.21-25.el7.x86_64
libdb-utils-5.3.21-25.el7.x86_64

How reproducible: Always

Steps to Reproduce:
1. Have a hypervisor crash.
2. Restart 389-ds-base in a virtual machine

Actual results:

389ds logs the following and crashes:

[05/Jan/2020:13:38:58.207238892 +0000] - NOTICE - dblayer_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.

Expected results:

389ds starts normally, or starts and emits a sensible error message.

Additional info:

Backtrace of crash:

[root@phoebe ~]# gdb /usr/sbin/ns-slapd
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/ns-slapd...Reading symbols from /usr/lib/debug/usr/sbin/ns-slapd.debug...done.
done.
(gdb) run -D /etc/dirsrv/slapd-phoebe -i /tmp/pid -d 0
Starting program: /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-phoebe -i /tmp/pid -d 0
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Program received signal SIGBUS, Bus error.
__env_faultmem (addr=<optimized out>, size=<optimized out>, created=<optimized out>, env=<optimized out>, env=<optimized out>)
    at ../../src/env/env_region.c:1502
1502					ret |= p[0];
Missing separate debuginfos, use: debuginfo-install audit-libs-2.8.5-4.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-gssapi-2.1.26-23.el7.x86_64 cyrus-sasl-lib-2.1.26-23.el7.x86_64 cyrus-sasl-md5-2.1.26-23.el7.x86_64 cyrus-sasl-plain-2.1.26-23.el7.x86_64 elfutils-libelf-0.176-2.el7.x86_64 elfutils-libs-0.176-2.el7.x86_64 glibc-2.17-292.el7.x86_64 gperftools-libs-2.6.1-1.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-37.el7_7.2.x86_64 libattr-2.4.46-13.el7.x86_64 libcap-2.22-10.el7.x86_64 libcap-ng-0.7.5-4.el7.x86_64 libcom_err-1.42.9-16.el7.x86_64 libevent-2.0.21-4.el7.x86_64 libgcc-4.8.5-39.el7.x86_64 libgcrypt-1.5.3-14.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libicu-50.2-3.el7.x86_64 libselinux-2.5-14.1.el7.x86_64 libstdc++-4.8.5-39.el7.x86_64 lz4-1.7.5-3.el7.x86_64 nspr-4.21.0-1.el7.x86_64 nss-3.44.0-7.el7_7.x86_64 nss-softokn-3.44.0-8.el7_7.x86_64 nss-softokn-freebl-3.44.0-8.el7_7.x86_64 nss-util-3.44.0-4.el7_7.x86_64 openldap-2.4.44-21.el7_6.x86_64 openssl-libs-1.0.2k-19.el7.x86_64 pam-1.1.8-22.el7.x86_64 pcre-8.32-17.el7.x86_64 sqlite-3.7.17-8.el7.x86_64 svrcore-4.1.3-2.el7.x86_64 systemd-libs-219-67.el7_7.2.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) bt full
#0  0x00007fffedc7ed60 in __env_faultmem (addr=<optimized out>, size=<optimized out>, created=<optimized out>, env=<optimized out>, env=<optimized out>)
    at ../../src/env/env_region.c:1502
        ret = 255
        p = 0x7fffe2020000 <Address 0x7fffe2020000 out of bounds>
        t = 0x7fffe210a4d8 <Address 0x7fffe210a4d8 out of bounds>
#1  0x00007fffedc7ff68 in __env_faultmem (env=0x55555605d7c0, env=0x55555605d7c0, created=<optimized out>, size=<optimized out>, addr=<optimized out>)
    at ../../src/env/env_region.c:1107
        rp = 0x7fffe7c56238
        ret = 0
        buf = "__db.003\000\001\000\000\000\000\000\000\000\211+VUU\000\000\000\246\232\357\026x"
        st = 
            {st_dev = 64768, st_ino = 9772386, st_nlink = 1, st_mode = 33152, st_uid = 99, st_gid = 99, __pad0 = 0, st_rdev = 0, st_size = 966656, st_blksize = 4096, st_blocks = 1888, st_atim = {tv_sec = 1578231263, tv_nsec = 898307277}, st_mtim = {tv_sec = 1577743377, tv_nsec = 563965884}, st_ctim = {tv_sec = 1577743377, tv_nsec = 563965884}, __unused = {0, 0, 0}}
#2  0x00007fffedc7ff68 in __env_region_attach (env=env@entry=0x55555605d7c0, infop=infop@entry=0x7fffffffda30, init=init@entry=0, max=max@entry=0)
    at ../../src/env/env_region.c:1107
        rp = 0x7fffe7c56238
        ret = 0
        buf = "__db.003\000\001\000\000\000\000\000\000\000\211+VUU\000\000\000\246\232\357\026x"
        st = 
            {st_dev = 64768, st_ino = 9772386, st_nlink = 1, st_mode = 33152, st_uid = 99, st_gid = 99, __pad0 = 0, st_rdev = 0, st_size = 966656, st_blksize = 4096, st_blocks = 1888, st_atim = {tv_sec = 1578231263, tv_nsec = 898307277}, st_mtim = {tv_sec = 1577743377, tv_nsec = 563965884}, st_ctim = {tv_sec = 1577743377, tv_nsec = 563965884}, __unused = {0, 0, 0}}
#3  0x00007fffedc80199 in __env_remove_env (env=env@entry=0x55555605d7c0) at ../../src/env/env_region.c:919
        dbenv = 0x5555562b8900
        renv = 0x7fffe7c56000
        infop = <optimized out>
        reginfo = 
          {env = 0x55555605d7c0, type = REGION_TYPE_MPOOL, id = 3, rp = 0x7fffe7c56238, name = 0x5555561259b0 "/var/lib/dirsrv/slapd-phoebe/db/__db.003", fhp = 0x555556305360, addr = 0x7fffe1f34000, head = 0x7fffe1f34000, primary = 0x0, max_alloc = 0, allocated = 0, mem = 0x0, mtx_alloc = 0, flags = 2}
        rp = 0x7fffe7c56238
        flags_orig = 0
        i = 2
#4  0x00007fffedc79a94 in __env_open (dbenv=0x5555562b8900, db_home=<optimized out>, flags=9987, mode=<optimized out>) at ../../src/env/env_open.c:201
        ip = 0x0
        env = 0x55555605d7c0
---Type <return> to continue, or q <return> to quit---
        orig_flags = 2048
        register_recovery = 0
        ret = 0
        t_ret = <optimized out>
#5  0x00007fffeb5d4091 in dblayer_start (li=0x55555605c8c0, dbmode=dbmode@entry=2) at ldap/servers/slapd/back-ldbm/dblayer.c:1490
        recover_flags = 9987
        return_value = <optimized out>
        priv = 0x555555fb6600
        pEnv = 0x55555626b000
        region_dir = 0x55555626b5e0 "/var/lib/dirsrv/slapd-phoebe/db"
        log_dir = <optimized out>
        open_flags = 10019
        mi = <optimized out>
        result = <optimized out>
#6  0x00007fffeb62ae97 in ldbm_back_start (pb=<optimized out>) at ldap/servers/slapd/back-ldbm/start.c:370
        li = 0x55555605c8c0
        home_dir = 0x0
        action = 0
        retval = <optimized out>
#7  0x00007ffff7939097 in plugin_call_func (list=0x55555606db80, operation=operation@entry=212, pb=0x555556192120, call_one=call_one@entry=1)
    at ldap/servers/slapd/plugin.c:2028
        n = <optimized out>
        func = 0x7fffeb62a640 <ldbm_back_start>
        rc = <optimized out>
        return_value = 0
        count = 0
#8  0x00007ffff793970b in plugin_call_one (pb=<optimized out>, operation=212, list=<optimized out>) at ldap/servers/slapd/plugin.c:1978
        enabled = <optimized out>
        satisfied = 1
        break_out = 0
        ret = <optimized out>
        total_plugins = 78
        i = <optimized out>
        value = 0x0
        index = <optimized out>
        plugins_started = 1
---Type <return> to continue, or q <return> to quit---
        shutdown_index = 31
        pb = <optimized out>
        config = 0x5555562ee000
        plugin_index = 60
        plugin_entry = <optimized out>
        plugin_head = 0x55555626b3e0
        the_plugin_type = <optimized out>
        num_plg_started = 52
        plugin = <optimized out>
        ep = <optimized out>
#9  0x00007ffff793970b in plugin_dependency_startall (operation=212, errmsg=<synthetic pointer>, plugin_list=0x0, argv=0x7fffffffe3b8, argc=7)
    at ldap/servers/slapd/plugin.c:1737
        enabled = <optimized out>
        satisfied = 1
        break_out = 0
        ret = <optimized out>
        total_plugins = 78
        i = <optimized out>
        value = 0x0
        index = <optimized out>
        plugins_started = 1
        shutdown_index = 31
        pb = <optimized out>
        config = 0x5555562ee000
        plugin_index = 60
        plugin_entry = <optimized out>
        plugin_head = 0x55555626b3e0
        the_plugin_type = <optimized out>
        num_plg_started = 52
        plugin = <optimized out>
        ep = <optimized out>
#10 0x00007ffff793970b in plugin_startall (argc=argc@entry=7, argv=argv@entry=0x7fffffffe3b8, plugin_list=plugin_list@entry=0x0)
    at ldap/servers/slapd/plugin.c:1950
#11 0x0000555555565a18 in main (argc=<optimized out>, argv=0x7fffffffe3b8) at ldap/servers/slapd/main.c:1141
        rc = 0
        sdn = 0x0
---Type <return> to continue, or q <return> to quit---
        return_value = <optimized out>
        mcfg = 
          {extraname = 0x0, slapd_exemode = 1, n_port = 389, i_port = 0, s_port = 636, myname = 0x555555fb2050 "ns-slapd", ldif_file = 0x0, ldif_files = 0, cmd_line_instance_name = 0x0, cmd_line_instance_names = 0x0, skip_db_protect_check = 0, db2ldif_include = 0x0, db2ldif_exclude = 0x0, ldif2db_removedupvals = 1, ldif2db_noattrindexes = 0, db2index_attrs = 0x0, ldif_printkey = 5, archive_name = 0x0, db2ldif_dump_replica = 0, db2ldif_dump_uniqueid = 1, ldif2db_generate_uniqueid = 1, ldif2db_namespaceid = 0x0, importexport_encrypt = 0, upgradedb_flags = 0, upgradednformat_dryrun = 0, is_quiet = 0, backuptools_verbose = 0, dbverify_verbose = 0, dbverify_dbdir = 0x0}
        slapdFrontendConfig = <optimized out>
        ports_info = 
          {n_port = 389, s_port = 636, n_listenaddr = 0x5555562a39e0, s_listenaddr = 0x5555562ac360, n_socket = 0x5555562ac420, i_listenaddr = 0x0, i_port = 0, i_socket = 0x0, s_socket = 0x5555562ac1d0}
        tp = 0x0
        m = <optimized out>
        notify = <optimized out>

Comment 2 thierry bordaz 2020-01-06 17:23:42 UTC

Hi Graham, 

It is expected that, after hypervisor crash, DS detects (at restart) a disorderly shutdown and run a recovery.
It looks the environment files (/var/lib/dirsrv/slapd-phoebe/db/__db*) are in bad state and prevent to open the environment and the recovery.

Is it reproducible ? If yes, could you remove env files __db* before attempting the startup/recovery ?
You can also try to run db_verify but it may also fail to open_env.

Comment 4 mreynolds 2020-01-14 19:30:16 UTC

Upstream ticket:

https://pagure.io/389-ds-base/issue/50599

Comment 7 Viktor Ashirov 2020-03-03 11:57:16 UTC

*** Bug 1610409 has been marked as a duplicate of this bug. ***

Comment 10 Viktor Ashirov 2020-03-20 11:11:32 UTC

Build tested: 389-ds-base-1.3.10.2-1.1asan.el7.x86_64

I've installed a fresh instance, stopped it. I've opened __db.001 and put some garbage data in it. On the next start server crashed since the env files were corrupted, but the next start was successful since the __db.* files were recreated.

Marking as VERIFIED.

Comment 14 errata-xmlrpc 2020-09-29 19:46:56 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (389-ds-base bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3894

Note You need to log in before you can comment on or make changes to this bug.