Bug 1705320 - libmagic uses exit() instead of _exit() after failed exec()
Summary: libmagic uses exit() instead of _exit() after failed exec()
Alias: None
Product: Fedora
Classification: Fedora
Component: file
Version: 30
Hardware: x86_64
OS: Linux
Target Milestone: ---
Assignee: Kamil Dudka
QA Contact: Fedora Extras Quality Assurance
Depends On:
TreeView+ depends on / blocked
Reported: 2019-05-02 02:17 UTC by Eshin Kunishima
Modified: 2019-09-06 12:33 UTC (History)
11 users (show)

Fixed In Version: file-5.36-4.fc30
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2019-09-06 12:33:33 UTC
Type: Bug

Attachments (Terms of Use)

Description Eshin Kunishima 2019-05-02 02:17:29 UTC
Description of problem:
when I was building RPM package, rpmbuild suddenly crashed with segmentation fault. rpmdb seems to be broken at the time.

Version-Release number of selected component (if applicable):
    [root@localhost ~]# dnf --version
    [root@localhost ~]# rpm --version
    RPM version

How reproducible:
since my environment was broken so I didn't check reproducible yet.

Steps to Reproduce:
1. dnf install autoconf automake libtool wget libtirpc-devel rpm-build zlib-devel libuuid-devel libattr-devel libblkid-devel libselinux-devel libudev-devel libaio-devel parted lsscsi ksh openssl-devel elfutils-libelf-devel libffi-devel
2. git clone https://github.com/zfsonlinux/zfs.git
3. cd zfs && ./autogen.sh && ./configure --with-config=srpm && make -s -j$(nproc) && make -j1 pkg-utils rpm-dkms

Actual results:
rpmbuild suddenly crashed and rpmdb was broken

Expected results:
keep rpmdb correctly even if rpmbuild crashed

Additional info:

    [root@localhost ~]# dmesg
    [ 2695.839862] rpmbuild[6682]: segfault at 7f429950a531 ip 00007f429adfd4f9 sp 00007ffdb8e9b600 error 6 in libdb-5.3.so[7f429ad41000+14f000]
    [ 2695.839870] Code: d1 48 8b 4d 48 48 89 54 08 60 48 29 d8 48 89 75 48 48 89 43 60 48 8b 43 68 48 83 f8 ff 0f 84 a6 00 00 00 48 8b 53 70 48 29 c2 <48> 89 54 03 70 48 8b 43 70 48 8b 53 68 48 01 d8 48 01 10 83 ad b0

    [root@localhost ~]# dnf 
    error: rpmdb: BDB0113 Thread/process 6682/139924035643584 failed: BDB1507 Thread died in Berkeley DB library
    error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
    error: cannot open Packages index using db5 -  (-30973)
    error: cannot open Packages database in /var/lib/rpm
    Error: Error: rpmdb open failed

    [root@localhost ~]# rpm -qa
    (stucked. it filled up 1 CPU)

Comment 1 Panu Matilainen 2019-05-15 13:51:48 UTC
Never, ever, build software as root. dnf install build-deps obviously needs to be done as root but the rest should be done as a regular user, that'll help keeping your system safe from more than just rpmdb breakage.

Obviously that's not an excuse for rpmbuild to crash, but its segfaulting somewhere inside libdb and with complex make & other scripts running as root and building custom kernel modules, it's anybodys guess what's going on in there. If you can come up with a minimal reproducer involving just rpmbuild on non-tainted system then we can have a look.

Comment 2 Eshin Kunishima 2019-05-15 17:23:59 UTC
Thank you for your feedback. I Installed Fedora 30 Server using Minimal Install packages group to KVM and build packages with "root" user to reproduce this issue. same as the previous time, rpmbuild crashed with SEGV and its database was broken. the least, building packages with a "regular user" have never cause SEGV for the past couple of weeks.

        [root@localhost ~]# uname -a
        Linux localhost.localdomain 5.0.14-300.fc30.x86_64 #1 SMP Thu May 9 10:43:38 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

        [root@localhost ~]# dnf install autoconf automake libtool wget libtirpc-devel rpm-build zlib-devel libuuid-devel libattr-devel libblkid-devel libselinux-devel libudev-devel libaio-devel parted lsscsi ksh openssl-devel elfutils-libelf-devel libffi-devel make git
        [root@localhost ~]# git clone --depth 1 https://github.com/zfsonlinux/zfs.git
        [root@localhost ~]# ./autogen.sh && ./configure --with-config=srpm && make -s -j$(nproc)

        (there was no issue at this time. rpm worked correctly.)

        [root@localhost ~]# make -j1 pkg-utils rpm-dkms
        ... trimmed ...

        (the first-time build seems to be successful and rpm still worked. but SEGV happened on running the command again.)

        [root@localhost ~]# make -j1 pkg-utils rpm-dkms
        ... trimmed ...
        Executing(%doc): /bin/sh -e /tmp/zfs-build-root-tMXwDbTy/TMP/rpm-tmp.wNzcPV
        + umask 022
        + cd /tmp/zfs-build-root-tMXwDbTy/BUILD
        + cd zfs-0.8.0
        + DOCDIR=/tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/doc/libzfs2-devel
        + export LC_ALL=C
        + LC_ALL=C
        + export DOCDIR
        + /usr/bin/mkdir -p /tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/doc/libzfs2-devel
        + cp -pr AUTHORS /tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/doc/libzfs2-devel
        + cp -pr COPYRIGHT /tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/doc/libzfs2-devel
        + cp -pr LICENSE /tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/doc/libzfs2-devel
        + cp -pr NOTICE /tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/doc/libzfs2-devel
        + cp -pr README.md /tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/doc/libzfs2-devel
        + exit 0
        Provides: libnvpair1-devel libuutil1-devel libzfs2-devel = 0.8.0-rc5.fc30 libzfs2-devel(x86-64) = 0.8.0-rc5.fc30 libzpool2-devel pkgconfig(libzfs) = 0.8.0 pkgconfig(libzfs_core) = 0.8.0
        Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
        Requires: /usr/bin/pkg-config libnvpair.so.1()(64bit) libuutil.so.1()(64bit) libzfs.so.2()(64bit) libzfs_core.so.1()(64bit) libzpool.so.2()(64bit) pkgconfig(libzfs_core)
        Obsoletes: zfs-devel
        Processing files: zfs-test-0.8.0-rc5.fc30.x86_64

        (rpmbuild crashed with SEGV and rpmdb was died.)

        [root@localhost ~]# coredumpctl info
                PID: 32527 (rpmbuild)
                UID: 0 (root)
                GID: 0 (root)
                Signal: 11 (SEGV)
            Timestamp: Thu 2019-05-16 01:16:51 JST (9min ago)
        Command Line: rpmbuild --define _tmppath /tmp/zfs-build-root-tMXwDbTy/TMP --define _topdir /tmp/zfs-build-root-tMXwDbTy --define _without_debug 1 --define _without_debug_kmem 1 --define _without_debug_kmem_tracking 1 --define _without_debuginfo 1 --define _without_asan 1 --define _initconfdir /etc/sysconfig --rebuild zfs-0.8.0-rc5.fc30.src.rpm
            Executable: /usr/bin/rpmbuild
        Control Group: /user.slice/user-0.slice/session-1.scope
                Unit: session-1.scope
                Slice: user-0.slice
            Session: 1
            Owner UID: 0 (root)
            Boot ID: 01846aef47f049dba4ae5437f6fde42b
            Machine ID: 202697992ccb430badd4ea635e176cee
            Hostname: localhost.localdomain
            Storage: /var/lib/systemd/coredump/core.rpmbuild.0.01846aef47f049dba4ae5437f6fde42b.32527.1557937011000000.lz4
            Message: Process 32527 (rpmbuild) of user 0 dumped core.

                        Stack trace of thread 32527:
                        #0  0x00007f6d105b64f9 n/a (libdb-5.3.so)
                        #1  0x00007f6d105b663a __lock_id_free (libdb-5.3.so)
                        #2  0x00007f6d105ff9ed __env_refresh (libdb-5.3.so)
                        #3  0x00007f6d105ffed0 __env_close (libdb-5.3.so)
                        #4  0x00007f6d106003a7 __env_close_pp (libdb-5.3.so)
                        #5  0x00007f6d108f0718 n/a (librpm.so.8)
                        #6  0x00007f6d108fa1e1 n/a (librpm.so.8)
                        #7  0x00007f6d108fd16d n/a (librpm.so.8)
                        #8  0x00007f6d106e86c0 __run_exit_handlers (libc.so.6)
                        #9  0x00007f6d106e8800 exit (libc.so.6)
                        #10 0x00007f6d102384a7 file_zmagic (libmagic.so.1)
                        #11 0x00007f6d1023f043 file_buffer (libmagic.so.1)
                        #12 0x00007f6d1022cc7e n/a (libmagic.so.1)
                        #13 0x00007f6d109752b6 rpmfcClassify (librpmbuild.so.8)
                        #14 0x00007f6d10975dcb n/a (librpmbuild.so.8)
                        #15 0x00007f6d10967003 n/a (librpmbuild.so.8)
                        #16 0x00007f6d1096002c n/a (librpmbuild.so.8)
                        #17 0x000055ab7527880d n/a (rpmbuild)
                        #18 0x000055ab75278a6b n/a (rpmbuild)
                        #19 0x000055ab75277d31 n/a (rpmbuild)
                        #20 0x00007f6d106d1f33 __libc_start_main (libc.so.6)
                        #21 0x000055ab75277dde n/a (rpmbuild)

        [root@localhost coredump]# gdb /usr/bin/rpmbuild core.rpmbuild.0.01846aef47f049dba4ae5437f6fde42b.32527.1557937011000000

        ... trimmed ...

        Core was generated by `rpmbuild --define _tmppath /tmp/zfs-build-root-tMXwDbTy/TMP --define _topdir /t'.
        Program terminated with signal SIGSEGV, Segmentation fault.
        #0  0x00007f6d105b64f9 in __lock_freelocker_int (region=region@entry=0x7f6d0ef826b0, sh_locker=sh_locker@entry=0x7f6d0efb4e78, reallyfree=reallyfree@entry=1, lt=<optimized out>, lt=<optimized out>) at ../../src/lock/lock_id.c:507
        507			SH_TAILQ_REMOVE(&region->lockers, sh_locker,
        (gdb) bt
        #0  0x00007f6d105b64f9 in __lock_freelocker_int (region=region@entry=0x7f6d0ef826b0, sh_locker=sh_locker@entry=0x7f6d0efb4e78, reallyfree=reallyfree@entry=1, lt=<optimized out>, lt=<optimized out>) at ../../src/lock/lock_id.c:507
        #1  0x00007f6d105b663a in __lock_id_free (env=env@entry=0x55ab76900580, sh_locker=0x7f6d0efb4e78) at ../../src/lock/lock_id.c:213
        #2  0x00007f6d105ff9ed in __env_refresh (dbenv=dbenv@entry=0x55ab768ffd50, orig_flags=orig_flags@entry=0, rep_check=rep_check@entry=0) at ../../src/env/env_open.c:754
        #3  0x00007f6d105ffed0 in __env_close (dbenv=0x55ab768ffd50, flags=<optimized out>) at ../../src/env/env_open.c:651
        #4  0x00007f6d106003a7 in __env_close_pp (dbenv=0x55ab768ffd50, flags=<optimized out>) at ../../src/env/env_open.c:579
        #5  0x00007f6d108f0718 in db_fini (dbhome=0x55ab76903170 "/var/lib/rpm", rdb=0x55ab768ffb50) at backend/db3.c:179
        #6  db3_dbiClose (dbi=0x55ab768ff8c0, flags=<optimized out>) at backend/db3.c:756
        #7  0x00007f6d108fa1e1 in dbiForeach (del=1, func=<optimized out>, ndbi=<optimized out>, dbis=0x55ab768ffc70) at rpmdb.c:378
        #8  rpmdbClose (db=0x55ab768ffb50) at rpmdb.c:411
        #9  0x00007f6d108fd16d in rpmdbClose (db=<optimized out>) at rpmdb.c:391
        #10 rpmAtExit () at rpmdb.c:327
        #11 0x00007f6d106e86c0 in __run_exit_handlers (status=status@entry=1, listp=0x7f6d1086e738 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
        #12 0x00007f6d106e8800 in __GI_exit (status=status@entry=1) at exit.c:139
        #13 0x00007f6d102384a7 in uncompressbuf (n=0x7ffeb5626aa8, newch=0x7ffeb5626aa0, old=<optimized out>, method=1, bytes_max=1048576, fd=<optimized out>) at compress.c:728
        #14 file_zmagic (ms=ms@entry=0x55ab7691d6a0, b=b@entry=0x7ffeb5626b40, name=name@entry=0x55ab76dea6e0 "/tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/zfs/zfs-tests/tests/functional/history/sparc.migratedpool.DAT.Z") at compress.c:250
        #15 0x00007f6d1023f043 in file_buffer (ms=ms@entry=0x55ab7691d6a0, fd=fd@entry=15, inname=inname@entry=0x55ab76dea6e0 "/tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/zfs/zfs-tests/tests/functional/history/sparc.migratedpool.DAT.Z", buf=buf@entry=0x55ab76dfede0, nb=163879)
            at funcs.c:249
        #16 0x00007f6d1022cc7e in file_or_fd (ms=ms@entry=0x55ab7691d6a0, inname=inname@entry=0x55ab76dea6e0 "/tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/zfs/zfs-tests/tests/functional/history/sparc.migratedpool.DAT.Z", fd=15, fd@entry=0) at magic.c:509
        #17 0x00007f6d1022d320 in magic_file (ms=ms@entry=0x55ab7691d6a0, inname=inname@entry=0x55ab76dea6e0 "/tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/zfs/zfs-tests/tests/functional/history/sparc.migratedpool.DAT.Z") at magic.c:398
        #18 0x00007f6d109752b6 in rpmfcClassify (fc=fc@entry=0x55ab76920050, argv=0x55ab76c4e2f0, fmode=fmode@entry=0x55ab76caa5e0) at rpmfc.c:1104
        #19 0x00007f6d10975dcb in rpmfcGenerateDepends (spec=spec@entry=0x55ab76908b00, pkg=pkg@entry=0x55ab769395c0) at rpmfc.c:1426
        #20 0x00007f6d10967003 in processBinaryFiles (spec=spec@entry=0x55ab76908b00, pkgFlags=0, didInstall=didInstall@entry=4, test=test@entry=0) at files.c:3114
        #21 0x00007f6d1096002c in buildSpec (buildArgs=0x55ab7527c980 <rpmBTArgs>, spec=0x55ab76908b00, what=2975) at build.c:261
        #22 0x000055ab7527880d in buildForTarget (ts=<optimized out>, arg=<optimized out>, ba=0x55ab7527c980 <rpmBTArgs>) at rpmbuild.c:517
        #23 0x000055ab75278a6b in build (ts=0x55ab768febb0, arg=0x55ab76907cd0 "/tmp/zfs-build-root-tMXwDbTy/SPECS/zfs.spec", rcfile=0x0, ba=0x55ab7527c980 <rpmBTArgs>) at rpmbuild.c:545
        #24 0x000055ab75277d31 in main (argc=<optimized out>, argv=<optimized out>) at rpmbuild.c:681

Comment 3 Eshin Kunishima 2019-05-15 19:01:08 UTC
according to the backtrace, this issue seems to caused by missing de-compression tool "uncompress" included in "ncompress" package to decompress .Z file. execvp(3) returned error and forked process died with exit(3), in result, rpmAtExit inherited from a parent process was called. is this situation being assumed in rpmbuild?

Comment 4 Panu Matilainen 2019-05-16 05:58:36 UTC
Heh, oookkaay. Thanks for the traceback and analysis, that actually does make sense. 

I had no idea libmagic forks external helpers, but it's no wonder if libdb crashes trying to shut down a database created in another process. It's not exactly an expected usage pattern, but shouldn't be hard to prevent either. 

Arguably there's a bug in libmagic too, since it's a forked child whose exec() failed it should arguably use _exit() instead of exit() to avoid this kind of thing from happening.

Comment 5 Panu Matilainen 2019-05-16 06:39:42 UTC
Seems the libmagic side has (mostly) been fixed already just a few days ago:

Comment 6 Panu Matilainen 2019-08-29 11:20:55 UTC
On a second thought, rpm cannot really protect itself from arbitrary fork()+exit() from either underlying libraries or API users, reassigning to file which is the troublemaker here. Like noted above, this is at least mostly fixed upstream already, but I don't know about status in Fedora.

Comment 7 Kamil Dudka 2019-08-30 15:29:06 UTC
dist-git commit: https://src.fedoraproject.org/rpms/file/c/136eea21

Comment 8 Fedora Update System 2019-08-30 15:48:11 UTC
FEDORA-2019-83fa1cfd0f has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-83fa1cfd0f

Comment 9 Fedora Update System 2019-08-31 01:04:00 UTC
file-5.36-4.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-83fa1cfd0f

Comment 10 Eshin Kunishima 2019-08-31 04:27:12 UTC
I've tested same condition with testing packages. As a result, the rpmdb breakage issue seems to be fixed. Thank you for your response.

Comment 11 Kamil Dudka 2019-09-02 10:26:52 UTC
Perfect.  Thank you for testing it!

Comment 12 Fedora Update System 2019-09-06 12:33:33 UTC
file-5.36-4.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.