Hide Forgot
Description of problem: when I was building RPM package, rpmbuild suddenly crashed with segmentation fault. rpmdb seems to be broken at the time. Version-Release number of selected component (if applicable): [root@localhost ~]# dnf --version 4.2.2 [root@localhost ~]# rpm --version RPM version 4.14.2.1 How reproducible: since my environment was broken so I didn't check reproducible yet. https://github.com/zfsonlinux/zfs/wiki/Custom-Packages#dkms Steps to Reproduce: 1. dnf install autoconf automake libtool wget libtirpc-devel rpm-build zlib-devel libuuid-devel libattr-devel libblkid-devel libselinux-devel libudev-devel libaio-devel parted lsscsi ksh openssl-devel elfutils-libelf-devel libffi-devel 2. git clone https://github.com/zfsonlinux/zfs.git 3. cd zfs && ./autogen.sh && ./configure --with-config=srpm && make -s -j$(nproc) && make -j1 pkg-utils rpm-dkms Actual results: rpmbuild suddenly crashed and rpmdb was broken Expected results: keep rpmdb correctly even if rpmbuild crashed Additional info: [root@localhost ~]# dmesg ...trimmed... [ 2695.839862] rpmbuild[6682]: segfault at 7f429950a531 ip 00007f429adfd4f9 sp 00007ffdb8e9b600 error 6 in libdb-5.3.so[7f429ad41000+14f000] [ 2695.839870] Code: d1 48 8b 4d 48 48 89 54 08 60 48 29 d8 48 89 75 48 48 89 43 60 48 8b 43 68 48 83 f8 ff 0f 84 a6 00 00 00 48 8b 53 70 48 29 c2 <48> 89 54 03 70 48 8b 43 70 48 8b 53 68 48 01 d8 48 01 10 83 ad b0 ...trimmed... [root@localhost ~]# dnf error: rpmdb: BDB0113 Thread/process 6682/139924035643584 failed: BDB1507 Thread died in Berkeley DB library error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: cannot open Packages index using db5 - (-30973) error: cannot open Packages database in /var/lib/rpm Error: Error: rpmdb open failed [root@localhost ~]# rpm -qa (stucked. it filled up 1 CPU)
Never, ever, build software as root. dnf install build-deps obviously needs to be done as root but the rest should be done as a regular user, that'll help keeping your system safe from more than just rpmdb breakage. Obviously that's not an excuse for rpmbuild to crash, but its segfaulting somewhere inside libdb and with complex make & other scripts running as root and building custom kernel modules, it's anybodys guess what's going on in there. If you can come up with a minimal reproducer involving just rpmbuild on non-tainted system then we can have a look.
Thank you for your feedback. I Installed Fedora 30 Server using Minimal Install packages group to KVM and build packages with "root" user to reproduce this issue. same as the previous time, rpmbuild crashed with SEGV and its database was broken. the least, building packages with a "regular user" have never cause SEGV for the past couple of weeks. [root@localhost ~]# uname -a Linux localhost.localdomain 5.0.14-300.fc30.x86_64 #1 SMP Thu May 9 10:43:38 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux [root@localhost ~]# dnf install autoconf automake libtool wget libtirpc-devel rpm-build zlib-devel libuuid-devel libattr-devel libblkid-devel libselinux-devel libudev-devel libaio-devel parted lsscsi ksh openssl-devel elfutils-libelf-devel libffi-devel make git [root@localhost ~]# git clone --depth 1 https://github.com/zfsonlinux/zfs.git [root@localhost ~]# ./autogen.sh && ./configure --with-config=srpm && make -s -j$(nproc) (there was no issue at this time. rpm worked correctly.) [root@localhost ~]# make -j1 pkg-utils rpm-dkms ... trimmed ... (the first-time build seems to be successful and rpm still worked. but SEGV happened on running the command again.) [root@localhost ~]# make -j1 pkg-utils rpm-dkms ... trimmed ... Executing(%doc): /bin/sh -e /tmp/zfs-build-root-tMXwDbTy/TMP/rpm-tmp.wNzcPV + umask 022 + cd /tmp/zfs-build-root-tMXwDbTy/BUILD + cd zfs-0.8.0 + DOCDIR=/tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/doc/libzfs2-devel + export LC_ALL=C + LC_ALL=C + export DOCDIR + /usr/bin/mkdir -p /tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/doc/libzfs2-devel + cp -pr AUTHORS /tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/doc/libzfs2-devel + cp -pr COPYRIGHT /tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/doc/libzfs2-devel + cp -pr LICENSE /tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/doc/libzfs2-devel + cp -pr NOTICE /tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/doc/libzfs2-devel + cp -pr README.md /tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/doc/libzfs2-devel + exit 0 Provides: libnvpair1-devel libuutil1-devel libzfs2-devel = 0.8.0-rc5.fc30 libzfs2-devel(x86-64) = 0.8.0-rc5.fc30 libzpool2-devel pkgconfig(libzfs) = 0.8.0 pkgconfig(libzfs_core) = 0.8.0 Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 Requires: /usr/bin/pkg-config libnvpair.so.1()(64bit) libuutil.so.1()(64bit) libzfs.so.2()(64bit) libzfs_core.so.1()(64bit) libzpool.so.2()(64bit) pkgconfig(libzfs_core) Obsoletes: zfs-devel Processing files: zfs-test-0.8.0-rc5.fc30.x86_64 (rpmbuild crashed with SEGV and rpmdb was died.) [root@localhost ~]# coredumpctl info PID: 32527 (rpmbuild) UID: 0 (root) GID: 0 (root) Signal: 11 (SEGV) Timestamp: Thu 2019-05-16 01:16:51 JST (9min ago) Command Line: rpmbuild --define _tmppath /tmp/zfs-build-root-tMXwDbTy/TMP --define _topdir /tmp/zfs-build-root-tMXwDbTy --define _without_debug 1 --define _without_debug_kmem 1 --define _without_debug_kmem_tracking 1 --define _without_debuginfo 1 --define _without_asan 1 --define _initconfdir /etc/sysconfig --rebuild zfs-0.8.0-rc5.fc30.src.rpm Executable: /usr/bin/rpmbuild Control Group: /user.slice/user-0.slice/session-1.scope Unit: session-1.scope Slice: user-0.slice Session: 1 Owner UID: 0 (root) Boot ID: 01846aef47f049dba4ae5437f6fde42b Machine ID: 202697992ccb430badd4ea635e176cee Hostname: localhost.localdomain Storage: /var/lib/systemd/coredump/core.rpmbuild.0.01846aef47f049dba4ae5437f6fde42b.32527.1557937011000000.lz4 Message: Process 32527 (rpmbuild) of user 0 dumped core. Stack trace of thread 32527: #0 0x00007f6d105b64f9 n/a (libdb-5.3.so) #1 0x00007f6d105b663a __lock_id_free (libdb-5.3.so) #2 0x00007f6d105ff9ed __env_refresh (libdb-5.3.so) #3 0x00007f6d105ffed0 __env_close (libdb-5.3.so) #4 0x00007f6d106003a7 __env_close_pp (libdb-5.3.so) #5 0x00007f6d108f0718 n/a (librpm.so.8) #6 0x00007f6d108fa1e1 n/a (librpm.so.8) #7 0x00007f6d108fd16d n/a (librpm.so.8) #8 0x00007f6d106e86c0 __run_exit_handlers (libc.so.6) #9 0x00007f6d106e8800 exit (libc.so.6) #10 0x00007f6d102384a7 file_zmagic (libmagic.so.1) #11 0x00007f6d1023f043 file_buffer (libmagic.so.1) #12 0x00007f6d1022cc7e n/a (libmagic.so.1) #13 0x00007f6d109752b6 rpmfcClassify (librpmbuild.so.8) #14 0x00007f6d10975dcb n/a (librpmbuild.so.8) #15 0x00007f6d10967003 n/a (librpmbuild.so.8) #16 0x00007f6d1096002c n/a (librpmbuild.so.8) #17 0x000055ab7527880d n/a (rpmbuild) #18 0x000055ab75278a6b n/a (rpmbuild) #19 0x000055ab75277d31 n/a (rpmbuild) #20 0x00007f6d106d1f33 __libc_start_main (libc.so.6) #21 0x000055ab75277dde n/a (rpmbuild) [root@localhost coredump]# gdb /usr/bin/rpmbuild core.rpmbuild.0.01846aef47f049dba4ae5437f6fde42b.32527.1557937011000000 ... trimmed ... Core was generated by `rpmbuild --define _tmppath /tmp/zfs-build-root-tMXwDbTy/TMP --define _topdir /t'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f6d105b64f9 in __lock_freelocker_int (region=region@entry=0x7f6d0ef826b0, sh_locker=sh_locker@entry=0x7f6d0efb4e78, reallyfree=reallyfree@entry=1, lt=<optimized out>, lt=<optimized out>) at ../../src/lock/lock_id.c:507 507 SH_TAILQ_REMOVE(®ion->lockers, sh_locker, (gdb) bt #0 0x00007f6d105b64f9 in __lock_freelocker_int (region=region@entry=0x7f6d0ef826b0, sh_locker=sh_locker@entry=0x7f6d0efb4e78, reallyfree=reallyfree@entry=1, lt=<optimized out>, lt=<optimized out>) at ../../src/lock/lock_id.c:507 #1 0x00007f6d105b663a in __lock_id_free (env=env@entry=0x55ab76900580, sh_locker=0x7f6d0efb4e78) at ../../src/lock/lock_id.c:213 #2 0x00007f6d105ff9ed in __env_refresh (dbenv=dbenv@entry=0x55ab768ffd50, orig_flags=orig_flags@entry=0, rep_check=rep_check@entry=0) at ../../src/env/env_open.c:754 #3 0x00007f6d105ffed0 in __env_close (dbenv=0x55ab768ffd50, flags=<optimized out>) at ../../src/env/env_open.c:651 #4 0x00007f6d106003a7 in __env_close_pp (dbenv=0x55ab768ffd50, flags=<optimized out>) at ../../src/env/env_open.c:579 #5 0x00007f6d108f0718 in db_fini (dbhome=0x55ab76903170 "/var/lib/rpm", rdb=0x55ab768ffb50) at backend/db3.c:179 #6 db3_dbiClose (dbi=0x55ab768ff8c0, flags=<optimized out>) at backend/db3.c:756 #7 0x00007f6d108fa1e1 in dbiForeach (del=1, func=<optimized out>, ndbi=<optimized out>, dbis=0x55ab768ffc70) at rpmdb.c:378 #8 rpmdbClose (db=0x55ab768ffb50) at rpmdb.c:411 #9 0x00007f6d108fd16d in rpmdbClose (db=<optimized out>) at rpmdb.c:391 #10 rpmAtExit () at rpmdb.c:327 #11 0x00007f6d106e86c0 in __run_exit_handlers (status=status@entry=1, listp=0x7f6d1086e738 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108 #12 0x00007f6d106e8800 in __GI_exit (status=status@entry=1) at exit.c:139 #13 0x00007f6d102384a7 in uncompressbuf (n=0x7ffeb5626aa8, newch=0x7ffeb5626aa0, old=<optimized out>, method=1, bytes_max=1048576, fd=<optimized out>) at compress.c:728 #14 file_zmagic (ms=ms@entry=0x55ab7691d6a0, b=b@entry=0x7ffeb5626b40, name=name@entry=0x55ab76dea6e0 "/tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/zfs/zfs-tests/tests/functional/history/sparc.migratedpool.DAT.Z") at compress.c:250 #15 0x00007f6d1023f043 in file_buffer (ms=ms@entry=0x55ab7691d6a0, fd=fd@entry=15, inname=inname@entry=0x55ab76dea6e0 "/tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/zfs/zfs-tests/tests/functional/history/sparc.migratedpool.DAT.Z", buf=buf@entry=0x55ab76dfede0, nb=163879) at funcs.c:249 #16 0x00007f6d1022cc7e in file_or_fd (ms=ms@entry=0x55ab7691d6a0, inname=inname@entry=0x55ab76dea6e0 "/tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/zfs/zfs-tests/tests/functional/history/sparc.migratedpool.DAT.Z", fd=15, fd@entry=0) at magic.c:509 #17 0x00007f6d1022d320 in magic_file (ms=ms@entry=0x55ab7691d6a0, inname=inname@entry=0x55ab76dea6e0 "/tmp/zfs-build-root-tMXwDbTy/BUILDROOT/zfs-0.8.0-rc5.fc30.x86_64/usr/share/zfs/zfs-tests/tests/functional/history/sparc.migratedpool.DAT.Z") at magic.c:398 #18 0x00007f6d109752b6 in rpmfcClassify (fc=fc@entry=0x55ab76920050, argv=0x55ab76c4e2f0, fmode=fmode@entry=0x55ab76caa5e0) at rpmfc.c:1104 #19 0x00007f6d10975dcb in rpmfcGenerateDepends (spec=spec@entry=0x55ab76908b00, pkg=pkg@entry=0x55ab769395c0) at rpmfc.c:1426 #20 0x00007f6d10967003 in processBinaryFiles (spec=spec@entry=0x55ab76908b00, pkgFlags=0, didInstall=didInstall@entry=4, test=test@entry=0) at files.c:3114 #21 0x00007f6d1096002c in buildSpec (buildArgs=0x55ab7527c980 <rpmBTArgs>, spec=0x55ab76908b00, what=2975) at build.c:261 #22 0x000055ab7527880d in buildForTarget (ts=<optimized out>, arg=<optimized out>, ba=0x55ab7527c980 <rpmBTArgs>) at rpmbuild.c:517 #23 0x000055ab75278a6b in build (ts=0x55ab768febb0, arg=0x55ab76907cd0 "/tmp/zfs-build-root-tMXwDbTy/SPECS/zfs.spec", rcfile=0x0, ba=0x55ab7527c980 <rpmBTArgs>) at rpmbuild.c:545 #24 0x000055ab75277d31 in main (argc=<optimized out>, argv=<optimized out>) at rpmbuild.c:681
according to the backtrace, this issue seems to caused by missing de-compression tool "uncompress" included in "ncompress" package to decompress .Z file. execvp(3) returned error and forked process died with exit(3), in result, rpmAtExit inherited from a parent process was called. is this situation being assumed in rpmbuild?
Heh, oookkaay. Thanks for the traceback and analysis, that actually does make sense. I had no idea libmagic forks external helpers, but it's no wonder if libdb crashes trying to shut down a database created in another process. It's not exactly an expected usage pattern, but shouldn't be hard to prevent either. Arguably there's a bug in libmagic too, since it's a forked child whose exec() failed it should arguably use _exit() instead of exit() to avoid this kind of thing from happening.
Seems the libmagic side has (mostly) been fixed already just a few days ago: https://github.com/file/file/commit/d360318c69df73b12a7d7011ae278fb6165f1692
On a second thought, rpm cannot really protect itself from arbitrary fork()+exit() from either underlying libraries or API users, reassigning to file which is the troublemaker here. Like noted above, this is at least mostly fixed upstream already, but I don't know about status in Fedora.
dist-git commit: https://src.fedoraproject.org/rpms/file/c/136eea21
FEDORA-2019-83fa1cfd0f has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-83fa1cfd0f
file-5.36-4.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-83fa1cfd0f
I've tested same condition with testing packages. As a result, the rpmdb breakage issue seems to be fixed. Thank you for your response.
Perfect. Thank you for testing it!
file-5.36-4.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.