Created attachment 1286216 [details] script to prepare the system for upgrade Upgrade from f25 to f26 gets stuck when latest libdb is used. This is log from my upgrade on ppc64le with libdb-5.3.28-21.fc25.ppc64le installed ... D: ========== +++ crypto-policies-20160921-4.gitf3018dd.fc25 noarch-linux 0x0 D: erase: crypto-policies-20160921-4.gitf3018dd.fc25.noarch has 46 files D: Plugin: calling hook psm_pre in selinux plugin D: skip 100644 1 ( 0, 0) 2856 /usr/share/man/man8/update-crypto-policies.8.gz D: skip 100644 1 ( 0, 0) 26432 /usr/share/licenses/crypto-policies/COPYING.LESSER D: skip 040755 2 ( 0, 0) 4096 /usr/share/licenses/crypto-policies D: skip 100644 1 ( 0, 0) 70 /usr/share/crypto-policies/reload-cmds.sh D: skip 100644 1 ( 0, 0) 606 /usr/share/crypto-policies/default-config D: skip 100644 1 ( 0, 0) 87 /usr/share/crypto-policies/LEGACY/openssl.txt D: skip 100644 1 ( 0, 0) 431 /usr/share/crypto-policies/LEGACY/nss.txt D: skip 100644 1 ( 0, 0) 142 /usr/share/crypto-policies/LEGACY/krb5.txt D: skip 100644 1 ( 0, 0) 312 /usr/share/crypto-policies/LEGACY/java.txt D: skip 100644 1 ( 0, 0) 438 /usr/share/crypto-policies/LEGACY/gnutls28.txt D: skip 100644 1 ( 0, 0) 496 /usr/share/crypto-policies/LEGACY/gnutls.txt D: skip 100644 1 ( 0, 0) 64 /usr/share/crypto-policies/LEGACY/bind.txt D: skip 100644 1 ( 0, 0) 103 /usr/share/crypto-policies/FUTURE/openssl.txt D: skip 100644 1 ( 0, 0) 400 /usr/share/crypto-policies/FUTURE/nss.txt D: skip 100644 1 ( 0, 0) 111 /usr/share/crypto-policies/FUTURE/krb5.txt D: skip 100644 1 ( 0, 0) 386 /usr/share/crypto-policies/FUTURE/java.txt D: skip 100644 1 ( 0, 0) 420 /usr/share/crypto-policies/FUTURE/gnutls28.txt D: skip 100644 1 ( 0, 0) 465 /usr/share/crypto-policies/FUTURE/gnutls.txt D: skip 100644 1 ( 0, 0) 105 /usr/share/crypto-policies/FUTURE/bind.txt D: skip 100644 1 ( 0, 0) 160 /usr/share/crypto-policies/EMPTY/openssl.txt D: skip 100644 1 ( 0, 0) 145 /usr/share/crypto-policies/EMPTY/nss.txt D: skip 100644 1 ( 0, 0) 22 /usr/share/crypto-policies/EMPTY/krb5.txt D: skip 100644 1 ( 0, 0) 594 /usr/share/crypto-policies/EMPTY/java.txt D: skip 100644 1 ( 0, 0) 36 /usr/share/crypto-policies/EMPTY/gnutls28.txt D: skip 100644 1 ( 0, 0) 36 /usr/share/crypto-policies/EMPTY/gnutls.txt D: skip 100644 1 ( 0, 0) 123 /usr/share/crypto-policies/EMPTY/bind.txt D: skip 100644 1 ( 0, 0) 97 /usr/share/crypto-policies/DEFAULT/openssl.txt D: skip 100644 1 ( 0, 0) 418 /usr/share/crypto-policies/DEFAULT/nss.txt D: skip 100644 1 ( 0, 0) 125 /usr/share/crypto-policies/DEFAULT/krb5.txt D: skip 100644 1 ( 0, 0) 345 /usr/share/crypto-policies/DEFAULT/java.txt D: skip 100644 1 ( 0, 0) 420 /usr/share/crypto-policies/DEFAULT/gnutls28.txt D: skip 100644 1 ( 0, 0) 465 /usr/share/crypto-policies/DEFAULT/gnutls.txt D: skip 100644 1 ( 0, 0) 70 /usr/share/crypto-policies/DEFAULT/bind.txt D: skip 040755 6 ( 0, 0) 4096 /usr/share/crypto-policies D: skip 100755 1 ( 0, 0) 2362 /usr/bin/update-crypto-policies D: skip 040755 2 ( 0, 0) 4096 /etc/crypto-policies/local.d D: skip 100644 1 ( 0, 0) 606 /etc/crypto-policies/config D: skip 120777 1 ( 0, 0) 46 /etc/crypto-policies/back-ends/openssl.config D: skip 120777 1 ( 0, 0) 42 /etc/crypto-policies/back-ends/nss.config D: skip 120777 1 ( 0, 0) 43 /etc/crypto-policies/back-ends/krb5.config D: skip 120777 1 ( 0, 0) 43 /etc/crypto-policies/back-ends/java.config D: skip 120777 1 ( 0, 0) 47 /etc/crypto-policies/back-ends/gnutls28.config D: skip 120777 1 ( 0, 0) 45 /etc/crypto-policies/back-ends/gnutls.config D: skip 120777 1 ( 0, 0) 43 /etc/crypto-policies/back-ends/bind.config D: skip 040755 2 ( 0, 0) 4096 /etc/crypto-policies/back-ends D: skip 040755 5 ( 0, 0) 4096 /etc/crypto-policies fdio: 2 writes, 759 total bytes in 0.000020 secs D: --- h# 186 crypto-policies-20160921-4.gitf3018dd.fc25.noarch D: adding "crypto-policies" to Name index. D: adding 46 entries to Basenames index. D: adding "Unspecified" to Group index. D: adding 8 entries to Requirename index. D: adding 2 entries to Providename index. D: adding 14 entries to Dirnames index. D: adding 1 entries to Installtid index. D: adding 1 entries to Sigmd5 index. D: adding "9f33e91a20fa838c5e85706350d2b9c7e5d4ac2c" to Sha1header index. D: ========== +++ krb5-libs-1.14.4-7.fc25 ppc64le-linux 0x2 D: erase: krb5-libs-1.14.4-7.fc25.ppc64le has 41 files D: Plugin: calling hook psm_pre in selinux plugin D: %triggerun(krb5-libs-1.15.1-8.fc26.ppc64le): scriptlet start D: %triggerun(krb5-libs-1.15.1-8.fc26.ppc64le): execv(/bin/sh) pid 3867 D: Plugin: calling hook scriptlet_fork_post in selinux plugin D: setexecfilecon: (/bin/sh) ++ rpm -q --qf '%{VERSION}\n' krb5-libs ++ sort -V ++ tr -d '\n' ++ head -n 1 + old_ver=1.14.4 ++ rpm -q --qf '%{RELEASE}\n' krb5-libs ++ sort -V ++ head -n 1 ++ tr -d '\n' + old_rel=7.fc25 + old_rel=7 + [[ 1.14.4 < 1.15.1 ]] + grep -q 'includedir /etc/krb5.conf.d' /etc/krb5.conf + exit 0 D: %triggerun(krb5-libs-1.15.1-8.fc26.ppc64le): waitpid(3867) rc 3867 status 0 Cleanup : krb5-libs-1.14.4-7.fc25.ppc64le 613/706 D: skip 040755 2 ( 0, 0) 4096 /var/kerberos/krb5/user D: skip 040755 3 ( 0, 0) 4096 /var/kerberos/krb5 D: skip 040755 3 ( 0, 0) 4096 /var/kerberos D: skip 100644 1 ( 0, 0) 13970 /usr/share/man/man5/krb5.conf.5.gz D: skip 100644 1 ( 0, 0) 1176 /usr/share/man/man5/k5login.5.gz D: skip 100644 1 ( 0, 0) 1207 /usr/share/man/man5/k5identity.5.gz D: skip 100644 1 ( 0, 0) 39 /usr/share/man/man5/.k5login.5.gz D: skip 100644 1 ( 0, 0) 42 /usr/share/man/man5/.k5identity.5.gz D: skip 100644 1 ( 0, 0) 410 /usr/share/locale/en_US/LC_MESSAGES/mit-krb5.mo D: skip 100644 1 ( 0, 0) 61301 /usr/share/licenses/krb5-libs/LICENSE D: skip 040755 2 ( 0, 0) 4096 /usr/share/licenses/krb5-libs D: skip 100644 1 ( 0, 0) 15120 /usr/share/doc/krb5-libs/README D: skip 100644 1 ( 0, 0) 61301 /usr/share/doc/krb5-libs/NOTICE D: skip 040755 2 ( 0, 0) 4096 /usr/share/doc/krb5-libs D: skip 100755 1 ( 0, 0)135408 /usr/lib64/libkrb5support.so.0.1 D: skip 120777 1 ( 0, 0) 21 /usr/lib64/libkrb5support.so.0 D: skip 100755 1 ( 0, 0)1199384 /usr/lib64/libkrb5.so.3.3 D: skip 120777 1 ( 0, 0) 14 /usr/lib64/libkrb5.so.3 D: skip 100755 1 ( 0, 0) 68576 /usr/lib64/libkrad.so.0.0 D: skip 120777 1 ( 0, 0) 14 /usr/lib64/libkrad.so.0 D: skip 100755 1 ( 0, 0)135224 /usr/lib64/libkdb5.so.8.0 D: skip 120777 1 ( 0, 0) 14 /usr/lib64/libkdb5.so.8 D: skip 100755 1 ( 0, 0)267296 /usr/lib64/libk5crypto.so.3.1 D: skip 120777 1 ( 0, 0) 18 /usr/lib64/libk5crypto.so.3 D: skip 100755 1 ( 0, 0)202424 /usr/lib64/libgssrpc.so.4.2 D: skip 120777 1 ( 0, 0) 16 /usr/lib64/libgssrpc.so.4 D: skip 100755 1 ( 0, 0)402960 /usr/lib64/libgssapi_krb5.so.2.2 D: skip 120777 1 ( 0, 0) 21 /usr/lib64/libgssapi_krb5.so.2 D: skip 100755 1 ( 0, 0) 68584 /usr/lib64/krb5/plugins/tls/k5tls.so D: skip 040755 2 ( 0, 0) 4096 /usr/lib64/krb5/plugins/tls D: skip 040755 2 ( 0, 0) 4096 /usr/lib64/krb5/plugins/preauth D: skip 040755 2 ( 0, 0) 4096 /usr/lib64/krb5/plugins/libkrb5 D: skip 040755 2 ( 0, 0) 4096 /usr/lib64/krb5/plugins/kdb D: skip 040755 2 ( 0, 0) 4096 /usr/lib64/krb5/plugins/authdata D: skip 040755 7 ( 0, 0) 4096 /usr/lib64/krb5/plugins D: skip 040755 3 ( 0, 0) 4096 /usr/lib64/krb5 D: skip 120777 1 ( 0, 0) 42 /etc/krb5.conf.d/crypto-policies D: skip 040755 2 ( 0, 0) 4096 /etc/krb5.conf.d D: skip 100644 1 ( 0, 0) 677 /etc/krb5.conf D: skip 040755 2 ( 0, 0) 4096 /etc/gss/mech.d D: skip 040755 3 ( 0, 0) 4096 /etc/gss D: %postun(krb5-libs-1.14.4-7.fc25.ppc64le): scriptlet start D: %postun(krb5-libs-1.14.4-7.fc25.ppc64le): execv(/sbin/ldconfig) pid 3879 D: Plugin: calling hook scriptlet_fork_post in selinux plugin D: setexecfilecon: (/sbin/ldconfig) D: %postun(krb5-libs-1.14.4-7.fc25.ppc64le): waitpid(3879) rc 3879 status 0 Version-Release number of selected component (if applicable): libdb-5.3.28-21.fc25 libdb-5.3.28-21.fc26 How reproducible: 100% Steps to Reproduce: - installed F-25 "minimal" from "netinstall" iso (so system is GA + updates) - updated libdb to libdb-5.3.28-21.fc25.ppc64le (downloaded from ppc koji) - created a side repo for F-26 libdb containing libdb-5.3.28-21.fc26.ppc64le (downloaded from primary koji) - run "dnf distro-sync --releasever 26" with the side repo enabled - and got a hung after ... Cleanup : crypto-policies-20160921-4.gitf3018dd.fc25.noarch 614/708 Cleanup : krb5-libs-1.14.4-7.fc25.ppc64le 615/708 Actual results: upgrade is stuck Expected results: upgrade is finished Additional info: We definitely see the problem on ppc64le, but we will check ppc64 (and others) too.
Thanks Dan for providing the log! Since the upgrade process fails not long after touching the rpmdb via krb5-libs it would seem the environment got rebuilt at that point even with all checks in place via -21.fc25 (check for libdb's lock and check for rpm's transaction log)
Keep in mind it works successfully when I upgrade without krb5-libs by adding the option: exclude=krb5-libs see description of https://bugzilla.redhat.com/show_bug.cgi?id=1443415#c0
And I got a hung on ppc64 as well, but on a different place (much earlier) ... D: %triggerpostun(systemd-231-15.fc25.ppc64): running <lua> scriptlet. D: %triggerpostun(systemd-233-3.fc26.ppc64): running <lua> scriptlet. fdio: 2 writes, 162 total bytes in 0.000018 secs D: %postun(ebtables-2.0.10-21.fc25.ppc64): scriptlet start D: %postun(ebtables-2.0.10-21.fc25.ppc64): execv(/bin/sh) pid 4240 D: Plugin: calling hook scriptlet_fork_post in selinux plugin D: setexecfilecon: (/bin/sh) + '[' 1 -ge 1 ']' + systemctl try-restart ebtables.service + /sbin/ldconfig D: %postun(ebtables-2.0.10-21.fc25.ppc64): waitpid(4240) rc 4240 status 0 Cleanup : e2fsprogs-libs-1.43.3-1.fc25.ppc64 517/730 D: --- h# 256 ebtables-2.0.10-21.fc25.ppc64 D: adding "ebtables" to Name index. D: adding 38 entries to Basenames index. D: adding "System Environment/Base" to Group index. D: adding 40 entries to Requirename index. D: adding 26 entries to Providename index. D: adding 1 entries to Conflictname index. D: adding 9 entries to Dirnames index. D: adding 1 entries to Installtid index. D: adding 1 entries to Sigmd5 index. D: adding "a43cf5611f45537556cc133fe32f99b6c945057a" to Sha1header index. D: ========== +++ e2fsprogs-libs-1.43.3-1.fc25 ppc64-linux 0x2 D: erase: e2fsprogs-libs-1.43.3-1.fc25.ppc64 has 4 files D: Plugin: calling hook psm_pre in selinux plugin D: skip 100755 1 ( 0, 0)532864 /usr/lib64/libext2fs.so.2.4 D: skip 120777 1 ( 0, 0) 16 /usr/lib64/libext2fs.so.2 D: skip 100755 1 ( 0, 0) 70320 /usr/lib64/libe2p.so.2.3 D: skip 120777 1 ( 0, 0) 13 /usr/lib64/libe2p.so.2 D: %postun(e2fsprogs-libs-1.43.3-1.fc25.ppc64): scriptlet start D: %postun(e2fsprogs-libs-1.43.3-1.fc25.ppc64): execv(/sbin/ldconfig) pid 4246 D: Plugin: calling hook scriptlet_fork_post in selinux plugin D: setexecfilecon: (/sbin/ldconfig) D: %postun(e2fsprogs-libs-1.43.3-1.fc25.ppc64): waitpid(4246) rc 4246 status 0
I think I have found the root of the problem. In the case the libdb -21 update would be installed before the upgrade to F26 takes place and the environment gets recreated with new libdb internal structures (through another rpm/dnf command call), scriplets run during the upgrade could still access the rpmdb even though glibc's pthread internals were changed... I have modified libdb's checks to be more restrictive and return a failure when a possible pthread version mismatch is encountered and libdb cannot safely rebuild its environment. Dan, Guy, could you try reproducing the issue with this build? https://koji.fedoraproject.org/koji/taskinfo?taskID=19926708
My ppc64 test looks good now, it went over the e2fsprogs-libs-1.43.3-1.fc25.ppc64 cleanup and issues the expecetd warnings at krb5-libs ... Cleanup : libmount-2.28.2-2.fc25.ppc64 610/706 Cleanup : libblkid-2.28.2-2.fc25.ppc64 611/706 Cleanup : crypto-policies-20160921-4.gitf3018dd.fc25.noarch 612/706 error: db5 error(-30969) from dbenv->open: BDB0091 DB_VERSION_MISMATCH: Database environment version mismatch error: cannot open Packages index using db5 - (-30969) error: cannot open Packages database in /var/lib/rpm error: db5 error(-30969) from dbenv->open: BDB0091 DB_VERSION_MISMATCH: Database environment version mismatch error: cannot open Packages index using db5 - (-30969) error: cannot open Packages database in /var/lib/rpm error: db5 error(-30969) from dbenv->open: BDB0091 DB_VERSION_MISMATCH: Database environment version mismatch error: cannot open Packages index using db5 - (-30969) error: cannot open Packages database in /var/lib/rpm error: db5 error(-30969) from dbenv->open: BDB0091 DB_VERSION_MISMATCH: Database environment version mismatch error: cannot open Packages index using db5 - (-30969) error: cannot open Packages database in /var/lib/rpm Cleanup : krb5-libs-1.14.4-7.fc25.ppc64 613/706 Cleanup : coreutils-8.25-17.fc25.ppc64 614/706 ...
If I update my f25 ppc64le VM with your libdb -22 before the upgrade to f26, everything works well. "dnf -y --releasever=26 system-upgrade download" and "dnf system-upgrade reboot" are ok and the system is upgraded successfully. Thanks
Petr: could this issue also be the cause of the RPM database corruption that people are now reporting sometimes happens after updating to -21 on F24/F25? See https://bugzilla.redhat.com/show_bug.cgi?id=1397087 discussion...
No, I do not think so. This should only be caused by updating to glibc (with new internal structures) after a separate transaction updating libdb to -21 is already completed. I have yet to look into what happened with the F24/25 updates but afaics there should not be any reason to drop the current patches since the issue happens after the update goes through and is easily fixable with "rpmdb --rebuilddb" ...
I would've liked to discuss it with you, but it only became clear there were still issues on Friday evening (your time) and it didn't seem practical to do nothing until today. For a stable release of an operating system, "oh don't worry that you can't install anything and your system is printing terrifying errors, you can just run this cryptic command as root at a console to fix it" really isn't a good look. Given that we had to make a call with the people who were around at the time, trying to prevent this happening to more people seemed like the best idea. Fixing upgrades to a pre-release seems substantially less important than not causing highly alarming bugs for people running stable releases.
Fixing this still needs some more work. Plain out failing when a possible pthread version mismatch is encountered (which is based on comparing the modify timestamp of libpthread.so file with what is saved in libdb's environment) makes the dnf update process fail in the "verify" step even though it should be able to access the environment as it still has the old version of libpthread loaded.
libdb-5.3.28-24.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-2b68e14594
libdb-5.3.28-24.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-014d67fa9d
libdb-5.3.28-24.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-372bb1edb3
libdb-5.3.28-24.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-2b68e14594
The fixed version on F24 ppc64le is not available, the version available in updates-testing is libdb-5.3.28-21.fc24.ppc64le.
(In reply to Jakub Vavra from comment #15) > The fixed version on F24 ppc64le is not available, the version available in > updates-testing is libdb-5.3.28-21.fc24.ppc64le. I'm building f25+f24 updates-testing in ppc koji now, should be available soon for download
With the packages libdb-5.3.28-24.fc24.ppc64le.rpm and libdb-utils-5.3.28-24.fc24.ppc64le.rpm built and provided by pkubat the upgrade worked fine.
Upgrade from f25->f26 completed on armhfp with libdb-5.3.28-24.
with f26 compose 20170701 and fc24: 4.11.6-101.fc24.ppc64le or fc25: 4.11.6-201.fc25.ppc64le still hang in cleanup phase.
to complete my previous comment #19, the libdb versions used when failure is: === $grep -HnE libdb ~/Desktop/upgrade_run-dnf.rpm.log /home/normand/Desktop/upgrade_run-dnf.rpm.log:14:Jul 02 01:37:44 INFO Upgraded: libdb-5.3.28-21.fc24.ppc64le /home/normand/Desktop/upgrade_run-dnf.rpm.log:43:Jul 02 01:38:17 INFO Upgraded: libdb-utils-5.3.28-21.fc24.ppc64le /home/normand/Desktop/upgrade_run-dnf.rpm.log:66:Jul 02 01:38:26 INFO Cleanup: libdb-utils-5.3.28-14.fc24.ppc64le /home/normand/Desktop/upgrade_run-dnf.rpm.log:78:Jul 02 01:38:26 INFO Cleanup: libdb-5.3.28-14.fc24.ppc64le /home/normand/Desktop/upgrade_run-dnf.rpm.log:140:Jul 02 01:42:13 INFO Upgraded: libdb-5.3.28-21.fc26.ppc64le /home/normand/Desktop/upgrade_run-dnf.rpm.log:248:Jul 02 01:42:13 INFO Upgraded: libdb-utils-5.3.28-21.fc26.ppc64le ===
libdb-5.3.28-24.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
last tested ppc64/ppc64le compose 20170707 still hangs while cleanup similar as previous comment #19. And the grep of libdb is similar as comment #20 with older version versus expected libdb-5.3.28-24 of comment #21
(In reply to Michel Normand from comment #22) > last tested ppc64/ppc64le compose 20170707 still hangs while cleanup > similar as previous comment #19. > And the grep of libdb is similar as comment #20 with older version > versus expected libdb-5.3.28-24 of comment #21 I am not sure if I understand the comment correctly but this issue is fixed in libdb-5.3.28-24. If you are trying to update to libdb-5.3.28-21 instead (as per comment 20) then yes the update will most likely hang. As for the updates for F24/25, I have submitted them to stable earlier today so they should be available soon.
libdb-5.3.28-24.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.
libdb-5.3.28-24.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.
The mirrors for f24 are not yet in sync today, will have to wait more time. (same pb for f25) === [michel@fc24ppc64le ~]$ dnf list upgrade libdb Last metadata expiration check: 0:00:13 ago on Wed Jul 12 07:42:16 2017. Installed Packages libdb.ppc64le 5.3.28-14.fc24 @anaconda Available Packages libdb.ppc64le 5.3.28-21.fc24 updates ===