Seen on a customer's RHEL3 system which was just upgraded via anaconda from U2 to U3. The upgrade appears to have successfully completed. After rebooting once the upgrade was finished, some additional packages were installed. While installing these packages, the rpmdb appears to get very confused: Script addpkgs: tbed2 [270]$ cat addpkgs rpm -Uvh anacron-2.3-25.1*.rpm rpm -Uvh authd-1.4.1-1.rhel3*.rpm rpm -Uvh diskdumputils-0.4.0-1*.rpm rpm -Uvh eal3-certification-0.7-1*.rpm rpm -Uvh eal3-certification-doc-0.7-1*.rpm rpm -Uvh evolution-connector-1.4.7-5*.rpm rpm -Uvh finger-server-0.17-18*.rpm rpm -Uvh gnome-python2-applet-1.99.14-5*.rpm rpm -Uvh ImageMagick-perl-5.5.6-5.1*.rpm rpm -Uvh nss_db-2.2-20.4*.rpm rpm -Uvh openssl096b-0.9.6b-16*.rpm rpm -Uvh pxe-0.1-36*.rpm rpm -Uvh samba-swat-3.0.6-2.3E*.rpm rpm -Uvh up2date-update-4.2.38-1*.rpm Output of addpkgs: [root@fdl2 tbed2]# ./addpkgs warning: anacron-2.3-25.1.i386.rpm: V3 DSA signature: NOKEY, key ID db42a60e Preparing... ########################################### [100%] package anacron-2.3-25.1 is already installed warning: authd-1.4.1-1.rhel3.i386.rpm: V3 DSA signature: NOKEY, key ID db42a60e Preparing... ########################################### [100%] package authd-1.4.1-1.rhel3 is already installed warning: diskdumputils-0.4.0-1.i386.rpm: V3 DSA signature: NOKEY, key ID db42a60e Preparing... ########################################### [100%] package diskdumputils-0.4.0-1 is already installed warning: eal3-certification-0.7-1.noarch.rpm: V3 DSA signature: NOKEY, key ID db42a60e Preparing... ########################################### [100%] package eal3-certification-0.7-1 is already installed warning: eal3-certification-doc-0.7-1.noarch.rpm: V3 DSA signature: NOKEY, key ID db42a60e Preparing... ########################################### [100%] package eal3-certification-doc-0.7-1 is already installed warning: evolution-connector-1.4.7-5.i386.rpm: V3 DSA signature: NOKEY, key ID db42a60e Preparing... ########################################### [100%] package evolution-connector-1.4.7-5 is already installed warning: finger-server-0.17-18.i386.rpm: V3 DSA signature: NOKEY, key ID db42a60e Preparing... ########################################### [100%] package finger-server-0.17-18 is already installed warning: gnome-python2-applet-1.99.14-5.i386.rpm: V3 DSA signature: NOKEY, key ID db42a60e Preparing... ########################################### [100%] package gnome-python2-applet-1.99.14-5 is already installed warning: ImageMagick-perl-5.5.6-5.1.i386.rpm: V3 DSA signature: NOKEY, key ID db42a60e Preparing... ########################################### [100%] package ImageMagick-perl-5.5.6-5.1 is already installed warning: nss_db-2.2-20.4.i386.rpm: V3 DSA signature: NOKEY, key ID db42a60e Preparing... ########################################### [100%] package nss_db-2.2-20.4 is already installed warning: openssl096b-0.9.6b-16.i386.rpm: V3 DSA signature: NOKEY, key ID db42a60e Preparing... ########################################### [100%] 1:openssl096b ########################################### [100%] rpmdb: page 0: illegal page type or format rpmdb: PANIC: Invalid argument rpmdb: fatal region error detected; run recovery rpmdb: fatal region error detected; run recovery rpmdb: fatal region error detected; run recovery error: db4 error(-30982) from dbcursor->c_get: DB_RUNRECOVERY: Fatal error, run database recovery error: error(-30982) getting "3.0.4-1" records from Requireversion index rpmdb: fatal region error detected; run recovery error: db4 error(-30982) from dbcursor->c_get: DB_RUNRECOVERY: Fatal error, run database recovery error: error(-30982) getting "4.0-1" records from Requireversion index rpmdb: fatal region error detected; run recovery error: db4 error(-30982) from dbcursor->c_close: DB_RUNRECOVERY: Fatal error, run database recovery rpmdb: fatal region error detected; run recovery error: db4 error(-30982) from db->sync: DB_RUNRECOVERY: Fatal error, run database recovery rpmdb: fatal region error detected; run recovery error: db4 error(-30982) from db->open: DB_RUNRECOVERY: Fatal error, run database recovery rpmdb: fatal region error detected; run recovery error: db4 error(-30982) from db->close: DB_RUNRECOVERY: Fatal error, run database recovery error: cannot open Provideversion index using db3 - (-30982) rpmdb: fatal region error detected; run recovery error: db4 error(-30982) from db->open: DB_RUNRECOVERY: Fatal error, run database recovery rpmdb: fatal region error detected; run recovery error: db4 error(-30982) from db->close: DB_RUNRECOVERY: Fatal error, run database recovery error: cannot open Installtid index using db3 - (-30982) rpmdb: fatal region error detected; run recovery error: db4 error(-30982) from db->open: DB_RUNRECOVERY: Fatal error, run database recovery rpmdb: fatal region error detected; run recovery error: db4 error(-30982) from db->close: DB_RUNRECOVERY: Fatal error, run database recovery error: cannot open Sigmd5 index using db3 - (-30982) rpmdb: fatal region error detected; run recovery error: db4 error(-30982) from db->open: DB_RUNRECOVERY: Fatal error, run database recovery rpmdb: fatal region error detected; run recovery error: db4 error(-30982) from db->close: DB_RUNRECOVERY: Fatal error, run database recovery error: cannot open Sha1header index using db3 - (-30982) rpmdb: fatal region error detected; run recovery error: db4 error(-30982) from db->open: DB_RUNRECOVERY: Fatal error, run database recovery rpmdb: fatal region error detected; run recovery error: db4 error(-30982) from db->close: DB_RUNRECOVERY: Fatal error, run database recovery error: cannot open Filemd5s index using db3 - (-30982) rpmdb: fatal region error detected; run recovery error: db4 error(-30982) from db->sync: DB_RUNRECOVERY: Fatal error, run database recovery rpmdb: fatal region error detected; run recovery error: db4 error(-30982) from db->sync: DB_RUNRECOVERY: Fatal error, run database recovery rpmdb: fatal region error detected; run recovery error: db4 error(-30982) from db->sync: DB_RUNRECOVERY: Fatal error, run database recovery rpmdb: fatal region error detected; run recovery error: db4 error(-30982) from db->sync: DB_RUNRECOVERY: Fatal error, run database recovery rpmdb: fatal region error detected; run recovery
After seeing this problem, the customer ran: rpm --rebuilddb which prints: rpmdb: fatal region error detected; run recovery error: db4 error(-30982) from dbenv->open: DB_RUNRECOVERY: Fatal error, run database recovery error: cannot open Packages index
This looks similar to bug #122565, which the reported closed just after opening it since the problem disappeared after rebooting. We tried rebooting the machine, which allows 'rpm -qa' to work again, but when the 'addpkgs' script is run again, the same error happens.
We tried rebooting the machine again and immediately ran 'rpm --rebuilddb' This worked, and then the 'addpkgs' script ran successfully. So the question is, can we figure out how the problem happened in the first place? The Issue Tracker ticket has a sysreport attachment if that is useful.
Rebooting does rm -f /var/lib/rpm/__db* which probably would have fixed the problem without doing --rebuilddb. Could you grab a copy of the the database please cd /var/lib tar czvf /tmp/rpmdb-132758.tar.gz ./rpm so that I can attempt to reproduce the problem? A pointer (i.e. URL), rather than an attachment will be needed, as bugzilla won't handle rpmdb size attachments.
Sure, I will ask the customer for a copy of the rpmdb. I hope it's still useful to you after running --rebuilddb.
jbj: I've uploaded the rpmdb to the Issue Tracker ticket and ftp://people.redhat.com/bfox/.jsc-ips/rpmdb-132758.tar.gz Please let me know if you need any more information.
I've downloaded the rpmdb, checked that all SHA1 digests verify, and looked for header data loss by attempting rpm -Va --nofiles. That verifies that the rpmdb is intact and useful, about all that can be done after --rebuilddb.