Bug 132758 (IT_49604)

Summary: RPM database confusion
Product: Red Hat Enterprise Linux 3 Reporter: Brent Fox <bfox>
Component: rpmAssignee: Jeff Johnson <jbj>
Status: CLOSED WORKSFORME QA Contact: Mike McLean <mikem>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.0CC: nobody+pnasrat, peggy.proffitt, tao
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2004-09-22 12:37:34 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Brent Fox 2004-09-16 18:45:37 UTC
Seen on a customer's RHEL3 system which was just upgraded via anaconda
from U2 to U3.  The upgrade appears to have successfully completed.  

After rebooting once the upgrade was finished, some additional
packages were installed.  While installing these packages, the rpmdb
appears to get very confused:

Script addpkgs:

tbed2 [270]$ cat addpkgs
rpm -Uvh anacron-2.3-25.1*.rpm
rpm -Uvh authd-1.4.1-1.rhel3*.rpm
rpm -Uvh diskdumputils-0.4.0-1*.rpm
rpm -Uvh eal3-certification-0.7-1*.rpm
rpm -Uvh eal3-certification-doc-0.7-1*.rpm
rpm -Uvh evolution-connector-1.4.7-5*.rpm
rpm -Uvh finger-server-0.17-18*.rpm
rpm -Uvh gnome-python2-applet-1.99.14-5*.rpm
rpm -Uvh ImageMagick-perl-5.5.6-5.1*.rpm
rpm -Uvh nss_db-2.2-20.4*.rpm
rpm -Uvh openssl096b-0.9.6b-16*.rpm
rpm -Uvh pxe-0.1-36*.rpm
rpm -Uvh samba-swat-3.0.6-2.3E*.rpm
rpm -Uvh up2date-update-4.2.38-1*.rpm



Output of addpkgs:

[root@fdl2 tbed2]# ./addpkgs 
warning: anacron-2.3-25.1.i386.rpm: V3 DSA signature: NOKEY, key ID
db42a60e
Preparing...               
########################################### [100%]
        package anacron-2.3-25.1 is already installed
warning: authd-1.4.1-1.rhel3.i386.rpm: V3 DSA signature: NOKEY, key ID
db42a60e
Preparing...               
########################################### [100%]
        package authd-1.4.1-1.rhel3 is already installed
warning: diskdumputils-0.4.0-1.i386.rpm: V3 DSA signature: NOKEY, key
ID db42a60e
Preparing...               
########################################### [100%]
        package diskdumputils-0.4.0-1 is already installed
warning: eal3-certification-0.7-1.noarch.rpm: V3 DSA signature: NOKEY,
key ID db42a60e
Preparing...               
########################################### [100%]
        package eal3-certification-0.7-1 is already installed
warning: eal3-certification-doc-0.7-1.noarch.rpm: V3 DSA signature:
NOKEY, key ID db42a60e
Preparing...               
########################################### [100%]
        package eal3-certification-doc-0.7-1 is already installed
warning: evolution-connector-1.4.7-5.i386.rpm: V3 DSA signature:
NOKEY, key ID db42a60e
Preparing...               
########################################### [100%]
        package evolution-connector-1.4.7-5 is already installed
warning: finger-server-0.17-18.i386.rpm: V3 DSA signature: NOKEY, key
ID db42a60e
Preparing...               
########################################### [100%]
        package finger-server-0.17-18 is already installed
warning: gnome-python2-applet-1.99.14-5.i386.rpm: V3 DSA signature:
NOKEY, key ID db42a60e
Preparing...               
########################################### [100%]
        package gnome-python2-applet-1.99.14-5 is already installed
warning: ImageMagick-perl-5.5.6-5.1.i386.rpm: V3 DSA signature: NOKEY,
key ID db42a60e
Preparing...               
########################################### [100%]
        package ImageMagick-perl-5.5.6-5.1 is already installed
warning: nss_db-2.2-20.4.i386.rpm: V3 DSA signature: NOKEY, key ID
db42a60e
Preparing...               
########################################### [100%]
        package nss_db-2.2-20.4 is already installed
warning: openssl096b-0.9.6b-16.i386.rpm: V3 DSA signature: NOKEY, key
ID db42a60e
Preparing...               
########################################### [100%]
   1:openssl096b           
########################################### [100%]
rpmdb: page 0: illegal page type or format
rpmdb: PANIC: Invalid argument
rpmdb: fatal region error detected; run recovery
rpmdb: fatal region error detected; run recovery
rpmdb: fatal region error detected; run recovery
error: db4 error(-30982) from dbcursor->c_get: DB_RUNRECOVERY: Fatal
error, run database recovery
error: error(-30982) getting "3.0.4-1" records from Requireversion index
rpmdb: fatal region error detected; run recovery
error: db4 error(-30982) from dbcursor->c_get: DB_RUNRECOVERY: Fatal
error, run database recovery
error: error(-30982) getting "4.0-1" records from Requireversion index
rpmdb: fatal region error detected; run recovery
error: db4 error(-30982) from dbcursor->c_close: DB_RUNRECOVERY: Fatal
error, run database recovery
rpmdb: fatal region error detected; run recovery
error: db4 error(-30982) from db->sync: DB_RUNRECOVERY: Fatal error,
run database recovery
rpmdb: fatal region error detected; run recovery
error: db4 error(-30982) from db->open: DB_RUNRECOVERY: Fatal error,
run database recovery
rpmdb: fatal region error detected; run recovery
error: db4 error(-30982) from db->close: DB_RUNRECOVERY: Fatal error,
run database recovery
error: cannot open Provideversion index using db3 -  (-30982)
rpmdb: fatal region error detected; run recovery
error: db4 error(-30982) from db->open: DB_RUNRECOVERY: Fatal error,
run database recovery
rpmdb: fatal region error detected; run recovery
error: db4 error(-30982) from db->close: DB_RUNRECOVERY: Fatal error,
run database recovery
error: cannot open Installtid index using db3 -  (-30982)
rpmdb: fatal region error detected; run recovery
error: db4 error(-30982) from db->open: DB_RUNRECOVERY: Fatal error,
run database recovery
rpmdb: fatal region error detected; run recovery
error: db4 error(-30982) from db->close: DB_RUNRECOVERY: Fatal error,
run database recovery
error: cannot open Sigmd5 index using db3 -  (-30982)
rpmdb: fatal region error detected; run recovery
error: db4 error(-30982) from db->open: DB_RUNRECOVERY: Fatal error,
run database recovery
rpmdb: fatal region error detected; run recovery
error: db4 error(-30982) from db->close: DB_RUNRECOVERY: Fatal error,
run database recovery
error: cannot open Sha1header index using db3 -  (-30982)
rpmdb: fatal region error detected; run recovery
error: db4 error(-30982) from db->open: DB_RUNRECOVERY: Fatal error,
run database recovery
rpmdb: fatal region error detected; run recovery
error: db4 error(-30982) from db->close: DB_RUNRECOVERY: Fatal error,
run database recovery
error: cannot open Filemd5s index using db3 -  (-30982)
rpmdb: fatal region error detected; run recovery
error: db4 error(-30982) from db->sync: DB_RUNRECOVERY: Fatal error,
run database recovery
rpmdb: fatal region error detected; run recovery
error: db4 error(-30982) from db->sync: DB_RUNRECOVERY: Fatal error,
run database recovery
rpmdb: fatal region error detected; run recovery
error: db4 error(-30982) from db->sync: DB_RUNRECOVERY: Fatal error,
run database recovery
rpmdb: fatal region error detected; run recovery
error: db4 error(-30982) from db->sync: DB_RUNRECOVERY: Fatal error,
run database recovery
rpmdb: fatal region error detected; run recovery

Comment 1 Brent Fox 2004-09-16 18:47:31 UTC
After seeing this problem, the customer ran:

rpm --rebuilddb

which prints:

rpmdb: fatal region error detected; run recovery
error: db4 error(-30982) from dbenv->open: DB_RUNRECOVERY: Fatal
error, run database recovery
error: cannot open Packages index

Comment 2 Brent Fox 2004-09-16 18:50:36 UTC
This looks similar to bug #122565, which the reported closed just
after opening it since the problem disappeared after rebooting.  

We tried rebooting the machine, which allows 'rpm -qa' to work again,
but when the 'addpkgs' script is run again, the same error happens.

Comment 3 Brent Fox 2004-09-16 19:33:26 UTC
We tried rebooting the machine again and immediately ran 'rpm
--rebuilddb'  This worked, and then the 'addpkgs' script ran successfully.

So the question is, can we figure out how the problem happened in the
first place?  

The Issue Tracker ticket has a sysreport attachment if that is useful.

Comment 4 Jeff Johnson 2004-09-17 03:48:45 UTC
Rebooting does
    rm -f /var/lib/rpm/__db*
which probably would have fixed the problem without doing
--rebuilddb.

Could you grab a copy of the the database please
    cd /var/lib
    tar czvf /tmp/rpmdb-132758.tar.gz ./rpm
so that I can attempt to reproduce the problem?

A pointer (i.e. URL), rather than an attachment will be needed,
as bugzilla won't handle rpmdb size attachments.

Comment 5 Brent Fox 2004-09-17 16:15:47 UTC
Sure, I will ask the customer for a copy of the rpmdb.  I hope it's
still useful to you after running --rebuilddb.

Comment 6 Brent Fox 2004-09-20 17:53:11 UTC
jbj: I've uploaded the rpmdb to the Issue Tracker ticket and
ftp://people.redhat.com/bfox/.jsc-ips/rpmdb-132758.tar.gz  

Please let me know if you need any more information.

Comment 7 Jeff Johnson 2004-09-22 12:37:34 UTC
I've downloaded the rpmdb, checked that all SHA1 digests
verify, and looked for header data loss by attempting
rpm -Va --nofiles. That verifies that the rpmdb is
intact and useful, about all that can be done after --rebuilddb.