Bug 90560 - RPM database error: How to recover safely and fully
Summary: RPM database error: How to recover safely and fully
Alias: None
Product: Red Hat Linux
Classification: Retired
Component: rpm   
(Show other bugs)
Version: 9
Hardware: All Linux
Target Milestone: ---
Assignee: Jeff Johnson
QA Contact: Mike McLean
: 90589 (view as bug list)
Depends On:
TreeView+ depends on / blocked
Reported: 2003-05-09 16:59 UTC by J.M. Arago
Modified: 2007-04-18 16:53 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2003-05-14 16:39:30 UTC
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

Description J.M. Arago 2003-05-09 16:59:53 UTC
Description of problem:
   How to Repair an RPM database safely after:
   rpmdb: fatal region error detected; run recovery
   error: db4 error(-30982): Fatal error, run database recovery

Version-Release number of selected component (if applicable):
   RPM version 4.2

How reproducible:
   Unknown in my /var/lib/rpm/Packages database (Fatal error)
   Yes in rpmdb-redhat-9-0.20030313

Steps to Reproduce:
1. chdir /usr/lib/rpmdb/i386-redhat-linux/redhat
2. db_verify Packages
Actual results:
   db_verify: region error detected; run recovery.

Expected results:
   No errors, database ok

Additional info:
   chdir /var/lib/rpm ; rpm --rebuilddb  --> trimmed to 75% my Packages file
   (see attached text file with more info)

Comment 1 Jeff Johnson 2003-05-09 23:13:42 UTC
You appear to have two problems.

Before doing anything else, save a copy of /var/lib/rpm
    cd /var/lib
    tar czvf rpmdb.tar.gz rpm

First eliminate stale locks by doing
    rm -f /var/lib/rpm/__db*

What does
    /usr/lib/rpm/rpmdb_verify /var/lib/rpm/Packages
say? If rpmdb_verify gives a recovery needed (or any other) error, then
the fix is to dump and reload the file by doing:
    cd /var/lib/rpm
    mv Packages Packages-ORIG
    /usr/lib/rpm/rpmdb_dump Packages-ORIG | /usr/lib/rpm/rpmdb_load Packages

After Packages passes rpmdb_verify, then a rpm --rebuilddb will
regenerate the indices.

DOes that fix?

Comment 2 Jeff Johnson 2003-05-09 23:15:17 UTC
*** Bug 90589 has been marked as a duplicate of this bug. ***

Comment 3 J.M. Arago 2003-05-10 11:01:41 UTC
It worked, but with my trimmed database (reduced Packages file after "rpm
--rebuild" I did yesterday).

But it works now with "/usr/lib/rpm/rpmdb_verify /var/lib/rpm/Packages", as well
as "/usr/lib/rpm/rpmdb_verify usr/lib/rpmdb/i386-redhat-linux/redhat/Packages"
(both without errors or warnings).

Yesterday I was doing "db_verify" and "db_dump", "db_load", in fact using the
old versions (in the path):
-r-xr-xr-x  1 root  root      5548 feb  5 22:09 /usr/bin/db_archive
-r-xr-xr-x  1 root  root      7092 feb  5 22:09 /usr/bin/db_checkpoint
-r-xr-xr-x  1 root  root      6768 feb  5 22:09 /usr/bin/db_deadlock
-r-xr-xr-x  1 root  root     10008 feb  5 22:09 /usr/bin/db_dump
-r-xr-xr-x  1 root  root     51864 feb  5 22:09 /usr/bin/db_dump185
-r-xr-xr-x  1 root  root     15924 feb  5 22:09 /usr/bin/db_load
-r-xr-xr-x  1 root  root      6284 feb  5 22:09 /usr/bin/db_printlog
-r-xr-xr-x  1 root  root      6564 feb  5 22:09 /usr/bin/db_recover
-r-xr-xr-x  1 root  root     19140 feb  5 22:09 /usr/bin/db_stat
-r-xr-xr-x  1 root  root      5548 feb  5 22:09 /usr/bin/db_upgrade
-r-xr-xr-x  1 root  root      5704 feb  5 22:09 /usr/bin/db_verify
# /usr/bin/db_verify -V
Sleepycat Software: Berkeley DB 4.0.14: (November 18, 2001)

Instead of the new ones (not in path):
-rwxr-xr-x  1 rpm   rpm   6948 feb 27 22:24 /usr/lib/rpm/rpmdb_deadlock
-rwxr-xr-x  1 rpm   rpm  11156 feb 27 22:24 /usr/lib/rpm/rpmdb_dump
-rwxr-xr-x  1 rpm   rpm  18032 feb 27 22:24 /usr/lib/rpm/rpmdb_load
-rwxr-xr-x  1 rpm   rpm  24748 feb 27 22:24 /usr/lib/rpm/rpmdb_stat
-rwxr-xr-x  1 rpm   rpm  39652 feb 27 22:24 /usr/lib/rpm/rpmdb_svc
-rwxr-xr-x  1 rpm   rpm   7248 feb 27 22:24 /usr/lib/rpm/rpmdb_verify
# /usr/lib/rpm/rpmdb_verify -V
Sleepycat Software: Berkeley DB 4.1.24: (September 13, 2002)

It seems to me that the upgrade to RPM-4.2 in RH-9 does not erase the old
"/usr/bin/db_*" binaries of DB-4.0.14, neither installs the new binaries
"/usr/lib/rpm/rpmdb_*" of DB-4.1.24 in the user/root path.

I don't know if leaving the old "/usr/bin/db_*" binaries is harmful, as far as I
don't use them, but use the new "/usr/lib/rpm/rpmdb_*" ones.

On other hand, why the "db_recover" and other binaries are not in the new version?

Second, I still get the following error after erasing the __db.*, verifying ok
with /usr/lib/rpm/rpmdb_verify, and:
   # rpm --rebuilddb
   error: db4 error(16) from dbenv->remove: Dispositivo o recurso ocupado
(device or resource busy).

This means that --rebuilddb does not work well with the new databases?
Or perhaps it is not needed since the __db.* index files are generated
otherwise? (in this case the outdated option should not be deprecated?).

Finally, I still have the task of recovering my full initial rpm database, as I
told you in my first e-mail, that I update hereafter:

Now I am able to query only 726 rpm's in my trimmed database, so that I have
lost about 247 rpm's. Since my Packages file was reduced from 37376000 bytes to
26710016 bytes, it is to about 75 % (similar to 973 to 726), I assume that I
have lost the info about the missing rpm's.

Now, before going to the painful task of "reloading" all my missed rpm in the
database, I have 1 specific question left:

1. May I just "copy" the Packages file from a similar computer at home where I
have installed the same packages, more or less at the same times (since I
updated to RH-9 a few weeks ago in both machines), and just try to reload a few
of the rpms newly installed at this machine?.

A final remark is that I am afraid of not fixing my rpm database, because I
could get false dependencies faults in the future, or other troubles. Don't you
agree on this?

Comment 4 J.M. Arago 2003-05-14 16:39:30 UTC
I have solved my problem by copying my full /var/lib/rpm directory with the one
in my home machine with quite similar packages. Thus I had to update only a few
different packages in both machines.

I have no idea why my /var/lib/rpm/Packages file got corrupted with a "region
fatal error", neither why when I did "rpm --rebuilddb" my Packages file was
trimmed and 25% of the packages were lost there.

Since I did not save my corrupted Packages file I can not check if the alternate
procedure, (given by jbj above:
  cd /var/lib/rpm ; rm -f __db.* ; mv Packages Packages-ORIG ;
/usr/lib/rpm/rpmdb_dump Packages-ORIG | /usr/lib/rpm/rpmdb_load Packages ) would
have worked to recover my damaged Packages database.

On other hand, I found that the binaries /usr/bin/db_* , in the default path,
that were installed under RedHat-9 in the db4-utils-4.0.14-20.i386.rpm, are from
a lower version 4.0.14 of Berkeley DB4, while the binaries /usr/lib/rpm/rpmdb_*,
not in the standard path, were installed by rpm-devel-4.2-0.69.i386.rpm, and use
a newer version DB 4.1.24. So that doing "db_verify Packages" results in a
region error found in the new rpm-4.2 generated databases. Probably the use of
"db_dump" and "db_load" also has some potential problems.

I regard this report as NOT-A-BUG, resolving it, but I recommend Red Hat to
improve or update the documentation for "rpm --rebuilddb" and "rpmdb
--rebuilddb" man pages, giving there the recommendation to save the rpm database
directory "/var/lib/rpm/" before doing a "rebuilddb", deleting the "__db.00?"
files and running /usr/lib/rpm/rpmdb_dump and _load, as well as
"/usr/lib/rpm/rpmdb_verify /var/lib/rpm/Packages", instead of the "db_verify".

It would also be helpful to have some info in these man pages about the
maintenance of the "/var/log/rpmpkgs" files, with the full list of the installed
rpms, updated by the "/etc/cron.daily/rpm" file, and saved weekly by the
"/etc/logrotate.d/rpm" file, as distributed in rpm-4.2.

To have compatible versions of DB4 in future RH distros, would help to avoid
such problems.

Many thanks to Jeff Johnson (jbj@redhat.com) for your quick attention.

Comment 5 Mike H. 2003-05-15 18:54:15 UTC
I first noticed the rpm db corruption problem after I first installed rh9 and 
then ran the up2date client. After the up2date client stalled out and crashed, 
the rpm db has been corrupted and giving me troubles sense. Why does rpmdb --
rebuild always say device unavailable or give a small error message? 

Note You need to log in before you can comment on or make changes to this bug.