Bug 213963 (yum-segfault)

Can you get a backtrace from the segfault (ulimit -c unlimited, and then run gdb
on the core)?

Can you please explain me (or link me to an explanation) what commands do I need
to run, as I haven't ever used gdb?

Btw, a similar bug in some way: bug 214110.

This problem exists with FC5 it happens much more frequently when using apt-get. 
One other thing to note, the more memory in the computer the less frequently you will 
see the segfault. Also, another point to note, it happens much much less frequently 
when running rpm alone, next in frequency of occurrence is yum, and then apt-get. 
This has been going on for about a month and a half. I believe that this started when 
the new memory handling was introduced into the 2.6.18 kernel. However, I'm not a 
programmer so I really don't know.

(In reply to comment #3)
> it happens much much less frequently when running rpm alone

Could you please do the following: Uninstall apt/yum/smart, repair the rpmdb,
make a backup of /var/lib/rpm and then stress-test the system with the rpm
commands you use in some kind of loop (installing/erasing a package or whatever
you use which triggers this bug).

If it does break in this scenario, then it's not a yum bug anymore, but either
rpm or kernel related issue. I would then recommend to move the bug to the rpm
component, then please tar up both the backup and the now broken /var/lib/rpm
and upload them somewhere (don't attach to this bug report, these are several MB
worth of data) and post the URL to this bug for rpm experts to look at and
diagnose the issue. If you don't have somewhere to upload it, contact me in PM
first and I'll set something up for you.

I just noticed that Eli had forgotten to add himself to the Cc - I just added
you (and myself), can you please check the previous comment in this bug for some
instuctions? Thanks.

Axel, if it happens in rpm, in apt-get in yum then it cannot be a yum bug. It
would be the lowest common denominator of the three which is certainly rpm -
more specifically rpm-lib or the db.

A damage in the rpmdb can disclose itself at later invocation, so in order to
identify that rpm used *alone* already exhibits this bug it needs to be run in
an otherwise isolated environment, e.g. isolated in the sense that neither yum,
apt or any higher level depsolver messed with the rpmdb in any way and perhaps
in a way that was not even transparent to the user like cron/daemons/applets and
so on.

That's why the testing in a clean environment as described in comment #4 is
neccessary to outrune any cross-depsolver influences. Because should Eli or
Vedran find that rpm alone is not able to nuke the rpmdb even though they stress
the system, then it's back to the higher level depsolvers. Personally I'd place
my bets on rpm/kernel issues (given lots of other bug reports I've seen
recently), but the testing will show for sure.

FWIW I did the same on my setups and couldn't provoke rpm to eat my rpmdb. But
neither did yum/apt/smart choke my rpmdb, so I'm out of the game as a test
vector. ;)

Outrune? Make that outrule ... :/

I've copied this over, from my posting at bug

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=211254

OK... Here's the situation. I don't like yum and if I can avoid using it I do. And I have 
been avoiding it for several years. I don't invoke the automatic scripts, because I like to 
do kernel updates manually. And, on occasion I like to change runlevels in order to 
ensure that I get trouble free updates (another story).

I deal with 5 fc computers. Three of them are fc6 and 2 are fc5. The fc6 computers 
function as desktops while the fc5 function as servers.

FC6 computers Memory Composistion
------------------------------------------------
1 Gig
512 MBytes
256 MBytes

FC5 computers Memory Composition
----------------------------------------------
512 MBytes
256 MBytes

The only computer that is giving me consistent headaches is the 256 FC5 machine. It is 
utilizing serveral services. It provides gateway, dns, mail gateway, ftp, http, and ssh 
services. So, it utilizes a considerable amount of memory, but, I have never, until 
recently had a problem with apt-get. In order to fix the problems, for the last month or 
so, I've had to:

rm -f /var/lib/rpm/__db*
rm -f /var/lib/apt/*.bin
rm -f /var/cache/apt/lists/*.*

Then, reboot the computer run rpm --rebuilddb

then

apt-get update
apt-get upgrade

Usually worked but a real pain in the butt.

A couple of days ago, that didn't work either. So, I tried yum, because I need to keep 
the system patched, especially with security updates since this computer constantly 
faces the internet.

First invocation, yum segfaulted. After simply running

rm -f /var/lib/rpm/__db*

And ran yum update again, everything worked the way it was supposed to. However, I 
started getting packages from repos I did not want the packages to come from. And 
this is why I prefer apt-get to yum. I love the pinning feature available in apt-get. It 
saves my brain a great deal of confusion.

Anyway. On the other machines I am still able to run apt-get. When I see the segfault, 
simply running

rm -f /var/cache/rpm/*.bin fixes the problem.

apt-get update
apt-get upgrade

works.

I have never had a problem like this until recently. Which makes me suspect the 
upgrading of critical libraries as the main culprit. This brings me back to my suspicion 
that the main instigator is a kernel update. Why, because, if I recall correctly, one of the 
updates to the 2.6.18 kernel changed the way the kernel manages memory and if I'm 
not mistaken, a segfault is somekind of screwup in memory.

The long and the short of this is that this is not an easy "ahhah, there's the
NULL pointer dereference" type of bug, don't expect it to be fixed "just like
that". Something in causing corruption in apt main datastructure (which is a
memorymapped cachefile) and whether it's the combination of 2.6.18 kernel +
mmap() use + low memory + apt-rpm's usage patters or something else remains to
be seen.

I'm reinstalling my 32bit testbox now and try to see if I can (eventually)
reproduce it by limiting available memory.

Oops, comment #10 was intended for bug 211254, not here.

*** Bug 216161 has been marked as a duplicate of this bug. ***

*** Bug 216398 has been marked as a duplicate of this bug. ***

*** Bug 216493 has been marked as a duplicate of this bug. ***

*** Bug 216725 has been marked as a duplicate of this bug. ***

*** Bug 216281 has been marked as a duplicate of this bug. ***

I would like to add that I am observing the same phenomenon. Besides
random crashes yum also sometimes just hangs. Deleting
/var/lib/rpm/__db* and rpm --rebuilddb fixes the problem temporarily
(for one day or so). The problem started right after a fresh install
of FC6. I think it is a major issue. Before I was running FC4 on this machine and
the problem never happened.

My system is otherwise very stable. 

Is anyone of the people with an issue running yum on systems with small memory?
For people using apt on low-memory machines there seems to exist an issue. Maybe
the same is true for the people seeing the yum problems.

I searched for bug reports regarding strange rpmdb behaviour today and I
encountered this one, which seems to be related to my problem.

Today, after a couple of days, I did my routine 'yum update' and yum crashed
instantly with a traceback indicating rpmdb problems. I was guessing my rpmdb
had a problem, so tried rpm --verifydb, which gave no output (I'm presuming this
is ok?), then I deleted __db* files in /var/lib/rpm and run rpm --rebuilddb. At
first attempt, rpm --rebuilddb gave the same error as yum, namely:
rpmdb: seek: 2261270528 0 2: Invalid argument 
error: db4 error(22) from dbcursor->c_get: Invalid argument

However, at the second and all subsequent runs of rpm --rebuilddb, the command
gave no output, so I tried running it in verbose mode and it produced:
[tadej@tlinux-stable rpm]$ sudo rpm -vv --rebuilddb
D: rebuilding database /var/lib/rpm into /var/lib/rpmrebuilddb.17346
D: creating directory /var/lib/rpmrebuilddb.17346
D: opening old database with dbapi 3
D: opening  db environment /var/lib/rpm/Packages create:cdb:mpool
D: opening  db index       /var/lib/rpm/Packages rdonly mode=0x0
D: locked   db index       /var/lib/rpm/Packages
D: opening new database with dbapi 3
D: opening  db environment /var/lib/rpmrebuilddb.17346/Packages create:mpool
D: opening  db index       /var/lib/rpmrebuilddb.17346/Packages create mode=0x42
D: closed   db index       /var/lib/rpm/Packages
D: closed   db environment /var/lib/rpm/Packages
D: closed   db index       /var/lib/rpmrebuilddb.17346/Packages
D: closed   db environment /var/lib/rpmrebuilddb.17346/Packages
D: removing directory /var/lib/rpmrebuilddb.17346
D: May free Score board((nil))

Everything seems normal to me.

Here is my directory listing of /var/lib/rpm (are the sizes of other files here
ok?):
[tadej@tlinux-stable rpm]$ ls -al
total 22184
drwxr-xr-x  2 rpm  rpm      4096 Nov 30 21:02 .
drwxr-xr-x 26 root root     4096 Nov 30 21:02 ..
-rw-r--r--  1 rpm  rpm  10567680 Nov 24 15:54 Basenames
-rw-r--r--  1 rpm  rpm     12288 Nov 24 15:54 Conflictname
-rw-r--r--  1 rpm  rpm   2916352 Nov 24 15:54 Dirnames
-rw-r--r--  1 rpm  rpm  10391552 Nov 24 15:55 Filemd5s
-rw-r--r--  1 rpm  rpm     32768 Nov 24 15:54 Group
-rw-r--r--  1 rpm  rpm     36864 Nov 24 15:54 Installtid
-rw-r--r--  1 rpm  rpm     86016 Nov 24 15:54 Name
-rw-r--r--  1 rpm  rpm     12288 Nov 24 15:54 Packages
-rw-r--r--  1 rpm  rpm    651264 Nov 24 15:54 Providename
-rw-r--r--  1 rpm  rpm    176128 Nov 24 15:54 Provideversion
-rw-r--r--  1 rpm  rpm     12288 Jun 21  2005 Pubkeys
-rw-r--r--  1 rpm  rpm    561152 Nov 24 15:54 Requirename
-rw-r--r--  1 rpm  rpm    389120 Nov 24 15:54 Requireversion
-rw-r--r--  1 rpm  rpm    294912 Nov 24 15:54 Sha1header
-rw-r--r--  1 rpm  rpm    159744 Nov 24 15:54 Sigmd5
-rw-r--r--  1 rpm  rpm     12288 Nov 22 12:45 Triggername


I don't know hot to fix my rpmdb, even 'manually', so I'm unable to use rpm (and
yum) at all.

Any pointers would be really appreciated! (Maybe I should fill a new bug report?)

My setup details:
Compaq Evo N800v laptop with Pentium 4 Mobile cpu and 512MB of ram.
Up2date FC6 installation (fresh install of RHL9 and upgraded with each FC
release to date using anaconda cd installer, no problems ever).
My rpm -q command doesn't work, but yum version is 3.0.1-2.fc6 and rpm version
is the one that ships with FC6.

I have seen the problem on a FC5 machine with 512MB of RAM. I have also seen the
problem on a FC5 machine with less (160MB IIRC).

My bug is a possible duplicate: bug #213986

Regarding the relation to "low memory", my FC has about 86MB RAM (It's a Xen VM)

I don't think it is low memory -- if yum has problems to run on laptop with
500MB physical RAM and 1GB of swap, then it has a problem.

Just a quick follow up to my previous comment (comment #19):

Trying to get my rpmdb back, I noticed that my Packages file is suspiciously
small (only 12288 bytes).
So, I ran '/usr/lib/rpm/rpmdb_dump Packages' and I get: 
VERSION=3
format=bytevalue
type=hash
db_pagesize=4096
HEADER=END
DATA=END

No data at all!

Is there still a way to get my rpmdb back up after this?
I still have no ideas how 'yum update ...' managed to remove all data from my
Packages file.

Segafualts and loss of data are likely due to removing an rpmdb environment
without correcting other problems in the rpmdb.

FYI: Most rpmdb "hangs" are now definitely fixed by purging stale read locks when opening
a database environment in rpm-4.4.8-0.4. There's more todo, but I'm quite sure that a
large class of problems with symptoms of "hang" are now corrected.

Detecting damaged by verifying when needed is well automated in rpm-4.4.8-0.4. Automatically 
correcting all possible damage is going to take more work, but a large class of problems is likely
already fixed in rpm-4.4.8-0.8 as well.

UPSTREAM

*** Bug 217760 has been marked as a duplicate of this bug. ***

*** Bug 218206 has been marked as a duplicate of this bug. ***

*** Bug 218353 has been marked as a duplicate of this bug. ***

*** Bug 218439 has been marked as a duplicate of this bug. ***

*** Bug 218535 has been marked as a duplicate of this bug. ***

*** Bug 218677 has been marked as a duplicate of this bug. ***

*** Bug 218869 has been marked as a duplicate of this bug. ***

*** Bug 219007 has been marked as a duplicate of this bug. ***

Just a quick follow up to my previous posts:
I managed to recover my lost Packages file using /var/log/rpmpkgs file, which
had information about all installed rpms on my system.
To make the process easier, I wrote the shell script attached bellow.

Now, I'm using yum without problems.

Note: This script should only be used if you have an empty /var/lib/rpm/Packages
file.

Created attachment 143505 [details]
/var/lib/rpm/Packages recovery shell script

rpmq just crashed on my FC5 system (I could have opened a new bug, but it would
just have been marked a duplicate of this one):

Reading symbols from /usr/lib/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /usr/lib/libkrb5.so.3...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /usr/lib/libk5crypto.so.3...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /usr/lib/libkrb5support.so.0...done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libcom_err.so.2...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libexpat.so.0...done.
Loaded symbols for /lib/libexpat.so.0
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libsetrans.so.0...done.
Loaded symbols for /lib/libsetrans.so.0
#0  __memp_fget_rpmdb (dbmfp=0x8273838, pgnoaddr=0xbf88a51c, flags=0,
addrp=0xbf88a4f8) at ../db/dist/../mp/mp_fget.c:190
190     ../db/dist/../mp/mp_fget.c: No such file or directory.
        in ../db/dist/../mp/mp_fget.c
(gdb) bt
#0  __memp_fget_rpmdb (dbmfp=0x8273838, pgnoaddr=0xbf88a51c, flags=0,
addrp=0xbf88a4f8) at ../db/dist/../mp/mp_fget.c:190
#1  0x003f2850 in __db_goff_rpmdb (dbp=0x8273538, dbt=0x827c11c, tlen=3448,
pgno=14859, bpp=0x8274c84, bpsz=0x8274c8c)
    at ../db/dist/../db/db_overflow.c:147
#2  0x003fa15d in __db_ret_rpmdb (dbp=0x8273538, h=0xb7d0bf84, indx=1,
dbt=0x827c11c, memp=0x8274c84, memsize=0x8274c8c)
    at ../db/dist/../db/db_ret.c:50
#3  0x003e548d in __db_c_get_rpmdb (dbc_arg=0x8274c38, key=0x827c104,
data=0x827c11c, flags=Variable "flags" is not available.
) at ../db/dist/../db/db_cam.c:778
#4  0x003eb946 in __db_c_get_pp_rpmdb (dbc=0x8274c38, key=0x827c104,
data=0x827c11c, flags=28)
    at ../db/dist/../db/db_iface.c:1741
#5  0x0037b946 in db3cget (dbi=0x8272f90, dbcursor=0x8274c38, key=0x827c104,
data=0x827c11c, flags=28) at db3.c:612
#6  0x0037666d in rpmdbNextIterator (mi=0x827c0e8) at rpmdb.h:591
#7  0x00152d5f in rpmtsFindPubkey (ts=0x8272bb0) at rpmts.c:376
#8  0x001538c6 in verifyDSASignature (ts=0x8272bb0, t=Variable "t" is not available.
) at signature.c:1460
#9  0x0015505c in rpmVerifySignature (ts=0x8272bb0, result=0xbf88a9b8 "Header V3
DSA signature: ") at signature.c:1518
#10 0x0012cf45 in headerCheck (ts=0x8272bb0, uh=0x8275da8, uc=17880,
msg=0xbf89aa70) at package.c:632
#11 0x003760ff in rpmdbNextIterator (mi=0x8274bb8) at rpmdb.c:2280
#12 0x0014431f in rpmgiNext (gi=0x8272de8) at rpmgi.c:504
#13 0x00135a2b in rpmgiShowMatches (qva=0x1a0680, ts=0x8272bb0) at query.c:372
#14 0x00136032 in rpmQueryVerify (qva=0x1a0680, ts=0x8272bb0, arg=0x0) at
query.c:448
#15 0x00136e9e in rpmcliArgIter (ts=0x8272bb0, qva=0x1a0680, argv=0x0) at
query.c:728
#16 0x00137002 in rpmcliQuery (ts=0x8272bb0, qva=0x1a0680, argv=0x0) at query.c:807
#17 0x08049afc in main (argc=5, argv=0xbf89bd24) at ./rpmqv.c:804
#18 0x00b3f4e4 in __libc_start_main () from /lib/libc.so.6
#19 0x08049251 in _start ()
(gdb) 

Could this be triggered by http://article.gmane.org/gmane.linux.kernel/477324 ?

According to Panu (bug # 211254 comment 55) the 2.6.18 packaged kernels behave
like the 2.6.19 example given in the article above. It seems to directly affect
apt, but could also affect mmaps in in db4/rpm.

*** Bug 221946 has been marked as a duplicate of this bug. ***

*** Bug 221997 has been marked as a duplicate of this bug. ***

*** Bug 222238 has been marked as a duplicate of this bug. ***

*** Bug 222450 has been marked as a duplicate of this bug. ***

*** Bug 222535 has been marked as a duplicate of this bug. ***

*** Bug 223030 has been marked as a duplicate of this bug. ***

*** Bug 222849 has been marked as a duplicate of this bug. ***

*** Bug 212333 has been marked as a duplicate of this bug. ***

*** Bug 223734 has been marked as a duplicate of this bug. ***

The current kernel in FC6 (2.6.19.2 based) fixes the issue mentioned in comment
#36) and cured the apt issue. In case this bug is trigged by the same mmap
problem it should have vanished on FC6 and the latest kernel.

E.g. is there anyone still seeing this problem on FC6 with kernel
2.6.19-1.2895.fc6 (after having fixed the rpmdb under 2.6.19, of course)?

*** Bug 225376 has been marked as a duplicate of this bug. ***

The problem mentioned in comment #36 is a user mode bug. That cannot be fixed by
using a different kernel, but the probability of running into the problem might
be reduced.

How is a change in mmap(2) behavior in linux kernels a "user mode bug". The problem
is demonstrably fixed by using later kernels.

And how exactly can the probability of a broken mmap implementation be
meaningfully reduced? Sure mmap does not have to be used, but then tar
or even shar might be a better choice of package manager under those
confitions.

*** Bug 227242 has been marked as a duplicate of this bug. ***

*** Bug 215901 has been marked as a duplicate of this bug. ***

*** Bug 214110 has been marked as a duplicate of this bug. ***

*** Bug 215198 has been marked as a duplicate of this bug. ***

*** Bug 223548 has been marked as a duplicate of this bug. ***

*** Bug 208674 has been marked as a duplicate of this bug. ***

*** Bug 212504 has been marked as a duplicate of this bug. ***

*** Bug 215180 has been marked as a duplicate of this bug. ***

*** Bug 215184 has been marked as a duplicate of this bug. ***

*** Bug 219919 has been marked as a duplicate of this bug. ***

*** Bug 224514 has been marked as a duplicate of this bug. ***

*** Bug 228463 has been marked as a duplicate of this bug. ***

*** Bug 216026 has been marked as a duplicate of this bug. ***

One of mine was marked as a duplicate of this one.  To update the situation. 

After dozens of rebuilds of the RPM DB one of the system updates I did fixed the
problem or I was not duplicating it. The machine I first filed the problem on
was one I had run FC3 on for years without a problem. I had 2 FC5 machines both
of which ran without any problems. I wiped out the FC3 partition and put FC6 on
the machine and that was when the RPM problems arose. I'd heard references by
people in fedora forums too the problem before that from early adopters of FC6.
I was surprised when I searched bugzilla not too see the problem reported before
so created a new bug report. 

Since then I upgraded the other two machines to FC6 and the problem did not
present itself. Hardware failure caused me to replace two of the machines
recently. One with a 64 bit brand new machine which I put FC7 64 bit  on. The
other I cobbled together with parts from the two failed machines and installed a
fresh FC6 32 bit install. The FC6 machine is running with 512 megs of Ram and
has so far had no rpm problems. The FC7 machine has a corrupted rpm DB and major
problems. So much so it's not really a usable machine yet as I cannot get so
much essential software on it. 

The difference seems to be that I concentrated on building the FC7 machine and
the mass updates seem to be the fault. 

Both machines, imediately after install I did a yum update from the command line
and patched the machine. Both had no problems with this. 

Both machines from the KDE menu I used add/remove software to install kyum and
yumextender. 

On the FC6 machine I grabbed packages 3-10 at a time. On the FC7 machine I
selected over 100 packages to install. I then ran into a conflict with a couple
of the selected libs. I removed the offending packages and installed normally. 

Then I installed the freshrpms rpm to grab additional packages. On the FC6
machine no problem.  On the FC7 machine this failed miserably. I got a complaint
about not being able to get an eclusive lock on the rpm db. Which was strange as
I was not running anything that should have touched the rpm db. It appeared to
install anyway.  When I attempted to install packages from the command line yum
would stall then fail attempting to contact freshrpms. Yet when I pulled up Kyum
and yumextender the repository didn't exist. Attempts to install software from
the repository hung up the command line yum. It would die attempting to contact
the mirror. I tried to uninstall the freshrpms rpm from the command line but the
rpm database said it didn't exist. I removed the locks did a rpm clean all and a
rpm makecache. Neither helped. 

Finally I was able to force the rpm from the command line to reinstall and all
worked well for several hours. Then it failed again. A second time I was able to
force the freshrpms to reinstall and it worked well for one package then died
again. 

On both machines one of the FIRST things I did was kill the yumupdate service
and make sure it NEVER started again. 

I am about to rebuild the yum database but expect to have the same problems I
did on an earlier FC6 machine with rpm corruption problems. 

My biggest question was what was locking the rpm DB? I did a ps and there was no
trace of rpm anything in the process list. No file level locks existed on any of
the files in /var/lib/rpm. 

I have installed these yum plugins on both machines. 
kernel-module
donloadonly
mer-conf 
alloweddowngrade
cangelog
installonyn
fedorakmod
security
presto
ski-broken
fastestmirror

*** Bug 248273 has been marked as a duplicate of this bug. ***

Draciron, please don't mix up F7 issues into this one, it has it's own set of
bugs such as #245389 and #203938 (common case here, fedorakmod plugin).

This bug is about FC6 (and to some extent FC5 on certain period) where rpm and
anything using it was segfaulting all over the place. Evidence strongly suggests
it was caused by the mmap() bug in kernels 2.6.18-19 which was fixed in kernel
2.6.19-1.2895.fc6 and later. To summarize:

1) upgrade to kernel 2.6.19-1.2895 or later (with rpm2cpio if need be)
2) reboot to the new kernel (thus clearing any stale rpmdb locks)
3) (optionally) run rpm --rebuilddb
4) you shouldn't see this problem anymore

In other words, this has been fixed long long time ago but not in rpm because it
never really was an rpm bug.

*** Bug 228221 has been marked as a duplicate of this bug. ***

*** Bug 229622 has been marked as a duplicate of this bug. ***

*** Bug 234111 has been marked as a duplicate of this bug. ***

Time to close this monster... not a bug in rpm or yum but in certain kernel
versions' mmap() implementation.

Hi,

if it's fixed in kernel >= 2.6.19-1.2895, what about RHEL? Is it fixing in its
kernel, too? Thanks!

RHEL never had this bug (except for RHEL 5 beta possibly) so there's nothing to
fix there. 

*** Bug 213043 has been marked as a duplicate of this bug. ***