Bug 86047 - RPM segfaults on install of db4
Summary: RPM segfaults on install of db4
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat Linux
Classification: Retired
Component: rpm
Version: 7.3
Hardware: All
OS: Linux
medium
high
Target Milestone: ---
Assignee: Jeff Johnson
QA Contact: Mike McLean
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2003-03-13 05:24 UTC by Stephen Rasku
Modified: 2007-04-18 16:51 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2003-04-14 15:09:49 UTC
Embargoed:


Attachments (Terms of Use)
strace output (292.54 KB, text/plain)
2003-03-31 05:08 UTC, Stephen Rasku
no flags Details
strace output after restarting nscd (292.80 KB, text/plain)
2003-04-03 03:49 UTC, Stephen Rasku
no flags Details
RPM output after manual install (7.82 KB, text/plain)
2003-04-03 17:25 UTC, Stephen Rasku
no flags Details
strace output after manual install (36.28 KB, text/plain)
2003-04-03 17:25 UTC, Stephen Rasku
no flags Details
Output from rpm -ivv after removing packages.rpm (7.82 KB, text/plain)
2003-04-03 18:51 UTC, Stephen Rasku
no flags Details
strace output after removing packages.rpm (36.28 KB, text/plain)
2003-04-03 18:52 UTC, Stephen Rasku
no flags Details
strace output after removing macros.db1 (22.56 KB, text/plain)
2003-04-03 19:03 UTC, Stephen Rasku
no flags Details

Description Stephen Rasku 2003-03-13 05:24:15 UTC
Description of problem:

I get a segmentation fault when I try and install the rawhide RPM for db4.  db1,
db2, db3, are all already installed.

Version-Release number of selected component (if applicable):

db2-2.4.14-10
db1-1.85-8
db3-3.3.11-6
db4-4.0.14-20
rpm-4.0.4-7x.18

How reproducible:

100%

Steps to Reproduce:
1.rpm -ivh db4-4.0.14-20.i386.rpm
    
Actual results:

Segmentation fault.


Expected results:

db4 should get installed.

Comment 1 Jeff Johnson 2003-03-13 19:44:26 UTC
Almost all segfaults in rpm are due to bad
headers from database.

Have you tried --rebuilddb?

Comment 2 Stephen Rasku 2003-03-21 04:19:37 UTC
--rebuilddb has no effect.  I still get a segfault when I try and install after
I rebuild.

Comment 3 Jeff Johnson 2003-03-22 20:15:25 UTC
Can you attach pointer (i.e. URL, attachments won't work)
to a tarball of your database
    cd /var/lib
     tar czvf /tmp/rpmdb-86047.tar.gz rpm
and I'll take a look?

Comment 4 Stephen Rasku 2003-03-26 20:05:10 UTC
Go to:

    http://richmond.tgivan.com/rpmdb-86047.tar.gz

md5sum:

865c59fdb46f82c379db0c24bed29836  /tmp/rpmdb-86047.tar.gz


Comment 5 Jeff Johnson 2003-03-28 19:54:58 UTC
Your database looks fine, header signatures/digests verify
after doing
    rpm --import RPM-GPG-KEY

Can you attach strace and stderr output of
    strace -o /tmp/xxx /bin/rpm -Uvv some_package_that_segfaults_here

Comment 6 Stephen Rasku 2003-03-31 05:06:49 UTC
This appears to be a problem with the rawhide glibc RPM.  I had a pristine
installation of 7.3 and after installing:

glibc-2.3.2-11.9.i386.rpm
glibc-common-2.3.2-11.9.i386.rpm
glibc-devel-2.3.2-11.9.i386.rpm
binutils-2.13.90.0.18-9.i386.rpm

I can't install any more RPMS.  This is the output from:

strace -o /tmp/strace.out /bin/rpm -ivv db4-4.0.14-21.i386.rpm:

D: ============== db4-4.0.14-21.i386.rpm
D: Expected size:      4626475 = lead(96)+sigs(180)+pad(4)+data(4626195)
D:   Actual size:      4626475
D: opening  db environment /var/lib/rpm/Packages create:mpool
D: opening  db index       /var/lib/rpm/Packages create mode=0x42
D: locked   db index       /var/lib/rpm/Packages
D:      added binary package [0]
D: found 0 source and 1 binary packages
D: ========== +++ db4-4.0.14-21
D: opening  db index       /var/lib/rpm/Depends create mode=0x42
D: opening  db index       /var/lib/rpm/Basenames create mode=0x42
D:  Requires: /sbin/ldconfig                                YES (db files)
D:  Requires: R /sbin/ldconfig                              YES (cached)
D: opening  db index       /var/lib/rpm/Providename create mode=0x42
D:  Requires: libc.so.6                                     YES (db provides)
D:  Requires: libc.so.6(GLIBC_2.0)                          YES (db provides)
D:  Requires: libc.so.6(GLIBC_2.1)                          YES (db provides)
D:  Requires: libc.so.6(GLIBC_2.1.3)                        YES (db provides)
D:  Requires: libc.so.6(GLIBC_2.2)                          YES (db provides)
D:  Requires: libc.so.6(GLIBC_2.3)                          YES (db provides)
D:  Requires: libc.so.6(GLIBC_2.3.2)                        YES (db provides)
D:  Requires: libpthread.so.0                               YES (db provides)
D:  Requires: libpthread.so.0(GLIBC_2.0)                    YES (db provides)
D:  Requires: libpthread.so.0(GLIBC_2.2)                    YES (db provides)
D:  Requires: libpthread.so.0(GLIBC_2.3.2)                  YES (db provides)
D:   NO     A rpmlib(CompressedFileNames) <= 3.0.4-1    B
rpmlib(VersionedDependencies) = 3.0.3-1
D:   YES    A rpmlib(CompressedFileNames) <= 3.0.4-1    B
rpmlib(CompressedFileNames) = 3.0.4-1
D:  Requires: rpmlib(CompressedFileNames) <= 3.0.4-1        YES (rpmlib provides)
D:   NO     A rpmlib(PayloadFilesHavePrefix) <= 4.0-1   B
rpmlib(VersionedDependencies) = 3.0.3-1
D:   NO     A rpmlib(PayloadFilesHavePrefix) <= 4.0-1   B
rpmlib(CompressedFileNames) = 3.0.4-1
D:   NO     A rpmlib(PayloadFilesHavePrefix) <= 4.0-1   B rpmlib(PayloadIsBzip2)
= 3.0.5-1
D:   YES    A rpmlib(PayloadFilesHavePrefix) <= 4.0-1   B
rpmlib(PayloadFilesHavePrefix) = 4.0-1
D:  Requires: rpmlib(PayloadFilesHavePrefix) <= 4.0-1       YES (rpmlib provides)
D: opening  db index       /var/lib/rpm/Conflictname create mode=0x42
D: closed   db index       /var/lib/rpm/Depends
D: ========== recording tsort relations
D: ========== tsorting packages (order, #predecessors, #succesors, tree, depth)
D:     0    0    1    0    0  db4-4.0.14-21
D: installing binary packages
D: opening  db index       /var/lib/rpm/Name create mode=0x42
D: Expected size:      4626475 = lead(96)+sigs(180)+pad(4)+data(4626195)
D:   Actual size:      4626475
D:   install: db4-4.0.14-21 has 13 files, test = 0


Comment 7 Stephen Rasku 2003-03-31 05:08:29 UTC
Created attachment 90796 [details]
strace output

Comment 8 Jeff Johnson 2003-04-02 23:50:18 UTC
This line indicates you are not running nscd:

connect(9, {sin_family=AF_UNIX, path="/var/run/.nscd_socket"}, 110) = -1 ENOENT
(No such file or directory)

Can you start nscd by doing
    service nscd restart
and try repeating the failure?

Comment 9 Stephen Rasku 2003-04-03 03:47:46 UTC
The failure still occurs:

D: ============== db4-4.0.14-21.i386.rpm
D: Expected size:      4626475 = lead(96)+sigs(180)+pad(4)+data(4626195)
D:   Actual size:      4626475
D: opening  db environment /var/lib/rpm/Packages create:mpool
D: opening  db index       /var/lib/rpm/Packages create mode=0x42
D: locked   db index       /var/lib/rpm/Packages
D:      added binary package [0]
D: found 0 source and 1 binary packages
D: ========== +++ db4-4.0.14-21
D: opening  db index       /var/lib/rpm/Depends create mode=0x42
D: opening  db index       /var/lib/rpm/Basenames create mode=0x42
D:  Requires: /sbin/ldconfig                                YES (db files)
D:  Requires: R /sbin/ldconfig                              YES (cached)
D: opening  db index       /var/lib/rpm/Providename create mode=0x42
D:  Requires: libc.so.6                                     YES (db provides)
D:  Requires: libc.so.6(GLIBC_2.0)                          YES (db provides)
D:  Requires: libc.so.6(GLIBC_2.1)                          YES (db provides)
D:  Requires: libc.so.6(GLIBC_2.1.3)                        YES (db provides)
D:  Requires: libc.so.6(GLIBC_2.2)                          YES (db provides)
D:  Requires: libc.so.6(GLIBC_2.3)                          YES (db provides)
D:  Requires: libc.so.6(GLIBC_2.3.2)                        YES (db provides)
D:  Requires: libpthread.so.0                               YES (db provides)
D:  Requires: libpthread.so.0(GLIBC_2.0)                    YES (db provides)
D:  Requires: libpthread.so.0(GLIBC_2.2)                    YES (db provides)
D:  Requires: libpthread.so.0(GLIBC_2.3.2)                  YES (db provides)
D:   NO     A rpmlib(CompressedFileNames) <= 3.0.4-1    B
rpmlib(VersionedDependencies) = 3.0.3-1
D:   YES    A rpmlib(CompressedFileNames) <= 3.0.4-1    B
rpmlib(CompressedFileNames) = 3.0.4-1
D:  Requires: rpmlib(CompressedFileNames) <= 3.0.4-1        YES (rpmlib provides)
D:   NO     A rpmlib(PayloadFilesHavePrefix) <= 4.0-1   B
rpmlib(VersionedDependencies) = 3.0.3-1
D:   NO     A rpmlib(PayloadFilesHavePrefix) <= 4.0-1   B
rpmlib(CompressedFileNames) = 3.0.4-1
D:   NO     A rpmlib(PayloadFilesHavePrefix) <= 4.0-1   B rpmlib(PayloadIsBzip2)
= 3.0.5-1
D:   YES    A rpmlib(PayloadFilesHavePrefix) <= 4.0-1   B
rpmlib(PayloadFilesHavePrefix) = 4.0-1
D:  Requires: rpmlib(PayloadFilesHavePrefix) <= 4.0-1       YES (rpmlib provides)
D: opening  db index       /var/lib/rpm/Conflictname create mode=0x42
D: closed   db index       /var/lib/rpm/Depends
D: ========== recording tsort relations
D: ========== tsorting packages (order, #predecessors, #succesors, tree, depth)
D:     0    0    1    0    0  db4-4.0.14-21
D: installing binary packages
D: opening  db index       /var/lib/rpm/Name create mode=0x42
D: Expected size:      4626475 = lead(96)+sigs(180)+pad(4)+data(4626195)
D:   Actual size:      4626475
D:   install: db4-4.0.14-21 has 13 files, test = 0

Comment 10 Stephen Rasku 2003-04-03 03:49:02 UTC
Created attachment 90858 [details]
strace output after restarting nscd

Comment 11 Jeff Johnson 2003-04-03 15:55:57 UTC
Good, nscd is running, that eliminates the LDAP segfault:
   connect(9, {sin_family=AF_UNIX, path="/var/run/.nscd_socket"}, 110) = 0

Hmmm, the strace still indicates a failure looking up a user name
like "root" (afaict, tricky).

Can you try reinstalling rpm manually? There are rpm-4.0.5-1.7x packages
at
    ftp://ftp.rpm.org/pub/rpm/test-4.0.5

Download (don't forget popt :-) into /var/tmp, and install manually
by doing (as root)
    mkdir /var/tmp/xxx
    cd /var/tmp/xxx
    for f in ../{rpm,popt}-*.rpm; do
        rpm2cpio $f | cpio -dim
    done
    find . -type d -exec chmod 755 {} \;
    tar cf - . | (cd /; tar xvf -)

Verify the manual install by doing
    rpm -qa

and then try repeating the same command, with strace please.

Comment 12 Stephen Rasku 2003-04-03 17:24:28 UTC
This is a bit different but it seems to have broken the database:

[root@host db4]# rpm -q rpm
error: cannot open Name index using db1 - Invalid argument (22)

[root@h24-84-188-188 db4]# rpm -qa
<no output>

Attachments to follow.

Comment 13 Stephen Rasku 2003-04-03 17:25:18 UTC
Created attachment 90871 [details]
RPM output after manual install

Comment 14 Stephen Rasku 2003-04-03 17:25:47 UTC
Created attachment 90872 [details]
strace output after manual install

Comment 15 Jeff Johnson 2003-04-03 17:38:23 UTC
Bingo, "db1" shouldn't be there.

What does "ls -al /var/lib/rpm" say?

Move the *.rpm files out of there, try again.
If the *.rpm files reappear (and are small), nuke.

BTW, is this a Red Hat Linux system or something else?

Comment 16 Stephen Rasku 2003-04-03 18:49:20 UTC
This is a pristine Red Hat 7.3 install with the changes noted in comment #6. 
These are the contents of /var/lib/rpm:

total 30169
drwxr-xr-x    2 rpm      rpm          1024 Apr  3 09:22 .
drwxr-xr-x   13 root     root         1024 Mar 27 21:07 ..
-rw-r--r--    1 rpm      rpm       5378048 Apr  2 19:46 Basenames
-rw-r--r--    1 rpm      rpm         12288 Apr  2 19:46 Conflictname
-rw-r--r--    1 rpm      rpm        778240 Mar 30 20:54 Dirnames
-rw-r--r--    1 rpm      rpm       5255168 Mar 30 20:54 Filemd5s
-rw-r--r--    1 rpm      rpm         24576 Mar 30 20:54 Group
-rw-r--r--    1 rpm      rpm         16384 Mar 30 20:54 Installtid
-rw-r--r--    1 rpm      rpm         45056 Apr  2 19:46 Name
-rw-r--r--    1 rpm      rpm      20582400 Mar 30 20:54 Packages
-rw-r--r--    1 root     root            8 Apr  3 09:22 packages.rpm
-rw-r--r--    1 rpm      rpm        180224 Apr  2 19:46 Providename
-rw-r--r--    1 rpm      rpm         65536 Mar 30 20:54 Provideversion
-rw-r--r--    1 rpm      rpm        237568 Mar 30 20:54 Requirename
-rw-r--r--    1 rpm      rpm        135168 Mar 30 20:54 Requireversion
-rw-r--r--    1 rpm      rpm         90112 Mar 30 20:54 Sha1header
-rw-r--r--    1 rpm      rpm         40960 Mar 30 20:54 Sigmd5
-rw-r--r--    1 rpm      rpm         12288 Mar 30 20:53 Triggername

Results after moving packages.rpm to follow.  It's not segfaulting any more but
I can't query the database or install the db4 rpm because -- I think -- is that
it isn't reading the database properly:

[root@host db4]# rpm -qa
error: cannot open Packages index using db1 - No such file or directory (2)

I assume that when you said "nuke" you meant the *.rpm files and not the whole
/var/lib/rpm directory.


Comment 17 Stephen Rasku 2003-04-03 18:51:39 UTC
Created attachment 90877 [details]
Output from rpm -ivv after removing packages.rpm

Why is it using db1?

Comment 18 Stephen Rasku 2003-04-03 18:52:17 UTC
Created attachment 90878 [details]
strace output after removing packages.rpm

Comment 19 Jeff Johnson 2003-04-03 18:57:55 UTC
Yes, just nuke /var/lib/rpm/packages.rpm.

rpm has support for db3 first, then db1, with lazy creation.
so the db3 open is failing, fallback is to db2.

what needs figgering is why db3 open is failing.

Ah there it is:
    open("/etc/rpm/macros.db1", O_RDONLY)   = 3
    ...
    read(3, "%_dbapi\t\t1\n", 8192)         = 11

Try
    rm /etc/rpm/macros.db1
lather, rinse, repeat ;-)


Comment 20 Stephen Rasku 2003-04-03 19:02:18 UTC
OK, it looks like it is now trying to open using db3 but still doesn't like it:

D: ============== db4-4.0.14-21.i386.rpm
D: Expected size:      4626475 = lead(96)+sigs(180)+pad(4)+data(4626195)
D:   Actual size:      4626475
D: opening  db environment /var/lib/rpm/Packages create:mpool
rpmdb: write: 0xbfffc0c0, 8192: Invalid argument
error: db4 error(22) from dbenv->open: Invalid argument
D: opening  db index       /var/lib/rpm/Packages create mode=0x42
error: cannot open Packages index using db3 - Invalid argument (22)
error: cannot open Packages database in /var/lib/rpm
D: found 0 source and 0 binary packages

strace output to follow.

Comment 21 Stephen Rasku 2003-04-03 19:03:28 UTC
Created attachment 90879 [details]
strace output after removing macros.db1

Comment 22 Jeff Johnson 2003-04-03 19:10:20 UTC
Now we're getting someplace.

The EINVAL is because the kernel cabal is inventing
semantics for O_DIRECT which is used by Berkeley DB
when opening certain files. The O_DIRECT flag used
to be (and still is for "stock" Red Hat ix86 kernels) silently
ignored, but now returns EINVAL.

So, what kernel are you running?

Comment 23 Stephen Rasku 2003-04-03 19:25:14 UTC
kernel-2.4.18-27.7.x

Comment 24 Jeff Johnson 2003-04-03 19:45:15 UTC
Well here's where the EINVAL is coming from, trying to
create (carefully, there are file races etc, with O_DIRECT)
the file /var/lib/rpm/__db.001:

stat64("/var/lib/rpm/__db.001", 0xbfffe0a0) = -1 ENOENT (No such file or directory)
open("/var/lib/rpm/__db.001", O_RDWR|O_CREAT|O_EXCL|O_DIRECT|O_LARGEFILE, 0644) = 3
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
open("/var/lib/rpm/__db.001", O_RDWR|O_CREAT|O_DIRECT|O_LARGEFILE, 0644) = 4
fcntl64(4, F_SETFD, FD_CLOEXEC)         = 0
_llseek(4, 0, [0], SEEK_END)            = 0
_llseek(4, 0, [0], SEEK_CUR)            = 0
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192) = -1
EINVAL (Invalid argument)

Presumably this is the "stock" Red Hat kernel-2.4.18-27.7.x package?


Comment 25 Stephen Rasku 2003-04-03 19:46:51 UTC
Yes.

Comment 26 Jeff Johnson 2003-04-03 20:05:16 UTC
OK, off to kernel for remedy. I believe there's a magic
incantation to disable ext3 from returning EINVAL when
given O_DIRECT, there's a possiblility that kernel pkg
is mis-built, otherwise bounce back to rpm ...

Comment 27 Stephen Rasku 2003-04-03 21:38:01 UTC
Re-assigning to kernel as per comment 26.

Comment 28 Arjan van de Ven 2003-04-03 21:55:16 UTC
assigning back to rpm; the kernel behaves as per specification; rpm does
assumptions beyond and incompatible with the specification.

Comment 29 Jeff Johnson 2003-04-14 15:09:49 UTC
Heh, "kernel specifications" my ass.

O_DIRECT usage is removed in rpm-4.0.5, rpm-4.1.1 and rpm-4.2 packages,
currently available at
    ftp://ftp.rpm.org/pub/rpm/test-{4.1.1,4.2}

Note carefully that the packages have been rebuilt with the same version.

Install manually as before, then reinstall using rpm -Uvh, if
necessary.

Reopen this bug if you still have problems.

Comment 30 Stephen Rasku 2003-05-23 20:00:17 UTC
Finally got around to testing this.  Looks good.


Note You need to log in before you can comment on or make changes to this bug.