Bug 185694 - rpm --erase fails with memory alloc returned NULL on custom package
rpm --erase fails with memory alloc returned NULL on custom package
Status: CLOSED DUPLICATE of bug 484094
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: rpm (Show other bugs)
4.0
i386 Linux
medium Severity medium
: ---
: ---
Assigned To: Panu Matilainen
Mike McLean
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2006-03-16 17:23 EST by Dirk Robinson
Modified: 2010-02-04 04:17 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-02-04 04:17:30 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
An rpm with 88700 empty files (9.07 MB, application/octet-stream)
2006-03-22 13:15 EST, Dirk Robinson
no flags Details

  None (edit)
Description Dirk Robinson 2006-03-16 17:23:26 EST
Description of problem:
I packaged a circuit simulation program by Mentor Graphics for distribution by
our RHN server at our university. The package deploys and installs, but when I
try to erase or upgrade it, rpm crashes with something like:
memory alloc (2116098924 bytes) returned NULL.

The last few lines from rpm -vv -e are:
D: sanity checking 1 elements
D: computing 88749 file fingerprints
D: computing file dispositions
D: opening  db index       /var/lib/rpm/Basenames create mode=0x42
memory alloc (2116098924 bytes) returned NULL.

Version-Release number of selected component (if applicable):
RHEL 4, RPM version 4.3.3

How reproducible:
Similar errors occur on systems with 512MB or 1GB ram installed;

Comments:
Yes, the package has 88749 file entries. 14978 of them are directories. I have
had success with another Mentor program that had 61650 files, 4078 of which were
directories.

Thanks,
Dirk
Comment 1 Jeff Johnson 2006-03-17 20:19:19 EST
Got a ptr to the package so that I can reproduce the problem?
Comment 2 Dirk Robinson 2006-03-22 13:16:01 EST
Created attachment 126491 [details]
An rpm with 88700 empty files

I have been able to reproduce the bug with this rpm package. It is the same as
the original, except all of the files are replaced with a zero-length file.
Comment 3 Jeff Johnson 2006-03-22 21:20:38 EST
Thanks for the package. I've reproduced, digging now ...
Comment 4 Jeff Johnson 2006-03-22 21:21:43 EST
jbj@wellfleet rpm]$ sudo rpm -evv manyfiles
D: opening  db environment /var/lib/rpm/Packages cdb:mpool:joinenv
D: opening  db index       /var/lib/rpm/Packages rdonly mode=0x0
D: locked   db index       /var/lib/rpm/Packages
D: opening  db index       /var/lib/rpm/Name rdonly mode=0x0
D:  read h#    7699 Header SHA1 digest: OK (d14c51732731cff28ed397b4d5f0018ae342c90c)
D: ========== --- manyfiles-1.1-1 i386/linux 0x0
D: opening  db index       /var/lib/rpm/Requirename rdonly mode=0x0
D: ========== recording tsort relations
D: ========== tsorting packages (order, #predecessors, #succesors, tree, Ldepth, Rbreadth)
D:     0    0    0    1    0    0 -manyfiles-1.1-1.i386
D: closed   db index       /var/lib/rpm/Requirename
D: closed   db index       /var/lib/rpm/Name
D: closed   db index       /var/lib/rpm/Packages
D: closed   db environment /var/lib/rpm/Packages
D: opening  db environment /var/lib/rpm/Packages cdb:mpool:joinenv
D: opening  db index       /var/lib/rpm/Packages create mode=0x42
D: mounted filesystems:
D:     i        dev    bsize       bavail       iavail mount point
D:     0 0x00000302     4096       376558      8485949 rw /
D:     1 0x00000003     4096            0           -1 rw /proc
D:     2 0x00000000     4096            0           -1 rw /sys
D:     3 0x0000000b     4096            0           -1 rw /dev/pts
D:     4 0x00000301     1024        66686        26034 rw /boot
D:     5 0x00000012     4096       128959       128958 rw /dev/shm
D:     6 0x00000805     4096      1399597     13533670 rw /S
D:     7 0x00000013     4096            0           -1 rw /proc/sys/fs/binfmt_misc
D:     8 0x00000014     4096            0           -1 rw /var/lib/nfs/rpc_pipefs
D:     9 0x00000015     4096            0           -1 rw /net
D:    10 0x00000100     1024          128           15 rw /home/irlp/local
D: sanity checking 1 elements
D: running pre-transaction scripts
D: computing 88749 file fingerprints
D: computing file dispositions
D: opening  db index       /var/lib/rpm/Basenames create mode=0x42
memory alloc (1871229540 bytes) returned NULL.
Comment 5 Jeff Johnson 2006-03-22 21:32:15 EST
OK, you have many many duplicated basenames:
[root@wellfleet tmp]# rpm -qp --qf '[%{basenames}\n]' manyfiles-1.1-1.i386.rpm | wc
  88749   88754 1150966
[root@wellfleet tmp]# time rpm -qp --qf '[%{basenames}\n]' manyfiles-1.1-1.i386.rpm | sort -u | wc
  26741   26746  483313

real    9m26.862s
user    5m4.939s
sys     3m45.778s

That's the longest package query I've ever seen as well ;-)

There's a possible quick and dirty fix to rpm to handle this number of duplicated basenames.

Checking ...
Comment 6 Jeff Johnson 2006-03-22 21:43:29 EST
... yah, this quick and dirty patch permits
    rpm -evv manyfiles
to succeed with a memory footprint of ~52Mb:
Index: rpmdb/rpmdb.c
===============================================================
====
RCS file: /cvs/devel/rpm/rpmdb/rpmdb.c,v
retrieving revision 1.120.2.14
diff -u -b -B -w -p -r1.120.2.14 rpmdb.c
--- rpmdb/rpmdb.c       17 Mar 2006 21:23:39 -0000      1.120.2.14
+++ rpmdb/rpmdb.c       23 Mar 2006 02:45:25 -0000
@@ -3386,6 +3386,7 @@ static struct skipDir_s {
     _skip("/usr/lib/locale"),
     _skip("/usr/src"),
     _skip("/lib/modules"),
+    _skip("/opt/manyfiles"),
     { 0, NULL }
 };
 

If you *must* deal with a package containing this number of files with duplicated basenames,
then I suggest patching rpm, replacing "/opt/manyfiles" as appropriate for your actual
install path. rpm was not designed to scale for 88749 files ...
Comment 7 Jeff Johnson 2006-03-22 21:45:08 EST
FWIW, my test was with rpm-4.4.6-0.7, but the RHEL 4 rpm has the same problem,
and I expect a similar quick and dirty patch will "fix".
Comment 8 Dirk Robinson 2006-03-23 16:57:16 EST
Thanks for looking into it, Jeff. I decided to just change the spec file so that
it uses the unfavored appoach of uncompressing a tarball in the %post script. My
main goal is for easy distribution with our RHN server.
-Dirk
Comment 9 Jeff Johnson 2006-04-04 07:52:36 EDT
WORKSFORME.
Comment 10 Jeff Johnson 2007-04-01 15:34:06 EDT
This bug is likel fixed (or at least alleviated) in rpm cvs, will be in rpm-4.4.9-0.3 when built.

UPSTREAM
Comment 11 Harry He 2009-05-15 04:24:51 EDT
I also met such issue when installing and uninstalling large packages with many files, I still don't understand why it need so much memory (about 900M). 

How can I verify it now? is there any patches available before official release?
Comment 12 Panu Matilainen 2010-02-04 04:17:30 EST

*** This bug has been marked as a duplicate of bug 484094 ***

Note You need to log in before you can comment on or make changes to this bug.