Description of problem: I packaged a circuit simulation program by Mentor Graphics for distribution by our RHN server at our university. The package deploys and installs, but when I try to erase or upgrade it, rpm crashes with something like: memory alloc (2116098924 bytes) returned NULL. The last few lines from rpm -vv -e are: D: sanity checking 1 elements D: computing 88749 file fingerprints D: computing file dispositions D: opening db index /var/lib/rpm/Basenames create mode=0x42 memory alloc (2116098924 bytes) returned NULL. Version-Release number of selected component (if applicable): RHEL 4, RPM version 4.3.3 How reproducible: Similar errors occur on systems with 512MB or 1GB ram installed; Comments: Yes, the package has 88749 file entries. 14978 of them are directories. I have had success with another Mentor program that had 61650 files, 4078 of which were directories. Thanks, Dirk
Got a ptr to the package so that I can reproduce the problem?
Created attachment 126491 [details] An rpm with 88700 empty files I have been able to reproduce the bug with this rpm package. It is the same as the original, except all of the files are replaced with a zero-length file.
Thanks for the package. I've reproduced, digging now ...
jbj@wellfleet rpm]$ sudo rpm -evv manyfiles D: opening db environment /var/lib/rpm/Packages cdb:mpool:joinenv D: opening db index /var/lib/rpm/Packages rdonly mode=0x0 D: locked db index /var/lib/rpm/Packages D: opening db index /var/lib/rpm/Name rdonly mode=0x0 D: read h# 7699 Header SHA1 digest: OK (d14c51732731cff28ed397b4d5f0018ae342c90c) D: ========== --- manyfiles-1.1-1 i386/linux 0x0 D: opening db index /var/lib/rpm/Requirename rdonly mode=0x0 D: ========== recording tsort relations D: ========== tsorting packages (order, #predecessors, #succesors, tree, Ldepth, Rbreadth) D: 0 0 0 1 0 0 -manyfiles-1.1-1.i386 D: closed db index /var/lib/rpm/Requirename D: closed db index /var/lib/rpm/Name D: closed db index /var/lib/rpm/Packages D: closed db environment /var/lib/rpm/Packages D: opening db environment /var/lib/rpm/Packages cdb:mpool:joinenv D: opening db index /var/lib/rpm/Packages create mode=0x42 D: mounted filesystems: D: i dev bsize bavail iavail mount point D: 0 0x00000302 4096 376558 8485949 rw / D: 1 0x00000003 4096 0 -1 rw /proc D: 2 0x00000000 4096 0 -1 rw /sys D: 3 0x0000000b 4096 0 -1 rw /dev/pts D: 4 0x00000301 1024 66686 26034 rw /boot D: 5 0x00000012 4096 128959 128958 rw /dev/shm D: 6 0x00000805 4096 1399597 13533670 rw /S D: 7 0x00000013 4096 0 -1 rw /proc/sys/fs/binfmt_misc D: 8 0x00000014 4096 0 -1 rw /var/lib/nfs/rpc_pipefs D: 9 0x00000015 4096 0 -1 rw /net D: 10 0x00000100 1024 128 15 rw /home/irlp/local D: sanity checking 1 elements D: running pre-transaction scripts D: computing 88749 file fingerprints D: computing file dispositions D: opening db index /var/lib/rpm/Basenames create mode=0x42 memory alloc (1871229540 bytes) returned NULL.
OK, you have many many duplicated basenames: [root@wellfleet tmp]# rpm -qp --qf '[%{basenames}\n]' manyfiles-1.1-1.i386.rpm | wc 88749 88754 1150966 [root@wellfleet tmp]# time rpm -qp --qf '[%{basenames}\n]' manyfiles-1.1-1.i386.rpm | sort -u | wc 26741 26746 483313 real 9m26.862s user 5m4.939s sys 3m45.778s That's the longest package query I've ever seen as well ;-) There's a possible quick and dirty fix to rpm to handle this number of duplicated basenames. Checking ...
... yah, this quick and dirty patch permits rpm -evv manyfiles to succeed with a memory footprint of ~52Mb: Index: rpmdb/rpmdb.c =============================================================== ==== RCS file: /cvs/devel/rpm/rpmdb/rpmdb.c,v retrieving revision 1.120.2.14 diff -u -b -B -w -p -r1.120.2.14 rpmdb.c --- rpmdb/rpmdb.c 17 Mar 2006 21:23:39 -0000 1.120.2.14 +++ rpmdb/rpmdb.c 23 Mar 2006 02:45:25 -0000 @@ -3386,6 +3386,7 @@ static struct skipDir_s { _skip("/usr/lib/locale"), _skip("/usr/src"), _skip("/lib/modules"), + _skip("/opt/manyfiles"), { 0, NULL } }; If you *must* deal with a package containing this number of files with duplicated basenames, then I suggest patching rpm, replacing "/opt/manyfiles" as appropriate for your actual install path. rpm was not designed to scale for 88749 files ...
FWIW, my test was with rpm-4.4.6-0.7, but the RHEL 4 rpm has the same problem, and I expect a similar quick and dirty patch will "fix".
Thanks for looking into it, Jeff. I decided to just change the spec file so that it uses the unfavored appoach of uncompressing a tarball in the %post script. My main goal is for easy distribution with our RHN server. -Dirk
WORKSFORME.
This bug is likel fixed (or at least alleviated) in rpm cvs, will be in rpm-4.4.9-0.3 when built. UPSTREAM
I also met such issue when installing and uninstalling large packages with many files, I still don't understand why it need so much memory (about 900M). How can I verify it now? is there any patches available before official release?
*** This bug has been marked as a duplicate of bug 484094 ***