Bug 438675 - rpmbuild segfaults on huge directory tree
Summary: rpmbuild segfaults on huge directory tree
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: rpm
Version: 4.8
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Panu Matilainen
QA Contact: Petr Sklenar
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2008-03-24 12:00 UTC by Adam Stokes
Modified: 2018-10-20 00:22 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2009-05-18 20:04:32 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
reproducer (761 bytes, text/x-rpm-spec)
2008-03-24 12:00 UTC, Adam Stokes
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2009:0951 0 normal SHIPPED_LIVE rpm bug fix update 2009-05-18 13:23:21 UTC

Description Adam Stokes 2008-03-24 12:00:29 UTC
Description of problem:
when asked to package a huge directory tree that contains lots of files with the
same name, rpmbuild will eventually segfault in the "Processing files" stage.

We have a test case, which gives:

rpmbuild -ba .../
...
+ cd /var/tmp/test-root
+ python /var/tmp/test-root/createDirs.py
Processing files: test-0-0

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 182894245600 (LWP 1567)]
0x000000342142e21c in compressFilelist (h=0x527120) at legacy.c:355
355             baseName = strrchr(fileNames[i], '/') + 1;
(gdb) bt
#0  0x000000342142e21c in compressFilelist (h=0x527120) at legacy.c:355
#1  0x0000003421a0c1f6 in genCpioListAndHeader (fl=0x7fbfffb370, fip=0x518e70,
   h=0x527120, isSrc=0) at files.c:1342
#2  0x0000003421a0e500 in processBinaryFiles (spec=0x5282a0,
   installSpecialDoc=4, test=0) at files.c:2156
#3  0x0000003421a08a5c in buildSpec (ts=0x527e90, spec=0x5282a0, what=223,
   test=0) at build.c:349
#4  0x0000000000402e18 in ?? ()
#5  0x00000000004035b1 in ?? ()
#6  0x0000000000404157 in ?? ()
#7  0x000000342071c3fb in __libc_start_main () from /lib64/tls/libc.so.6
#8  0x0000000000402ada in ?? ()
#9  0x0000007fbffff938 in ?? ()
#10 0x000000000000001c in ?? ()
#11 0x0000000000000004 in ?? ()
#12 0x0000007fbffffb61 in ?? ()
#13 0x0000007fbffffb73 in ?? ()
#14 0x0000007fbffffb77 in ?? ()
#15 0x0000007fbffffb7b in ?? ()
#16 0x0000000000000000 in ?? ()
(gdb) print i
$1 = 0
(gdb) print fileNames
$2 = (char **) 0x2aace97010
(gdb) print fileNames[i]
$3 = 0x2aadcdd010 "/src"
(gdb)

Notes:

* memory usage increases slowly while the program runs, to several 100MB
* trying to trigger the issue via a lower "ulimit -v" at 100MB steps: 100-400MB
are handled gracefully, 500MB shows the issue but brings no significant speedup.

* symbols above are as on RHEL4.6/x86_64, rpm-build-4.3.3-23_nonptl

* The use of python in the .spec is just to speed up directory creation, the bug
equally happens on a prepopulated tree, or via shell. Directory tree
deletion/creation of course takes ages, since on ext3...

* this actually gets triggered in production for one of our experiments build
system, where the use of same-name files cannot be avoided easily.

Version-Release number of selected component (if applicable):
rpm-build-4.3.3-23_nonptl

How reproducible:
100%

Steps to Reproduce:
1. see attached spec file
2.
3.
  
Actual results:
Segfault

Expected results:
No segfault

Additional info:
https://lists.dulug.duke.edu/pipermail/rpm-maint/2007-October/000553.html

Comment 1 Adam Stokes 2008-03-24 12:00:30 UTC
Created attachment 298882 [details]
reproducer

Comment 2 Panu Matilainen 2008-03-25 05:29:08 UTC
Yup, fixed upstream already:
https://lists.dulug.duke.edu/pipermail/rpm-maint/2007-December/000652.html

Easy to backport, and needed for RHEL 5 too.

Comment 3 RHEL Program Management 2008-09-05 17:24:06 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Comment 5 Jindrich Novy 2008-10-16 12:25:26 UTC
Fix for this issue is now applied in CVS.

Comment 7 Petr Sklenar 2009-01-29 11:48:40 UTC
Hello,
old version segfaulting like in Comment #0 but new version doesn't work neither. It has something to common with bug: 

Bug 462539 -  Unable to create immutable header region. When building 110MB rpm from simple tgz
--This^ bug couldn't be fixed, bug 462539#c6

I used reproducer spec file:

# rpmbuild -ba huge-dir-tree.spec

Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.41642
+ umask 022
+ cd /usr/src/redhat/BUILD
+ LANG=C
+ export LANG
+ unset DISPLAY
+ exit 0
Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.41642
+ umask 022
+ cd /usr/src/redhat/BUILD
+ LANG=C
+ export LANG
+ unset DISPLAY
+ exit 0
Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.41642
+ umask 022
+ cd /usr/src/redhat/BUILD
+ LANG=C
+ export LANG
+ unset DISPLAY
+ rm -rf /var/tmp/huge-dir-tree-root
+ '[' '!' -d /var/tmp/huge-dir-tree-root/ ']'
+ mkdir -p /var/tmp/huge-dir-tree-root/
+ cat
+ cd /var/tmp/huge-dir-tree-root
+ python /var/tmp/huge-dir-tree-root/createDirs.py
Processing files: huge-dir-tree-0-0
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Checking for unpackaged file(s): /usr/lib/rpm/check-files /var/tmp/huge-dir-tree-root
Wrote: /usr/src/redhat/SRPMS/huge-dir-tree-0-0.src.rpm
error: Unable to create immutable header region.


RPM build errors:
    Unable to create immutable header region.

---------------

what's the fix for this bug:
not segfaulting but another issue
or
successful built - which couldn't be done ?

Comment 8 Petr Sklenar 2009-01-30 12:30:13 UTC
I was able to built hugedirtree with lower amount of files - 130 000, but then I wasn't able to uninstalled that package, filled a bug 483235

Comment 9 Petr Sklenar 2009-02-04 15:50:19 UTC
Is it possible to test it by customer? Please can you provide customer latest packages for RHEL4.8?

Comment 10 J.H.M. Dassen (Ray) 2009-02-25 09:51:59 UTC
The customer has been provided with test packages and has provided the
following feedback:
	I confirm that rpm-4.3.3-32 addresses our reproducer. However, the
	new "rpm-devel-4.3.3-32_nonptl.i386" RPM has picked up two new
	dependencies, on beecrypt-devel and elfutils-libelf-devel - these
	might be packaging errors (why would the executable depend on some
	header files?)

Comment 11 Panu Matilainen 2009-02-25 10:03:07 UTC
The executable doesn't require headers, and the executable also doesn't come from rpm-devel package (whose dependencies are correct now) but rpm-build package :)

Comment 12 Issue Tracker 2009-02-26 14:26:56 UTC
Hello Jan,

You wrote:
> I confirm that rpm-4.3.3-32 addresses our reproducer.

That's good to know. Thank you for your feedback. 

> However, the new "rpm-devel-4.3.3-32_nonptl.i386" RPM has picked up
two new
> dependencies, on beecrypt-devel and elfutils-libelf-devel - these might
be
> packaging errors (why would the executable depend on some header
files?)

Erm, the rpm executable is in the plain "rpm" package (not in
"rpm-devel"),
and that package does not depend on any -devel packages. 

As for the rpm-devel package depending on other -devel packages, there's
a
change here indeed. Having a -devel package depending on another -devel
package isn't surprising to me (after all, in order to use the devel
package
to build code, you generally need to have the headers of libraries it
depends
on installed), but perhaps that's my Debian bias talking - I haven't
done
enough RPM building to know what the consensus is in RPM land.

Are you OK with writing this off as an artefact of the test packages, or
do
you want me to dig deeper here?

Kind regards,
Ray


Category set to: Devel Tools
Internal Status set to 'Waiting on Customer'
Status set to: Waiting on Client
Interim resolution set to 'Custom Package'

This event sent from IssueTracker by rdassen 
 issue 141055

Comment 13 Chris Ward 2009-03-13 14:02:41 UTC
~~ Attention Partners!  ~~
RHEL 4.8Beta has been released on partners.redhat.com. There should
be a fix present, which addresses this bug. Please test and report back results on this OtherQA Partner bug at your earliest convenience.

If you encounter any issues, please set the bug back to the ASSIGNED state and
describe any issues you encountered. If you have found a NEW bug, clone this bug and describe the issues you've encountered. Further questions can be directed to your Red Hat Partner Manager.

If you have VERIFIED the bug fix. Please select your PartnerID from the Verified field above. Please leave a comment with your test results details. Include which arches tested, package version and any applicable logs.

 - Red Hat QE Partner Management

Comment 14 Chris Ward 2009-03-25 12:26:46 UTC
Setting to verified based on Customer Verification results in comment #10. 

If there are any additional issues that need to be addressed, please clone this bug and make a new request. If it is found that this bug has not really been resolved, please reset to ASSIGNED state and describe the issues you are encountering.

Comment 17 errata-xmlrpc 2009-05-18 20:04:32 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-0951.html


Note You need to log in before you can comment on or make changes to this bug.