Bug 734838 - createrepo crashes on long file lists
Summary: createrepo crashes on long file lists
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Fedora
Classification: Fedora
Component: createrepo
Version: 15
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Luke Macken
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 578256
TreeView+ depends on / blocked
 
Reported: 2011-08-31 15:17 UTC by Daniel Mach
Modified: 2016-09-20 02:42 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-08-07 16:54:00 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Daniel Mach 2011-08-31 15:17:12 UTC
I ran createrepo on a file list with 30k RPMs.
It failed with :
Traceback (most recent call last):
  File "/usr/share/createrepo/genpkgmetadata.py", line 278, in <module>
    main(sys.argv[1:])
  File "/usr/share/createrepo/genpkgmetadata.py", line 252, in main
    mdgen.doPkgMetadata()
  File "/usr/lib/python2.7/site-packages/createrepo/__init__.py", line 396, in doPkgMetadata
    self.writeMetadataDocs(packages)
  File "/usr/lib/python2.7/site-packages/createrepo/__init__.py", line 631, in writeMetadataDocs
    stderr=subprocess.PIPE)
  File "/usr/lib64/python2.7/subprocess.py", line 672, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1202, in _execute_child
    raise child_exception
OSError: [Errno 7] Argument list too long

The traceback wasn't actually thrown, I added 'raise' to createrepo code to see what happened there.

Wouldn't be possible to feed /usr/share/createrepo/worker.py in a different way?
It could read file list from stdin, for example.

Comment 1 seth vidal 2011-08-31 16:36:21 UTC
I can probably make a tempfile per worker and let them read the pkgs out of that. It means less back-and-forth via stdin and less likelihood for the pipe interface of python to do something horrible.

I'll have a patch today - do you think you can test it on your 30K pkgs case to see if it makes everything happy?

Comment 3 Daniel Mach 2011-08-31 18:54:47 UTC
(In reply to comment #1)
> I'll have a patch today - do you think you can test it on your 30K pkgs case to
> see if it makes everything happy?

Thanks for the patch.
I'll test it either tomorrow morning or early next week.

Comment 4 Daniel Mach 2011-09-01 06:45:59 UTC
It fails on undefined self._worker_tmp_path.
You should use worker_tmp_path instead.


btw, I'm getting only this message:
Spawning worker 0 with 29588 pkgs
Worker 0: reading packages/fakechroot/2.9/24.5.el6/i686/fakechroot-libs-2.9-24.5.el6.i686.rpm

Is it intentional? I'd expect all processed packages to be printed.
Createrepo seems to be stuck, but it's too early to confirm that. I'll send you more details later (or create a new bug since it looks like another issue).

Comment 5 seth vidal 2011-09-01 13:08:13 UTC
Did you apply the patch ontop of what's in f16/rawhide or did you take the whole snapshot?

Comment 6 Daniel Mach 2011-09-03 15:31:06 UTC
I see... I applied patch on the latest F16 build. That explains why it didn't work.

Comment 7 Fedora End Of Life 2012-08-07 16:54:02 UTC
This message is a notice that Fedora 15 is now at end of life. Fedora
has stopped maintaining and issuing updates for Fedora 15. It is
Fedora's policy to close all bug reports from releases that are no
longer maintained. At this time, all open bugs with a Fedora 'version'
of '15' have been closed as WONTFIX.

(Please note: Our normal process is to give advanced warning of this
occurring, but we forgot to do that. A thousand apologies.)

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, feel free to reopen
this bug and simply change the 'version' to a later Fedora version.

Bug Reporter: Thank you for reporting this issue and we are sorry that
we were unable to fix it before Fedora 15 reached end of life. If you
would still like to see this bug fixed and are able to reproduce it
against a later version of Fedora, you are encouraged to click on
"Clone This Bug" (top right of this page) and open it against that
version of Fedora.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

The process we are following is described here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping


Note You need to log in before you can comment on or make changes to this bug.