I ran createrepo on a file list with 30k RPMs. It failed with : Traceback (most recent call last): File "/usr/share/createrepo/genpkgmetadata.py", line 278, in <module> main(sys.argv[1:]) File "/usr/share/createrepo/genpkgmetadata.py", line 252, in main mdgen.doPkgMetadata() File "/usr/lib/python2.7/site-packages/createrepo/__init__.py", line 396, in doPkgMetadata self.writeMetadataDocs(packages) File "/usr/lib/python2.7/site-packages/createrepo/__init__.py", line 631, in writeMetadataDocs stderr=subprocess.PIPE) File "/usr/lib64/python2.7/subprocess.py", line 672, in __init__ errread, errwrite) File "/usr/lib64/python2.7/subprocess.py", line 1202, in _execute_child raise child_exception OSError: [Errno 7] Argument list too long The traceback wasn't actually thrown, I added 'raise' to createrepo code to see what happened there. Wouldn't be possible to feed /usr/share/createrepo/worker.py in a different way? It could read file list from stdin, for example.
I can probably make a tempfile per worker and let them read the pkgs out of that. It means less back-and-forth via stdin and less likelihood for the pipe interface of python to do something horrible. I'll have a patch today - do you think you can test it on your 30K pkgs case to see if it makes everything happy?
patch push upstream http://createrepo.baseurl.org/gitweb?p=createrepo.git;a=commitdiff;h=fcb7fa19a488fd60c27b912fe2bbc16b614decb8
(In reply to comment #1) > I'll have a patch today - do you think you can test it on your 30K pkgs case to > see if it makes everything happy? Thanks for the patch. I'll test it either tomorrow morning or early next week.
It fails on undefined self._worker_tmp_path. You should use worker_tmp_path instead. btw, I'm getting only this message: Spawning worker 0 with 29588 pkgs Worker 0: reading packages/fakechroot/2.9/24.5.el6/i686/fakechroot-libs-2.9-24.5.el6.i686.rpm Is it intentional? I'd expect all processed packages to be printed. Createrepo seems to be stuck, but it's too early to confirm that. I'll send you more details later (or create a new bug since it looks like another issue).
Did you apply the patch ontop of what's in f16/rawhide or did you take the whole snapshot?
I see... I applied patch on the latest F16 build. That explains why it didn't work.
This message is a notice that Fedora 15 is now at end of life. Fedora has stopped maintaining and issuing updates for Fedora 15. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At this time, all open bugs with a Fedora 'version' of '15' have been closed as WONTFIX. (Please note: Our normal process is to give advanced warning of this occurring, but we forgot to do that. A thousand apologies.) Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, feel free to reopen this bug and simply change the 'version' to a later Fedora version. Bug Reporter: Thank you for reporting this issue and we are sorry that we were unable to fix it before Fedora 15 reached end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged to click on "Clone This Bug" (top right of this page) and open it against that version of Fedora. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping