Bug 975220 - createrepo 0.9.9 fails with MemoryError
createrepo 0.9.9 fails with MemoryError
Status: CLOSED INSUFFICIENT_DATA
Product: Beaker
Classification: Community
Component: scheduler (Show other bugs)
0.12
Unspecified Unspecified
unspecified Severity high (vote)
: ---
: ---
Assigned To: Dan Callaghan
tools-bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-17 16:27 EDT by Stefanie Forrester
Modified: 2014-08-21 20:42 EDT (History)
13 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-08-21 20:42:57 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Stefanie Forrester 2013-06-17 16:27:24 EDT
Description of problem:
Any time beaker runs createrepo, it creates temporary files in /var/tmp which are not cleaned up. The files accumulate until the system disk is full, and beaker ceases to function. Downgrading to createrepo-0.9.8-4.el6.noarch has mitigated the problem for now.

Version-Release number of selected component (if applicable):
createrepo-0.9.9-17.el6.noarch, createrepo-0.9.9-18.el6.noarch, beaker-0.12.1-1.el6eng.noarch

How reproducible:
Every time.

Steps to Reproduce:
1. Update to a newer version of createrepo (newer than 0.9.8-4).
2. Run beakerd, watch /var/tmp for files.

Actual results:
/var/tmp fills with temporary files until the disk is 100% full.

Expected results:
/var/tmp should get cleaned up periodically. 

Additional info:
Comment 1 Dan Callaghan 2013-06-17 20:30:59 EDT
If createrepo is leaking files to /var/tmp for no reason then it's really a createrepo bug. But I will look into this further.
Comment 2 Dan Callaghan 2013-06-17 22:35:15 EDT
It seems like the actual problem is that createrepo was using too much memory and bailing out without cleaning up its temp files in /var/tmp. So the disk filling up was just a side-effect of that.

2013-06-17 11:50:37,757 bkr.server.xmlrpccontroller ERROR Error handling XML-RPC method
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/bkr/server/xmlrpccontroller.py", line 54, in RPC2
    response = self.process_rpc(method,params)
  File "/usr/lib/python2.6/site-packages/bkr/server/xmlrpccontroller.py", line 43, in process_rpc
    response = obj(*params)
  File "<string>", line 3, in upload
  File "/usr/lib/python2.6/site-packages/turbogears/identity/conditions.py", line 249, in require
    return fn(self, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/bkr/server/tasks.py", line 205, in upload
    Task.update_repo()
  File "/usr/lib/python2.6/site-packages/bkr/server/model.py", line 6351, in update_repo
    % (retcode, output))
ValueError: createrepo failed with exit status 1:
Traceback (most recent call last):
  File "/usr/share/createrepo/genpkgmetadata.py", line 291, in <module>
    main(sys.argv[1:])
  File "/usr/share/createrepo/genpkgmetadata.py", line 265, in main
    mdgen.doPkgMetadata()
  File "/usr/lib/python2.6/site-packages/createrepo/__init__.py", line 412, in doPkgMetadata
    self.writeMetadataDocs(packages)
  File "/usr/lib/python2.6/site-packages/createrepo/__init__.py", line 567, in writeMetadataDocs
    self.flfile.write(old_po.xml_dump_filelists_metadata())
  File "/usr/lib/python2.6/site-packages/yum/packages.py", line 1215, in xml_dump_filelists_metadata
    msg += misc.to_unicode(self._dump_files())
MemoryError
Comment 3 Dan Callaghan 2013-06-18 03:28:08 EDT
Our production Beaker instance has 20382 task RPMs totalling 2.3GB. I tried experimenting with running createrepo on RHEL6 against a copy of the task RPMs from production, using RLIMIT_AS 1500000000:

ulimit -v 1464843
/usr/bin/time createrepo -q --update --checksum sha .

With createrepo-0.9.8-4.el6.noarch:

13.28user 0.57system 0:14.57elapsed 95%CPU (0avgtext+0avgdata 2757840maxresident)k
4560inputs+17360outputs (0major+173449minor)pagefaults 0swaps

With createrepo-0.9.9-17.el6.noarch:

88.45user 2.73system 1:38.56elapsed 92%CPU (0avgtext+0avgdata 865264maxresident)k
334392inputs+326152outputs (0major+87653minor)pagefaults 0swaps

So with createrepo 0.9.9 the memory usage appears to actually be *lower*, but that's just because the real work is done in a worker process now. I'm not sure of a good way to find the max resident size of the worker process.

But neither version is hitting MemoryError in my environment. (Note that GNU time has a bug [1] where the max resident size is misreported by a factor of 4 so the max resident size for createrepo 0.9.8 is 673MB.) So I'm not sure why it would be using more memory when it runs in the production environment.

[1] https://groups.google.com/forum/?fromgroups#!topic/gnu.utils.help/u1MOsHL4bhg
Comment 4 Dan Callaghan 2013-06-18 07:04:56 EDT
I have had absolutely no success in reproducing this MemoryError from createrepo, even when run inside the Beaker application in mod_wsgi with a copy of the task library from our production Beaker.

I can run createrepo 0.9.9 successfully with ulimit -v as low as 520000. At 500000 I get the following stack trace which looks completely different than the one above:

Traceback (most recent call last):
  File "/usr/share/createrepo/genpkgmetadata.py", line 291, in <module>
    main(sys.argv[1:])
  File "/usr/share/createrepo/genpkgmetadata.py", line 269, in main
    mdgen.doRepoMetadata()
  File "/usr/lib/python2.6/site-packages/createrepo/__init__.py", line 1003, in doRepoMetadata
    compressFile(resultpath, result_compressed, compress_type)
  File "/usr/lib/python2.6/site-packages/createrepo/utils.py", line 108, in compressFile
    bzipFile(source, dest)
  File "/usr/lib/python2.6/site-packages/createrepo/utils.py", line 58, in bzipFile
    destination = bz2.BZ2File(dest, 'w', compresslevel=9)
MemoryError

Also, Beaker was running happily for several days with createrepo 0.9.9 (including accept new task uploads into the task library) before things first went bad at 2013-06-17 08:30:36 UTC.

So there must have been some other complicating factor which was causing createrepo to get MemoryError. I'm out of ideas though.
Comment 5 David Kutálek 2013-06-18 08:35:09 EDT
Dan, I met this problem a few days sooner - on Saturday June 15 at about 6 UTC:

php-CoreOS-php-security-CVE-2007-1285-1.5-3.noarch.rpm
Exception: <Fault 1: '<type \'exceptions.ValueError\'>:createrepo failed with exit status 1:\nTraceback (most recent call last):\n  File "/usr/share/createrepo/genpkgmetadata.py", line 291, in <module>\n    main(sys.argv[1:])\n  File "/usr/share/createrepo/genpkgmetadata.py", line 265, in main\n    mdgen.doPkgMetadata()\n  File "/usr/lib/python2.6/site-packages/createrepo/__init__.py", line 412, in doPkgMetadata\n    self.writeMetadataDocs(packages)\n  File "/usr/lib/python2.6/site-packages/createrepo/__init__.py", line 567, in writeMetadataDocs\n    self.flfile.write(old_po.xml_dump_filelists_metadata())\n  File "/usr/lib/python2.6/site-packages/yum/packages.py", line 1215, in xml_dump_filelists_metadata\n    msg += misc.to_unicode(self._dump_files())\nMemoryError\n'>
make: *** [bkradd] Error 1


I was doing make bkradd (ie adding task package) for 38 tests, last 4 failed with this error. About an hour or two later, I tried two of them once more manually and it again failed. 

Same day about 10 hours later (about 16 UTC) I tried loading beaker web gui but got repeatedly 500 Internal server error - not sure whether this is connected issue though. I decided to not file a ticket/bug because of weekend :-o.
Comment 6 Lalatendu Mohanty 2013-06-18 09:36:52 EDT
I am too getting this create repo issue along with "500 Internal server error -"

+ '[' -d /home/msvbhat/rpmbuild/BUILDROOT/rhs-tests-rhs-tests-beaker-rhs-gluster-qe-libs-dev-vbhat-1.32.9-0.x86_64 ']'
+ rm -rf /home/msvbhat/rpmbuild/BUILDROOT/rhs-tests-rhs-tests-beaker-rhs-gluster-qe-libs-dev-vbhat-1.32.9-0.x86_64
+ exit 0
/mnt/testarea/rhts-build-CflehXkC/install /work/rhs-tests/beaker/rhs/gluster-qe-libs
/home/msvbhat/.beaker_client/config not found, using /etc/beaker/client.conf
rhs-tests-rhs-tests-beaker-rhs-gluster-qe-libs-dev-vbhat-1.32.9-0.noarch.rpm
Exception: <Fault 1: '<type \'exceptions.ValueError\'>:createrepo failed with exit status 1:\nTraceback (most recent call last):\n  File "/usr/share/createrepo/genpkgmetadata.py", line 291, in <module>\n    main(sys.argv[1:])\n  File "/usr/share/createrepo/genpkgmetadata.py", line 265, in main\n    mdgen.doPkgMetadata()\n  File "/usr/lib/python2.6/site-packages/createrepo/__init__.py", line 412, in doPkgMetadata\n    self.writeMetadataDocs(packages)\n  File "/usr/lib/python2.6/site-packages/createrepo/__init__.py", line 567, in writeMetadataDocs\n    self.flfile.write(old_po.xml_dump_filelists_metadata())\n  File "/usr/lib/python2.6/site-packages/yum/packages.py", line 1215, in xml_dump_filelists_metadata\n    msg += misc.to_unicode(self._dump_files())\nMemoryError\n'>
make: *** [bkradd] Error 1


Please note that we are in heavy development of beaker tasks and this greatly affects our ability to develop automation.  If the logs are filling up the system can we be sure to implement a workaround that doesn't affect our ability to add/update packages
Comment 7 Lalatendu Mohanty 2013-06-18 09:55:30 EDT
Just now I tried to upload again a new package and it worked fine. May be somebody cleaned temp files in /var/tmp in production, so it worked. Thanks
Comment 8 Stefanie Forrester 2013-06-18 12:03:48 EDT
Dan, so you were unable to duplicate this issue on another beaker instance? Do you think this is an environment-specific error, rather than a createrepo bug? I'll do some testing on stage and possibly even set up another beaker instance just to compare. It'll take time though... we have a lot of other priorities, and a temporary fix is in place.

Lalatendu, I cleaned up the files yesterday and downgraded the createrepo package. It's stable now and shouldn't cause any more problems. That'll hold while we figure out the root cause of the issue.
Comment 9 Dan Callaghan 2013-06-18 17:57:34 EDT
(In reply to David Kutálek from comment #5)

This is a useful data point, thanks David. The server logs were unfortunately truncated at 2013-06-16 23:36:49 (I guess in an attempt to recover disk space) so I didn't see your failure from 2013-06-15. But nevertheless there were some successful task uploads between 2013-06-16 23:36:49 and 2013-06-17 08:30:36 so there must have been some other factor which came into play causing the MemoryErrors.

(In reply to Stefanie Forrester from comment #8)
> Dan, so you were unable to duplicate this issue on another beaker instance?
> Do you think this is an environment-specific error, rather than a createrepo
> bug?

The only bug I can see with createrepo is that it uses 600MB of memory to generate 18MB of repodata, which is not very efficient. It doesn't leak anything in /var/tmp *unless* it dies in the middle with MemoryError.

So there is definitely something environment-specific on beaker-02 which is causing either:
(a) createrepo to use far more memory when run on beaker-02 than on my systems; or
(b) RLIMIT_AS to be set lower than I think it is; or
(c) malloc to return NULL in createrepo for some reason other than exceeding RLIMIT_AS.

I'm pretty sure it has to be (a) but I can't come up with any reasons why it would be happening.
Comment 13 Dan Callaghan 2014-08-21 20:42:57 EDT
Closing this since we could never reproduce the MemoryError in createrepo, and we have been successfully using createrepo_c (which uses much less memory and is faster) in production for several months now instead.

Note You need to log in before you can comment on or make changes to this bug.