Bug 916794

Description of problem:

I setup a new pulp on a m1.large instance (~7.5GB RAM and no swap), after syncing/deleting a repo I attempting to run: "pulp-admin orphan list".  It ran for 3 minutes then failed with a "MemoryError".

Note:  I did not restart pulp after the sync.  I ran the sync then delete, then orphan list.




Version-Release number of selected component (if applicable):
# rpm -qa | grep pulp
m2crypto-0.21.1.pulp-8.el6.x86_64
python-oauth2-1.5.170-3.pulp.el6.noarch
pulp-puppet-plugins-2.0.8-1.noarch
python-pulp-rpm-extension-2.0.8-1.noarch
pulp-rpm-admin-extensions-2.0.8-1.noarch
python-isodate-0.5.0-1.pulp.el6.noarch
python-pulp-puppet-common-2.0.8-1.noarch
python-rhsm-1.8.0-1.pulp.el6.x86_64
pulp-server-2.0.8-1.noarch
pulp-rpm-plugins-2.0.8-1.noarch
pulp-selinux-2.0.8-1.noarch
python-pulp-client-lib-2.0.8-1.noarch
python-pulp-bindings-2.0.8-1.noarch
pulp-admin-client-2.0.8-1.noarch
pulp-puppet-admin-extensions-2.0.8-1.noarch
python-pulp-common-2.0.8-1.noarch
python-pulp-rpm-common-2.0.8-1.noarch
mod_wsgi-3.3-4.pulp.el6.x86_64
pulp-builtins-admin-extensions-2.0.8-1.noarch


How reproducible:


Steps to Reproduce:
1) Adjust Pulp to use 4 threads for syncing
# cat /usr/lib/pulp/plugins/importers/yum_importer/yum_importer.conf 
{
    "num_threads": 4
}
2) Synced a small repo of 3 packages
  http://jmatthews.fedorapeople.org/pulp_unittest/
3) Synced RHEL 6.2 X86_64 from RH CDN.
 pulp-admin rpm repo create --repo-id rhel6_x86_64 --feed-ca ./cdn.redhat.com-chain.crt --feed-cert ./1359391926_4512.crt --feed-key ./1359391926_4512.key  --feed  https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.2/x86_64/os

4) Deleted Repo RHEL 6.2 X86_64
5) Run: pulp-admin orphan list
Note: This was run after Repo was deleted and task showed:
 pulp-admin repo tasks list --repo-id rhel6_x864
+----------------------------------------------------------------------+
                                 Tasks
+----------------------------------------------------------------------+

Operations:  delete
Resources:   rhel6_x864 (repository)
State:       Successful
Start Time:  2013-02-28T22:05:08Z
Finish Time: 2013-02-28T22:05:10Z
Result:      N/A
Task Id:     6580dc7f-8d62-4d6a-a592-0e3f17014e49


[root@ec2-50-17-33-189 ~]# pulp-admin repo list
+----------------------------------------------------------------------+
                              Repositories
+----------------------------------------------------------------------+

Id:                 pulp_unittest
Display Name:       pulp_unittest
Description:        None
Content Unit Count: 61

  
Actual results:
# time pulp-admin orphan list &> orphan.list

real    3m12.158s
user    2m1.903s
sys     0m5.497s



# cat orphan.list 
An unexpected error has occurred. More information can be found in the client
log file ~/.pulp/admin.log.


2013-02-28 17:12:49,023 - ERROR - Client-side exception occurred
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/client/extensions/core.py", line 478, in run
    exit_code = Cli.run(self, args)
  File "/usr/lib/python2.6/site-packages/okaara/cli.py", line 921, in run
    exit_code = command_or_section.execute(self.prompt, remaining_args)
  File "/usr/lib/python2.6/site-packages/pulp/client/extensions/extensions.py", line 224, in execute
    return self.method(*arg_list, **clean_kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/client/commands/unit.py", line 104, in run
    orphans = self.context.server.content_orphan.orphans().response_body
  File "/usr/lib/python2.6/site-packages/pulp/bindings/content.py", line 27, in orphans
    return self.server.GET(self.PATH)
  File "/usr/lib/python2.6/site-packages/pulp/bindings/server.py", line 84, in GET
    return self._request('GET', path, queries)
  File "/usr/lib/python2.6/site-packages/pulp/bindings/server.py", line 139, in _request
    self.api_responses_logger.info("Response body :\n %s\n" % json.dumps(response_body, indent=2))
  File "/usr/lib64/python2.6/json/__init__.py", line 237, in dumps
    **kw).encode(obj)
  File "/usr/lib64/python2.6/json/encoder.py", line 368, in encode
    return ''.join(chunks)
MemoryError


Additional info:

Note memory usage is quite high in main wsgi process.

from top:
25291 apache    20   0 6246m 4.6g 1888 S  0.9 63.0 208:46.29 httpd  

from pmap -d:
mapped: 6526976K    writeable/private: 5324176K    shared: 1004K

from ps auxf:
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

mongodb  24472  3.6  0.7 1797896 58268 ?       Sl   10:41  14:44 /usr/bin/mongod --quiet -f /etc/mongodb.conf run
root     25289  0.0  0.0 206560  3608 ?        Ss   13:06   0:04 /usr/sbin/httpd
apache   25291 82.2 64.6 6526976 4943052 ?     Sl   13:06 208:44  \_ (wsgi:pulp)    
apache   25292  0.2  0.0 209052  6812 ?        S    13:06   0:45  \_ /usr/sbin/httpd
apache   25293  0.2  0.0 209052  6804 ?        S    13:06   0:45  \_ /usr/sbin/httpd
apache   25294  0.2  0.0 209048  6808 ?        S    13:06   0:44  \_ /usr/sbin/httpd
apache   25295  0.2  0.0 209052  6804 ?        S    13:06   0:44  \_ /usr/sbin/httpd
apache   25296  0.3  2.3 380820 178628 ?       S    13:06   0:56  \_ /usr/sbin/httpd
apache   25297  0.2  0.0 209052  6804 ?        S    13:06   0:44  \_ /usr/sbin/httpd
apache   25298  0.2  0.0 209052  6804 ?        S    13:06   0:45  \_ /usr/sbin/httpd
apache   25299  0.3  1.5 319988 117740 ?       S    13:06   0:55  \_ /usr/sbin/httpd



Sync output.
cat /usr/lib/pulp/plugins/importers/yum_importer/yum_importer.conf 
{
    "num_threads": 4
}

# time pulp-admin rpm repo sync run --repo-id rhel6_x864
+----------------------------------------------------------------------+
                 Synchronizing Repository [rhel6_x864]
+----------------------------------------------------------------------+

This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.

Downloading metadata...
[\]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       7281/7281 items
Delta RPMs: 0/0 items
Tree Files: 6/6 items
Files:      0/0 items
... completed

Importing errata...
[\]
... completed

Importing package groups/categories...
[/]
... completed

Publishing packages...
[==================================================] 100%
Packages: 7281/7281 items
... completed

Publishing distributions...
[==================================================] 100%
Distributions: 6/6 items
... completed

Generating metadata
[/]
... completed

Publishing repository over HTTPS
[-]
... completed


real    174m3.440s
user    4m27.888s
sys     0m19.161s