Bug 1122987 - Document Qpid Consumer Scalability Limitation
Summary: Document Qpid Consumer Scalability Limitation
Alias: None
Product: Pulp
Classification: Retired
Component: documentation
Version: 2.4 Beta
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 2.5.0
Assignee: Brian Bouterse
QA Contact: Preethi Thomas
Depends On:
TreeView+ depends on / blocked
Reported: 2014-07-24 14:14 UTC by Brian Bouterse
Modified: 2014-11-24 21:33 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2014-11-24 21:33:36 UTC

Attachments (Terms of Use)

Description Brian Bouterse 2014-07-24 14:14:37 UTC
During some scalability testing that involved Pulp, the following issue was observed [0].

In Pulp environments that have a large number of consumers, the qpidd broker may run out of file descriptors while reading or writing to its journal files. I've created an upstream issue in Qpid's issue tracker[1] for them to address this limitation within Qpid, however Pulp needs to document the issue and its workaround.

Refer to the BZ [0] for a recap of the issue, but basically qpidd can run out of file descriptors in environments with lots of consumers. The workaround is to increase the file descriptor limit that qpidd uses by putting an entry in  /etc/security/limit.conf for qpidd.

This note should be put into the troubleshooting guide [2], and also in the Rally story US2818 [3] about documenting the scalability of Pulp.

[0]:  https://bugzilla.redhat.com/show_bug.cgi?id=1122055
[1]:  https://issues.apache.org/jira/browse/QPID-5924
[2]:  http://pulp-user-guide.readthedocs.org/en/latest/troubleshooting.html
[3]:  https://rally1.rallydev.com/#/7675694122d/detail/userstory/19528332567

Comment 1 Brian Bouterse 2014-07-30 15:36:25 UTC
A patch has been committed upstream that makes Qpid's use of file descriptors as efficient as it could be without using a pool of file descriptors. It's not clear which upstream Qpid release will include this Fix, but it will likely be Qpid 0.30. Once a Fix Version is identified upstream, the docs should identify which versions do and do not contain this issue.

Comment 2 Brian Bouterse 2014-07-31 20:23:51 UTC
This fix is confirmed to be included in the Qpid 0.30 release. The Pulp docs should indicate that.

Comment 3 Brian Bouterse 2014-09-04 13:58:05 UTC
In addition to the file descriptors problem, the Qpid developers identified another limiting resource, kernel AIO contexts. They commented on it here [0], and point to a white paper [1] about resolving this issue.

[0]:  https://bugzilla.redhat.com/show_bug.cgi?id=1124906#c5
[1]:  https://bugzilla.redhat.com/attachment.cgi?id=930496

Comment 4 Brian Bouterse 2014-10-07 20:48:42 UTC
PR available at:  https://github.com/pulp/pulp/pull/1213

Comment 5 Brian Bouterse 2014-10-08 13:45:28 UTC
Merged to 2.5-testing. I did not merge forward.

Comment 6 Preethi Thomas 2014-10-14 13:50:14 UTC

Note You need to log in before you can comment on or make changes to this bug.