During some scalability testing that involved Pulp, the following issue was observed [0]. In Pulp environments that have a large number of consumers, the qpidd broker may run out of file descriptors while reading or writing to its journal files. I've created an upstream issue in Qpid's issue tracker[1] for them to address this limitation within Qpid, however Pulp needs to document the issue and its workaround. Refer to the BZ [0] for a recap of the issue, but basically qpidd can run out of file descriptors in environments with lots of consumers. The workaround is to increase the file descriptor limit that qpidd uses by putting an entry in /etc/security/limit.conf for qpidd. This note should be put into the troubleshooting guide [2], and also in the Rally story US2818 [3] about documenting the scalability of Pulp. [0]: https://bugzilla.redhat.com/show_bug.cgi?id=1122055 [1]: https://issues.apache.org/jira/browse/QPID-5924 [2]: http://pulp-user-guide.readthedocs.org/en/latest/troubleshooting.html [3]: https://rally1.rallydev.com/#/7675694122d/detail/userstory/19528332567
A patch has been committed upstream that makes Qpid's use of file descriptors as efficient as it could be without using a pool of file descriptors. It's not clear which upstream Qpid release will include this Fix, but it will likely be Qpid 0.30. Once a Fix Version is identified upstream, the docs should identify which versions do and do not contain this issue.
This fix is confirmed to be included in the Qpid 0.30 release. The Pulp docs should indicate that.
In addition to the file descriptors problem, the Qpid developers identified another limiting resource, kernel AIO contexts. They commented on it here [0], and point to a white paper [1] about resolving this issue. [0]: https://bugzilla.redhat.com/show_bug.cgi?id=1124906#c5 [1]: https://bugzilla.redhat.com/attachment.cgi?id=930496
PR available at: https://github.com/pulp/pulp/pull/1213
Merged to 2.5-testing. I did not merge forward.
verified