Bug 1122987

Summary:	Document Qpid Consumer Scalability Limitation
Product:	[Retired] Pulp	Reporter:	Brian Bouterse <bmbouter>
Component:	documentation	Assignee:	Brian Bouterse <bmbouter>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Preethi Thomas <pthomas>
Severity:	low	Docs Contact:
Priority:	low
Version:	2.4 Beta	CC:	pthomas, rbarlow, skarmark
Target Milestone:	---	Keywords:	Triaged
Target Release:	2.5.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-11-24 21:33:36 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Brian Bouterse 2014-07-24 14:14:37 UTC

During some scalability testing that involved Pulp, the following issue was observed [0].

In Pulp environments that have a large number of consumers, the qpidd broker may run out of file descriptors while reading or writing to its journal files. I've created an upstream issue in Qpid's issue tracker[1] for them to address this limitation within Qpid, however Pulp needs to document the issue and its workaround.

Refer to the BZ [0] for a recap of the issue, but basically qpidd can run out of file descriptors in environments with lots of consumers. The workaround is to increase the file descriptor limit that qpidd uses by putting an entry in  /etc/security/limit.conf for qpidd.

This note should be put into the troubleshooting guide [2], and also in the Rally story US2818 [3] about documenting the scalability of Pulp.


[0]:  https://bugzilla.redhat.com/show_bug.cgi?id=1122055
[1]:  https://issues.apache.org/jira/browse/QPID-5924
[2]:  http://pulp-user-guide.readthedocs.org/en/latest/troubleshooting.html
[3]:  https://rally1.rallydev.com/#/7675694122d/detail/userstory/19528332567

Comment 1 Brian Bouterse 2014-07-30 15:36:25 UTC

A patch has been committed upstream that makes Qpid's use of file descriptors as efficient as it could be without using a pool of file descriptors. It's not clear which upstream Qpid release will include this Fix, but it will likely be Qpid 0.30. Once a Fix Version is identified upstream, the docs should identify which versions do and do not contain this issue.

Comment 2 Brian Bouterse 2014-07-31 20:23:51 UTC

This fix is confirmed to be included in the Qpid 0.30 release. The Pulp docs should indicate that.

Comment 3 Brian Bouterse 2014-09-04 13:58:05 UTC

In addition to the file descriptors problem, the Qpid developers identified another limiting resource, kernel AIO contexts. They commented on it here [0], and point to a white paper [1] about resolving this issue.

[0]:  https://bugzilla.redhat.com/show_bug.cgi?id=1124906#c5
[1]:  https://bugzilla.redhat.com/attachment.cgi?id=930496

Comment 4 Brian Bouterse 2014-10-07 20:48:42 UTC

PR available at:  https://github.com/pulp/pulp/pull/1213

Comment 5 Brian Bouterse 2014-10-08 13:45:28 UTC

Merged to 2.5-testing. I did not merge forward.

Comment 6 Preethi Thomas 2014-10-14 13:50:14 UTC

verified