Bug 1124906 - [linearstore] qpid fails to recover large numbers of queues - too many open files
Summary: [linearstore] qpid fails to recover large numbers of queues - too many open f...
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp
Version: Development
Hardware: Unspecified
OS: Unspecified
Target Milestone: 3.0
: ---
Assignee: Kim van der Riet
QA Contact: Eric Sammons
Depends On:
TreeView+ depends on / blocked
Reported: 2014-07-30 15:13 UTC by Brian Bouterse
Modified: 2015-01-21 12:54 UTC (History)
6 users (show)

Fixed In Version: qpid-cpp-0.22-45
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2015-01-21 12:54:30 UTC

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Apache JIRA QPID-5924 None None None Never

Description Brian Bouterse 2014-07-30 15:13:30 UTC
This BZ is designed to track the resolution of JIRA 5924 from upstream to ensure it makes it into the MRG product. See the upstream issue for its full description.

[0]:  https://issues.apache.org/jira/browse/QPID-5924

Comment 1 Kim van der Riet 2014-07-30 15:57:46 UTC
Fixed in r.1614665 [https://svn.apache.org/r1614665]

See upstream QPID-5924 (link above) for comments.

Comment 3 Kim van der Riet 2014-08-12 13:00:41 UTC
The number of queues you may be able to create may vary from system to system depending on the version of RHEL and the available hardware resources - I am not familiar with exactly what happens here. IFAIR, the default max per user is 1024 file handles, but this can be extended by the root user.

Before the fix, the linearstore recovery was using 2 file handles per queue - one for normal write use after recovery and the other used for reading the contents of the file. After recovery was complete, all of the read file handles were closed together, and only one file handle per queue remained open.

It should be possible to test this behavior as follows:

1. Using a client on another machine, create as many queues as possible until you exhaust the available file handles on the broker. If the default is 1024, then you should be able to create about 1000 queues. It might be a good idea to put at least one message on each queue. At this point, the broker should be using 1 file handle per queue.

2. Stop the broker and then restart it. If the broker is using 2 file handles per queue, it should only be able to recover approx. half the queues (approx. 500) before failing.

After the fix, the  broker should be able to recover all the queues. As the messages are removed, the file handle for that queue will be opened, but you should be able to remove all messages (or perhaps run out of handles very close the last queue).

You can use ulimit to see the number of file handles and change the limit, but this test should not depend on an absolute number of handles being available.

Comment 6 Valiantsina Hubeika 2014-08-26 10:57:42 UTC
verified on 

Note You need to log in before you can comment on or make changes to this bug.