Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1420483 - [linearstore] Enhancements to linearstore options
Summary: [linearstore] Enhancements to linearstore options
Keywords:
Status: POST
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp
Version: Development
Hardware: Unspecified
OS: Linux
unspecified
medium
Target Milestone: ---
: ---
Assignee: Kim van der Riet
QA Contact: Messaging QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-02-08 19:53 UTC by Kim van der Riet
Modified: 2020-11-04 23:25 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Apache JIRA QPID-7666 0 None None None 2017-02-08 20:17:36 UTC

Description Kim van der Riet 2017-02-08 19:53:13 UTC
In applications where large numbers of persistent queues are created on a single broker, control over the default and per-queue store options are important so that resources are not wasted or become exhausted prematurely.

There are two resources of concern:
1. Memory allocated per queue for buffering;
2. Disk resources consumed for journal files.

Currently, the default setting for memory allocation is 1024kiB/queue (ie 32 pages of 32kiB). The only default setting available is the page size (using the wcache-page-size option), and the number of pages is set internally according to the following table:

wcache-page-size | wcache-num-pages | total mmap
     (kiB)       | (set interanlly) |  (kiB)
-----------------+------------------+-----------
             4   |         64       |    256
             8   |         64       |    512
            16   |         32       |    512
            32*  |         32       |   1024
            64   |         16       |   1024
           128   |          8       |   1024
* default

This gives in effect only 3 per-queue memory allocation options: 256kiB, 512kiB and 1024kiB.

Currently, the values of wcache-page-size is limited to the powers of 2 between 4 and 128.

It is suggested that the wcache-num-pages value be set as as an option rather than internally, which would give the user much greater control over the per-queue memory footprint and the buffer geometry (which can be optimized for the disk hardware and queue load characteristics).

In addition, the upper value caps should be removed (as they are arbitrary limits) and the default value of wcache-page-size should be revised.

Further, it would be valuable to add these parameters to qpid-config so that if a queue is created using this tool, the default values for the store parameters can be overidden on a per-queue basis.

Comment 1 Kim van der Riet 2017-02-08 21:30:18 UTC
Fixed (https://git-wip-us.apache.org/repos/asf?p=qpid-cpp.git;h=34cfb7f)

* Added option --wcache-num-pages, removed internal table/formulas for determining this value and connected to option.

* Set default for --wcache-num-pages to 16.

* Changed default for --wcache-page-size from 32 to 16.

* Added wiring to set the values of wcache-page-size and wcache-num-pages from qpid-config by looking for the values "qpid.wcache-page-size" and "qpid.wcache-num-pages" in the queue create arguments. This will allow qpid-config to use these as keys to set these values on queue creation.

* Added options --wcache-page-size and --wcache-num-pages to qpid-config as queue create options. This allows these parameters to be set when a queue is created using qpid-config.

* Tidy-up of some linearstore messages. In particular, when the broker is stopped, the linearstore no longer uses message
[Store] info Linear Store: Journal "<queue-name>": Destroyed
which may be confusing (and cause some unnecessary panic to users), but rather
[Store] info Linear Store: Journal "<queue-name>": Stopped

* When a queue is created using parameters other than the defaults, these are now reported at info level logging.


Note You need to log in before you can comment on or make changes to this bug.