In applications where large numbers of persistent queues are created on a single broker, control over the default and per-queue store options are important so that resources are not wasted or become exhausted prematurely.
There are two resources of concern:
1. Memory allocated per queue for buffering;
2. Disk resources consumed for journal files.
Currently, the default setting for memory allocation is 1024kiB/queue (ie 32 pages of 32kiB). The only default setting available is the page size (using the wcache-page-size option), and the number of pages is set internally according to the following table:
wcache-page-size | wcache-num-pages | total mmap
(kiB) | (set interanlly) | (kiB)
4 | 64 | 256
8 | 64 | 512
16 | 32 | 512
32* | 32 | 1024
64 | 16 | 1024
128 | 8 | 1024
This gives in effect only 3 per-queue memory allocation options: 256kiB, 512kiB and 1024kiB.
Currently, the values of wcache-page-size is limited to the powers of 2 between 4 and 128.
It is suggested that the wcache-num-pages value be set as as an option rather than internally, which would give the user much greater control over the per-queue memory footprint and the buffer geometry (which can be optimized for the disk hardware and queue load characteristics).
In addition, the upper value caps should be removed (as they are arbitrary limits) and the default value of wcache-page-size should be revised.
Further, it would be valuable to add these parameters to qpid-config so that if a queue is created using this tool, the default values for the store parameters can be overidden on a per-queue basis.
* Added option --wcache-num-pages, removed internal table/formulas for determining this value and connected to option.
* Set default for --wcache-num-pages to 16.
* Changed default for --wcache-page-size from 32 to 16.
* Added wiring to set the values of wcache-page-size and wcache-num-pages from qpid-config by looking for the values "qpid.wcache-page-size" and "qpid.wcache-num-pages" in the queue create arguments. This will allow qpid-config to use these as keys to set these values on queue creation.
* Added options --wcache-page-size and --wcache-num-pages to qpid-config as queue create options. This allows these parameters to be set when a queue is created using qpid-config.
* Tidy-up of some linearstore messages. In particular, when the broker is stopped, the linearstore no longer uses message
[Store] info Linear Store: Journal "<queue-name>": Destroyed
which may be confusing (and cause some unnecessary panic to users), but rather
[Store] info Linear Store: Journal "<queue-name>": Stopped
* When a queue is created using parameters other than the defaults, these are now reported at info level logging.