Bug 1089652 - [RFE]: Configuration option for linear store to delete the used journal files instead of recycling them.
Summary: [RFE]: Configuration option for linear store to delete the used journal files...
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp
Version: 3.0
Hardware: All
OS: Linux
Target Milestone: 3.1
: ---
Assignee: Kim van der Riet
QA Contact: Zdenek Kraus
Depends On:
TreeView+ depends on / blocked
Reported: 2014-04-21 10:55 UTC by Christos Triantafyllidis
Modified: 2019-05-20 11:10 UTC (History)
8 users (show)

Fixed In Version: qpid-cpp-0.30-4
Doc Type: Enhancement
Doc Text:
Because customers required used journal files to be cleaned of all legacy user data, and returned to the pool for re-use, a linearstore file overwrite option is implemented by adding the --overwrite-before-return option flag when starting the broker (with the linearstore module loaded). When using the --overwrite-before-return option, the store overwrites the data portion of the file (the entire file excluding the header) with `\0`, which erases the previous contents of the file. This will prevent any legacy data from existing in the Empty File Pool (EFP). Using this option will have a performance impact because it takes time to overwrite each file. This option should not be used if security considerations do not require it.
Clone Of:
[Needinfo] Awaiting Docs text from Kim. Bumped for info 6th Mar.
Last Closed: 2015-04-14 13:47:57 UTC
Target Upstream Version:

Attachments (Terms of Use)
Patch proposal (incomplete) (27.21 KB, patch)
2014-07-22 13:32 UTC, Pavel Moravec
no flags Details | Diff

System ID Priority Status Summary Last Updated
Apache JIRA QPID-6043 None None None Never
Red Hat Product Errata RHEA-2015:0805 normal SHIPPED_LIVE Red Hat Enterprise MRG Messaging 3.1 Release 2015-04-14 17:45:54 UTC

Comment 2 Justin Ross 2014-06-16 15:12:27 UTC
Kim, please assess.

Comment 3 Kim van der Riet 2014-06-19 15:42:43 UTC
This should be relatively easy to implement.

If security is the issue, then another option would be to overwrite the entire file before replacing it into the EFP. This is probably more expensive than deleting the file and re-creating it later. However, when deleting only, there is no guarantee that a new file will not contain some of the old data as the kernel does not wipe old file data on delete.

Comment 6 Pavel Moravec 2014-07-22 13:32:31 UTC
Created attachment 919932 [details]
Patch proposal (incomplete)

Patch proposal, though incomplete.

What remains to do:
1) storing and recovering the option. Currently broker restart forgets to set the option to the queue's journal. I attempted to fix it in MessageStoreImpl but without a luck.

2) Parsing property value: see cpp/src/qpid/linearstore/MessageStoreImpl.cpp, lines starting "qpid::framing::FieldTable::ValuePtr value =". Currently the only accepted values are "0" or "true". The problem here is, args_ provides the value as string everytime (so we have to translate it to bool).

3) Think about if the trick with EmptyFilePool::createEmptyFile used is safe.

4) JournalImpl.cpp has some updates to _mgmtObject that is in commented out code - it is worth having mgmtObject updated to be able to see what queues have this option (un)set.

Comment 8 Kim van der Riet 2014-08-25 21:10:03 UTC
Proposed solution: r.1620426

This solution adds an option --overwrite-before-return which causes each used journal file to be completely overwritten with /0 chars before being placed back into the EFP.

To test, start broker using --truncate yes. Then send and receive a number of messages to a queue:

./qpid-send --address "tq ; {create: always, node: {durable: true}}" --tx 0 --messages 10000 --content-size 1000 --durable yes

./qpid-receive --address "tq" --print-headers no --print-content no --messages 10000

Once this is complete, there should be a number of files in the EFP:
tree <store-dir>

The contents can be checked with a simple script like this:
for f in <store-dir>/p001/efp/2048k/*.jrnl; do echo $f; hexdump -C -n 8192 $f; done

If the --overwrite-before-return option is used, then the files should all be blank except for the header:

00000000  51 4c 53 66 02 00 00 00  00 00 00 00 00 00 00 00  |QLSf............|
00000010  00 00 00 00 00 00 00 00  01 00 01 00 00 00 00 00  |................|
00000020  00 08 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

otherwise data will be present starting at address 00001000.

Comment 17 Zdenek Kraus 2014-12-18 14:13:52 UTC
This was tested on RHEL 6 i686 and x86_64 with following packages:

Fix work as expected.


Comment 18 Zdenek Kraus 2014-12-22 13:31:58 UTC
Fixing Assignee and QA contact Fields, sorry for that.

Comment 19 Jared MORGAN 2015-01-09 04:28:25 UTC
Hey Kim, is there docs text available for this ticket considering that it was customer-initiated (based on the Customer Portal ticket)?

Comment 22 Kim van der Riet 2015-03-17 14:39:32 UTC
I apologize for taking so long to get to this! Doc text provided.

Comment 25 errata-xmlrpc 2015-04-14 13:47:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.