Description of problem: If consumer with message selector is trying to receive paged messages from queue then no message is received. How reproducible: 1. Start EAP 6.4.0.DR11 server with queue InQueue 2. Send 100 messages to InQueue so some messages are paged - 50 messages have string property "color" set "RED" - 50 messages have string property "color" set "GREEN" 3. Try to receive messages with consumer with message selector "color = 'RED'" from InQueue 4. Check that 50 messages were received Result: Only not paged messages are received. To reproduce the problem follow those steps: Clone our testsuite from git: git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git Go to eap-tests-hornetq/scripts and run groovy script PrepareServers.groovy with -DEAP_VERSION=6.4.0.DR11 parameter (groovy at least 2.2.1+ must be used): groovy -DEAP_VERSION=6.4.0.DR11 PrepareServers.groovy (Script will prepare 4 servers - server1..4 in the directory where are you currently standing.) Export these paths to server directories and mcast address: export JBOSS_HOME_1=$PWD/server1/jboss-eap export JBOSS_HOME_2=$PWD/server2/jboss-eap export JBOSS_HOME_3=$PWD/server3/jboss-eap export JBOSS_HOME_4=$PWD/server4/jboss-eap export MCAST_ADDR=235.3.4.5 And finally go to jboss-hornetq-testsuite/ in our testsuite and run: mvn clean install -Dtest=FilteringQueueTestCase#testSimpleFilterClient Expected results: Messages which Additional info:
Why there are 4 servers to this test? This is about redistribution, right? Redistribution with selectors won't work in certain cases at all.. Including Paging to the picture would make it even worse.
In this test only one server is used. There is no redistribution. 4 servers must be setup for the test framework to run properly. But in this test just one server is used.
Did you push your commit? or did you do this on any other branch other than master? I just looked here and I can't find FilteringQueueTestCase anywhere on the codebase: clebert@powerbox eap-tests-hornetq]$ git up origin ˇCurrent branch master is up to date. [clebert@powerbox eap-tests-hornetq]$ git grep FilteringQueueTestCase (empty result) If I ran the test of course I would get this: INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 5.694 s [INFO] Finished at: 2014-12-03T14:38:44-05:00 [INFO] Final Memory: 30M/327M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project qa-hornetq-testsuite: No tests were executed! (Set -DfailIfNoTests=false to ignore this error.) -> [Help 1] [
In particular what would I need to make a similar test? You are sending 100 messages: 50 with red, 50 with green What is the page-size so the messages will start to page.. and what is the size of each message that you are sending? I wanted to create a testcase for my own codebase so it would be easier to see what's going on.
I don't think this is a page-scenario that we want to support anyways... I'm having a de-ja-vou sensation but I believe we have been through a similar issue. You would receive all the messages if you were having both consumers active as the system would be depaging accordingly. We don't want to do a full scan on the system to satisfy a filter. If you had subscriptions with filters then it's all ok... anyways... send me more information on the test and I will take an additional look if it would be possible to fix and what would be the cost on doing it.
ahhhhh... you are not receiving the one with RED which was on TOP!!! Got it!! Sorry about that! I couldn't run your test but I should be able to write my own.. if I can run it I will clear the needinfo
I have written this following test: https://gist.github.com/clebertsuconic/5a2852b6c5850fce7b94 If I send 100 Message, start to page at the 10th message (or any other place).. all the messages with red are received. I'm acking on every received message. If I start receiving green instead I wouldn't receive since the system won't do a scan on paging to satisfy a queue. (it would work on browsing AFAIK). Is that what you're seeing.. what's the difference on your test?
Sorry I really forgot to push it. It's pushed right now. Yes, this is what I see.
I'm treating this with a limited fix that I believe it would fix your issue... When depaging on Queues / Regular consumers, we can't depage > max-size. However I have seen a case where it was depaging even though it had space in memory. Another case I'm fixing is Browsing was not going beyond the memory. On your case you had a transaction where you were consuming the entire red set. if the entire set was beyond > max-size you would also starve.
Fixed by upgrade BZ1184141
Returning to assigned state as FilteringQueueTestCase#testSimpleFilterClient test is failing with EAP 6.4.0.ER2 (HornetQ 2.3.25.Final) @Clebert I've modified the test to commit each received message because in comment #9 you've mentioned that I should receive the entire red set because it's bigger than max-size-bytes and it cannot be depaged to memory. Still only 9 messages (from 50) are received. It appears that newly depaged messages does not hold condition in selector. In documentation [1] there is text: "Consumers with selectors will also navigate through the page-files and it will ignore messages that don't match the criteria." I understand it that only messages which match selector will be depaged. This is not regression against EAP 6.3.0. [1] http://documentation-devel.engineering.redhat.com/site/documentation/en-US/JBoss_Enterprise_Application_Platform/6.4/html-single/Administration_and_Configuration_Guide/#sect-Paging
I don't think the test is correct. it's a consumer over a regular consumer and you first need to depage regular messages. If you used a browser it would work all the way to the end.
To have additional regular consumer which consumes all or just green messages from queue can be easily missed by users. Basic understanding of developer will be: "when I have consumer with selector on queue it will read all messages matching the selector from queue". He'll most likely not realize that when paging is active then those paged messages witch match his selector won't be read unless there is another consumer which reads the "other" messages.
It's an anti-pattern anyways... we can't depage the whole system in order to satisfy a single consumer. it would OME. if you had a regular consumer as well you would eventually consume messages. Doing this kind of thing you are making message be like a DB and we don't want to do that.
I see, performance impact would be huge. If there is nothing more what could be done, could you close it as won't fix then? I'll create new bz for documentation so there is no confusion.
As any message system it's not possible to make a scan over a large set of data to satisfy browsing. Providing a fix for this would mean to do a full scan every time, what's impossible to satisfy. I have improved how depage works to satisfy filters as they enter the queue and as they are depaged.