Bug 1193793 - [QA](6.4.z) After failback cachedCommands are never cleaned
Summary: [QA](6.4.z) After failback cachedCommands are never cleaned
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: HornetQ
Version: 6.4.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: CR1
: EAP 6.4.5
Assignee: Ryan Emerson
QA Contact: Miroslav Novak
URL:
Whiteboard:
Depends On:
Blocks: 1235745 1247277
TreeView+ depends on / blocked
 
Reported: 2015-02-18 08:45 UTC by Miroslav Novak
Modified: 2017-01-17 11:38 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-01-17 11:38:51 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Apache JIRA ARTEMIS-152 0 None None None Never
Red Hat Issue Tracker HORNETQ-1484 0 Major Resolved Improve log output to explain why cached commands are repeated. 2016-01-05 03:48:35 UTC

Description Miroslav Novak 2015-02-18 08:45:16 UTC
Description of problem:

If failover/failback happens multiple times then there are lots of log messages:
...
09:24:24,333 INFO  [org.hornetq.jms.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=2533423e-b745-11e4-b5d4-cb40437956fb) HQ121003: JMS Server Manager Running cached command for destroyConnectionFactory for RemoteConnectionFactory
...
09:24:24,334 INFO  [org.hornetq.jms.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=2533423e-b745-11e4-b5d4-cb40437956fb) HQ121003: JMS Server Manager Running cached command for destroyConnectionFactory for RemoteConnectionFactory


in backup server log. Number of those log messages increases with each failover.

Steps to Reproduce (dedicated topology with shared store):
    Start the live.
    Start the backup.
    Kill-9 the live. Backup becomes live.
    Start the live. Live becomes live, backup becomes backup.
    Kill-9 the live. Backup becomes live again.
    ...

Actual results:
Cached commands are never cleared and only added to list after failback.

Expected results:
Clear cached commands after start of backup server.

Comment 1 Ryan Emerson 2015-07-03 09:18:44 UTC
This is a consequence of https://issues.jboss.org/browse/HORNETQ-1047.  

To improve user experience, the log output has been updated to explain that this is expected behaviour.  

2.3.25.x PR: https://github.com/hornetq/hornetq/pull/2030
2.4.x PR: https://github.com/hornetq/hornetq/pull/2031
Upstream PR: https://github.com/apache/activemq-artemis/pull/59

Comment 2 JBoss JIRA Server 2015-07-03 09:18:53 UTC
Ryan Emerson <remerson> updated the status of jira HORNETQ-1484 to Coding In Progress

Comment 3 Ryan Emerson 2015-07-09 15:59:03 UTC
2.3.x PR: https://github.com/hornetq/hornetq/pull/2034

Comment 5 JBoss JIRA Server 2015-07-13 14:20:58 UTC
Ryan Emerson <remerson> updated the status of jira HORNETQ-1484 to Resolved

Comment 6 Ondřej Kalman 2015-07-31 09:58:00 UTC
Problem is, that this log is stacking and stacking and stacking. After few failovers it can be really long and user will be spammed with useless information. So this solution is not fix.

Comment 7 Ryan Emerson 2015-07-31 10:07:43 UTC
Ondřej would an acceptable solution be for the logging level to be changed to DEBUG with the new message explaining that the repeated commands are expected? This is how the log is now handled upstream.

Comment 8 Ondřej Kalman 2015-07-31 10:12:47 UTC
I think, that it would be better if that "cache" which holds this command history is  cleaned after failback.

Comment 9 Ryan Emerson 2015-07-31 10:14:53 UTC
This is not possible because of https://issues.jboss.org/browse/HORNETQ-1047.  Clearing the cache will cause a regression.

Comment 10 Ondřej Kalman 2015-07-31 10:20:22 UTC
And what about making this cache history as set. If you avoid duplications, problem will be fixed without regression.

Comment 11 Ryan Emerson 2015-07-31 14:19:35 UTC
I don't believe avoiding duplication is possible in this case because it is necessary for the backup server to execute the cached commands in the same order that they were originally executed, so that the state of the failed live server can be reproduced.  The commands stored in the cache can create/remove topics, therefore the order of execution is important. For example, if we did not respect duplicate commands then the following workflow would not be reproducible:

JMSServerManager.createTopic("example" ...)
JMSServerManager.removeTopicFromJNDI("example")
JMSServerManager.createTopic("example" ...) // Identical topic to above

Comment 12 Ondřej Kalman 2015-08-03 12:13:05 UTC
Ok, in this case is acceptable to change logging level to DEBUG.

Comment 15 JBoss JIRA Server 2015-08-05 09:21:07 UTC
Ryan Emerson <remerson> updated the status of jira HORNETQ-1484 to Reopened

Comment 20 JBoss JIRA Server 2015-09-16 09:06:04 UTC
Ryan Emerson <remerson> updated the status of jira HORNETQ-1484 to Resolved

Comment 21 Ondřej Kalman 2015-11-02 16:05:11 UTC
Verified with 6.4.5.CP.CR1

Comment 22 Petr Penicka 2017-01-17 11:38:51 UTC
Retroactively bulk-closing issues from released EAP 6.4 cumulative patches.


Note You need to log in before you can comment on or make changes to this bug.