Bug 860292

Summary:	we no longer have JMS messages persisted to the DB in AS7
Product:	[Other] RHQ Project	Reporter:	John Mazzitelli <mazz>
Component:	Alerts	Assignee:	Nobody <nobody>
Status:	NEW ---	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	4.5	CC:	hrupp, jshaughn
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description John Mazzitelli 2012-09-25 13:41:13 UTC

As part of the AS7 port (bug 851268) we no longer are able to persist the JMS messages to the database (HornetQ does not support this behavior, unlike JBossMQ in JBossAS 4.2.3).

Because the RHQ Servers are not required to communicate with each other (they don't even have to have connectivity between each other), we cannot assume user environments can cluster HornetQ servers and we don't want to introduce yet another server component to the RHQ deployment environment (like a HornetQ server).

So what this means is that when we send a JMS message (when alert conditions are true or when drift processing has to get done), the RHQ Server that sent the JMS message will be the same RHQ Server that will process the JMS message.

Comment 1 John Sanda 2012-09-25 14:16:10 UTC

We used JMS for drift to provide asynchronous request handling for uploading files from the agent to the server. The new, asynchronous methods in EJB 3.1  which is part of JEE 6 may be sufficient. It would also be a big simplification over the current JMS-based implementation. With this alternative approach as with the newer, HornetQ JMS approach, the same server will have to handle uploading the file content.

If we want the ability for any server to handle the file upload or if asynchronous methods are not sufficient for whatever reasons, we could consider using Infinispan as a replacement for the current JMS-based implementation. There are a lot of different designs we could consider, but at a high level, we could use embedded, distributed caches in each RHQ server. This of course means server2server communication. If we decide that is not an option, we could consider using the stand alone hotrod server, though embedded gives us more options like using the distributed executor framework.

Comment 2 Jay Shaughnessy 2012-09-25 14:38:44 UTC

It should be noted the "singleton" queue that we got using jbossmq with db persistence was more a side-effect than a standard messaging system feature, and not something we should expect from HornetQ or another messaging manager.

Keeping HornetQ as it is built into AS7 causes minimal disruption.  Each RHQ server will have its messages persisted into HornetQ's high-speed journal.  The journal is *only* used for recovery at startup.  Even if several servers share the same directory (on a shared disk) they will not share the messages.

The end effect is that messages delivered to ServerX will be processed by ServerX. If ServerX goes down with undelivered messages they will get processed when the server comes up.  Obviously, for a single-server env there is no change in behavior.  For HA envs there could be a delay in alerting if in fact the alert condition message was the final alert condition to match for a set of alert conditions. Similarly, requested drift files may need to wait until the server returns.

Personally, the change in behavior seems of minor impact to me.  If we fel a need to keep the previous behavior we will likely need to build our own db-backed mechanism with some sort of polling/locking mechanism running on each server.