Bug 1014099

Summary: WARN HQ222180: There are no queues bound into Dead Letter Address jms.queue.DLQ. Messages will ... during start
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Martin Svehla <msvehla>
Component: JMSAssignee: Jeff Mesnil <jmesnil>
Status: CLOSED CURRENTRELEASE QA Contact: Martin Svehla <msvehla>
Severity: medium Docs Contact: Russell Dickenson <rdickens>
Priority: unspecified    
Version: 6.2.0, 6.3.0CC: csuconic, jmesnil, kkhan, mnovak, msvehla, smumford
Target Milestone: DR3   
Target Release: EAP 6.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
This release of JBoss EAP 6 includes an enhancement to the messaging subsystem configuration which now includes two JMS queues for the dead letter queue and the expiry queue. In older releases the default address settings were configured to send expired messages to an expiry address and undelivered messages to a dead letter address. However there were no queues bound to theses address and it resulted in message loss preventing administrators to examine these messages and diagnostic potential application issues. Expired messages will now be stored in the DLQ queue and undelivered messages will now be stored in the ExpiryQueue.
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-06-28 15:26:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1074682    

Description Martin Svehla 2013-10-01 12:20:44 UTC
Description of problem:
Default standalone-full.xml and standalone-full-ha.xml have DLQ jms.queue.DLQ and expiry queue jms.queue.ExpiryQueue set in address settings for # address. However, none of these queues are deployed. This should be consistent.


Version-Release number of selected component (if applicable):
EAP 6.2.0.ER3, HornetQ 2.3.8.Final


Expected results:
Either remove DLQ/expiry address settings or deploy both queues in default configurations.

Comment 1 Miroslav Novak 2014-03-06 12:05:36 UTC
There is WARN log during start of EAP 6.3.0.DR2 server. Clean start is requirement for EAP 6.x.

From server.log:
WARN  [org.hornetq.core.server] (ServerService Thread Pool -- 58) HQ222180: There are no queues bound into Dead Letter Address jms.queue.DLQ. Messages will be dropped if you use the Dead Letter Address until you deploy a destination on this address

There should not be any warning during start. This happens for "full" and "full-ha" profile.

When standalone-full-ha.xml is modified and DLQ is deployed:
<jms-destinations>     
...               
   <jms-queue name="DLQ">
      <entry name="java:/jms/queue/DLQ"/>
      <durable>true</durable>
    </jms-queue>
...
</jms-destinations>

then another warning is logged:

12:49:36,964 WARN  [org.hornetq.core.server] (ServerService Thread Pool -- 62) HQ222179: There are no queues bound into expiry address jms.queue.ExpiryQueue. Messages will be dropped if you use expiration until you deploy a destination on this address

this time the warning is complaining by missing ExpireQueue which does NOT disappear when:

<jms-queue name="ExpiryQueue">
   <entry name="java:/jms/queue/ExpiryQueue"/>
   <durable>true</durable>
</jms-queue>

is added to configuration which seems to be a bug. Goal is to modify default configuration so no warning is logged.

Comment 2 Jeff Mesnil 2014-03-06 14:53:46 UTC
This issue is the consequence of https://bugzilla.redhat.com/show_bug.cgi?id=1014103 where HornetQ warns if either the DLQ or the Expiry Queue is missing but the messaging subsystem has not been updated to configure them explicitly.

Comment 3 Clebert Suconic 2014-03-06 15:05:16 UTC
We have added the log as requested ...

if the queue is not deployed.. just deploy it.. otherwise the Warning has to be valid.


The only scenario where this is a bug, is if the queues are deployed... if they are not.. this is a won't fix.

Comment 4 Jeff Mesnil 2014-03-06 15:34:54 UTC
I confirm this is a bug in HornetQ.

The warning about missing EpxiryQueue appears... when the ExpiryQueue is deployed.

The logic is flawed: any queue that is deployed (or loaded from the journal) *before* the expiry queue will log the warning.

Comment 5 Jeff Mesnil 2014-03-06 15:39:12 UTC
Clebert, when you create the PR to upgrade HornetQ with the fix, please make sure to add the DLQ and ExpiryQueue to the messaging subsystem configuration so that the warnings will not be logged:

<jms-destinations>
    <jms-queue name="DLQ">
        <entry name="java:/jms/queue/DLQ"/>
        <durable>true</durable>
    </jms-queue>
    <jms-queue name="ExpiryQueue">
        <entry name="java:/jms/queue/ExpiryQueue"/>
        <durable>true</durable>
    </jms-queue>
</jms-destinations>

Comment 6 Clebert Suconic 2014-03-06 15:45:03 UTC
I saw that as well..I was running the server here.

I will change the logic from https://bugzilla.redhat.com/show_bug.cgi?id=1014103 to show a warning at the first message routed, or the first consumer created on the queue.

Comment 7 Clebert Suconic 2014-03-06 20:11:26 UTC
scratch my last comment. the issue is on Expiry only because of the code for management.

I'm adding tests and fixing it.

Comment 8 Clebert Suconic 2014-03-07 22:10:28 UTC
@Jeff: I was able to fix what I could here at https://github.com/hornetq/hornetq/pull/1571


However, the server doesn't guarantee the order of the creates...


Say if I make this on the config, the server will create the queues on a random order:

                <jms-destinations>
                    <jms-queue name="DDD">
                        <entry name="java:/DDD"/>
                        <durable>true</durable>
                    </jms-queue>
                    <jms-queue name="DLQ">
                        <entry name="java:/jms/queue/DLQ"/>
                        <durable>true</durable>
                    </jms-queue>
                    <jms-queue name="ExpiryQueue">
                        <entry name="java:/jms/queue/ExpiryQueue"/>
                        <durable>true</durable>
                    </jms-queue>
                </jms-destinations>


I made the HornetQ to defer initialization after loading the journal, what would make this an issue only at the first start with an empty journal.

17:10:02,936 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 58) HQ221003: trying to deploy queue jms.queue.DDD
17:10:02,959 WARN  [org.hornetq.core.server] (ServerService Thread Pool -- 58) HQ222179: There are no queues bound into Expiry Address jms.queue.ExpiryQueue, from queue jms.queue.DDD (address jms.queue.DDD). Messages will be dropped if you use expiration until you deploy a destination on this address
17:10:02,960 WARN  [org.hornetq.core.server] (ServerService Thread Pool -- 58) HQ222180: There are no queues bound into Dead Letter Address jms.queue.DLQ, from queue jms.queue.DDD (address jms.queue.DDD). Messages will be dropped if you use the Dead Letter Address until you deploy a destination on this address
17:10:02,987 INFO  [org.jboss.as.messaging] (ServerService Thread Pool -- 58) JBAS011601: Bound messaging object to jndi name java:/DDD
17:10:02,996 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-11) JBAS010406: Registered connection factory java:/JmsXA
17:10:03,000 INFO  [org.jboss.as.messaging] (ServerService Thread Pool -- 61) JBAS011601: Bound messaging object to jndi name java:/ConnectionFactory
17:10:03,001 INFO  [org.jboss.as.messaging] (ServerService Thread Pool -- 60) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/RemoteConnectionFactory
17:10:03,002 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 59) HQ221003: trying to deploy queue jms.queue.ExpiryQueue
17:10:03,003 INFO  [org.jboss.as.messaging] (ServerService Thread Pool -- 59) JBAS011601: Bound messaging object to jndi name java:/jms/queue/ExpiryQueue
17:10:03,003 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 62) HQ221003: trying to deploy queue jms.queue.DLQ
17:10:03,005 INFO  [org.jboss.as.messaging] (ServerService Thread Pool -- 62) JBAS011601: Bound messaging object to jndi name java:/jms/queue/DLQ

Comment 9 Miroslav Novak 2014-03-10 08:26:06 UTC
One of the base requirements for EAP 6 is to have clean start/stop. There cannot be any warnings.

Comment 10 Jeff Mesnil 2014-03-10 11:30:04 UTC
We will not introduce queue ordering (and slowing down the server bootstrap) just to fix this warning issue.

At this point, I believe the warning commit should be reverted from HornetQ and handled by WildFly. We have all necessary in WildFly management model to check whether there are queues bound for the expiry & dead-letter address when adding/updating an address-setting.

Comment 11 Clebert Suconic 2014-03-10 20:22:58 UTC
https://github.com/jbossas/jboss-eap/pull/1037

Comment 12 Clebert Suconic 2014-03-10 20:23:49 UTC
@Jmesnil: Feel free to squash my PR into yours.. I have created the BZ update:


https://bugzilla.redhat.com/show_bug.cgi?id=1074682

Comment 13 Jeff Mesnil 2014-03-11 10:14:02 UTC
The issue WFLY-3094 will fix this issue the proper way:

1. add ExpiryQueue and DLQ to full configuration
2. log a warning if a address-settings define an expiry-address / dead-letter-address that is not bound to any resources

PR for upstream wildfly: https://github.com/wildfly/wildfly/pull/6015
PR for jboss-eap branch: https://github.com/jbossas/jboss-eap/pull/1044

Comment 14 Miroslav Novak 2014-03-13 10:07:57 UTC
There are no warnings during start with default configuration now which is ok. But when I remove queues:

<jms-queue name="ExpiryQueue">
   <entry name="java:/jms/queue/ExpiryQueue"/>
</jms-queue>
<jms-queue name="DLQ">
   <entry name="java:/jms/queue/DLQ"/>
</jms-queue>

and there is defined:
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>

in address-settings then no warning occurs as Jeff has in 2. in comment #13.

Returning to assigned state.

I'll remove blocker ? flag since this is just a minor issue now. Main problem with warnings during start with default configuration is fixed.

Comment 15 Miroslav Novak 2014-03-13 10:31:17 UTC
The warning are at the top of server.log. I missed it at the first look. Thanks Jeff for pointing this out. Setting as verified.