Bug 912688

Created attachment 699417 [details]
reproducer

I did a typo in 5. step. There should be:
5. start jms producer by "sh start-producer.sh 192.168.40.1 1000"

Created attachment 699440 [details]
reproducer second try

Tried to reproduce here with the reproducer.. Everything ok here.

I need to verify our environment. Looks that this is an environment issue which happens only sometimes.

This shouldnt be "NEW" issues.

I checked the source code of org.hornetq.ra.inflow.HornetQMessageHandler and it looks like that something set endpoint to null between lines 302-313:
public void onMessage(final ClientMessage message) {
...
         endpoint.beforeDelivery(HornetQActivation.ONMESSAGE); // line 302
         
         beforeDelivery = true;
         msg.doBeforeReceive();

         //In the transacted case the message must be acked *before* onMessage is called

         if (transacted)         {
            message.acknowledge();
         }

        ((MessageListener)endpoint).onMessage(msg); // line  313 - this were NPE is thrown
...
}

This happens only sometimes but looks as a problematic issue. I'll increase severity.

Created attachment 718092 [details]
trace logs from second server with mdb

Where's the source for MDB1?

Created attachment 730342 [details]
MdbWithRemoteOutQueueToContaniner1.java

Sources of MDB attached (MdbWithRemoteOutQueueToContaniner1.java)

Can you try again with hornetq CR2? I couldn't manage to recreate the environment. 

You probably already have it and can do it quickly there. Looking at the source code the endpoint is only set to null after a shutdown or an undeploy, and I don't see that happening on your testcase. or am I wrong?

This happens only sometimes on different platforms and JDKs. Tomorrow we'll get EAP 6.1.0.ER4 (HQ 2.3.0.CR2) I'll try to reproduce this issue with this build.

Anyway I'm unable to say what is causing this issue.

Ive tried to recreate this issue with EAP 6.1.0.ER4 (HQ 2.3.0.CR2) to no avail, it all works ok. Looking at the code this can only happen if the MDB activation is stopped, this would either be on server shutdown or if there was some unhandled exception in the MDB and the Application server decided to remove the MDB (usually after n failures altho I am not sure). I am assuming its the latter but with out being able to reproduce it i can't say for sure or find out the underlying issue. I have looked at the logs and it does look like the RA can't connect properly so it may be an edge case issue due to a weird env.

Can you provide an environment that shows this issue so we can progress?

I did some experiments and noticed that both of the servers must be bound to IP alias.

To create IP alias for eth1 interface use commands (under root):
ifconfig eth1:0 192.168.40.1 up
ifconfig eth1:1 192.168.40.2 up

and then start servers by:
sh start-server1.sh 192.168.40.1
sh start-server2.sh 192.168.40.2

and client which sends messages to InQueue to 192.168.40.1:
sh start-producer.sh 192.168.40.1 1000

This is regression against EAP 6.0.1 where this worked without problem.

i have followed these instructions and run the test successfully many times without errors. I can only assume that its environmental.

From discussion with Andy it looks that there was another node on network creating a cluster with first server. Second server with MDB tried to connect to this node in cluster which had not deployed InQueue and failed. This caused that HornetQMessageHandler.teardown was called and set endpoint to null.

Miroslav, is there anything else that needs to be done on this issue, or should it be resolved?

We couldn't replicate this issue at all...


But anyway, the changes I have done this week to fix SpecJMS will highly probably fix any possibility of this issue. The issue on SpecJMS was related to out of order of messages, and some issues with closing the consumers. Those fixes should eliminate any possibility of this happening.


If you have a way to check this issue, please checkout master from hornetq and take a look after replacing the jars and client jar. 


Or wait until we release master, which I already talked to Andy as he's taking care of now.

With Andy we were able to reproduce this issue in one of the workstations. Problem was that there was one more server on the network which was creating cluster with jms server and mdb server tried to connect to this other node:

Apr 08 11:55:04 <ataylor>	the issue is that the RA is connecting to both nodes
Apr 08 11:55:26 <ataylor>	one doesnt have the queue so it fails so we have a semi initialised RA
Apr 08 11:55:39 <ataylor>	which is why the endpoint is torn down and set to bnull
Apr 08 11:55:57 <ataylor>	but the succesful consumers are still receiving messages
Apr 08 11:56:45 <ataylor>	i will check further as we didnt used to load balance across nodes, so not sure why it is now
Apr 08 11:59:56 <mnovak>	from conf of server2 the HornetQ RA should connect only to server1...yes, the cluster thing is probably making a mess here
Apr 08 12:00:32 <ataylor>	yes, its a mis configuration, altho we could handle the teardown and retry better
Apr 08 12:01:01 <ataylor>	basically wre tear down all the consumers and recreat ethem, even the ones that are connected ok
Apr 08 12:01:07 <ataylor>	so it all goes haywire

So are we closing as not a bug?

From the Clebert's comment above it seems that he did some changes which will avoid this NPE. I'll be able to verify this fix in EAP 6.1.0.ER6.

@Clebert 
Could you update the status of this bz, please?

Switched to ASSIGNED. Please don't forget to use the right statuses.

@Miroslav: How can I update the status on something we weren't able to replicate?


AFAIK Andy Taylor determined there was also a config issue.


And we had some issues with ordering on the queue....


I don't have any extra data on this.

I would say.. this is not a bug...

From irc discussion dev was able to replicate and investigate this issue on 08th of April. (comment 23)

This appears as a regression against EAP 6.0.1. Could you provide some info why there is a change in behavior?

@miro. I thought we had already confirmed that this was happening because there was a rogue server on the lab?

Yes, but it's not happening with previous version. EAP 6.0.1 could handle such situation without problem.

(In reply to comment #29)
> Yes, but it's not happening with previous version. EAP 6.0.1 could handle
> such situation without problem.

Are you saying that you had rogue servers running every time you ran the test, I don't think so. This is not an issue, the issue happened because there were other servers running on the machine that was causing issues. Please close

There was a situation when close and messages stilll coming through... done thorugh SpecJMS tests...


I don't think this will happen again...


In any case, make sure you test it with isolated clusters.. it will be hard to duplicate any test scenario you have.

Based on feedback from dev this bz will be closed. New enhancement bz#959641 where this behavior can be improved was created.