Hide Forgot
There is a Known Issue in this release of JBoss EAP 6 wherein the remote naming lookup fails intermittently on an IBM JDK with the following exception: ---- org.jboss.remoting3.NotOpenException: Endpoint is not open. ---- This issue appears to present when the client uses the IBM JDK. In instances where the server is on the IBM JDK and the and the client uses another JDK, the issue does not present. The issue is being investigated for a resolution.
=============== Server config: =============== <security-realms> <security-realm name="SSLRealm"> <server-identities> <ssl> <keystore path="/home/jmartisk/Workspace/eap-tests-ejb/manualmode/target/test-classes/ssl/jbossServer.keystore" keystore-password="JBossPassword"/> </ssl> </server-identities> <authentication> <truststore path="/home/jmartisk/Workspace/eap-tests-ejb/manualmode/target/test-classes/ssl/jbossServer.keystore" keystore-password="JBossPassword"/> </authentication> </security-realm> </security-realms> <subsystem xmlns="urn:jboss:domain:remoting:1.1"> <connector name="remoting-connector" socket-binding="remoting" security-realm="SSLRealm"/> </subsystem> =============== Client code (first the appropriate keystores path/password are set): =============== Properties env = new Properties(); env.put(Context.INITIAL_CONTEXT_FACTORY, "org.jboss.naming.remote.client.InitialContextFactory"); env.put(Context.PROVIDER_URL, "remote://127.0.0.1:4447"); env.put("jboss.naming.client.ejb.context", true); env.put("jboss.naming.client.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT", "false"); env.put("jboss.naming.client.connect.options.org.xnio.Options.SSL_STARTTLS", "true"); env.put("jboss.naming.client.connect.options.org.xnio.Options.SSL_ENABLED", "true"); env.put("jboss.naming.client.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS", "false"); env.put("jboss.naming.client.connect.options.org.xnio.Options.SASL_DISALLOWED_MECHANISMS", "false"); env.put("jboss.naming.client.remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED","true"); InitialContext ctx = new InitialContext(env); ctx.lookup .... =============== Error: =============== In all cases, I am getting this error during JNDI lookup: javax.naming.NamingException: Failed to connect to any server. Servers tried: [remote://127.0.0.1:4447] at org.jboss.naming.remote.client.HaRemoteNamingStore.failOverSequence(HaRemoteNamingStore.java:200) at org.jboss.naming.remote.client.HaRemoteNamingStore.namingStore(HaRemoteNamingStore.java:131) at org.jboss.naming.remote.client.HaRemoteNamingStore.namingOperation(HaRemoteNamingStore.java:112) at org.jboss.naming.remote.client.HaRemoteNamingStore.lookup(HaRemoteNamingStore.java:223) at org.jboss.naming.remote.client.RemoteContext.lookup(RemoteContext.java:79) at org.jboss.naming.remote.client.RemoteContext.lookup(RemoteContext.java:83) at javax.naming.InitialContext.lookup(InitialContext.java:422) at org.jboss.qa.ejb.backwardscompatibility.manualmode.ssl.SSLEJBRemoteClientTestCase.testStatefulBeanByte(SSLEJBRemoteClientTestCase.java:260) [on IBM JDK 6 - server side log] Remoting "config-based-naming-client-endpoint" read-1, WRITE: TLSv1 Handshake, length = 81 Finalizer thread, called closeOutbound() Finalizer thread, closeOutboundInternal() Finalizer thread, SEND TLSv1 ALERT: warning, description = close_notify Finalizer thread, WRITE: TLSv1 Alert, length = 2 Finalizer thread, called closeInbound() Finalizer thread, closeInboundInternal() Finalizer thread, closeOutboundInternal() Finalizer thread, called closeOutbound() Finalizer thread, closeOutboundInternal() 08:00:56,946 ERROR [org.jboss.remoting.remote.connection] (Remoting "dev106" read-1) JBREM000200: Remote connection failed: javax.net.ssl.SSLException: Received close_notify during handshake [on IBM JDK 7] .. the "TEST: " on beginning of a line means that the line is from client's log, others are from server Remoting "config-based-naming-client-endpoint" write-1, WRITE: TLSv1 Handshake, length = 48 Remoting "config-based-naming-client-endpoint" write-1, WRITE: TLSv1 Application Data, length = 47 Remoting "config-based-naming-client-endpoint" task-3, WRITE: TLSv1 Application Data, length = 14 main, WRITE: TLSv1 Application Data, length = 38 Remoting "config-based-naming-client-endpoint" task-1, WRITE: TLSv1 Application Data, length = 15 Remoting "config-based-naming-client-endpoint" read-1, WRITE: TLSv1 Application Data, length = 11 main, WRITE: TLSv1 Application Data, length = 19 main, WRITE: TLSv1 Application Data, length = 12 main, WRITE: TLSv1 Application Data, length = 179 main, WRITE: TLSv1 Application Data, length = 12 Finalizer thread, WRITE: TLSv1 Application Data, length = 5 Finalizer thread, called closeOutbound() Finalizer thread, closeOutboundInternal() Finalizer thread, SEND TLSv1 ALERT: warning, description = close_notify Finalizer thread, WRITE: TLSv1 Alert, length = 32 TEST: 15:17:56,357 ERROR [org.jboss.naming.remote.protocol.v1.RemoteNamingStoreV1$MessageReceiver:240] Channel end notification received, closing channel Channel ID ae5865b5 (outbound) of Remoting connection 4793eaad to localhost/127.0.0.1:4447 Finalizer thread, WRITE: TLSv1 Application Data, length = 5 Finalizer thread, called closeOutbound() Finalizer thread, closeOutboundInternal() Finalizer thread, SEND TLSv1 ALERT: warning, description = close_notify Finalizer thread, WRITE: TLSv1 Alert, length = 32 TEST: 15:17:56,360 INFO [org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver:394] EJBCLIENT000016: Channel Channel ID aec87ac8 (outbound) of Remoting connection cc8e4279 to localhost/127.0.0.1:4447 can no longer process messages TEST: 15:17:56,363 ERROR [org.jboss.naming.remote.protocol.v1.RemoteNamingStoreV1$MessageReceiver:240] Channel end notification received, closing channel Channel ID ccf4ed3a (outbound) of Remoting connection cc8e4279 to localhost/127.0.0.1:4447 Finalizer thread, READ: TLSv1 Alert, length = 32 Finalizer thread, RECV TLSv1 ALERT: warning, close_notify Finalizer thread, closeInboundInternal() Finalizer thread, closeOutboundInternal() Finalizer thread, called closeInbound() Finalizer thread, closeInboundInternal() Finalizer thread, closeOutboundInternal() Finalizer thread, called closeOutbound() Finalizer thread, closeOutboundInternal() 15:17:56,374 INFO [org.jboss.as.naming] (Remoting "dhcp-4-145" task-1) JBAS011806: Channel end notification received, closing channel Channel ID 2e5865b5 (inbound) of Remoting connection 1c2ce53f to null 15:17:56,373 ERROR [org.jboss.as.naming] (pool-2-thread-2) JBAS011809: Failed to send exception response to client: org.jboss.remoting3.NotOpenException: Writes closed at org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:108) [jboss-remoting-3.2.15.GA-redhat-1.jar:3.2.15.GA-redhat-1] at org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:297) [jboss-remoting-3.2.15.GA-redhat-1.jar:3.2.15.GA-redhat-1] at org.jboss.naming.remote.protocol.v1.WriteUtil.write(WriteUtil.java:59) at org.jboss.naming.remote.protocol.v1.WriteUtil.writeExceptionResponse(WriteUtil.java:81) at org.jboss.naming.remote.protocol.v1.RemoteNamingServerV1$MessageReciever$1.sendIOException(RemoteNamingServerV1.java:97) at org.jboss.naming.remote.protocol.v1.RemoteNamingServerV1$MessageReciever$1.run(RemoteNamingServerV1.java:86) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1156) [rt.jar:1.7.0] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:626) [rt.jar:1.7.0] at java.lang.Thread.run(Thread.java:780) [vm.jar:1.7.0] =============== Used JDKs =============== java version "1.6.0" Java(TM) SE Runtime Environment (build pxi3260sr13ifix-20130303_02(SR13+IV37419)) IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux x86-32 jvmxi3260sr13-20130114_134867 (JIT enabled, AOT enabled) J9VM - 20130114_134867 JIT - r9_20130108_31100 GC - 20121212_AA) JCL - 20130303_02 java version "1.7.0" Java(TM) SE Runtime Environment (build pxa6470sr4ifix-20130305_01(SR4+IV37419)) IBM J9 VM (build 2.6, JRE 1.7.0 Linux amd64-64 Compressed References 20130205_137358 (JIT enabled, AOT enabled) J9VM - R26_Java726_SR4_20130205_1656_B137358 JIT - r11.b03_20130131_32403 GC - R26_Java726_SR4_20130205_1656_B137358_CMPRSS J9CL - 20130205_137358) JCL - 20130303_01 based on Oracle 7u13-b08 Exactly the same setup works with all other JDKs, just not with IBM JDK 6/7. Also, the 'ejb' protocol works with IBM JDK, only 'remote' doesn't.
I think this has to do with this issue https://issues.jboss.org/browse/AS7-5967
Looking at those log in this bugzilla the Finalizer thread is triggering the close of the remoting channel held by the remote naming context, so yes it does indeed look like https://issues.jboss.org/browse/AS7-5967
This error seems intermittent, I have been able to run the test successfully a few times with this config, but most of the time, it fails. That is probably another reason to think that it is indeed related to AS7-5967.
Passing back, I originally took ownership as the initial description sounded as though there was something specific regarding our use of SSL and the IBM JDK - however subsequent comments now suggest this is a general IBM JDK issue so not specific to SSL.
The issue seems to be on client side: When the client uses IBM JDK and server is on some other JDK, it doesn't work. When the client uses some other JDK and server is on IBM JDK, it works perfectly.
Added a draft Release Note based on my limited understanding from this ticket, the linked JIRA and the IBM.com ticket. Please review for technical accuracy and comment or update the text as required.
"When the client uses IBM JDK and server is on any JDK, including IBM." - the JDK of the server doesn't matter. That is the right version. My previous comment might be confusing, sorry for that.
Thanks Jan. The release note text above seems to reflect this. I assume you updated it (or I just got extremely lucky in my original wording; I can't remember what I wrote verbatim). Unless there's any other feedback received within two hours of this time-stamp, this Release Note will be marked for inclusion in the document.
Assigning jpai@redhat.com EJB issues to david.lloyd@redhat.com. Please re-assign to Cheng or others as needed.
David Lloyd <david.lloyd@redhat.com> updated the status of jira WFLY-1149 to Resolved
I did a preliminary test backporting the remote-naming fix manually to 1.0 branch and it does seem to resolve the issue.
All seems to be working well in EAP 6.4.0.DR10.