Bug 923836 - remote:// protocol does not work over SSL with IBM JDK
Summary: remote:// protocol does not work over SSL with IBM JDK
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: EJB, Naming
Version: 6.1.0
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: DR10
: EAP 6.4.0
Assignee: David M. Lloyd
QA Contact: Jan Martiska
URL:
Whiteboard:
Depends On: JBPAPP6-1680 1162232
Blocks: 1099440 1194960
TreeView+ depends on / blocked
 
Reported: 2013-03-20 14:52 UTC by Jan Martiska
Modified: 2019-08-19 12:43 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
There is a Known Issue in this release of JBoss EAP 6 wherein the remote naming lookup fails intermittently on an IBM JDK with the following exception: ---- org.jboss.remoting3.NotOpenException: Endpoint is not open. ---- This issue appears to present when the client uses the IBM JDK. In instances where the server is on the IBM JDK and the and the client uses another JDK, the issue does not present. The issue is being investigated for a resolution.
Clone Of:
: 1194960 (view as bug list)
Environment:
Last Closed:
Type: Bug


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
JBoss Issue Tracker WFLY-1149 Major Closed Naming lookup intermittently fails on IBM JDK due to org.jboss.remoting3.NotOpenException: Endpoint is not open. 2018-08-21 16:57:35 UTC

Description Jan Martiska 2013-03-20 14:52:11 UTC
===============
Server config:
===============

<security-realms>
	<security-realm name="SSLRealm">
                <server-identities>
                    <ssl>
                        <keystore path="/home/jmartisk/Workspace/eap-tests-ejb/manualmode/target/test-classes/ssl/jbossServer.keystore" keystore-password="JBossPassword"/>
                    </ssl>
                </server-identities>
                <authentication>
                    <truststore path="/home/jmartisk/Workspace/eap-tests-ejb/manualmode/target/test-classes/ssl/jbossServer.keystore" keystore-password="JBossPassword"/>
                </authentication>
            </security-realm>
</security-realms>

<subsystem xmlns="urn:jboss:domain:remoting:1.1">
            <connector name="remoting-connector" socket-binding="remoting" security-realm="SSLRealm"/>
</subsystem>

===============
Client code (first the appropriate keystores path/password are set):
===============

Properties env = new Properties();
env.put(Context.INITIAL_CONTEXT_FACTORY, "org.jboss.naming.remote.client.InitialContextFactory");
env.put(Context.PROVIDER_URL, "remote://127.0.0.1:4447");
env.put("jboss.naming.client.ejb.context", true);
env.put("jboss.naming.client.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT", "false");
env.put("jboss.naming.client.connect.options.org.xnio.Options.SSL_STARTTLS", "true");
env.put("jboss.naming.client.connect.options.org.xnio.Options.SSL_ENABLED", "true");
env.put("jboss.naming.client.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS", "false");
env.put("jboss.naming.client.connect.options.org.xnio.Options.SASL_DISALLOWED_MECHANISMS", "false");
env.put("jboss.naming.client.remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED","true");
InitialContext ctx = new InitialContext(env);
ctx.lookup ....

===============
Error:
===============

In all cases, I am getting this error during JNDI lookup:
javax.naming.NamingException: Failed to connect to any server. Servers tried: [remote://127.0.0.1:4447]
        at org.jboss.naming.remote.client.HaRemoteNamingStore.failOverSequence(HaRemoteNamingStore.java:200)
        at org.jboss.naming.remote.client.HaRemoteNamingStore.namingStore(HaRemoteNamingStore.java:131)
        at org.jboss.naming.remote.client.HaRemoteNamingStore.namingOperation(HaRemoteNamingStore.java:112)
        at org.jboss.naming.remote.client.HaRemoteNamingStore.lookup(HaRemoteNamingStore.java:223)
        at org.jboss.naming.remote.client.RemoteContext.lookup(RemoteContext.java:79)
        at org.jboss.naming.remote.client.RemoteContext.lookup(RemoteContext.java:83)
        at javax.naming.InitialContext.lookup(InitialContext.java:422)
        at org.jboss.qa.ejb.backwardscompatibility.manualmode.ssl.SSLEJBRemoteClientTestCase.testStatefulBeanByte(SSLEJBRemoteClientTestCase.java:260)


[on IBM JDK 6 - server side log]
Remoting "config-based-naming-client-endpoint" read-1, WRITE: TLSv1 Handshake, length = 81
Finalizer thread, called closeOutbound()
Finalizer thread, closeOutboundInternal()
Finalizer thread, SEND TLSv1 ALERT:  warning, description = close_notify
Finalizer thread, WRITE: TLSv1 Alert, length = 2
Finalizer thread, called closeInbound()
Finalizer thread, closeInboundInternal()
Finalizer thread, closeOutboundInternal()
Finalizer thread, called closeOutbound()
Finalizer thread, closeOutboundInternal()
08:00:56,946 ERROR [org.jboss.remoting.remote.connection] (Remoting "dev106" read-1) JBREM000200: Remote connection failed: javax.net.ssl.SSLException: Received close_notify during handshake

[on IBM JDK 7] .. the "TEST: " on beginning of a line means that the line is from client's log, others are from server
Remoting "config-based-naming-client-endpoint" write-1, WRITE: TLSv1 Handshake, length = 48
Remoting "config-based-naming-client-endpoint" write-1, WRITE: TLSv1 Application Data, length = 47
Remoting "config-based-naming-client-endpoint" task-3, WRITE: TLSv1 Application Data, length = 14
main, WRITE: TLSv1 Application Data, length = 38
Remoting "config-based-naming-client-endpoint" task-1, WRITE: TLSv1 Application Data, length = 15
Remoting "config-based-naming-client-endpoint" read-1, WRITE: TLSv1 Application Data, length = 11
main, WRITE: TLSv1 Application Data, length = 19
main, WRITE: TLSv1 Application Data, length = 12
main, WRITE: TLSv1 Application Data, length = 179
main, WRITE: TLSv1 Application Data, length = 12
Finalizer thread, WRITE: TLSv1 Application Data, length = 5
Finalizer thread, called closeOutbound()
Finalizer thread, closeOutboundInternal()
Finalizer thread, SEND TLSv1 ALERT:  warning, description = close_notify
Finalizer thread, WRITE: TLSv1 Alert, length = 32
TEST: 15:17:56,357 ERROR [org.jboss.naming.remote.protocol.v1.RemoteNamingStoreV1$MessageReceiver:240] Channel end notification received, closing channel Channel ID ae5865b5 (outbound) of Remoting connection 4793eaad to localhost/127.0.0.1:4447
Finalizer thread, WRITE: TLSv1 Application Data, length = 5
Finalizer thread, called closeOutbound()
Finalizer thread, closeOutboundInternal()
Finalizer thread, SEND TLSv1 ALERT:  warning, description = close_notify
Finalizer thread, WRITE: TLSv1 Alert, length = 32
TEST: 15:17:56,360 INFO  [org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver:394] EJBCLIENT000016: Channel Channel ID aec87ac8 (outbound) of Remoting connection cc8e4279 to localhost/127.0.0.1:4447 can no longer process messages
TEST: 15:17:56,363 ERROR [org.jboss.naming.remote.protocol.v1.RemoteNamingStoreV1$MessageReceiver:240] Channel end notification received, closing channel Channel ID ccf4ed3a (outbound) of Remoting connection cc8e4279 to localhost/127.0.0.1:4447
Finalizer thread, READ: TLSv1 Alert, length = 32
Finalizer thread, RECV TLSv1 ALERT:  warning, close_notify
Finalizer thread, closeInboundInternal()
Finalizer thread, closeOutboundInternal()
Finalizer thread, called closeInbound()
Finalizer thread, closeInboundInternal()
Finalizer thread, closeOutboundInternal()
Finalizer thread, called closeOutbound()
Finalizer thread, closeOutboundInternal()
15:17:56,374 INFO  [org.jboss.as.naming] (Remoting "dhcp-4-145" task-1) JBAS011806: Channel end notification received, closing channel Channel ID 2e5865b5 (inbound) of Remoting connection 1c2ce53f to null
15:17:56,373 ERROR [org.jboss.as.naming] (pool-2-thread-2) JBAS011809: Failed to send exception response to client: org.jboss.remoting3.NotOpenException: Writes closed
	at org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:108) [jboss-remoting-3.2.15.GA-redhat-1.jar:3.2.15.GA-redhat-1]
	at org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:297) [jboss-remoting-3.2.15.GA-redhat-1.jar:3.2.15.GA-redhat-1]
	at org.jboss.naming.remote.protocol.v1.WriteUtil.write(WriteUtil.java:59)
	at org.jboss.naming.remote.protocol.v1.WriteUtil.writeExceptionResponse(WriteUtil.java:81)
	at org.jboss.naming.remote.protocol.v1.RemoteNamingServerV1$MessageReciever$1.sendIOException(RemoteNamingServerV1.java:97)
	at org.jboss.naming.remote.protocol.v1.RemoteNamingServerV1$MessageReciever$1.run(RemoteNamingServerV1.java:86)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1156) [rt.jar:1.7.0]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:626) [rt.jar:1.7.0]
	at java.lang.Thread.run(Thread.java:780) [vm.jar:1.7.0]


===============
Used JDKs
===============
java version "1.6.0"
Java(TM) SE Runtime Environment (build pxi3260sr13ifix-20130303_02(SR13+IV37419))
IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux x86-32 jvmxi3260sr13-20130114_134867 (JIT enabled, AOT enabled)
J9VM - 20130114_134867
JIT  - r9_20130108_31100
GC   - 20121212_AA)
JCL  - 20130303_02

java version "1.7.0"
Java(TM) SE Runtime Environment (build pxa6470sr4ifix-20130305_01(SR4+IV37419))
IBM J9 VM (build 2.6, JRE 1.7.0 Linux amd64-64 Compressed References 20130205_137358 (JIT enabled, AOT enabled)
J9VM - R26_Java726_SR4_20130205_1656_B137358
JIT  - r11.b03_20130131_32403
GC   - R26_Java726_SR4_20130205_1656_B137358_CMPRSS
J9CL - 20130205_137358)
JCL - 20130303_01 based on Oracle 7u13-b08

Exactly the same setup works with all other JDKs, just not with IBM JDK 6/7. Also, the 'ejb' protocol works with IBM JDK, only 'remote' doesn't.

Comment 1 Jaikiran Pai 2013-03-22 12:35:27 UTC
I think this has to do with this issue https://issues.jboss.org/browse/AS7-5967

Comment 2 Jaikiran Pai 2013-03-22 12:37:24 UTC
Looking at those log in this bugzilla the Finalizer thread is triggering the close of the remoting channel held by the remote naming context, so yes it does indeed look like https://issues.jboss.org/browse/AS7-5967

Comment 3 Jan Martiska 2013-03-22 14:07:51 UTC
This error seems intermittent, I have been able to run the test successfully a few times with this config, but most of the time, it fails. That is probably another reason to think that it is indeed related to AS7-5967.

Comment 4 Darran Lofthouse 2013-03-25 10:08:27 UTC
Passing back, I originally took ownership as the initial description sounded as though there was something specific regarding our use of SSL and the IBM JDK - however subsequent comments now suggest this is a general IBM JDK issue so not specific to SSL.

Comment 5 Jan Martiska 2013-04-15 10:04:04 UTC
The issue seems to be on client side:
When the client uses IBM JDK and server is on some other JDK, it doesn't work.
When the client uses some other JDK and server is on IBM JDK, it works perfectly.

Comment 6 Scott Mumford 2013-05-15 22:52:49 UTC
Added a draft Release Note based on my limited understanding from this ticket, the linked JIRA and the IBM.com ticket.

Please review for technical accuracy and comment or update the text as required.

Comment 7 Jan Martiska 2013-05-16 09:41:33 UTC
"When the client uses IBM JDK and server is on any JDK, including IBM." - the JDK of the server doesn't matter. That is the right version. My previous comment might be confusing, sorry for that.

Comment 8 Scott Mumford 2013-05-16 23:44:49 UTC
Thanks Jan.

The release note text above seems to reflect this. I assume you updated it (or I just got extremely lucky in my original wording; I can't remember what I wrote verbatim).

Unless there's any other feedback received within two hours of this time-stamp, this Release Note will be marked for inclusion in the  document.

Comment 13 Dimitris Andreadis 2013-10-24 18:28:06 UTC
Assigning jpai@redhat.com EJB issues to david.lloyd@redhat.com. Please re-assign to Cheng or others as needed.

Comment 14 JBoss JIRA Server 2014-11-10 14:23:24 UTC
David Lloyd <david.lloyd@redhat.com> updated the status of jira WFLY-1149 to Resolved

Comment 17 Jan Martiska 2014-11-11 07:14:19 UTC
I did a preliminary test backporting the remote-naming fix manually to 1.0 branch and it does seem to resolve the issue.

Comment 19 Jan Martiska 2014-11-25 13:15:20 UTC
All seems to be working well in EAP 6.4.0.DR10.


Note You need to log in before you can comment on or make changes to this bug.