Bug 1120323 - Users cannot connect to remote jms topics within a gear
Summary: Users cannot connect to remote jms topics within a gear
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Containers
Version: 2.x
Hardware: All
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: Miciah Dashiel Butler Masters
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-07-16 18:12 UTC by Timothy Williams
Modified: 2018-12-09 18:10 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-01-08 20:28:44 UTC
jhonce: needinfo-


Attachments (Terms of Use)
Naming Connection Test (1.68 KB, application/gzip)
2014-07-16 18:12 UTC, Timothy Williams
no flags Details

Description Timothy Williams 2014-07-16 18:12:47 UTC
Created attachment 918492 [details]
Naming Connection Test

Description of problem:
Users are unable to connect to remote jms topics within a gear. On any other openshift installation, this works without issue. However, openshift.com appears to be blocking it in some way.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Create an external jms topic
2. Attempt to connect to and read from the jms topic.
3.

Actual results:
List failed: javax.naming.NamingException: Failed to create remoting connection [Root exception is java.lang.RuntimeException: Operation failed with status WAITING]

Expected results:
Connected

Additional info:

We are using jboss-client.jar to test this, as well as a small script Eric Rich wrote. I have attached the script to this email. It is run using the following:
-=~~~~~~~~~~~~~~~~~~~~~~~~~~=-
# java -cp $JBOSS_HOME/bin/client/jboss-client.jar:. NamingTest HOST:PORT USER_NAME PASSWORD SEARCH_STRING

Example: java -cp $JBOSS_HOME/bin/client/jboss-client.jar:. NamingTest 127.0.0.1:4447 admin redhat1234* ""
-=~~~~~~~~~~~~~~~~~~~~~~~~~~=-

Run from my local machine (the customer has no password on this topic):

$ java -cp jboss-client.jar:. NamingTest jms.example.com:38054 RemoteConnectionFactorory "" ""
Trying to Connect to jms.example.com:38054
Authenticating as: RemoteConnectionFactorory
Using Password:
Searching for:
Jul 15, 2014 6:30:25 PM org.xnio.Xnio <clinit>
INFO: XNIO Version 3.0.4.GA-redhat-1
Jul 15, 2014 6:30:25 PM org.xnio.nio.NioXnio <clinit>
INFO: XNIO NIO Implementation Version 3.0.4.GA-redhat-1
Jul 15, 2014 6:30:25 PM org.jboss.remoting3.EndpointImpl <clinit>
INFO: JBoss Remoting version 3.2.8.GA-redhat-1
jms: javax.naming.Context
Jul 15, 2014 6:30:26 PM org.jboss.naming.remote.protocol.v1.RemoteNamingStoreV1$MessageReceiver handleEnd
ERROR: Channel end notification received, closing channel Channel ID 856ee5c5 (outbound) of Remoting connection 08373b5c to jms.example.com/50.19.127.108:38054

We were able to connect.

Run from openshift.com application:

$ java -cp jboss-client.jar:. NamingTest jms.example.com:38054 RemoteConnectionFactory "" "" ""
Trying to Connect to jms.example.com:38054
Authenticating as: RemoteConnectionFactory
Using Password:
Searching for:
Jul 15, 2014 6:02:06 PM org.xnio.Xnio <clinit>
INFO: XNIO Version 3.0.4.GA-redhat-1
Jul 15, 2014 6:02:06 PM org.xnio.nio.NioXnio <clinit>
INFO: XNIO NIO Implementation Version 3.0.4.GA-redhat-1
Jul 15, 2014 6:02:06 PM org.jboss.remoting3.EndpointImpl <clinit>
INFO: JBoss Remoting version 3.2.8.GA-redhat-1
List failed: javax.naming.NamingException: Failed to create remoting connection [Root exception is java.lang.RuntimeException: Operation failed with status WAITING]

We were unable to connect. This error we receive is a general error that means either the host, port, user, or password are incorrect as we timed out waiting to connect.

Run from an application on the internal IT openshift instance:

java -cp jboss-client.jar:. NamingTest jms.example.com:38054 RemoteConnectionFactory "" ""
Trying to Connect to jms.exmaple.com:38054
Authenticating as: RemoteConnectionFactory
Using Password:
Searching for:
Jul 15, 2014 6:06:45 PM org.xnio.Xnio <clinit>
INFO: XNIO Version 3.0.4.GA-redhat-1
Jul 15, 2014 6:06:45 PM org.xnio.nio.NioXnio <clinit>
INFO: XNIO NIO Implementation Version 3.0.4.GA-redhat-1
Jul 15, 2014 6:06:45 PM org.jboss.remoting3.EndpointImpl <clinit>
INFO: JBoss Remoting version 3.2.8.GA-redhat-1
jms: javax.naming.Context
Jul 15, 2014 6:06:47 PM org.jboss.naming.remote.protocol.v1.RemoteNamingStoreV1$MessageReceiver handleEnd
ERROR: Channel end notification received, closing channel Channel ID ec8cf4a5 (outbound) of Remoting connection 5b142447 to jms.example.com/50.19.127.108:38054

We were able to connect.

Comment 1 Jhon Honce 2014-07-16 18:33:13 UTC
Port 38054 is not open for outbound connections.

Comment 2 Ben Parees 2014-07-18 18:14:40 UTC
Additional info:

Disabling SElinux resolves this issue.  With SELinux enabled, there is a denial of a node_bind attempt:
type=AVC msg=audit(1405716967.843:3208): avc:  denied  { node_bind } for  pid=14289 comm="java" scontext=unconfined_u:system_r:openshift_t:s0:c0,c1000 tcontext=system_u:object_r:node_t:s0:c1023 tclass=tcp_socket

And the full exception stack reported from java is:

javax.naming.NamingException: Failed to create remoting connection [Root exception is java.lang.RuntimeException: Operation failed with status WAITING]
at org.jboss.naming.remote.client.ClientUtil.namingException(ClientUtil.java:51)
at org.jboss.naming.remote.client.InitialContextFactory.getInitialContext(InitialContextFactory.java:151)
at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:307)
at javax.naming.InitialContext.init(InitialContext.java:242)
at javax.naming.InitialContext.<init>(InitialContext.java:216)
at NamingTest.getRemoteNamingInitialContext(NamingTest.java:43)
at NamingTest.main(NamingTest.java:14)
Caused by: java.lang.RuntimeException: Operation failed with status WAITING
at org.jboss.naming.remote.protocol.IoFutureHelper.get(IoFutureHelper.java:89)
at org.jboss.naming.remote.client.NamingStoreCache.getRemoteNamingStore(NamingStoreCache.java:68)
at org.jboss.naming.remote.client.InitialContextFactory.getOrCreateCachedNamingStore(InitialContextFactory.java:196)
at org.jboss.naming.remote.client.InitialContextFactory.getOrCreateNamingStore(InitialContextFactory.java:169)
at org.jboss.naming.remote.client.InitialContextFactory.getInitialContext(InitialContextFactory.java:134)
... 6 more
List failed: javax.naming.NamingException: Failed to create remoting connection [Root exception is java.lang.RuntimeException: Operation failed with status WAITING]

I can't figure out what in that stack is attempting (and failing) to bind... my guess is that something else somewhere is attempting to bind and silently failing, which later causes this failure... that's my best guess.  Possibly the CachedNamingStore.

However w/ selinux disabled and using netstat, I cannot find an additional port that's being opened/bound, so i'm still puzzled as to what exactly selinux is denying.

Comment 8 Miciah Dashiel Butler Masters 2016-01-08 20:28:44 UTC
For security reasons, we prefer not to open ports without need.  Please consider using port forwarding, as described in the following document, if you need to establish JMS connections into gears from an external source:

https://access.redhat.com/documentation/en-US/OpenShift_Online/2.0/html-single/User_Guide/index.html#sect-Port_Forwarding

Let us know if there is a problem with using port forwarding to satisfy your requirements.  If it turns out that we really cannot get around opening the port for direct access to JMS, we can re-open this Bugzilla report and look into amending security policy to permit access via port 38054.

Thanks!


Note You need to log in before you can comment on or make changes to this bug.