Description of problem: An application or user can no longer access the CLI in prod. Suspected selinux issue connecting from the user to OPENSHIFT_INTERNAL_IP:9999. This was working before the last prod update. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Create an AS or EAP instance 2. Try to connect from /opt/jboss-as-7.1.0/bin/jboss-cli.sh 3. Actual results: Cannot connect Expected results: Can connect Additional info: Here's the only entry at that time. I'm going to CC: dwalsh and see if he knows why this is being blocked. I'm sure he knows off the top of his head. time->Mon Dec 10 17:13:20 2012 type=SOCKADDR msg=audit(1355177600.083:4352685): saddr=02000000000000000000000000000000 type=SYSCALL msg=audit(1355177600.083:4352685): arch=40000003 syscall=102 success=no exit=-13 a0=2 a1=f77d11c0 a2=44a3e8 a3=f7605d28 items=0 ppid=21950 pid=21957 auid=5960 uid=5960 gid=5960 euid=5960 suid=5960 fsuid=5960 egid=5960 sgid=5960 fsgid=5960 tty=pts0 ses=208951 comm="java" exe="/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9/bin/java" subj=unconfined_u:system_r:openshift_t:s0:c5,c860 key=(null) type=AVC msg=audit(1355177600.083:4352685): avc: denied { node_bind } for pid=21957 comm="java" scontext=unconfined_u:system_r:openshift_t:s0:c5,c860 tcontext=system_u:object_r:node_t:s0:c1023 tclass=tcp_socket Dan, any ideas why these binds to the users IP address are being blocked? Tim On 12/10/2012 04:15 PM, William DeCoste wrote: > I can run your test or try to bind from the client whenever you > want. I just reran the client with a timestamp. The jboss cli > client tries to connect to the listening 127.11.164.1:9999 and > fails. > > [disconnected /] connect 127.11.164.1:9999 The controller is not > available at 127.11.164.1:9999 [disconnected /] exit > > [small-judcon.rhcloud.com data]\> netstat -aop --numeric-ports | > grep 9999 (Not all processes could be identified, non-owned process > info will not be shown, you would have to be root to see it all.) > tcp 0 0 127.11.164.1:9999 *:* LISTEN 25269/java > off (0.00/0/0) tcp 0 0 127.4.33.1:9999 *:* > LISTEN - off (0.00/0/0) tcp 0 0 > 127.9.211.1:9999 *:* LISTEN - off > (0.00/0/0) tcp 0 0 127.6.121.1:9999 *:* > LISTEN - off (0.00/0/0) tcp 0 0 > 127.6.13.1:9999 *:* LISTEN - off (0.00/0/0) > tcp 0 0 127.7.131.129:9999 *:* LISTEN - > off (0.00/0/0) tcp 0 0 127.3.188.129:9999 *:* > LISTEN - off (0.00/0/0) tcp 0 0 > 127.5.76.129:9999 *:* LISTEN - off > (0.00/0/0) tcp 0 0 127.10.135.129:9999 *:* > LISTEN - off (0.00/0/0) tcp 0 0 > 127.11.73.1:9999 *:* LISTEN - off > (0.00/0/0) tcp 0 0 127.3.243.129:9999 *:* > LISTEN - off (0.00/0/0) tcp 0 0 > 127.9.4.129:9999 *:* LISTEN - off > (0.00/0/0) [small-judcon.rhcloud.com data]\> date Mon Dec 10 > 17:13:56 EST 2012
https://issues.jboss.org/browse/AS7-6223
Created attachment 666373 [details] AS7 protocol patch
Created attachment 666374 [details] index file
Created attachment 666375 [details] module file
Added 3 files that will patch this issue: 1) Add the 3 files to .openshift/config/modules/org/jboss/as/protocol/main 2) Add "export JAVA_OPTS="... -Djboss.client_socket_bind_address=${OPENSHIFT_INTENRAL_IP}" to .openshift/actions_hooks/pre_start_jbossas_7 3) git add/commit/push
Created attachment 666445 [details] eap protocol patch
Added eap patch. Extract the tar into the root dir of the app and follow steps 2 and 3 above.
Created attachment 693530 [details] eap601 patch
This should be fixed with the upgrade to EAP6.1. Sending to QA
Will verify it when the EAP version upgrade to 6.1, now the version is 6.0, please refer to the following information: check the EAP version is 6.0 on devenv_3409 as below: # rhc cartridge list |grep jbosseap jbosseap-6.0 (*) JBoss Enterprise Application Platform 6.0 web
EAP was upgraded to 6.1 several weeks ago. The cartridge still says 6.0 but the underlying version is 6.1. You can see this in server.log.
According to comment 11, check it on devenv_3414, this issue is reproduced, and met error messages like "Failed to setup the history file: /var/lib/openshift/e22e36d6de0b11e29f9012313d1f9a29/.jboss-cli-history", found the '.jboss-cli-history' was not existing in app's home dir, please refer to the following results: Have use two ways to check this problem as below: 1. steps not follow comment 7 1) create an eap app rhc app create ceap00 jbosseap-6.0 2) ssh into this app and run jboss-cli.sh \> /opt/jboss-as-7.1.1.Final/bin/jboss-cli.sh Failed to setup the history file: /var/lib/openshift/e22e36d6de0b11e29f9012313d1f9a29/.jboss-cli-history java.io.FileNotFoundException: /var/lib/openshift/e22e36d6de0b11e29f9012313d1f9a29/.jboss-cli-history (Permission denied) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:212) at java.io.FileOutputStream.<init>(FileOutputStream.java:165) at java.io.FileWriter.<init>(FileWriter.java:90) at jline.History.setHistoryFile(History.java:45) at org.jboss.as.cli.impl.Console$Factory$1.setHistoryFile(Console.java:143) at org.jboss.as.cli.impl.CommandContextImpl.initBasicConsole(CommandContextImpl.java:269) at org.jboss.as.cli.impl.CommandContextImpl.<init>(CommandContextImpl.java:257) at org.jboss.as.cli.impl.CommandContextFactoryImpl.newCommandContext(CommandContextFactoryImpl.java:63) at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:224) at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:207) at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:34) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.modules.Module.run(Module.java:260) at org.jboss.modules.Main.main(Main.java:291) You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands. 2. steps follow comment 7 1) create an eap app rhc app create ceap00 jbosseap-6.0 2) Extract the tar into the root dir of this app gunzip eap601_protocol_patch.tar.gz tar xvf eap601_protocol_patch.tar -C /root/test/ceap00/ tar xvf eapPatch.tar -C /root/test/ceap00/ 3) add JAVA_OPTS env var echo 'export JAVA_OPTS="... -Djboss.client_socket_bind_address=${OPENSHIFT_INTENRAL_IP}"' > .openshift/action_hooks/pre_start_jbossas_7 OR echo 'export JAVA_OPTS="... -Djboss.client_socket_bind_address=${OPENSHIFT_JBOSSEAP_IP}"' > .openshift/action_hooks/pre_start_jbossas_7 4) push the changes git add . git commit -amp git push 5) ssh into this app and run jboss-cli.sh \> /opt/jboss-as-7.1.1.Final/bin/jboss-cli.sh Failed to setup the history file: /var/lib/openshift/e22e36d6de0b11e29f9012313d1f9a29/.jboss-cli-history java.io.FileNotFoundException: /var/lib/openshift/e22e36d6de0b11e29f9012313d1f9a29/.jboss-cli-history (Permission denied) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:212) at java.io.FileOutputStream.<init>(FileOutputStream.java:165) at java.io.FileWriter.<init>(FileWriter.java:90) at jline.History.setHistoryFile(History.java:45) at org.jboss.as.cli.impl.Console$Factory$1.setHistoryFile(Console.java:143) at org.jboss.as.cli.impl.CommandContextImpl.initBasicConsole(CommandContextImpl.java:269) at org.jboss.as.cli.impl.CommandContextImpl.<init>(CommandContextImpl.java:257) at org.jboss.as.cli.impl.CommandContextFactoryImpl.newCommandContext(CommandContextFactoryImpl.java:63) at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:224) at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:207) at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:34) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.modules.Module.run(Module.java:260) at org.jboss.modules.Main.main(Main.java:291) You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
I created an OpenShift app and tried to connect as I did previously. 1) ssh into the openshift app, then at the command line: $export JBOSS_HOME=/var/lib/openshift/51d474784382ec75ad000034/jbosseap $java -jar $JBOSS_HOME/jboss-modules.jar -mp $JBOSS_HOME/modules org.jboss.as.cli I got this: WARN: can't find jboss-cli.xml. Using default configuration values. You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands. [disconnected /] connect 23.20.226.65 The controller is not available at 23.20.226.65:9999: java.net.ConnectException: JBAS012144: Could not connect to remote://23.20.226.65:9999. The connection timed out: JBAS012144: Could not connect to remote://23.20.226.65:9999. The connection timed out [disconnected /]
The jboss-cli.sh script is now packaged, supported, and documented by the jboss-* cartridges as of https://github.com/openshift/origin-server/pull/3038. See the cartridge README.md files for details.
The bug has been fixed via Bug 980487 jboss-cli.sh can connect to the jboss instance. > jboss-cli.sh -c --controller=$OPENSHIFT_JBOSSAS_IP:9999 [standalone.252.129:9999 /] > jboss-cli.sh -c --controller=$OPENSHIFT_JBOSSEAP_IP:9999 [standalone.254.1:9999 /]
Is this available on OpenShift Online yet? I created a JBoss EAP app and still dont see this. One of the env vars is CARTRIDGE_VERSION_2=2. When will these changes be pushed to production - or can I get it now? Mark
Mark, The patch should go live early next month (August). In the meantime, you can work around the issue yourself by doing something similar to the patch: https://github.com/openshift/origin-server/commit/5f6dc4c4236b8892ae8849a40c73ad41bcd430ac From a shell in your application: $ export JAVA_OPTS="-Djboss.management.client_socket_bind_address=$OPENSHIFT_JBOSSAS_IP" $ /usr/share/jbossas/bin/jboss-cli.sh -c --controller=$OPENSHIFT_JBOSSEAP_IP:9999 You can expect to see an error about not being able to ~/.jboss-cli-history, but unless the JBoss CLI tool provides a way to configure the history file location (I couldn't find one) that will be the norm until my patch hits production (which creates that file in a writable location for you). Hope this helps.
Dan, yeah that's what I was trying. I found my issue (didnt update to the new env var name "OPENSHIFT_JBOSSEAP_IP" in my client app). I'm able to connect now Thanks Mark