Bug 1273690 - Can't connect to the javascript console for java pod which has jolokia agent running
Can't connect to the javascript console for java pod which has jolokia agent ...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Management Console (Show other bugs)
3.x
Unspecified Unspecified
unspecified Severity medium
: ---
: ---
Assigned To: Rob Davies
yapei
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-20 23:24 EDT by zhou ying
Modified: 2016-05-12 13:09 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-05-12 13:09:50 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
503_error (201.30 KB, image/png)
2015-10-20 23:24 EDT, zhou ying
no flags Details
log-from-all-in-one-instance (478 bytes, application/x-gzip)
2015-10-29 07:08 EDT, zhou ying
no flags Details
from-master (2.08 MB, application/x-gzip)
2015-11-03 00:16 EST, zhou ying
no flags Details

  None (edit)
Description zhou ying 2015-10-20 23:24:03 EDT
Created attachment 1084960 [details]
503_error

Description of problem:
When connect to the javascript console for java pod which has jolokia agent running,prompt:The connection to jolokia has failed with the following error, also check the javascript console for more details.
Error: 'net/http: TLS handshake timeout'
Trying to reach: 'https://172.17.0.3.:8778/jolokia/?maxDepth=7&maxCollectionSize=500&ignoreErrors=true&canonicalNameing=false'

Version-Release number of selected component (if applicable):
oc v1.0.6-801-ge74f43a
kubernetes v1.2.0-alpha.1-1107-g4c8e6f4
devenv-rhel7_2504

How reproducible:
always

Steps to Reproduce:
1. Login and create project;
2. Create java pod with command:
  `oc process -f http://central.maven.org/maven2/io/fabric8/jube/images/fabric8/quickstart-java-simple-mainclass/2.2.28/quickstart-java-simple-mainclass-2.2.28-kubernetes.json |oc create -f -`
3. Wait the pod running , and connect to the javascript console.

Actual results:
Get 503 error, cann't connect to the javascript console for java pod which has jolokia agent running

Expected results:
Connect correctly , show the  JVM trees and other java plugin details.

Additional info:
Comment 1 Eric Paris 2015-10-22 15:42:57 EDT
Can you run the following debug collection script, then upload the results to the BZ?  https://raw.githubusercontent.com/openshift/openshift-sdn/master/hack/debug.sh
Comment 2 zhou ying 2015-10-29 07:08 EDT
Created attachment 1087448 [details]
log-from-all-in-one-instance
Comment 3 Ben Bennett 2015-10-29 13:26:00 EDT
What machine did you run that script from?  It attempted to ssh and could not do so... do you have password-less ssh set up correctly (or can you run the ssh-agent to cache your passwords please).

Is 172.17.0.3 the correct address for the pod?  That doesn't look right... it looks like an IP address from the docker defaults.
Comment 4 zhou ying 2015-11-01 21:04:07 EST
I run the script in the ec2 instance, the ip:172.17.0.3 was from the docker defaults.
Comment 5 Ben Bennett 2015-11-02 09:26:45 EST
Can you please run the script from the master, and make sure that the master can ssh to the nodes without a password?
Comment 6 zhou ying 2015-11-03 00:16 EST
Created attachment 1088798 [details]
from-master
Comment 7 Ben Bennett 2015-11-03 09:51:55 EST
Okay, great.  There's lots of useful information there.  However, I don't understand how you have the versions you do... what version of openshift did you install?  And how did you install it?

Can you please attach the output from:
 - rpm -qf `which oc`
 - rpm -q atomic-openshift-clients
 - rpm -q atomic-openshift
 - rpm -q docker
 - lsmod

And finally, where are you trying to connect to the JS console from?  And external machine?  If so... how are you routing your traffic into the cluster?
Comment 13 Ben Bennett 2015-11-04 14:16:15 EST
So, after getting access to the cluster:

- I can ping 10.1.0.18 (the IP address of the pod running jolokia)
- I can run the JS console
- I can telnet to the port on the pod IP from a node, and something is listening
- But when I do: curl -v 'https://10.1.0.18:8778/jolokia/?maxDepth=7&maxCollectionSize=500&ignoreErrors=true&canonicalNaming=false' it just hangs after printing the cert information
- The "connect" link in the template section of his app reports "the connection to jolokia failed", as the reported noted, but with a completely different error "Forbidden" and a different URL (it has the node IP 10.1.0.18 rather than a service IP)
- From inside the pod (using the Java console): 'curl -v https://localhost:8778/jolokia' hangs similarly
- The log on the OpenShift web ui reports no errors (I just see it reporting a generated seed value)

Anyway, from all that... given that connectivity works from any node to the pod (pingable), and curl does the https handshake and then hangs (from any node, AND from the pod to 127.0.0.1), I can find nothing wrong with the networking.  I'd have to say there's something wrong with the Fabric8 thing running in the container, but I don't know how to debug that, or who can look at it.
Comment 15 stlewis@redhat.com 2015-11-04 17:12:58 EST
I think this just requires testing against an updated quickstarts that's set up to work with the secure connection, will have to find out from a colleague which one to test this with.
Comment 16 Ben Bennett 2015-11-05 13:13:22 EST
Based on Stan's comments above, and from email below, reassigning:
  I think they're testing against an old quickstart where jolokia isn't configured to work with the SSL connection that the console is using to connect to it.  Can't recall if the quickstarts have been updated yet or not, or if there's a better image for them to test with for now.
Comment 18 stlewis@redhat.com 2015-11-06 13:24:35 EST
Hmm, tested with a couple quickstarts on master, specifically this one:

https://github.com/fabric8io/ipaas-quickstarts/tree/master/quickstart/cdi/camel-jetty

No joy though, same behavior.  Will ask Roland Huss hopefully next week to take a look.  Definitely requires updated quickstarts, the old java base image used by previous versions of the quickstarts won't work with the console connection anymore as a secure connection is required now.
Comment 19 Roland Huss 2015-11-08 02:29:38 EST
Some comments:

* The current quickstarts are now in the Maven group 'io.fabric8.quickstarts'. However it seems that they are not pushed to Maven central anymore. I just opened an issue for that --> https://github.com/fabric8io/ipaas-quickstarts/issues/918

* Authentication is currently *not* switched on, neither for the quickstart images nor the fabric8 console. However, since OpenShift Origin 1.0.7 authentication is switched on for the OpenShift console to connect to JVM so that might fail. Authentication will be switched on asap (i.e. next week).
Comment 20 Roland Huss 2016-01-11 06:16:15 EST
I just checked the cdi-camel-jetty quickstart with the latest version and accessing the JVM with hawt.io works fine (with OpenShift authentication switched on now)

I checked this wish OpenShift Origin v1.1.0.1 and the 2.2.92 of the quickstarts in https://github.com/fabric8io/ipaas-quickstarts.

Could you please retry it again ?

As stated in https://github.com/fabric8io/ipaas-quickstarts/issues/918 the quickstarts are distributed now only as archetype, so the steps to check this is:


mvn archetype:generate \
      -DarchetypeGroupId=io.fabric8.archetypes \
      -DarchetypeArtifactId=cdi-camel-jetty-archetype \
      -DarchetypeVersion=2.2.92

oc login

mvn -Pf8-local-deploy
Comment 22 zhou ying 2016-01-13 01:08:33 EST
 Roland Huss:

 When I run `mvn -Pf8-local-deploy` always failed:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project test1: Execution default-compile of goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile failed: Plugin org.apache.maven.plugins:maven-compiler-plugin:3.3 or one of its dependencies could not be resolved: Failed to collect dependencies at org.apache.maven.plugins:maven-compiler-plugin:jar:3.3 -> org.codehaus.plexus:plexus-compiler-api:jar:2.5: Failed to read artifact descriptor for org.codehaus.plexus:plexus-compiler-api:jar:2.5: Could not transfer artifact org.codehaus.plexus:plexus-compiler:pom:2.5 from/to central (https://repo.maven.apache.org/maven2): Connection reset -> [Help 1]

I'm not clear with the process, could you please provider a json file like :
http://central.maven.org/maven2/io/fabric8/jube/images/fabric8/quickstart-java-simple-mainclass/2.2.28/quickstart-java-simple-mainclass-2.2.28-kubernetes.json

then I can use the json file to create resource on OpenShift.
Comment 23 Roland Huss 2016-01-13 08:48:39 EST
Seems like you can't access the (external) Maven repository. 

I can't provide a plain OpenShift deployment descriptor because we do neither release the docker images nor the json files anymore since some time. The process has changed completely since.

Even when I could provide a OpenShift descriptor, there is no image published which it can pull.

The official workflow is described here:

https://docs.openshift.com/enterprise/3.1/using_images/xpaas_images/fuse.html#simple-workflow-to-create-an-application-from-maven-archetype-catalog

We should get it running this way on your machine.

* What Maven and Java version are you using ?
* What is your platform ?
* Can you try 'mvn install' ? 'guess it will fail, too. With what error message ?
* Can you access https://repo.maven.apache.org/maven2 via browser or curl from the machine ?
Comment 24 zhou ying 2016-01-15 04:01:16 EST
My env:
[root@ip-172-18-8-186 test]# mvn -v
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T11:41:47-05:00)
Maven home: /usr/local/apache-maven-3.3.9
Java version: 1.8.0_65, vendor: Oracle Corporation
Java home: /usr/java/jdk1.8.0_65/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-229.7.2.el7.x86_64", arch: "amd64", family: "unix"

my platform is ec2 instance.

`mvn install` fail too, the logs :
[ERROR] Plugin org.apache.cxf:cxf-codegen-plugin:3.0.4.redhat-620133 or one of its dependencies could not be resolved: Could not find artifact org.apache.cxf:cxf-codegen-plugin:jar:3.0.4.redhat-620133 in central (https://repo.maven.apache.org/maven2) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException




[root@ip-172-18-8-186 test]# curl  https://repo.maven.apache.org/maven2
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
Comment 25 zhou ying 2016-01-19 03:33:21 EST
Hi:
   Today, I test on latest ose env, I use mvn to build image, then use the image to create app, the pod can connect well.

[root@dhcp-136-93 home]# mvn --version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: /usr/local/apache-maven-3.3.9
Java version: 1.8.0_65, vendor: Oracle Corporation
Java home: /usr/java/jdk1.8.0_65/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.0.4-301.fc22.x86_64", arch: "amd64", family: "unix"


Please change the status to ON_QA, then I can verify it. thanks!
Comment 26 zhou ying 2016-02-03 01:46:58 EST
Confirmed latest ose env, the issuse has fixed.

Note You need to log in before you can comment on or make changes to this bug.