Red Hat Bugzilla – Bug 1273690
Can't connect to the javascript console for java pod which has jolokia agent running
Last modified: 2016-05-12 13:09:50 EDT
Created attachment 1084960 [details] 503_error Description of problem: When connect to the javascript console for java pod which has jolokia agent running,prompt:The connection to jolokia has failed with the following error, also check the javascript console for more details. Error: 'net/http: TLS handshake timeout' Trying to reach: 'https://172.17.0.3.:8778/jolokia/?maxDepth=7&maxCollectionSize=500&ignoreErrors=true&canonicalNameing=false' Version-Release number of selected component (if applicable): oc v1.0.6-801-ge74f43a kubernetes v1.2.0-alpha.1-1107-g4c8e6f4 devenv-rhel7_2504 How reproducible: always Steps to Reproduce: 1. Login and create project; 2. Create java pod with command: `oc process -f http://central.maven.org/maven2/io/fabric8/jube/images/fabric8/quickstart-java-simple-mainclass/2.2.28/quickstart-java-simple-mainclass-2.2.28-kubernetes.json |oc create -f -` 3. Wait the pod running , and connect to the javascript console. Actual results: Get 503 error, cann't connect to the javascript console for java pod which has jolokia agent running Expected results: Connect correctly , show the JVM trees and other java plugin details. Additional info:
Can you run the following debug collection script, then upload the results to the BZ? https://raw.githubusercontent.com/openshift/openshift-sdn/master/hack/debug.sh
Created attachment 1087448 [details] log-from-all-in-one-instance
What machine did you run that script from? It attempted to ssh and could not do so... do you have password-less ssh set up correctly (or can you run the ssh-agent to cache your passwords please). Is 172.17.0.3 the correct address for the pod? That doesn't look right... it looks like an IP address from the docker defaults.
I run the script in the ec2 instance, the ip:172.17.0.3 was from the docker defaults.
Can you please run the script from the master, and make sure that the master can ssh to the nodes without a password?
Created attachment 1088798 [details] from-master
Okay, great. There's lots of useful information there. However, I don't understand how you have the versions you do... what version of openshift did you install? And how did you install it? Can you please attach the output from: - rpm -qf `which oc` - rpm -q atomic-openshift-clients - rpm -q atomic-openshift - rpm -q docker - lsmod And finally, where are you trying to connect to the JS console from? And external machine? If so... how are you routing your traffic into the cluster?
So, after getting access to the cluster: - I can ping 10.1.0.18 (the IP address of the pod running jolokia) - I can run the JS console - I can telnet to the port on the pod IP from a node, and something is listening - But when I do: curl -v 'https://10.1.0.18:8778/jolokia/?maxDepth=7&maxCollectionSize=500&ignoreErrors=true&canonicalNaming=false' it just hangs after printing the cert information - The "connect" link in the template section of his app reports "the connection to jolokia failed", as the reported noted, but with a completely different error "Forbidden" and a different URL (it has the node IP 10.1.0.18 rather than a service IP) - From inside the pod (using the Java console): 'curl -v https://localhost:8778/jolokia' hangs similarly - The log on the OpenShift web ui reports no errors (I just see it reporting a generated seed value) Anyway, from all that... given that connectivity works from any node to the pod (pingable), and curl does the https handshake and then hangs (from any node, AND from the pod to 127.0.0.1), I can find nothing wrong with the networking. I'd have to say there's something wrong with the Fabric8 thing running in the container, but I don't know how to debug that, or who can look at it.
I think this just requires testing against an updated quickstarts that's set up to work with the secure connection, will have to find out from a colleague which one to test this with.
Based on Stan's comments above, and from email below, reassigning: I think they're testing against an old quickstart where jolokia isn't configured to work with the SSL connection that the console is using to connect to it. Can't recall if the quickstarts have been updated yet or not, or if there's a better image for them to test with for now.
Hmm, tested with a couple quickstarts on master, specifically this one: https://github.com/fabric8io/ipaas-quickstarts/tree/master/quickstart/cdi/camel-jetty No joy though, same behavior. Will ask Roland Huss hopefully next week to take a look. Definitely requires updated quickstarts, the old java base image used by previous versions of the quickstarts won't work with the console connection anymore as a secure connection is required now.
Some comments: * The current quickstarts are now in the Maven group 'io.fabric8.quickstarts'. However it seems that they are not pushed to Maven central anymore. I just opened an issue for that --> https://github.com/fabric8io/ipaas-quickstarts/issues/918 * Authentication is currently *not* switched on, neither for the quickstart images nor the fabric8 console. However, since OpenShift Origin 1.0.7 authentication is switched on for the OpenShift console to connect to JVM so that might fail. Authentication will be switched on asap (i.e. next week).
I just checked the cdi-camel-jetty quickstart with the latest version and accessing the JVM with hawt.io works fine (with OpenShift authentication switched on now) I checked this wish OpenShift Origin v1.1.0.1 and the 2.2.92 of the quickstarts in https://github.com/fabric8io/ipaas-quickstarts. Could you please retry it again ? As stated in https://github.com/fabric8io/ipaas-quickstarts/issues/918 the quickstarts are distributed now only as archetype, so the steps to check this is: mvn archetype:generate \ -DarchetypeGroupId=io.fabric8.archetypes \ -DarchetypeArtifactId=cdi-camel-jetty-archetype \ -DarchetypeVersion=2.2.92 oc login mvn -Pf8-local-deploy
Roland Huss: When I run `mvn -Pf8-local-deploy` always failed: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project test1: Execution default-compile of goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile failed: Plugin org.apache.maven.plugins:maven-compiler-plugin:3.3 or one of its dependencies could not be resolved: Failed to collect dependencies at org.apache.maven.plugins:maven-compiler-plugin:jar:3.3 -> org.codehaus.plexus:plexus-compiler-api:jar:2.5: Failed to read artifact descriptor for org.codehaus.plexus:plexus-compiler-api:jar:2.5: Could not transfer artifact org.codehaus.plexus:plexus-compiler:pom:2.5 from/to central (https://repo.maven.apache.org/maven2): Connection reset -> [Help 1] I'm not clear with the process, could you please provider a json file like : http://central.maven.org/maven2/io/fabric8/jube/images/fabric8/quickstart-java-simple-mainclass/2.2.28/quickstart-java-simple-mainclass-2.2.28-kubernetes.json then I can use the json file to create resource on OpenShift.
Seems like you can't access the (external) Maven repository. I can't provide a plain OpenShift deployment descriptor because we do neither release the docker images nor the json files anymore since some time. The process has changed completely since. Even when I could provide a OpenShift descriptor, there is no image published which it can pull. The official workflow is described here: https://docs.openshift.com/enterprise/3.1/using_images/xpaas_images/fuse.html#simple-workflow-to-create-an-application-from-maven-archetype-catalog We should get it running this way on your machine. * What Maven and Java version are you using ? * What is your platform ? * Can you try 'mvn install' ? 'guess it will fail, too. With what error message ? * Can you access https://repo.maven.apache.org/maven2 via browser or curl from the machine ?
My env: [root@ip-172-18-8-186 test]# mvn -v Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T11:41:47-05:00) Maven home: /usr/local/apache-maven-3.3.9 Java version: 1.8.0_65, vendor: Oracle Corporation Java home: /usr/java/jdk1.8.0_65/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-229.7.2.el7.x86_64", arch: "amd64", family: "unix" my platform is ec2 instance. `mvn install` fail too, the logs : [ERROR] Plugin org.apache.cxf:cxf-codegen-plugin:3.0.4.redhat-620133 or one of its dependencies could not be resolved: Could not find artifact org.apache.cxf:cxf-codegen-plugin:jar:3.0.4.redhat-620133 in central (https://repo.maven.apache.org/maven2) -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException [root@ip-172-18-8-186 test]# curl https://repo.maven.apache.org/maven2 <html> <head><title>301 Moved Permanently</title></head> <body bgcolor="white"> <center><h1>301 Moved Permanently</h1></center> <hr><center>nginx</center> </body> </html>
Hi: Today, I test on latest ose env, I use mvn to build image, then use the image to create app, the pod can connect well. [root@dhcp-136-93 home]# mvn --version Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00) Maven home: /usr/local/apache-maven-3.3.9 Java version: 1.8.0_65, vendor: Oracle Corporation Java home: /usr/java/jdk1.8.0_65/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "4.0.4-301.fc22.x86_64", arch: "amd64", family: "unix" Please change the status to ON_QA, then I can verify it. thanks!
Confirmed latest ose env, the issuse has fixed.