Bug 1270491 - Failed in connecting to JMX console from openshift webconsole (QE V3 cluster only)
Failed in connecting to JMX console from openshift webconsole (QE V3 cluster ...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Management Console (Show other bugs)
3.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: stlewis@redhat.com
yapei
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-10 06:40 EDT by Xia Zhao
Modified: 2016-05-23 11:08 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-05-23 11:08:22 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Xia Zhao 2015-10-10 06:40:30 EDT
Description of problem:
Get error "The connection to jolokia has failed for an unknown reason, check the javascript console for more details." when attempt to connect to JMX console from browse pod page on openshift webconsole.

Version-Release number of selected component (if applicable):
oc v1.0.6-328-gdf1f19e
kubernetes v1.1.0-alpha.1-653-g86b4e77

Test Environment:
API: api.qe.openshift.com
Console: console.qe.openshift.com

How reproducible:
always

Steps to Reproduce:
1.Deploy java app by running below command, wait until pods are running:
oc create -f http://repo1.maven.org/maven2/io/fabric8/jube/images/fabric8/amq-broker/2.1.1/amq-broker-2.1.1-kubernetes.json`
2.Login web console
3.List pods on  "Browse" / "Pods" , choose the java app pod and click on the 'connect'  button

Actual results:
Not able to connect to JMX console, get error "The connection to jolokia has failed for an unknown reason, check the javascript console for more details." on browser

Expected results:
Should be able to connect to JMX console and view java components there


Additional info:
Found these error in pod logs:

INFO: class: org.apache.deltaspike.core.impl.exclude.GlobalAlternative activated=true
Oct 10, 2015 8:41:38 AM org.apache.deltaspike.core.util.ProjectStageProducer initProjectStage
INFO: Computed the following DeltaSpike ProjectStage: Production
PListStore:[/activemq-data/localhost/tmp_storage] started
java.lang.RuntimeException: java.io.IOException: Failed to create directory 'activemq-data/localhost/KahaDB'
    at org.apache.activemq.store.kahadb.KahaDBStore.size(KahaDBStore.java:1090)
    at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.size(KahaDBPersistenceAdapter.java:214)
    at org.apache.activemq.usage.StoreUsage.retrieveUsage(StoreUsage.java:51)
    at org.apache.activemq.usage.Usage.caclPercentUsage(Usage.java:279)
    at org.apache.activemq.usage.Usage.onLimitChange(Usage.java:184)
    at org.apache.activemq.usage.Usage.setLimit(Usage.java:168)
    at org.apache.activemq.broker.BrokerService.getSystemUsage(BrokerService.java:1104)
    at io.fabric8.amq.AMQBroker.doStart(AMQBroker.java:243)
    at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
    at io.fabric8.amq.Main.main(Main.java:34)
Caused by: java.io.IOException: Failed to create directory 'activemq-data/localhost/KahaDB'
    at org.apache.activemq.util.IOHelper.mkdirs(IOHelper.java:316)
    at org.apache.activemq.store.kahadb.MessageDatabase.createPageFile(MessageDatabase.java:2467)
    at org.apache.activemq.store.kahadb.MessageDatabase.getPageFile(MessageDatabase.java:2594)
    at org.apache.activemq.store.kahadb.KahaDBStore.size(KahaDBStore.java:1088)
    ... 9 more
Failed to Start AMQ_Broker
java.lang.RuntimeException: java.io.IOException: Failed to create directory 'activemq-data/localhost/KahaDB'
    at org.apache.activemq.store.kahadb.KahaDBStore.size(KahaDBStore.java:1090)[activemq-kahadb-store-5.11.1.jar:5.11.1]
    at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.size(KahaDBPersistenceAdapter.java:214)[activemq-kahadb-store-5.11.1.jar:5.11.1]
    at org.apache.activemq.usage.StoreUsage.retrieveUsage(StoreUsage.java:51)[activemq-broker-5.11.1.jar:5.11.1]
    at org.apache.activemq.usage.Usage.caclPercentUsage(Usage.java:279)[activemq-client-5.11.1.jar:5.11.1]
    at org.apache.activemq.usage.Usage.onLimitChange(Usage.java:184)[activemq-client-5.11.1.jar:5.11.1]
    at org.apache.activemq.usage.Usage.setLimit(Usage.java:168)[activemq-client-5.11.1.jar:5.11.1]
    at org.apache.activemq.broker.BrokerService.getSystemUsage(BrokerService.java:1104)[activemq-broker-5.11.1.jar:5.11.1]
    at io.fabric8.amq.AMQBroker.doStart(AMQBroker.java:243)[amq-broker-2.1.1.jar:]
    at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[activemq-client-5.11.1.jar:5.11.1]
    at io.fabric8.amq.Main.main(Main.java:34)[amq-broker-2.1.1.jar:]
Comment 2 Jessica Forrester 2015-10-11 16:05:47 EDT
Stan can you take a look?
Comment 3 chunchen 2015-10-12 03:15:21 EDT
Hi,Stan
  We met similar issue against OSE env before, the root cause was that the master is not a node. Please refer to the following old bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1230483
Comment 4 stlewis@redhat.com 2015-10-12 09:36:38 EDT
@chunchen, think this failure to connect is a different reason, failure to create it's database is a fatal error for the ActiveMQ broker, I would expect that it actually exited.

@Xia Zhao, any additional information in the javascript console?  Typically if the proxy fails to connect to the backend service you'd see a 503 which does get reported in the dialog.

Also, where did you find this particular image for the broker?  It's from last May!  I doubt it will work to be honest, try this one which was released a couple days ago:

http://search.maven.org/remotecontent?filepath=io/fabric8/ipaas/apps/amqbroker/2.2.46/amqbroker-2.2.46-kubernetes.json

This is the current source repo for these apps maintained by the fabric8 team, there's a few old artifacts in maven central now since there's been a lot of reorganization lately:

https://github.com/fabric8io/fabric8-ipaas
Comment 5 Xia Zhao 2015-10-12 23:12:36 EDT
Hi Stan,

Thank you for pointing out that the previous amq broker json file I used is out-of-date, I'll pay attention to this next time. 

On QE V3 cluster I replaced the test with this file:

http://search.maven.org/remotecontent?filepath=io/fabric8/ipaas/apps/amqbroker/2.2.46/amqbroker-2.2.46-kubernetes.json

and found the issue still repro. I didn't see any addtional info from the on-screen error message window and I disabled proxy for my browser during the test, the javascript console told:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://api.qe.openshift.com/api/v1/namespaces/p9/pods/amqbroker-ttme1:8778/proxy/jolokia/?maxDepth=7&maxCollectionSize=500&ignoreErrors=true&canonicalNaming=false. (Reason: missing token 'x-authorization' in CORS header 'Access-Control-Allow-Headers' from CORS preflight channel). <unknown>
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://api.qe.openshift.com/api/v1/namespaces/p9/pods/amqbroker-ttme1:8778/proxy/jolokia/?maxDepth=7&maxCollectionSize=500&ignoreErrors=true&canonicalNaming=false. (Reason: CORS request failed). <unknown>

The pod log still show the same error about "Failed to create directory 'activemq-data/localhost/KahaDB' ".

I also tested the same scenario on AWS instance(devenv-fedora_2444), and find the pod deployed by amqbroker-2.2.46-kubernetes.json can not be accessed, it had worked previously with amq-broker-2.1.1-kubernetes.json. 

More info about the failure on devenv-fedora_2444 with amqbroker-2.2.46-kubernetes.json:
1. Get on screen error with more info this time: "Error: 'dial tcp 172.17.0.3:8778: connection refused'
Trying to reach: 'http://172.17.0.3:8778/jolokia/?maxDepth=7&maxCollectionSize=500&ignoreErrors=true&canonicalNaming=false'"
2. Error in js console shows error code 503:
POST XHR https://ec2-52-23-153-64.compute-1.amazonaws.com:8443/api/v1/namespaces/p1/pods/amqbroker-d7y4p:8778/proxy/jolokia/ [HTTP/1.1 503 Service Unavailable 423ms]
3. The pod is running on web console, but the status is incorrect and can't get any logs by oc command line:
oc get po
NAME              READY     STATUS             RESTARTS   AGE
amqbroker-d7y4p   0/1       CrashLoopBackOff   1          15m
oc logs amqbroker-d7y4p
Pod "amqbroker-d7y4p" in namespace "p1": container "amqbroker" is in waiting state.


Please let me know if any other information is needed from my side.

Thanks,
Xia
Comment 6 stlewis@redhat.com 2015-10-13 08:51:39 EDT
(In reply to Xia Zhao from comment #5)
> Hi Stan,
> 
> Thank you for pointing out that the previous amq broker json file I used is
> out-of-date, I'll pay attention to this next time. 
> 
> On QE V3 cluster I replaced the test with this file:
> 
> http://search.maven.org/remotecontent?filepath=io/fabric8/ipaas/apps/
> amqbroker/2.2.46/amqbroker-2.2.46-kubernetes.json
> 
> and found the issue still repro. I didn't see any addtional info from the
> on-screen error message window and I disabled proxy for my browser during
> the test, the javascript console told:
> 
> Cross-Origin Request Blocked: The Same Origin Policy disallows reading the
> remote resource at
> https://api.qe.openshift.com/api/v1/namespaces/p9/pods/amqbroker-ttme1:8778/
> proxy/jolokia/
> ?maxDepth=7&maxCollectionSize=500&ignoreErrors=true&canonicalNaming=false.
> (Reason: missing token 'x-authorization' in CORS header
> 'Access-Control-Allow-Headers' from CORS preflight channel). <unknown>
> Cross-Origin Request Blocked: The Same Origin Policy disallows reading the
> remote resource at
> https://api.qe.openshift.com/api/v1/namespaces/p9/pods/amqbroker-ttme1:8778/
> proxy/jolokia/
> ?maxDepth=7&maxCollectionSize=500&ignoreErrors=true&canonicalNaming=false.
> (Reason: CORS request failed). <unknown>
> 

This shouldn't be present if that instance is running a build of master, maybe ensure you've cleared your browser's cache?  Just double-checked the origin repo and it shouldn't be sending that header.  That being said, I'm planning on pushing an update to the java console today.

> The pod log still show the same error about "Failed to create directory
> 'activemq-data/localhost/KahaDB' ".
> 
> I also tested the same scenario on AWS instance(devenv-fedora_2444), and
> find the pod deployed by amqbroker-2.2.46-kubernetes.json can not be
> accessed, it had worked previously with amq-broker-2.1.1-kubernetes.json. 
> 
> More info about the failure on devenv-fedora_2444 with
> amqbroker-2.2.46-kubernetes.json:
> 1. Get on screen error with more info this time: "Error: 'dial tcp
> 172.17.0.3:8778: connection refused'
> Trying to reach:
> 'http://172.17.0.3:8778/jolokia/
> ?maxDepth=7&maxCollectionSize=500&ignoreErrors=true&canonicalNaming=false'"
> 2. Error in js console shows error code 503:
> POST XHR
> https://ec2-52-23-153-64.compute-1.amazonaws.com:8443/api/v1/namespaces/p1/
> pods/amqbroker-d7y4p:8778/proxy/jolokia/ [HTTP/1.1 503 Service Unavailable
> 423ms]
> 3. The pod is running on web console, but the status is incorrect and can't
> get any logs by oc command line:
> oc get po
> NAME              READY     STATUS             RESTARTS   AGE
> amqbroker-d7y4p   0/1       CrashLoopBackOff   1          15m
> oc logs amqbroker-d7y4p
> Pod "amqbroker-d7y4p" in namespace "p1": container "amqbroker" is in waiting
> state.
 
Yeah, unfortunately it's quite possible for a pod to be "Running" but for the JVM to not actually be running.  From the console AFAIK I only really have the pod state to go on.

> 
> Please let me know if any other information is needed from my side.
> 
> Thanks,
> Xia
Comment 7 stlewis@redhat.com 2016-03-18 16:05:20 EDT
Added a fix to hide the Java link if the container isn't ready, we were just checking the pod state previously, PR is here -> https://github.com/openshift/origin/pull/8148
Comment 8 stlewis@redhat.com 2016-03-22 09:23:47 EDT
Whoops, totally forgot to set this to the right state...
Comment 9 Xia Zhao 2016-03-22 22:10:38 EDT
Verified on dev_preview int online environment, the JMX and Threads console behaved OK. Closing as fixed. Thanks!

Note You need to log in before you can comment on or make changes to this bug.