Bug 1480051

Summary: [starter][starter-us-east-1]Metrics route is not accessible
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: HawkularAssignee: Matt Wringe <mwringe>
Status: CLOSED NOTABUG QA Contact: Junqi Zhao <juzhao>
Severity: high Docs Contact:
Priority: high    
Version: unspecifiedCC: aos-bugs
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-10 13:36:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
"Application is not available" error for metrics route none

Description Junqi Zhao 2017-08-10 02:14:11 UTC 
Created attachment 1311512 [details]
"Application is not available" error for metrics route

Description of problem:
starter-us-east-1 env, Metrics route is not accessible."Application is not available" when accesssing metrics route

Command ***** oc get pod -n openshift-infra ***** result as below, hawkular-metrics pod is not reday:

NAME                         READY     STATUS              RESTARTS   AGE
hawkular-cassandra-1-csx0z   1/1       Running             0          1h
hawkular-cassandra-2-rxtzx   0/1       ContainerCreating   0          1h
hawkular-metrics-1kgwg       0/1       Running             9          1h
heapster-z0f0h               0/1       Running             50         7h


Version-Release number of selected component (if applicable):

OpenShift Master:v3.6.173.0.5 (online version 3.5.0.20)
Kubernetes Master:v1.6.1+5115d708d7 

How reproducible:
Always

Steps to Reproduce:
1. login https://metrics.starter-us-east-1.openshift.com/hawkular/metrics
2. 
3.

Actual results:
Application is not available shows in UI. 


Expected results:
Metrics route could be accessed

Additional info:
Comment 1 Matt Wringe 2017-08-10 13:36:58 UTC 
The route is not available because the Hawkular Metrics service doesn't have any running pods currently.

This is always the expected behaviour with the router when there are no running pods behind it, it should be clearly stated in the error message from the router when this happens.

The problem here is most likely because the Cassandra pod is stuck in the 'ContainerCreating' phase. Someone needs to fix that problem first (which is not related to metrics, this is likely caused by something like the PV that Cassandra needs cannot be mounted, or there isn't enough room on the node to deploy the pod)
Comment 2 Junqi Zhao 2017-08-11 01:49:42 UTC 
Metrics route could be accessed now.