Bug 1464332

Summary: [dedicated][aws][ded-stage-aws]Logging route is not accessible
Product: OpenShift Online Reporter: Junqi Zhao <juzhao>
Component: LoggingAssignee: Jeff Cantrill <jcantril>
Status: CLOSED CURRENTRELEASE QA Contact: Junqi Zhao <juzhao>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.xCC: aos-bugs, juzhao, pportant
Target Milestone: ---Keywords: OnlineDedicated
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1553654 (view as bug list) Environment:
Last Closed: 2017-11-09 18:53:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1484664    
Bug Blocks: 1553654    
Attachments:
Description Flags
Kibana ui -- Application is not Avaiable
none
kibana UI none

Description Junqi Zhao 2017-06-23 06:25:13 UTC
Created attachment 1290890 [details]
Kibana ui -- Application is not Avaiable

Description of problem:
logged in https://logs.ded-stage-aws.openshift.com/, it showed "Application is not available" error, checked pods' status, kibana pods were in CrashLoopBackOff status
***** oc get pod -n logging ***** result as below:
 
NAME                          READY     STATUS             RESTARTS   AGE
logging-curator-3-4hhtz       1/1       Running            5          55d
logging-es-6c763qto-6-vjxjr   1/1       Running            3          48d
logging-es-ma1p3rqj-6-q3tsr   1/1       Running            3          48d
logging-fluentd-0ct9v         1/1       Running            41         48d
logging-fluentd-1ngn6         1/1       Running            1          3d
logging-fluentd-39s73         1/1       Running            1          3d
logging-fluentd-4fg64         1/1       Running            1          3d
logging-fluentd-7tth0         1/1       Running            1          3d
logging-fluentd-gdtt2         1/1       Running            1          3d
logging-fluentd-glngj         1/1       Running            1          3d
logging-fluentd-gmpsv         1/1       Running            40         48d
logging-fluentd-jhtg3         1/1       Running            1          3d
logging-fluentd-mdxbq         1/1       Running            3          48d
logging-fluentd-xh0z4         1/1       Running            3          48d
logging-fluentd-xlv6s         1/1       Running            1          3d
logging-kibana-3-3whxd        1/2       CrashLoopBackOff   567        55d
logging-kibana-3-vzgdm        1/2       CrashLoopBackOff   562        55d

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Create one project and populate some logs.
2. login https://logs.ded-stage-aws.openshift.com/ to check logs.
3.

Actual results:
"Application is not available" error shows

Expected results:
Logging route should be accessible


Additional info:

Comment 1 Junqi Zhao 2017-06-23 06:26:15 UTC
Version:
OpenShift Master:v3.5.5.10
Kubernetes Master:v1.5.2+43a9be4

Comment 2 Jeff Cantrill 2017-06-28 22:09:53 UTC
This could be a number of reasons.  Given the version you describe it might be an open issue we have with the kibana-proxy which we have fixed and needs a backport.  Could you please provide additional information, some of which we recommend is here: 

https://github.com/openshift/origin-aggregated-logging/blob/master/docs/issues.md

Comment 3 Junqi Zhao 2017-06-29 00:45:10 UTC
ded-stage-aws environment is not ready now, will provide more info if same error happens again.

Comment 5 Jeff Cantrill 2017-07-26 15:10:58 UTC
Any update on this issue?

Comment 6 Junqi Zhao 2017-07-31 09:08:28 UTC
Tested today, Logging route is accessible now and log entries could be viewed on Kibana UI. See the snapshot

Testing env:

# oc version

oc v3.5.5.31
kubernetes v1.5.2+43a9be4
features: Basic-Auth GSSAPI Kerberos SPNEGO

# oc get pod -n logging

NAME                          READY     STATUS    RESTARTS   AGE
logging-curator-1-6zzz5       1/1       Running   0          9d
logging-es-t1442t5a-1-3zl40   1/1       Running   0          9d
logging-es-yem2xa4b-1-w2scf   1/1       Running   0          9d
logging-fluentd-184hj         1/1       Running   0          9d
logging-fluentd-2f1b8         1/1       Running   0          9d
logging-fluentd-fs0bz         1/1       Running   0          9d
logging-fluentd-ps3fb         1/1       Running   0          9d
logging-fluentd-xvpcn         1/1       Running   0          9d
logging-fluentd-z81c5         1/1       Running   0          9d
logging-kibana-1-qg2c8        2/2       Running   49         9d

Comment 7 Junqi Zhao 2017-07-31 09:08:59 UTC
Created attachment 1306902 [details]
kibana UI

Comment 8 Junqi Zhao 2017-07-31 09:15:51 UTC
Please change to ON-QA status, will close it once the status is changed.

Comment 9 Junqi Zhao 2017-08-24 05:51:39 UTC
Blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1484664

Comment 10 Junqi Zhao 2017-10-11 05:56:52 UTC
Logging route could be accessed now, although there is one defect
https://bugzilla.redhat.com/show_bug.cgi?id=1500585

Command ***** oc get pod -n logging ***** result as below:

NAME                          READY     STATUS    RESTARTS   AGE
logging-curator-2-8qmlj       1/1       Running   0          5d
logging-es-t1442t5a-2-qp6dr   1/1       Running   0          5d
logging-es-yem2xa4b-2-369dh   1/1       Running   0          5d
logging-fluentd-2zphx         1/1       Running   1          7d
logging-fluentd-4s861         1/1       Running   1          7d
logging-fluentd-8zgz8         1/1       Running   0          5d
logging-fluentd-cj401         1/1       Running   0          5d
logging-fluentd-gbjzp         1/1       Running   1          7d
logging-fluentd-mgkgl         1/1       Running   1          7d
logging-kibana-2-07xn9        2/2       Running   0          5d

env:
OpenShift Master:v3.6.173.0.21
Kubernetes Master:v1.6.1+5115d708d7

Comment 12 Junqi Zhao 2017-11-14 00:17:16 UTC
This issue is closed.