Bug 1464332 - [dedicated][aws][ded-stage-aws]Logging route is not accessible
[dedicated][aws][ded-stage-aws]Logging route is not accessible
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Logging (Show other bugs)
3.x
Unspecified Unspecified
unspecified Severity medium
: ---
: ---
Assigned To: Jeff Cantrill
Junqi Zhao
: OnlineDedicated
Depends On: 1484664
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-23 02:25 EDT by Junqi Zhao
Modified: 2017-11-13 19:17 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-11-09 13:53:24 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Kibana ui -- Application is not Avaiable (72.27 KB, image/png)
2017-06-23 02:25 EDT, Junqi Zhao
no flags Details
kibana UI (246.74 KB, image/png)
2017-07-31 05:08 EDT, Junqi Zhao
no flags Details

  None (edit)
Description Junqi Zhao 2017-06-23 02:25:13 EDT
Created attachment 1290890 [details]
Kibana ui -- Application is not Avaiable

Description of problem:
logged in https://logs.ded-stage-aws.openshift.com/, it showed "Application is not available" error, checked pods' status, kibana pods were in CrashLoopBackOff status
***** oc get pod -n logging ***** result as below:
 
NAME                          READY     STATUS             RESTARTS   AGE
logging-curator-3-4hhtz       1/1       Running            5          55d
logging-es-6c763qto-6-vjxjr   1/1       Running            3          48d
logging-es-ma1p3rqj-6-q3tsr   1/1       Running            3          48d
logging-fluentd-0ct9v         1/1       Running            41         48d
logging-fluentd-1ngn6         1/1       Running            1          3d
logging-fluentd-39s73         1/1       Running            1          3d
logging-fluentd-4fg64         1/1       Running            1          3d
logging-fluentd-7tth0         1/1       Running            1          3d
logging-fluentd-gdtt2         1/1       Running            1          3d
logging-fluentd-glngj         1/1       Running            1          3d
logging-fluentd-gmpsv         1/1       Running            40         48d
logging-fluentd-jhtg3         1/1       Running            1          3d
logging-fluentd-mdxbq         1/1       Running            3          48d
logging-fluentd-xh0z4         1/1       Running            3          48d
logging-fluentd-xlv6s         1/1       Running            1          3d
logging-kibana-3-3whxd        1/2       CrashLoopBackOff   567        55d
logging-kibana-3-vzgdm        1/2       CrashLoopBackOff   562        55d

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Create one project and populate some logs.
2. login https://logs.ded-stage-aws.openshift.com/ to check logs.
3.

Actual results:
"Application is not available" error shows

Expected results:
Logging route should be accessible


Additional info:
Comment 1 Junqi Zhao 2017-06-23 02:26:15 EDT
Version:
OpenShift Master:v3.5.5.10
Kubernetes Master:v1.5.2+43a9be4
Comment 2 Jeff Cantrill 2017-06-28 18:09:53 EDT
This could be a number of reasons.  Given the version you describe it might be an open issue we have with the kibana-proxy which we have fixed and needs a backport.  Could you please provide additional information, some of which we recommend is here: 

https://github.com/openshift/origin-aggregated-logging/blob/master/docs/issues.md
Comment 3 Junqi Zhao 2017-06-28 20:45:10 EDT
ded-stage-aws environment is not ready now, will provide more info if same error happens again.
Comment 5 Jeff Cantrill 2017-07-26 11:10:58 EDT
Any update on this issue?
Comment 6 Junqi Zhao 2017-07-31 05:08:28 EDT
Tested today, Logging route is accessible now and log entries could be viewed on Kibana UI. See the snapshot

Testing env:

# oc version

oc v3.5.5.31
kubernetes v1.5.2+43a9be4
features: Basic-Auth GSSAPI Kerberos SPNEGO

# oc get pod -n logging

NAME                          READY     STATUS    RESTARTS   AGE
logging-curator-1-6zzz5       1/1       Running   0          9d
logging-es-t1442t5a-1-3zl40   1/1       Running   0          9d
logging-es-yem2xa4b-1-w2scf   1/1       Running   0          9d
logging-fluentd-184hj         1/1       Running   0          9d
logging-fluentd-2f1b8         1/1       Running   0          9d
logging-fluentd-fs0bz         1/1       Running   0          9d
logging-fluentd-ps3fb         1/1       Running   0          9d
logging-fluentd-xvpcn         1/1       Running   0          9d
logging-fluentd-z81c5         1/1       Running   0          9d
logging-kibana-1-qg2c8        2/2       Running   49         9d
Comment 7 Junqi Zhao 2017-07-31 05:08 EDT
Created attachment 1306902 [details]
kibana UI
Comment 8 Junqi Zhao 2017-07-31 05:15:51 EDT
Please change to ON-QA status, will close it once the status is changed.
Comment 9 Junqi Zhao 2017-08-24 01:51:39 EDT
Blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1484664
Comment 10 Junqi Zhao 2017-10-11 01:56:52 EDT
Logging route could be accessed now, although there is one defect
https://bugzilla.redhat.com/show_bug.cgi?id=1500585

Command ***** oc get pod -n logging ***** result as below:

NAME                          READY     STATUS    RESTARTS   AGE
logging-curator-2-8qmlj       1/1       Running   0          5d
logging-es-t1442t5a-2-qp6dr   1/1       Running   0          5d
logging-es-yem2xa4b-2-369dh   1/1       Running   0          5d
logging-fluentd-2zphx         1/1       Running   1          7d
logging-fluentd-4s861         1/1       Running   1          7d
logging-fluentd-8zgz8         1/1       Running   0          5d
logging-fluentd-cj401         1/1       Running   0          5d
logging-fluentd-gbjzp         1/1       Running   1          7d
logging-fluentd-mgkgl         1/1       Running   1          7d
logging-kibana-2-07xn9        2/2       Running   0          5d

env:
OpenShift Master:v3.6.173.0.21
Kubernetes Master:v1.6.1+5115d708d7
Comment 12 Junqi Zhao 2017-11-13 19:17:16 EST
This issue is closed.

Note You need to log in before you can comment on or make changes to this bug.