Bug 1596483

Summary: [free-stg]user project index is not created
Product: OpenShift Online Reporter: Junqi Zhao <juzhao>
Component: LoggingAssignee: Jeff Cantrill <jcantril>
Status: CLOSED NOTABUG QA Contact: Junqi Zhao <juzhao>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.xCC: aos-bugs, juzhao, rmeggins
Target Milestone: ---Keywords: OnlineStarter
Target Release: 3.x   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-02 02:06:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
could not view user project logs on kibana UI -- free-stg
none
"Timed out while waiting for a green or yellow cluster state" in ES pod log none

Description Junqi Zhao 2018-06-29 05:56:39 UTC 
Created attachment 1455425 [details]
could not view user project logs on kibana UI -- free-stg

Description of problem:
free-stg cluster, created one project "java2" and created one pod there to populate logs, logged in kibana UI https://logs.free-stg.openshift.com/, error showed "Discover: [exception] The index 'project..empty-project.*' was not found. This could mean data has not yet been collected."

Checked kibana UI after 3 hours, it's still same error, user project logs could not be viewed from kibana.

Error in ES pod logs
ERR: Timed out while waiting for a green or yellow cluster state.
   * Try running sgadmin.sh with -icl and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
86

Not sure whether it is related to Bug 1596445 or not

Version-Release number of selected component (if applicable):
OpenShift Master:v3.10.9 
Kubernetes Master:v1.10.0+b81c8f8 
OpenShift Web Console:v3.10.9 

logging-curator:v3.10.9
oauth-proxy:v3.10.2
logging-elasticsearch:v3.10.2
logging-fluentd:v3.10.9
logging-kibana:v3.10.9
logging-auth-proxy:v3.10.9

How reproducible:
Always

Steps to Reproduce:
1. Create one project and pod to populate logs
$ oc new-project java
$ oc new-app --docker-image=docker.io/chunyunchen/java-mainclass:2.2.94-SNAPSHOT

2. After a few minutes, check logs from kibana UI
3.

Actual results:
error showed "Discover: [exception] The index 'project..empty-project.*' was not found. This could mean data has not yet been collected."
user project logs could not be viewed from kibana

Expected results:
user project logs could be viewed from kibana


Additional info:
Comment 1 Junqi Zhao 2018-06-29 05:58:29 UTC 
Created attachment 1455426 [details]
"Timed out while waiting for a green or yellow cluster state" in ES pod log
Comment 2 Jeff Cantrill 2018-06-29 16:36:41 UTC 
Please attach information about the indices state:

'oc exec -c elasticsearch $ESPOD -- indices'

Also, you may be able to work around by:

'oc exec -c elasticsearch $ESPOD -- es_seed_acl'
Comment 3 Rich Megginson 2018-06-29 16:40:16 UTC 
and the usual https://github.com/openshift/origin-aggregated-logging/blob/master/hack/logging-dump.sh output
Comment 4 Junqi Zhao 2018-07-02 02:06:27 UTC 
Re-tested today, user project index is created and could be found in kibana
Close it as NOTABUG