Created attachment 1455425 [details] could not view user project logs on kibana UI -- free-stg Description of problem: free-stg cluster, created one project "java2" and created one pod there to populate logs, logged in kibana UI https://logs.free-stg.openshift.com/, error showed "Discover: [exception] The index 'project..empty-project.*' was not found. This could mean data has not yet been collected." Checked kibana UI after 3 hours, it's still same error, user project logs could not be viewed from kibana. Error in ES pod logs ERR: Timed out while waiting for a green or yellow cluster state. * Try running sgadmin.sh with -icl and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates) 86 Not sure whether it is related to Bug 1596445 or not Version-Release number of selected component (if applicable): OpenShift Master:v3.10.9 Kubernetes Master:v1.10.0+b81c8f8 OpenShift Web Console:v3.10.9 logging-curator:v3.10.9 oauth-proxy:v3.10.2 logging-elasticsearch:v3.10.2 logging-fluentd:v3.10.9 logging-kibana:v3.10.9 logging-auth-proxy:v3.10.9 How reproducible: Always Steps to Reproduce: 1. Create one project and pod to populate logs $ oc new-project java $ oc new-app --docker-image=docker.io/chunyunchen/java-mainclass:2.2.94-SNAPSHOT 2. After a few minutes, check logs from kibana UI 3. Actual results: error showed "Discover: [exception] The index 'project..empty-project.*' was not found. This could mean data has not yet been collected." user project logs could not be viewed from kibana Expected results: user project logs could be viewed from kibana Additional info:
Created attachment 1455426 [details] "Timed out while waiting for a green or yellow cluster state" in ES pod log
Please attach information about the indices state: 'oc exec -c elasticsearch $ESPOD -- indices' Also, you may be able to work around by: 'oc exec -c elasticsearch $ESPOD -- es_seed_acl'
and the usual https://github.com/openshift/origin-aggregated-logging/blob/master/hack/logging-dump.sh output
Re-tested today, user project index is created and could be found in kibana Close it as NOTABUG