Bug 1603549

Created attachment 1460853 [details]
Screenshot of the kibana dashboard

Description of problem:
On a freshly installed v3.9.31 environment of OCP, kibana dashboard on clicking on Discover tab does not shows individual projects separately. All the indices are included under project.*

Here are the output of the indices from the cluster and the images used by the components of EFK

[root@master-2 ~]# oc exec logging-es-data-master-ixu321i3-2-j9c5q -- curl -s -k --cert /etc/elasticsearch/secret/admin-cert --key /etc/elasticsearch/secret/admin-key https://logging-es:9200/_cat/indices?v 
Defaulting container name to elasticsearch.
Use 'oc describe pod/logging-es-data-master-ixu321i3-2-j9c5q -n logging' to see all of the containers in this pod.
health status index                                                                        pri rep docs.count docs.deleted store.size pri.store.size 
green  open   project.kube-service-catalog.ae719622-7f96-11e8-8d01-fa163ec312db.2018.07.09   1   0     110219            0     66.6mb         66.6mb 
green  open   .searchguard.logging-es-data-master-ixu321i3                                   1   0          5            0     33.9kb         33.9kb 
green  open   project.kube-service-catalog.ae719622-7f96-11e8-8d01-fa163ec312db.2018.07.08   1   0     110258            0     66.8mb         66.8mb 
green  open   project.kube-service-catalog.ae719622-7f96-11e8-8d01-fa163ec312db.2018.07.19   1   0        211            0    268.8kb        268.8kb 
green  open   project.kube-service-catalog.ae719622-7f96-11e8-8d01-fa163ec312db.2018.07.07   1   0      69553            0     42.2mb         42.2mb 
green  open   project.one.083e6bf9-8b64-11e8-8d01-fa163ec312db.2018.07.19                    1   0         10            0     36.7kb         36.7kb 
green  open   project.logging.1817e4c7-7f96-11e8-8d01-fa163ec312db.2018.07.19                1   0        306            0    276.8kb        276.8kb 
green  open   .operations.2018.07.18                                                         3   0      14610            0     11.2mb         11.2mb 
green  open   .operations.2018.07.19                                                         1   0      46421            0     34.4mb         34.4mb 
green  open   project.test.66d83b3d-8b62-11e8-8d01-fa163ec312db.2018.07.19                   1   0         10            0     36.9kb         36.9kb 
green  open   .kibana                                                                        1   0          1            0      3.1kb          3.1kb 
green  open   .operations.2018.07.10                                                         3   0      50512            0     36.8mb         36.8mb 
green  open   .operations.2018.07.11                                                         3   0      68340            0     48.9mb         48.9mb 
green  open   project.kube-service-catalog.ae719622-7f96-11e8-8d01-fa163ec312db.2018.07.10   1   0      93673            0     57.1mb         57.1mb 
green  open   .operations.2018.07.12                                                         3   0      25079            0     25.5mb         25.5mb 
green  open   .kibana.a1881c06eec96db9901c7bbfe41c42a3f08e9cb4                               1   0          5            0     53.8kb         53.8kb 
[root@master-2 ~]# 

[root@master-2 ~]#  oc get po -o 'go-template={{range $pod := .items}}{{if eq $pod.status.phase "Running"}}{{range $container := $pod.spec.containers}}oc exec -c {{$container.name}} {{$pod.metadata.name}} -- find /root/buildinfo -name Dockerfile-openshift* | grep -o logging.* {{"\n"}}{{end}}{{end}}{{end}}' | bash -
logging-curator-v3.9.31-2
logging-elasticsearch-v3.9.31-2
logging-fluentd-v3.9.30-2
logging-fluentd-v3.9.30-2
logging-fluentd-v3.9.30-2
logging-fluentd-v3.9.30-2
logging-fluentd-v3.9.30-2
logging-kibana-v3.9.31-2
logging-auth-proxy-v3.9.31-2
[root@master-2 ~]#


Please check the screenshot to see what is seen when the kibana dashboard is accessed.


Version-Release number of selected component (if applicable):

# oc version
oc v3.9.31
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://openshift.internal.example.com:443
openshift v3.9.31
kubernetes v1.9.1+a0ce1bc657


How reproducible:

Steps to Reproduce:
1. Install a fresh cluster of v3.9.31
2. Install logging project and create a fresh project and run a deployment.
3. Check the elasticsearch for the indices and later check the kibana dashboard.

Actual results:
The projects are clubbed together under project.* tab in kibana

Expected results:
Individual projects should be shown under different tabs in kibana

Additional info: