Created attachment 1264941 [details] ES, Kibana, Fluentd logs Description of problem: Log entries are generated in ES after deployed logging stacks via ansible, but can not be found in kibana. Query elasticsearch directly, find fields like "kubernetes.container_name" and "@timestamp", but in kibana UI, fields are "kubernetes_container_name" and "time", it seems the data model is wrong. Version-Release number of selected component (if applicable): openshift-ansible-playbooks-3.5.39-1.git.0.cb12cdb.el7.noarch openshift-ansible-3.5.39-1.git.0.cb12cdb.el7.noarch image id: logging-elasticsearch 3.5.0 9b824bebeb36 logging-kibana 3.5.0 a6159c640977 logging-fluentd 3.5.0 32a4ac0a3e18 logging-curator 3.5.0 8cfcb23f26b6 logging-auth-proxy 3.5.0 139f7943475e How reproducible: Always Steps to Reproduce: 1. Deploy logging stacks via ansible 2. Log in Kibana UI to find logs. Actual results: Log entries are generated when querying elasticsearch directly, but logs can not be found in kibana. Expected results: Log entry can be found in Kibana UI. Additional info: Attached ansible inventory file,kibana,query elasticserch log, es pod and fluentd pod log. Attached the kibana UI snapshot
Created attachment 1264942 [details] ansible inventory file
Created attachment 1264944 [details] Kibana UI snapshot
This is a Regression and TestBlocker issue, blocks most of logging tests
The problem is that there are no JSON index pattern files for the common data model. ES/Kibana are using the default index pattern which is for pre-common data model (i.e. time field is "time" instead of "@timestamp", container fields are flattened like "kubernetes_pod_name" instead of "kubernetes.pod_name", etc.)
The default files seeded by the plugin are: https://github.com/fabric8io/openshift-elasticsearch-plugin/tree/master/src/main/resources/io/fabric8/elasticsearch/plugin/kibana Additionally, we can update kibana to load files different from the default without rebuilding the plugin or building a new image: 1. Configure the files to use [2] 2. Create a configmap with files from 1. 3. Mount the configmap into the podspec [1] https://github.com/fabric8io/openshift-elasticsearch-plugin/blob/master/src/main/java/io/fabric8/elasticsearch/plugin/ConfigurationSettings.java#L69-L71 Not sure we should consider this a blocker.
Correct pod to apply #5 is to the ES pods, not Kibana
@Jeff The latest image is the same as this defect submitted, please push newimages to repo. logging-elasticsearch 3.5.0 9b824bebeb36 8 days ago 399.4 MB logging-kibana 3.5.0 a6159c640977 2 weeks ago 342.4 MB logging-fluentd 3.5.0 32a4ac0a3e18 2 weeks ago 232.5 MB logging-curator 3.5.0 8cfcb23f26b6 3 weeks ago 211.1 MB logging-auth-proxy 3.5.0 139f7943475e 9 weeks ago 220 MB
Step 1: change viaq to generate index pattern files https://github.com/ViaQ/elasticsearch-templates/pull/36 once this PR is approved - step 2 is https://github.com/ViaQ/elasticsearch-templates/pull/36 then step 3 is to change the elasticsearch.yml in openshift-ansible
step 3 https://github.com/openshift/openshift-ansible/pull/3730
sorry, step 2 is https://github.com/openshift/origin-aggregated-logging/pull/357
Commit pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/5d0e257c59c55690eb22d46ea4c5cd8ebb2a2ec9 Bug 1434300 - Log entries are generated in ES after deployed logging stacks via ansible, but can not be found in kibana. https://bugzilla.redhat.com/show_bug.cgi?id=1434300 Add support for common data model index pattern files. Depends on https://github.com/ViaQ/elasticsearch-templates/pull/36
https://github.com/ViaQ/elasticsearch-templates/releases/tag/0.0.4 https://github.com/openshift/origin-aggregated-logging/commit/27f10829085ea4e1a0f8e596650fbb3952778eaf https://github.com/openshift/openshift-ansible/commit/7a401b8e12778b18bd4ebefbcc4b4f431e140e55 koji_builds: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=545824 repositories: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:rhaos-3.5-rhel-7-docker-candidate-20170322170113 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:3.5.0-10 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:3.5.0 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:latest brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.5
koji_builds: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=545890 repositories: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:rhaos-3.5-rhel-7-docker-candidate-20170322184721 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:3.5.0-12 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:3.5.0 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:latest brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.5 But you also need to use openshift-ansible from https://github.com/openshift/openshift-ansible with branch release-1.5 - the necessary ansible code has not been merged to master When you grab openshift-ansible, please confirm the following output: [root@ansible openshift-ansible]# git grep fabric8.elastic|cat roles/openshift_logging/files/elasticsearch-logging.yml: io.fabric8.elasticsear ch: ${PLUGIN_LOGLEVEL} roles/openshift_logging/templates/elasticsearch.yml.j2:io.fabric8.elasticsearch. authentication.users: ["system.logging.kibana", "system.logging.fluentd", "syste m.logging.curator", "system.admin"] roles/openshift_logging/templates/elasticsearch.yml.j2:io.fabric8.elasticsearch. kibana.mapping.app: /usr/share/elasticsearch/index_patterns/com.redhat.viaq-open shift.index-pattern.json roles/openshift_logging/templates/elasticsearch.yml.j2:io.fabric8.elasticsearch. kibana.mapping.ops: /usr/share/elasticsearch/index_patterns/com.redhat.viaq-open shift.index-pattern.json
used playbooks from https://github.com/openshift/openshift-ansible with branch release-1.5, and ansible is yum installed, version is ansible-2.2.1.0-2.el7.noarch. although the output of 'git grep fabric8.elastic|cat roles/openshift_logging/files/elasticsearch-logging.yml' is not the same with yours, log entries can be found in kibana now. see tha attached file Image id: openshift3/logging-elasticsearch 3.5.0 5ff198b5c68d 4 hours ago 399.4 MB openshift3/logging-kibana 3.5.0 a6159c640977 2 weeks ago 342.4 MB openshift3/logging-fluentd 3.5.0 32a4ac0a3e18 2 weeks ago 232.5 MB openshift3/logging-curator 3.5.0 8cfcb23f26b6 3 weeks ago 211.1 MB openshift3/logging-auth-proxy 3.5.0 139f7943475e 9 weeks ago 220 MB
Created attachment 1265583 [details] git grep fabric8.elastic output in my environment
The command is just # git grep fabric8.elastic|cat That's why the output is different But at any rate, good to see that Kibana is actually working
(In reply to Rich Megginson from comment #20) > The command is just > > # git grep fabric8.elastic|cat > > That's why the output is different > > But at any rate, good to see that Kibana is actually working Thanks, my output is the same as you provided now.
Commit pushed to master at https://github.com/openshift/openshift-ansible https://github.com/openshift/openshift-ansible/commit/c4fcf9ef11b4dfc27f8f1e509719864f0ef2d019 Bug 1434300 - Log entries are generated in ES after deployed logging stacks via ansible, but can not be found in kibana. https://bugzilla.redhat.com/show_bug.cgi?id=1434300 Add support for common data model index pattern files. Depends on https://github.com/ViaQ/elasticsearch-templates/pull/36 and https://github.com/openshift/origin-aggregated-logging/pull/357
I don't believe doc text is required for this bz, because it was only seen internally during 3.5 testing. This bz was caused by a bad fix for another customer facing issue https://bugzilla.redhat.com/show_bug.cgi?id=1426061 which does have doc text.
Closing since this never made it to customers.