Bug 1434300
| Summary: | Log entries are generated in ES after deployed logging stacks via ansible, but can not be found in kibana. | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Junqi Zhao <juzhao> | ||||||||||
| Component: | Logging | Assignee: | Rich Megginson <rmeggins> | ||||||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Junqi Zhao <juzhao> | ||||||||||
| Severity: | high | Docs Contact: | |||||||||||
| Priority: | high | ||||||||||||
| Version: | 3.5.0 | CC: | aos-bugs, juzhao, pweil, rmeggins, tdawson, xiazhao | ||||||||||
| Target Milestone: | --- | Keywords: | Regression | ||||||||||
| Target Release: | --- | ||||||||||||
| Hardware: | Unspecified | ||||||||||||
| OS: | Unspecified | ||||||||||||
| Whiteboard: | |||||||||||||
| Fixed In Version: | Doc Type: | No Doc Update | |||||||||||
| Doc Text: |
undefined
|
Story Points: | --- | ||||||||||
| Clone Of: | Environment: | ||||||||||||
| Last Closed: | 2017-04-03 16:24:58 UTC | Type: | Bug | ||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||
| Documentation: | --- | CRM: | |||||||||||
| Verified Versions: | Category: | --- | |||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
| Embargoed: | |||||||||||||
| Attachments: |
|
||||||||||||
Created attachment 1264942 [details]
ansible inventory file
Created attachment 1264944 [details]
Kibana UI snapshot
This is a Regression and TestBlocker issue, blocks most of logging tests The problem is that there are no JSON index pattern files for the common data model. ES/Kibana are using the default index pattern which is for pre-common data model (i.e. time field is "time" instead of "@timestamp", container fields are flattened like "kubernetes_pod_name" instead of "kubernetes.pod_name", etc.) The default files seeded by the plugin are: https://github.com/fabric8io/openshift-elasticsearch-plugin/tree/master/src/main/resources/io/fabric8/elasticsearch/plugin/kibana Additionally, we can update kibana to load files different from the default without rebuilding the plugin or building a new image: 1. Configure the files to use [2] 2. Create a configmap with files from 1. 3. Mount the configmap into the podspec [1] https://github.com/fabric8io/openshift-elasticsearch-plugin/blob/master/src/main/java/io/fabric8/elasticsearch/plugin/ConfigurationSettings.java#L69-L71 Not sure we should consider this a blocker. Correct pod to apply #5 is to the ES pods, not Kibana @Jeff The latest image is the same as this defect submitted, please push newimages to repo. logging-elasticsearch 3.5.0 9b824bebeb36 8 days ago 399.4 MB logging-kibana 3.5.0 a6159c640977 2 weeks ago 342.4 MB logging-fluentd 3.5.0 32a4ac0a3e18 2 weeks ago 232.5 MB logging-curator 3.5.0 8cfcb23f26b6 3 weeks ago 211.1 MB logging-auth-proxy 3.5.0 139f7943475e 9 weeks ago 220 MB Step 1: change viaq to generate index pattern files https://github.com/ViaQ/elasticsearch-templates/pull/36 once this PR is approved - step 2 is https://github.com/ViaQ/elasticsearch-templates/pull/36 then step 3 is to change the elasticsearch.yml in openshift-ansible sorry, step 2 is https://github.com/openshift/origin-aggregated-logging/pull/357 Commit pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/5d0e257c59c55690eb22d46ea4c5cd8ebb2a2ec9 Bug 1434300 - Log entries are generated in ES after deployed logging stacks via ansible, but can not be found in kibana. https://bugzilla.redhat.com/show_bug.cgi?id=1434300 Add support for common data model index pattern files. Depends on https://github.com/ViaQ/elasticsearch-templates/pull/36 https://github.com/ViaQ/elasticsearch-templates/releases/tag/0.0.4 https://github.com/openshift/origin-aggregated-logging/commit/27f10829085ea4e1a0f8e596650fbb3952778eaf https://github.com/openshift/openshift-ansible/commit/7a401b8e12778b18bd4ebefbcc4b4f431e140e55 koji_builds: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=545824 repositories: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:rhaos-3.5-rhel-7-docker-candidate-20170322170113 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:3.5.0-10 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:3.5.0 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:latest brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.5 koji_builds: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=545890 repositories: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:rhaos-3.5-rhel-7-docker-candidate-20170322184721 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:3.5.0-12 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:3.5.0 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:latest brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.5 But you also need to use openshift-ansible from https://github.com/openshift/openshift-ansible with branch release-1.5 - the necessary ansible code has not been merged to master When you grab openshift-ansible, please confirm the following output: [root@ansible openshift-ansible]# git grep fabric8.elastic|cat roles/openshift_logging/files/elasticsearch-logging.yml: io.fabric8.elasticsear ch: ${PLUGIN_LOGLEVEL} roles/openshift_logging/templates/elasticsearch.yml.j2:io.fabric8.elasticsearch. authentication.users: ["system.logging.kibana", "system.logging.fluentd", "syste m.logging.curator", "system.admin"] roles/openshift_logging/templates/elasticsearch.yml.j2:io.fabric8.elasticsearch. kibana.mapping.app: /usr/share/elasticsearch/index_patterns/com.redhat.viaq-open shift.index-pattern.json roles/openshift_logging/templates/elasticsearch.yml.j2:io.fabric8.elasticsearch. kibana.mapping.ops: /usr/share/elasticsearch/index_patterns/com.redhat.viaq-open shift.index-pattern.json used playbooks from https://github.com/openshift/openshift-ansible with branch release-1.5, and ansible is yum installed, version is ansible-2.2.1.0-2.el7.noarch. although the output of 'git grep fabric8.elastic|cat roles/openshift_logging/files/elasticsearch-logging.yml' is not the same with yours, log entries can be found in kibana now. see tha attached file Image id: openshift3/logging-elasticsearch 3.5.0 5ff198b5c68d 4 hours ago 399.4 MB openshift3/logging-kibana 3.5.0 a6159c640977 2 weeks ago 342.4 MB openshift3/logging-fluentd 3.5.0 32a4ac0a3e18 2 weeks ago 232.5 MB openshift3/logging-curator 3.5.0 8cfcb23f26b6 3 weeks ago 211.1 MB openshift3/logging-auth-proxy 3.5.0 139f7943475e 9 weeks ago 220 MB Created attachment 1265583 [details]
git grep fabric8.elastic output in my environment
The command is just # git grep fabric8.elastic|cat That's why the output is different But at any rate, good to see that Kibana is actually working (In reply to Rich Megginson from comment #20) > The command is just > > # git grep fabric8.elastic|cat > > That's why the output is different > > But at any rate, good to see that Kibana is actually working Thanks, my output is the same as you provided now. Commit pushed to master at https://github.com/openshift/openshift-ansible https://github.com/openshift/openshift-ansible/commit/c4fcf9ef11b4dfc27f8f1e509719864f0ef2d019 Bug 1434300 - Log entries are generated in ES after deployed logging stacks via ansible, but can not be found in kibana. https://bugzilla.redhat.com/show_bug.cgi?id=1434300 Add support for common data model index pattern files. Depends on https://github.com/ViaQ/elasticsearch-templates/pull/36 and https://github.com/openshift/origin-aggregated-logging/pull/357 I don't believe doc text is required for this bz, because it was only seen internally during 3.5 testing. This bz was caused by a bad fix for another customer facing issue https://bugzilla.redhat.com/show_bug.cgi?id=1426061 which does have doc text. Closing since this never made it to customers. |
Created attachment 1264941 [details] ES, Kibana, Fluentd logs Description of problem: Log entries are generated in ES after deployed logging stacks via ansible, but can not be found in kibana. Query elasticsearch directly, find fields like "kubernetes.container_name" and "@timestamp", but in kibana UI, fields are "kubernetes_container_name" and "time", it seems the data model is wrong. Version-Release number of selected component (if applicable): openshift-ansible-playbooks-3.5.39-1.git.0.cb12cdb.el7.noarch openshift-ansible-3.5.39-1.git.0.cb12cdb.el7.noarch image id: logging-elasticsearch 3.5.0 9b824bebeb36 logging-kibana 3.5.0 a6159c640977 logging-fluentd 3.5.0 32a4ac0a3e18 logging-curator 3.5.0 8cfcb23f26b6 logging-auth-proxy 3.5.0 139f7943475e How reproducible: Always Steps to Reproduce: 1. Deploy logging stacks via ansible 2. Log in Kibana UI to find logs. Actual results: Log entries are generated when querying elasticsearch directly, but logs can not be found in kibana. Expected results: Log entry can be found in Kibana UI. Additional info: Attached ansible inventory file,kibana,query elasticserch log, es pod and fluentd pod log. Attached the kibana UI snapshot