Bug 1493188
| Summary: | operations namespace logs not sent to .operations index | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Ruben Romero Montes <rromerom> | |
| Component: | Logging | Assignee: | Rich Megginson <rmeggins> | |
| Status: | CLOSED ERRATA | QA Contact: | Anping Li <anli> | |
| Severity: | high | Docs Contact: | ||
| Priority: | urgent | |||
| Version: | 3.6.0 | CC: | aos-bugs, myllynen, rmeggins, tlarsson | |
| Target Milestone: | --- | |||
| Target Release: | 3.6.z | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: |
Cause: The pattern for container logs in the journal field CONTAINER_NAME changed. The pattern was not matching for logs from pods in the "default", "openshift", or "openshift-infra" namespaces.
Consequence: Logs from these namespaces were being stored in indices matching "project.default.*" e.g. rather than ".operations.*".
Fix: The pattern matcher was fixed to match the correct pattern.
Result: logs from pods in the "default", "openshift", or "openshift-infra" namespaces are correctly written to the ".operations.*" indices.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1494310 (view as bug list) | Environment: | ||
| Last Closed: | 2017-10-25 13:06:40 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1494310 | |||
Here is my ansible inventory used: #Aggregated logging openshift_logging_install_logging=true openshift_logging_image_version=v3.6 openshift_logging_kibana_hostname=kibana.apps36.example.com openshift_logging_es_cluster_size=1 openshift_logging_es_memory_limit=1G Commit pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/e29f72f7e38e8e70b975f1284efe8514c1f32ac9 Bug 1493188 - operations namespace logs not sent to .operations index https://bugzilla.redhat.com/show_bug.cgi?id=1493188 The format of the CONTAINER_NAME field changed from k8s_container-name.container-hash_ ... to k8s_container-name_ .... which caused problems for the retag filter. Since we don't care about the value of container-hash in this context, just ignore it. Also added some test to ensure that operations namespace logs end up in es-ops in the .operations.* indices and nowhere else. The project logs can be list in .operations index with v3.6.173.0.39 @rich Let's use the v3.5 bug https://bugzilla.redhat.com/show_bug.cgi?id=1494310 to trace the v3.5 testing. Could you move it to ON_QA? Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:3049 |
Description of problem: After a fresh install of the Aggregated Logging logs coming from the operations projects (i.e. default, openshift-infra and openshift) are not sent to the .operations.* index Version-Release number of selected component (if applicable): $ oc version oc v3.6.173.0.5 $ oc get po -o 'go-template={{range $pod := .items}}{{range $container := $pod.spec.containers}}oc exec -c {{$container.name}} {{$pod.metadata.name}} -- find /root/buildinfo -name Dockerfile-openshift*{{"\n"}}{{end}}{{end}}' | bash - /root/buildinfo/Dockerfile-openshift3-logging-curator-v3.6.173.0.21-17 /root/buildinfo/Dockerfile-openshift3-logging-elasticsearch-v3.6.173.0.5-5 /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.6.173.0.21-17 /root/buildinfo/Dockerfile-openshift3-logging-fluentd-v3.6.173.0.21-17 /root/buildinfo/Dockerfile-openshift3-logging-kibana-v3.6.173.0.21-17 /root/buildinfo/Dockerfile-openshift3-logging-auth-proxy-v3.6.173.0.21-17 How reproducible: Always Steps to Reproduce: 1. Install a 3.6 cluster 2. Deploy EFK using ansible 3. Access Kibana and browse the .operations.* index pattern Actual results: Only logs from journald are shown Expected results: Logs from journald and infra projects to be shown Additional info: $ oc exec logging-es-data-master-moc45ja4-2-8q889 -- curl -s --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert --cacert /etc/elasticsearch/secret/admin-ca https://localhost:9200/_cat/indices?h=index project.jenkins.a985ce8a-90c7-11e7-af7f-5254004a7695.2017.09.19 .searchguard.logging-es-data-master-moc45ja4 project.logging.4d1af73a-90c3-11e7-a156-5254004a7695.2017.09.19 .kibana.d033e22ae348aeb5660fc2140aec35850c4da997 project.default.34aa1a30-90bc-11e7-a156-5254004a7695.2017.09.18 project.default.34aa1a30-90bc-11e7-a156-5254004a7695.2017.09.15 project.default.34aa1a30-90bc-11e7-a156-5254004a7695.2017.09.19 .operations.2017.09.15 .operations.2017.09.19 .kibana .operations.2017.09.18 project.logging.4d1af73a-90c3-11e7-a156-5254004a7695.2017.09.18