Bug 1722898 - Logging data from all projects are stored to .orphaned indexes with Elasticsearch
Summary: Logging data from all projects are stored to .orphaned indexes with Elasticse...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.1.z
Assignee: Rich Megginson
QA Contact: Anping Li
Whiteboard: 4.1.4
Depends On: 1722380 1724263
TreeView+ depends on / blocked
Reported: 2019-06-21 16:09 UTC by Rich Megginson
Modified: 2019-07-04 09:01 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Fluentd is unable to correctly determine the docker log driver. It thinks the log driver is journald when it is json-file. Fluentd then looks for the `CONTAINER_NAME` field in the record to hold the kubernetes metadata and it is not present. Consequence: Fluentd is not able to add kubernetes metadata to records. Records go to the .orphaned index. Fluentd spews lots of errors like this: [error]: record cannot use elasticsearch index na me type project_full: record is missing kubernetes field Fix: Fluentd should not rely on reading the docker configuration file to determine if the record contains kubernetes metadata. It should look at both the record tag and the record data and use whatever kubernetes metadata it finds there. Result: Fluentd can correctly add kubernetes metadata and assign records to the correct indices no matter which log driver docker is using. Records read from files under /var/log/containers/*.log will have a fluentd tag like kubernetes.var.log.containers.**. This applies both to CRI-O and docker file logs. Kubernetes records read from journald with CONTAINER_NAME will have a tag like journal.kubernetes.**. There is no CRI-O journald log driver yet, and it is not clear how those records will be represented, but hopefully they will follow the same CONTAINER_NAME convention, in which case they will Just Work.
Clone Of: 1722380
Last Closed: 2019-07-04 09:01:41 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift origin-aggregated-logging pull 1678 0 None closed [release-4.1] Bug 1722898: Logging data from all projects are stored to .orphaned indexes with Elasticsearch 2020-10-22 04:52:19 UTC
Red Hat Product Errata RHBA-2019:1635 0 None None None 2019-07-04 09:01:50 UTC

Comment 3 Anping Li 2019-06-27 10:47:20 UTC
The log weren't send to  .orphaned indexes, move bug to verfied.

Comment 5 errata-xmlrpc 2019-07-04 09:01:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.