Bug 1525415
Summary: | A lot of error messages in fluentd pod logs for deleted namespaces | ||||||
---|---|---|---|---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Anping Li <anli> | ||||
Component: | Logging | Assignee: | Jeff Cantrill <jcantril> | ||||
Status: | CLOSED ERRATA | QA Contact: | Anping Li <anli> | ||||
Severity: | high | Docs Contact: | |||||
Priority: | unspecified | ||||||
Version: | 3.5.1 | CC: | aos-bugs, ewolinet, jcantril, juzhao, mifiedle, nhosoi, pportant, rmeggins, wabouham | ||||
Target Milestone: | --- | ||||||
Target Release: | 3.6.z | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||
Doc Text: |
Cause: Messages for which the unique namespace id could not be determined could not be properly indexed.
Consequence: Messages could be lost and the error message appears in the logs
Fix: Modify the cache algorithm to provide the necessary data or default the value to 'orphaned'.
Result: The error message is resolved and messages are stored in an 'orphaned' index when a unique namespace id can not be determined.
|
Story Points: | --- | ||||
Clone Of: | 1493022 | Environment: | |||||
Last Closed: | 2018-04-12 05:59:59 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | 1493022 | ||||||
Bug Blocks: | |||||||
Attachments: |
|
Description
Anping Li
2017-12-13 09:52:40 UTC
Created attachment 1367223 [details]
logging dump file
Hit similar issue in v3.5. Note that, in my Env, only logging-curator/images/v3.5.5.31.48 is stage image.
OCP: atomic-openshift-3.5.5.31.48
Logging:
logging-kibana/images/3.5.0-51
logging-fluentd/images/3.5.0-46
logging-elasticsearch/images/3.5.0-58
logging-curator/images/v3.5.5.31.48
logging-auth-proxy/images/3.5.0-45
This change for fluent-plugin-kubernetes_metadata_filter already exists in 3.6. Its in all of our referenced Dockerfiles, puddles, and latest 3.6 build [4] [1] https://github.com/openshift/origin-aggregated-logging/blob/release-3.6/fluentd/Dockerfile.centos7#L38 [2] https://github.com/openshift/origin-aggregated-logging/blob/release-3.6/fluentd/Dockerfile#L36 [3] http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/RHAOS/AtomicOpenShift/3.6/latest/x86_64/os/Packages/rubygem-fluent-plugin-kubernetes_metadata_filter-1.0.1-1.el7.noarch.rpm [4] http://download-node-02.eng.bos.redhat.com/brewroot/packages/logging-fluentd-docker/v3.6.173.0.104/1/data/logs/x86_64-build.log I don't expect we would backport this fix to 3.5 You are right, @Jeff. Thanks! The environment variables in Dockerfile are not used for choosing the gem files. 14 ENV DATA_VERSION=1.6.0 \ .... 17 FLUENTD_FLATTEN_HASH=0.4.0-1 \ 18 FLUENTD_KUBE_METADATA=0.29.0-1 \ 19 FLUENTD_REWRITE_TAG=1.5.6-1 \ .... Probably, we could remove them... No error message about deleted namespaces with /openshift3/logging-fluentd: v3.6.173.0.104. so move bug to verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1106 |