Bug 1513408
| Summary: | Continuously error info "record cannot use elasticsearch index name type project_full" in fluentd pod logs if use system container docker. | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Junqi Zhao <juzhao> | |
| Component: | Logging | Assignee: | Jeff Cantrill <jcantril> | |
| Status: | CLOSED WONTFIX | QA Contact: | Junqi Zhao <juzhao> | |
| Severity: | low | Docs Contact: | ||
| Priority: | low | |||
| Version: | 3.7.0 | CC: | aos-bugs, gferrazs, jcantril, pweil, rmeggins, smunilla, veli-matti.leppanen | |
| Target Milestone: | --- | |||
| Target Release: | 3.7.z | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: |
Cause: Messages for which the unique namespace id could not be determined could not be properly indexed.
Consequence: Messages could be lost and the error message appears in the logs
Fix: Modify the cache algorithm to provide the necessary data or default the value to 'orphaned'.
Result: The error message is resolved and messages are stored in an 'orphaned' index when a unique namespace id can not be determined.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1551870 (view as bug list) | Environment: | ||
| Last Closed: | 2019-03-06 20:39:20 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1551870 | |||
Issue is not fixed, it is changed to ON_QA by errata, change back to MODIFIED now This fix is available in image logging-fluentd-docker-v3.7.26-1 and later: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=647573 Sam, can you help out Junqi? (In reply to Jeff Cantrill from comment #4) > This fix is available in image logging-fluentd-docker-v3.7.26-1 and later: > https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=647573 > > Sam, can you help out Junqi? Sam This is because issue is related to system container docker and issue is not fixed. see https://bugzilla.redhat.com/show_bug.cgi?id=1551870#c2 Tested with logging-fluentd/images/v3.7.42-2, issue is not fixed, use system container docker,there are still continuously info "record cannot use elasticsearch index name type project_full" Move to Assigned (In reply to Junqi Zhao from comment #7) > Tested with logging-fluentd/images/v3.7.42-2, issue is not fixed, use system > container docker,there are still continuously info "record cannot use > elasticsearch index name type project_full" > > Move to Assigned If you do _not_ use system container docker, does it work? That is, if you use "regular" docker, is the bug fixed? If so, I would like to move it to the bucket of bugs we have related to system container docker. (In reply to Rich Megginson from comment #8) > (In reply to Junqi Zhao from comment #7) > > Tested with logging-fluentd/images/v3.7.42-2, issue is not fixed, use system > > container docker,there are still continuously info "record cannot use > > elasticsearch index name type project_full" > > > > Move to Assigned > > If you do _not_ use system container docker, does it work? That is, if you > use "regular" docker, is the bug fixed? If so, I would like to move it to > the bucket of bugs we have related to system container docker. "regular" docker does not have this issue, this issue is only happen with system container docker |
Description of problem: Use system container docker,there are continuously info "record cannot use elasticsearch index name type project_full" in fluentd pod logs. This issue is only happen with system container docker, since its configure file is /etc/docker/container-daemon.json rather than /etc/docker/daemon.json or /etc/sysconfig/docker. Trace: 2017-11-15 10:19:25 +0000 [error]: record cannot use elasticsearch index name type project_full: record is missing kubernetes field: {"message"=>"2017-11-15 10:19:24 +0000 [error]: record cannot use elasticsearch index name type project_full: record is missing kubernetes field: {\"message\"=>\"2017-11-15T10:06:38.903902551Z AUDIT: id=\\\"968849aa-d7a7-45f2-9243-5ece6804e21f\\\" ip=\\\"10.130.0.1\\\" method=\\\"GET\\\" user=\\\"system:anonymous\\\" groups=\\\"\\\\\\\"system:unauthenticated\\\\\\\"\\\" as=\\\"<self>\\\" asgroups=\\\"<lookup>\\\" namespace=\\\"<none>\\\" uri=\\\"/healthz\\\"\\n\", \"level\"=>\"info\", \"hostname\"=>\"localhost.localdomain\", \"pipeline_metadata\"=>{\"collector\"=>{\"ipaddr4\"=>\"10.130.0.16\", \"ipaddr6\"=>\"fe80::c432:91ff:fe2f:d303\", \"inputname\"=>\"fluent-plugin-systemd\", \"name\"=>\"fluentd\", \"received_at\"=>\"2017-11-15T10:19:24.305099+00:00\", \"version\"=>\"0.12.39 1.6.0\"}}, \"@timestamp\"=>\"2017-11-15T10:06:38.904126+00:00\"}\n", "level"=>"info", "hostname"=>"localhost.localdomain", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"10.130.0.16", "ipaddr6"=>"fe80::c432:91ff:fe2f:d303", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2017-11-15T10:19:25.492302+00:00", "version"=>"0.12.39 1.6.0"}}, "@timestamp"=>"2017-11-15T10:19:24.308025+00:00"} 2017-11-15 10:19:25 +0000 [error]: record cannot use elasticsearch index name type project_full: record is missing kubernetes field: {"message"=>"2017-11-15 10:19:24 +0000 [error]: record cannot use elasticsearch index name type project_full: record is missing kubernetes field: {\"message\"=>\"2017-11-15T10:06:48.906137559Z AUDIT: id=\\\"73f3c8fa-f5a2-4160-b62f-747398429f3a\\\" ip=\\\"10.130.0.1\\\" method=\\\"GET\\\" user=\\\"system:anonymous\\\" groups=\\\"\\\\\\\"system:unauthenticated\\\\\\\"\\\" as=\\\"<self>\\\" asgroups=\\\"<lookup>\\\" namespace=\\\"<none>\\\" uri=\\\"/healthz\\\"\\n\", \"level\"=>\"info\", \"hostname\"=>\"localhost.localdomain\", \"pipeline_metadata\"=>{\"collector\"=>{\"ipaddr4\"=>\"10.130.0.16\", \"ipaddr6\"=>\"fe80::c432:91ff:fe2f:d303\", \"inputname\"=>\"fluent-plugin-systemd\", \"name\"=>\"fluentd\", \"received_at\"=>\"2017-11-15T10:19:24.305285+00:00\", \"version\"=>\"0.12.39 1.6.0\"}}, \"@timestamp\"=>\"2017-11-15T10:06:48.906337+00:00\"}\n", "level"=>"info", "hostname"=>"localhost.localdomain", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"10.130.0.16", "ipaddr6"=>"fe80::c432:91ff:fe2f:d303", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2017-11-15T10:19:25.492472+00:00", "version"=>"0.12.39 1.6.0"}}, "@timestamp"=>"2017-11-15T10:19:24.308032+00:00"} ****************************************************************************** # docker info | grep 'Logging Driver' Logging Driver: json-file # cat /etc/docker/daemon.json {} #cat /etc/sysconfig/docker OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false' Version-Release number of selected component (if applicable): logging componet version is v3.7.8-1 # openshift version openshift v3.7.8 kubernetes v1.7.6+a08f5eeb62 etcd 3.2.8 How reproducible: Always Steps to Reproduce: 1. Deploy logging 3.7 Actual results: continuously info "record cannot use elasticsearch index name type project_full" in fluentd pod logs Expected results: Should not be error in fluentd pod logs Additional info: ******************************************************************************* Work around is use the same "log-driver" in /etc/docker/daemon.json or /etc/sysconfig/docker with `docker info | grep 'Logging Driver'`