Bug 1632648
| Summary: | [3.11] Record is missing kubernetes field when use '--log-driver journald' in /etc/sysconfig/docker | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Anping Li <anli> | |
| Component: | Logging | Assignee: | Rich Megginson <rmeggins> | |
| Status: | CLOSED ERRATA | QA Contact: | Anping Li <anli> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | medium | |||
| Version: | 3.11.0 | CC: | anli, aos-bugs, jcantril, qitang, rmeggins | |
| Target Milestone: | --- | |||
| Target Release: | 3.11.z | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openshift3/ose-logging-fluentd:v3.11.23-1 | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: When using docker with log-driver journald, the setting in /etc/sysconfig/docker has changed to use --log-driver journald instead of --log-driver=journald
Consequence: Fluentd cannot detect that journald is being used, so assumes json-file, and cannot read any kubernetes metadata because it doesn't look for the journald CONTAINER_NAME field. The result is lots of fluentd errors.
Fix: Change the way Fluentd detects the docker log driver so that it looks for --log-driver journald in addition to --log-driver=journald.
Result: Fluentd can detect the docker log driver, and can correctly process Kubernetes container logs.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1638899 (view as bug list) | Environment: | ||
| Last Closed: | 2018-11-20 03:10:43 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1638899, 1638900, 1638902 | |||
|
Description
Anping Li
2018-09-25 10:03:31 UTC
(In reply to Anping Li from comment #0) > > Version-Release number of selected component (if applicable): > openshift3/logging-fluentd/images/v3.10.45-2 > openshift3/ose-logging-fluentd/images/v3.11.14-1 Which version are you reporting the issue against? @jeff, In v3.11, if confirmed , the bug should be cloned to v3.10. Is this a regression? Yes, It work prior. I cannot reproduce. I'm using the latest available 3.11 packages/images:
# docker images|grep fluentd
brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ose-logging-fluentd v3.11 98f5f5f31f7d 36 hours ago 286 MB
NOTE WELL: I do not have the 3.10 image you have listed above:
openshift3/logging-fluentd/images/v3.10.45-2
I'm not sure why there would be a 3.10 and a 3.11 fluentd image on the system at the same time. That seems suspicious to me.
Please reproduce, and please leave the system up and running for me to log into and examine.
This is what I have:
green open project.test-project.f6e6beb3-c0d2-11e8-8af8-5452005eafb9.2018.09.25 jbfxcnnXTSOxjBVg9THz7A 1 0 304 0 403.9kb 403.9kb
# oc exec -c elasticsearch $espod -- es_util --qu
ery=project.test-project.*/_search?q=kubernetes.namespace_name:test-project | jq
. |more
"hits": {
"total": 374,
...
"_index": "project.test-project.f6e6beb3-c0d2-11e8-8af8-5452005eafb9.201
Moving to 3.11.z stream: * see #c5 - unreproducible * we recommend users to use JSON driver * default cluster deployments use JSON driver * Logging CI regularly tests this driver with no failures at present This may have to be moved into the first z stream release if we are able to further isolate the issue. grr - the format of /etc/sysconfig/docker changed - instead of OPTIONS=' --selinux-enabled --log-driver=journald --signature-verification=False' it now uses OPTIONS=' --selinux-enabled --log-driver journald --signature-verification=False' Note the space instead of = after "--log-driver" :-( Commits pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/b2915fb5d0f425000e1926e0a5aaa6b2b14bdea7 Bug 1632648 - Record is missing kubernetes field when Docker log driver is journald https://bugzilla.redhat.com/show_bug.cgi?id=1632648 In /etc/sysconfig/docker, the log-driver may be specified as --log-driver=journald or --log-driver journald delimited by one or more whitespace https://github.com/openshift/origin-aggregated-logging/commit/3ad88ef01ee1ee7f2ae1be30f87792fd9e3c0e48 Merge pull request #1364 from richm/bug-1632648 Bug 1632648 - Record is missing kubernetes field when Docker log driver is journald Workarounds: - change the /etc/sysconfig/docker on each node - change it to use --log-driver=journald - oc edit ds/logging-fluentd - add env USE_JOURNAL true e.g. oc set env ds/logging-fluentd USE_JOURNAL=true Verified on logging-fluentd-v3.11.36-1. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:3537 |