Bug 1517605
| Summary: | [RFE] logging Support crio | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Anping Li <anli> |
| Component: | Logging | Assignee: | Jan Wozniak <jwozniak> |
| Status: | CLOSED ERRATA | QA Contact: | Anping Li <anli> |
| Severity: | high | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 3.7.0 | CC: | aos-bugs, erich, fkluknav, jcantril, jwozniak, mifiedle, pportant, pweil, rmeggins, wsun, xtian |
| Target Milestone: | --- | ||
| Target Release: | 3.9.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Enhancement | |
| Doc Text: |
Added a parser for cri-o formatted logs.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-03-28 14:13:03 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Anping Li
2017-11-27 04:39:43 UTC
I am able to reproduce and tried to do a little bit more troubleshooting. There appear to be two issues as correctly observed by QE 1) not overriding image environment variables from kubernetes - default for an environment variable "ES_HOST=localhost" is in the image [1] - kubernetes DC overrides the value [2] - the override is not correctly propagated to the environment of the container And when I tried to not provide the default "ES_HOST" in the image, kubernetes was able to set the env variable correctly through a DC 2) default logging format is 'text' and the cri-o system container doesn't have it configurable - https://www.mankier.com/8/crio#--log-format allows to set the log format to 'json' - but it appears to work only for the 'crio' daemon logs, not the container logs I think both could be potentially issues rather with crio than with logging alone as I think they go beyond logging [1] https://github.com/openshift/origin-aggregated-logging/blob/master/curator/Dockerfile.centos7#L8 [2] https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_logging_curator/templates/curator.j2#L68-L69 By grepping the code of cri-o in rhel 7.4, it does not seem to pass the --log-format flag to runc.
The only code touching log-format is:
368 switch c.GlobalString("log-format") {
369 case "text":
370 // retain logrus's default.
371 case "json":
372 logrus.SetFormatter(new(logrus.JSONFormatter))
373 default:
374 return fmt.Errorf("unknown log-format %q", c.GlobalString("log-format"))
375 }
There is a way to make our fluentd pipeline able to parse cri-o logs. A workaround until the cri-o container logs respect the --log-format command line option is described in: https://trello.com/c/ktGIxQGf/585-5-online-crio-fluentd-understands-the-cri-log-format-loggingepic-crio https://github.com/kubernetes/kubernetes/pull/54777 cri-o team is investigating the non-propagating of env variables https://github.com/kubernetes-incubator/cri-o/issues/1293 Ref changes from upstream: https://github.com/kubernetes/kubernetes/commit/70a0cdfa8e05ac47d7dd04b032ceb79bead3fb5f support for cri-o format in fluentd https://github.com/openshift/origin-aggregated-logging/pull/949 https://github.com/openshift/openshift-ansible/pull/7102 The crio works with logging-fluentd/images/v3.9.1 Moved to verified. The crio container logs can be collected. So no block on logging for our test/release now. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0489 |