Bug 1482002
| Summary: | Can't collect log entries due to fluentd error | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Xia Zhao <xiazhao> | ||||||
| Component: | Logging | Assignee: | Rich Megginson <rmeggins> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Xia Zhao <xiazhao> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | high | ||||||||
| Version: | 3.6.0 | CC: | aos-bugs, jwozniak, mifiedle, nhosoi, rmeggins | ||||||
| Target Milestone: | --- | Keywords: | Regression | ||||||
| Target Release: | 3.6.z | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: |
Cause: Fluentd could not write the files it uses for buffering records due to a problem converting values from ascii-8bit to utf-8.
Consequence: Fluentd emits a lot of errors and cannot add records to Elasticsearch.
Fix: Remove the patch that forced the utf-8 conversion.
Result: Fluentd can write ascii-8bit encoded files for its buffer.
|
Story Points: | --- | ||||||
| Clone Of: | |||||||||
| : | 1498999 (view as bug list) | Environment: | |||||||
| Last Closed: | 2017-10-25 13:04:36 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 1469859, 1498999 | ||||||||
| Attachments: |
|
||||||||
|
Description
Xia Zhao
2017-08-16 09:16:35 UTC
Created attachment 1314021 [details]
inventory file used for logging deployment
Created attachment 1314027 [details]
fluentd log
This bz is currently blocking logging tests on OCP 3.6.0 envs. the fluentd image in brew looks suspiciously small compared to the one freshly build from the branch rhaos-3.6-rhel-7. Perhaps incorrect build got pushed to brew? brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd v3.6 95dede9f3cb2 15 hours ago 235.1 MB local-reg:5000/openshift/logging-fluentd <none> 0ac973960bdb 4 hours ago 360.6 MB Can we log into the system? I want to look at the journal and see if I can find which record is causing this problem. *** Bug 1482532 has been marked as a duplicate of this bug. *** Installing logging with openshift_logging_image_version=v3.6.173.0.5 - this problem is seen. Installing logging with openshift_logging_image_version=v3.6.171 - this problem is NOT seen. (In reply to Mike Fiedler from comment #11) > Installing logging with openshift_logging_image_version=v3.6.173.0.5 - this > problem is seen. > > Installing logging with openshift_logging_image_version=v3.6.171 - this > problem is NOT seen. Right. Switching the buffer_type from "memory" to "file" happened after the version was bumped to v3.6.171. This is easy to reproduce with flexy. I'm thinking that the fluentd dependencies are conflicting - they are not up to date - once the 3.6 puddle is rebuilt I can build and test a new fluentd image. (In reply to Rich Megginson from comment #14) > This is easy to reproduce with flexy. > > I'm thinking that the fluentd dependencies are conflicting - they are not up > to date - once the 3.6 puddle is rebuilt I can build and test a new fluentd > image. This did not help :-( Now resorting to debugging the ruby code . . . The bug was introduced in logging-fluentd:v3.6.173.0.5-6 - logging-fluentd:v3.6.173.0.5-5 and earlier work. These are the commits between -5 and -6: http://pkgs.devel.redhat.com/cgit/rpms/logging-fluentd-docker/log/?h=rhaos-3.6-rhel-7 Impl fluentd file buffer. remove USE_MUX_CLIENT; mux service always check for k8s metadata fluentd 0.12.39; k8s filter 0.28.0; viaq 0.0.5 The error doesn't appear to be related to systemd input or elasticsearch output - I tried fluentd secure_forward with file buffer -> mux es with file buffer in both fluentd and mux I see the conversion error. So it must have something to do with file buffer, but I just don't know what it could be. Thanks, the fluentd is now back, checked with fluentd:v3.6.173.0.5-10 image that log entries can be collected and reflect on kibana. Set to verified. Image verified with: logging-fluentd v3.6.173.0.5-10 58ab4badc0b7 6 hours ago 235.1 MB Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:3049 |