Bug 1396433
| Summary: | Fluentd pop lots of dump an error event: error_class=NoMethodError error="undefined method `empty?' | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Peng Li <penli> | ||||
| Component: | Logging | Assignee: | Rich Megginson <rmeggins> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Xia Zhao <xiazhao> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 3.4.0 | CC: | aos-bugs, jeder, tdawson | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | aos-scalability-34 | ||||||
| Fixed In Version: | Doc Type: | No Doc Update | |||||
| Doc Text: |
This was a pre-release issue, no doc update required.
|
Story Points: | --- | ||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2017-01-18 12:55:48 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Peng Li
2016-11-18 10:07:20 UTC
Created attachment 1221804 [details]
fluentd log
Fixed upstream: https://github.com/openshift/origin-aggregated-logging/commit/d81917dbc1a0fdee0b7eaf42d9c5b5d6c88942d0 I found another case where this can happen. I'm working on a fix. It is quite easy to reproduce this issue: $ oc get pods ... logging-fluentd-hm0e7 1/1 Running 0 50m logging-kibana-1-wuup6 2/2 Running 0 5h ... $ oc exec logging-kibana-1-wuup6 -c kibana -- curl -s http://localhost:5601/aaaaaaaaaabbbbbbbbbb {"statusCode":404,"error":"Not Found"} $ oc logs logging-fluentd-hm0e7 ... 2016-11-18 21:11:29 +0000 [warn]: dump an error event: error_class=NoMethodError error="undefined method `empty?' for 404:Fixnum" tag="kubernetes.var.log.containers.logging-kibana-1-wuup6_logging_kibana-d4065dd584eaee0201b328afc7cb791e852298f83efc101eca7e172e9e06d82a.log" time=1479503489 record={"type"=>"response", "@timestamp"=>"2016-11-18T21:11:29+00:00", "method"=>"get", "statusCode"=>404, "message"=>"GET /aaaaaaaaaabbbbbbbbbb 404 5ms - 9.0B", "time"=>"2016-11-18T21:11:29.456730219Z", "docker"=>{"container_id"=>"d4065dd584eaee0201b328afc7cb791e852298f83efc101eca7e172e9e06d82a"}, "kubernetes"=>{"namespace_name"=>"logging", "pod_id"=>"b9cce867-ada7-11e6-a34d-fa163e43b587", "pod_name"=>"logging-kibana-1-wuup6", "labels"=>{"component"=>"kibana", "deployment"=>"logging-kibana-1", "deploymentconfig"=>"logging-kibana", "provider"=>"openshift"}, "host"=>"host-192-168-78-2.openstacklocal", "namespace_id"=>"528cc6fb-ada6-11e6-a34d-fa163e43b587", "container_name"=>"kibana"}, "hostname"=>"host-192-168-78-2.openstacklocal", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"172.17.0.6", "ipaddr6"=>"fe80::42:acff:fe11:6", "inputname"=>"fluent-plugin-in_tail", "name"=>"fluentd openshift", "received_at"=>"2016-11-18T21:11:29.456730219Z", "version"=>"0.12.29 1.4.0"}}} Patch submitted upstream: https://github.com/openshift/origin-aggregated-logging/pull/287 Commit pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/c0217b16d8bc361e2a3e07dd0307ca339e03c36f Bug 1396433 - Fluentd pop lots of dump an error event: error_class=NoMethodError error="undefined method `empty?' https://bugzilla.redhat.com/show_bug.cgi?id=1396433 Never test a value for `empty?` directly - always use the `isempty` method which can be safely used. Added tests to exercise the code with top level values and nested values that do not have the `empty?` method. Fix committed to rh repo: To ssh://rmeggins.redhat.com/rpms/logging-fluentd-docker a2c334f..1aa072d rhaos-3.4-rhel-7 -> rhaos-3.4-rhel-7 https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=525239 Images have been built. verified with latest fluentd and issue is gone. #oc exec logging-kibana-1-y9rci -c kibana -- curl -s http://localhost:5601/aaaaaaaaaabbbbbbbbbb {"statusCode":404,"error":"Not Found"} # oc logs logging-fluentd-ni7zd 2016-11-20 22:49:03 -0500 [info]: reading config file path="/etc/fluent/fluent.conf" Version: logging-fluentd 3.4.0 7b11a29c82c1 47 hours ago 231.7 MB Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0066 |