Description of problem: Fluentd generate many [warn]: dump an error event: error_class=NoMethodError error="undefined method `empty?' for 200:Fixnum" This is originally described at https://bugzilla.redhat.com/show_bug.cgi?id=1396019#c5, separate it so we can track it separately and this PR https://github.com/openshift/origin-aggregated-logging/pull/284 should solve this, but we can still see such error. Looks like it's not included by the latest image. Version-Release number of selected component (if applicable): openshift3/logging-fluentd 3.4.0 2cb15a5ae51e 18 hours ago 231.7 MB How reproducible: always Steps to Reproduce: 1. deploy logging stack. oc apply -n openshift -f https://raw.githubusercontent.com/openshift/origin-aggregated-logging/master/deployer/deployer.yaml oc project logging oc secrets new logging-deployer nothing=/dev/null oc new-app logging-deployer-account-template oadm policy add-cluster-role-to-user oauth-editor system:serviceaccount:logging:logging-deployer oadm policy add-scc-to-user privileged system:serviceaccount:logging:aggregated-logging-fluentd oadm policy add-cluster-role-to-user cluster-reader system:serviceaccount:logging:aggregated-logging-fluentd oadm policy add-cluster-role-to-user rolebinding-reader system:serviceaccount:logging:aggregated-logging-elasticsearch oc create configmap logging-deployer \ --from-literal kibana-hostname=kibana.$SUBDOMAIN \ --from-literal public-master-url=https://$MASTER:$PORT \ --from-literal es-cluster-size=1 \ --from-literal es-instance-ram=1G \ --from-literal es-pvc-size= \ --from-literal es-pvc-dynamic= \ --from-literal use-journal=false \ --from-literal enable-ops-cluster=false \ --from-literal kibana-ops-hostname=kibana-ops.$SUBDOMAIN oc label node -l registry=enabled logging-infra-fluentd=true --overwrite oc new-app logging-deployer-template -p IMAGE_PREFIX=$PREFIX -p IMAGE_VERSION=$VERSION -p MODE=install 2. check fluentd logs. 3. Actual results: see many error info. Expected results: no such error info. Additional info: log attached.
Created attachment 1221804 [details] fluentd log
Fixed upstream: https://github.com/openshift/origin-aggregated-logging/commit/d81917dbc1a0fdee0b7eaf42d9c5b5d6c88942d0
I found another case where this can happen. I'm working on a fix.
It is quite easy to reproduce this issue: $ oc get pods ... logging-fluentd-hm0e7 1/1 Running 0 50m logging-kibana-1-wuup6 2/2 Running 0 5h ... $ oc exec logging-kibana-1-wuup6 -c kibana -- curl -s http://localhost:5601/aaaaaaaaaabbbbbbbbbb {"statusCode":404,"error":"Not Found"} $ oc logs logging-fluentd-hm0e7 ... 2016-11-18 21:11:29 +0000 [warn]: dump an error event: error_class=NoMethodError error="undefined method `empty?' for 404:Fixnum" tag="kubernetes.var.log.containers.logging-kibana-1-wuup6_logging_kibana-d4065dd584eaee0201b328afc7cb791e852298f83efc101eca7e172e9e06d82a.log" time=1479503489 record={"type"=>"response", "@timestamp"=>"2016-11-18T21:11:29+00:00", "method"=>"get", "statusCode"=>404, "message"=>"GET /aaaaaaaaaabbbbbbbbbb 404 5ms - 9.0B", "time"=>"2016-11-18T21:11:29.456730219Z", "docker"=>{"container_id"=>"d4065dd584eaee0201b328afc7cb791e852298f83efc101eca7e172e9e06d82a"}, "kubernetes"=>{"namespace_name"=>"logging", "pod_id"=>"b9cce867-ada7-11e6-a34d-fa163e43b587", "pod_name"=>"logging-kibana-1-wuup6", "labels"=>{"component"=>"kibana", "deployment"=>"logging-kibana-1", "deploymentconfig"=>"logging-kibana", "provider"=>"openshift"}, "host"=>"host-192-168-78-2.openstacklocal", "namespace_id"=>"528cc6fb-ada6-11e6-a34d-fa163e43b587", "container_name"=>"kibana"}, "hostname"=>"host-192-168-78-2.openstacklocal", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"172.17.0.6", "ipaddr6"=>"fe80::42:acff:fe11:6", "inputname"=>"fluent-plugin-in_tail", "name"=>"fluentd openshift", "received_at"=>"2016-11-18T21:11:29.456730219Z", "version"=>"0.12.29 1.4.0"}}}
Patch submitted upstream: https://github.com/openshift/origin-aggregated-logging/pull/287
Commit pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/c0217b16d8bc361e2a3e07dd0307ca339e03c36f Bug 1396433 - Fluentd pop lots of dump an error event: error_class=NoMethodError error="undefined method `empty?' https://bugzilla.redhat.com/show_bug.cgi?id=1396433 Never test a value for `empty?` directly - always use the `isempty` method which can be safely used. Added tests to exercise the code with top level values and nested values that do not have the `empty?` method.
Fix committed to rh repo: To ssh://rmeggins.redhat.com/rpms/logging-fluentd-docker a2c334f..1aa072d rhaos-3.4-rhel-7 -> rhaos-3.4-rhel-7
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=525239 Images have been built.
verified with latest fluentd and issue is gone. #oc exec logging-kibana-1-y9rci -c kibana -- curl -s http://localhost:5601/aaaaaaaaaabbbbbbbbbb {"statusCode":404,"error":"Not Found"} # oc logs logging-fluentd-ni7zd 2016-11-20 22:49:03 -0500 [info]: reading config file path="/etc/fluent/fluent.conf" Version: logging-fluentd 3.4.0 7b11a29c82c1 47 hours ago 231.7 MB
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0066