FROM rhel7.2 MAINTAINER OpenShift Development <dev.redhat.com> ENV HOME=/opt/app-root/src \ PATH=/opt/app-root/src/bin:/opt/app-root/bin:$PATH \ RUBY_VERSION=2.0 \ FLUENTD_VERSION=0.12.31 \ GEM_HOME=/opt/app-root/src LABEL io.k8s.description="Fluentd container for collecting of docker container logs" \ io.k8s.display-name="Fluentd ${FLUENTD_VERSION}" \ io.openshift.expose-services="9200:http, 9300:http" \ io.openshift.tags="logging,elk,fluentd" # activesupport version 5.x requires ruby 2.2 # iproute needed for ip command to get ip addresses RUN rpmkeys --import file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release && \ yum install -y --setopt tsflags=nodocs \ gcc-c++ \ libcurl-devel \ make \ iproute \ hostname-3.13-3.el7.x86_64 \ ruby \ ruby-devel && \ yum clean all RUN mkdir -p ${HOME} && \ gem install -N --conservative --minimal-deps \ fluentd:${FLUENTD_VERSION} \ 'activesupport:<5' \ fluent-plugin-kubernetes_metadata_filter \ fluent-plugin-elasticsearch \ 'fluent-plugin-systemd:<0.1.0' \ systemd-journal \ fluent-plugin-rewrite-tag-filter \ fluent-plugin-secure-forward \ fluent-plugin-flatten-hash \ fluent-plugin-viaq_data_model ADD configs.d/ /etc/fluent/configs.d/ ADD run.sh generate_throttle_configs.rb ${HOME}/ RUN mkdir -p /etc/fluent/configs.d/{dynamic,user} && \ chmod 777 /etc/fluent/configs.d/dynamic && \ ln -s /etc/fluent/configs.d/user/fluent.conf /etc/fluent/fluent.conf WORKDIR ${HOME} USER 0 CMD ["sh", "run.sh"] That's the docker file we used to try to build a new fluentd images, but we are getting error building the ruby plugin ERROR: Error installing fluent-plugin-flatten-hash: serverengine requires Ruby version >= 2.1.0. That is based on the https://github.com/openshift/origin-aggregated-logging/tree/master/fluentd but instead of centos we updated it to rhel 7.2
Try removing fluent-plugin-flatten-hash, and remove fluentd/configs.d/openshift/filter-k8s-flatten-hash.conf, and remove any reference to flatten-hash in any other fluent.conf file. This is not needed any more with 3.4.
If that seems too risky, try adding 'serverengine:<2' \ to the list of gem install packages, just after 'activesupport:<5'
the image build process completed, but now we are seeing the following error when we started the fluentd pod # oc logs -f logging-fluentd-yle8g -n logging 2017-04-19 09:14:44 -0700 [info]: reading config file path="/etc/fluent/fluent.conf" 2017-04-19 09:15:03 -0700 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-19 09:15:03 -0700 error_class="TypeError" error="no implicit conversion of nil into String" plugin_id="object:1cfe238" 2017-04-19 09:15:03 -0700 [warn]: /opt/app-root/src/gems/fluent-plugin-elasticsearch-1.9.3/lib/fluent/plugin/out_elasticsearch_dynamic.rb:239:in `sub!' 2017-04-19 09:15:03 -0700 [warn]: /opt/app-root/src/gems/fluent-plugin-elasticsearch-1.9.3/lib/fluent/plugin/out_elasticsearch_dynamic.rb:239:in `expand_param' 2017-04-19 09:15:03 -0700 [warn]: /opt/app-root/src/gems/fluent-plugin-elasticsearch-1.9.3/lib/fluent/plugin/out_elasticsearch_dynamic.rb:129:in `block (2 levels) in write_objects' 2017-04-19 09:15:03 -0700 [warn]: /opt/app-root/src/gems/fluent-plugin-elasticsearch-1.9.3/lib/fluent/plugin/out_elasticsearch_dynamic.rb:124:in `each' 2017-04-19 09:15:03 -0700 [warn]: /opt/app-root/src/gems/fluent-plugin-elasticsearch-1.9.3/lib/fluent/plugin/out_elasticsearch_dynamic.rb:124:in `each_with_index' 2017-04-19 09:15:03 -0700 [warn]: /opt/app-root/src/gems/fluent-plugin-elasticsearch-1.9.3/lib/fluent/plugin/out_elasticsearch_dynamic.rb:124:in `block in write_objects' 2017-04-19 09:15:03 -0700 [warn]: /opt/app-root/src/gems/fluentd-0.12.31/lib/fluent/plugin/buf_memory.rb:67:in `feed_each' 2017-04-19 09:15:03 -0700 [warn]: /opt/app-root/src/gems/fluentd-0.12.31/lib/fluent/plugin/buf_memory.rb:67:in `msgpack_each' 2017-04-19 09:15:03 -0700 [warn]: /opt/app-root/src/gems/fluent-plugin-elasticsearch-1.9.3/lib/fluent/plugin/out_elasticsearch_dynamic.rb:120:in `write_objects' 2017-04-19 09:15:03 -0700 [warn]: /opt/app-root/src/gems/fluentd-0.12.31/lib/fluent/output.rb:490:in `write' 2017-04-19 09:15:03 -0700 [warn]: /opt/app-root/src/gems/fluentd-0.12.31/lib/fluent/buffer.rb:354:in `write_chunk' 2017-04-19 09:15:03 -0700 [warn]: /opt/app-root/src/gems/fluentd-0.12.31/lib/fluent/buffer.rb:333:in `pop' 2017-04-19 09:15:03 -0700 [warn]: /opt/app-root/src/gems/fluentd-0.12.31/lib/fluent/output.rb:342:in `try_flush' 2017-04-19 09:15:03 -0700 [warn]: /opt/app-root/src/gems/fluentd-0.12.31/lib/fluent/output.rb:149:in `run' 2017-04-19 09:15:03 -0700 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-19 09:15:05 -0700 error_class="TypeError" error="no implicit conversion of nil into String" plugin_id="object:1cfe238" 2017-04-19 09:15:03 -0700 [warn]: suppressed same stacktrace 2017-04-19 09:15:05 -0700 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-19 09:15:09 -0700 error_class="TypeError" error="no implicit conversion of nil into String" plugin_id="object:1cfe238" 2017-04-19 09:15:05 -0700 [warn]: suppressed same stacktrace 2017-04-19 09:15:09 -0700 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-19 09:15:17 -0700 error_class="TypeError" error="no implicit conversion of nil into String" plugin_id="object:1cfe238" 2017-04-19 09:15:09 -0700 [warn]: suppressed same stacktrace 2017-04-19 09:15:17 -0700 [warn]: temporarily failed to flush the buffer. next_retry=2017-04-19 09:15:34 -0700 error_class="TypeError" error="no implicit conversion of nil into String" plugin_id="object:1cfe238" 2017-04-19 09:15:17 -0700 [warn]: suppressed same stacktrace and adding 'serverengine:<2' \ to the list of gem install packages, just after 'activesupport:<5' without removing fluent-plugin-flatten-hash didn't work
Can I see the fluent.conf? oc get configmap logging-fluentd -o yaml Also the contents of the fluentd config dir: oc exec $fluentdpod -- ls -alrtFR /etc/fluent
oc exec logging-fluentd-p4jt8 -n logging -- ls -alrtFR /etc/fluent /etc/fluent: total 4 lrwxrwxrwx. 1 root root 38 Apr 19 09:31 fluent.conf -> /etc/fluent/configs.d/user/fluent.conf drwxr-xr-x. 5 root root 50 Apr 19 09:31 configs.d/ drwxrwxrwt. 3 root root 140 Apr 19 09:35 keys/ drwxr-xr-x. 51 root root 4096 Apr 19 09:35 ../ drwxr-xr-x. 4 root root 54 Apr 19 09:35 ./ /etc/fluent/configs.d: total 8 drwxrwxr-x. 2 root root 4096 Apr 19 09:29 openshift/ drwxr-xr-x. 5 root root 50 Apr 19 09:31 ./ drwxrwxrwx. 3 root root 4096 Apr 19 09:35 user/ drwxr-xr-x. 4 root root 54 Apr 19 09:35 ../ drwxrwxrwx. 2 root root 144 Apr 19 09:35 dynamic/ /etc/fluent/configs.d/openshift: total 64 -rwxrwxr-x. 1 root root 36 Apr 13 15:30 system.conf* -rwxrwxr-x. 1 root root 242 Apr 13 15:30 output-operations.conf* -rwxrwxr-x. 1 root root 151 Apr 13 15:30 output-applications.conf* -rwxrwxr-x. 1 root root 2135 Apr 13 15:30 filter-syslog-record-transform.conf* -rwxrwxr-x. 1 root root 4578 Apr 13 15:30 filter-retag-journal.conf* -rwxrwxr-x. 1 root root 216 Apr 13 15:30 filter-kibana-transform.conf* -rwxrwxr-x. 1 root root 1390 Apr 13 15:30 filter-k8s-record-transform.conf* -rwxrwxr-x. 1 root root 314 Apr 13 15:30 filter-k8s-meta.conf* -rwxrwxr-x. 1 root root 64 Apr 13 15:30 filter-exclude-journal-debug.conf* -rwxrwxr-x. 1 root root 4061 Apr 13 15:30 README* -rwxrwxr-x. 1 root root 615 Apr 19 09:28 es-copy-config.conf* -rwxrwxr-x. 1 root root 648 Apr 19 09:28 es-ops-copy-config.conf* -rwxrwxr-x. 1 root root 558 Apr 19 09:28 output-es-ops-config.conf* -rwxrwxr-x. 1 root root 528 Apr 19 09:29 output-es-config.conf* drwxrwxr-x. 2 root root 4096 Apr 19 09:29 ./ drwxr-xr-x. 5 root root 50 Apr 19 09:31 ../ /etc/fluent/configs.d/user: total 4 drwxr-xr-x. 5 root root 50 Apr 19 09:31 ../ lrwxrwxrwx. 1 root root 27 Apr 19 09:35 throttle-config.yaml -> ..data/throttle-config.yaml lrwxrwxrwx. 1 root root 18 Apr 19 09:35 fluent.conf -> ..data/fluent.conf lrwxrwxrwx. 1 root root 31 Apr 19 09:35 ..data -> ..4984_19_04_09_35_40.734909615/ drwxr-xr-x. 2 root root 51 Apr 19 09:35 ..4984_19_04_09_35_40.734909615/ drwxrwxrwx. 3 root root 4096 Apr 19 09:35 ./ /etc/fluent/configs.d/user/..4984_19_04_09_35_40.734909615: total 8 -rw-r--r--. 1 root root 0 Apr 19 09:35 throttle-config.yaml -rw-r--r--. 1 root root 877 Apr 19 09:35 fluent.conf drwxrwxrwx. 3 root root 4096 Apr 19 09:35 ../ drwxr-xr-x. 2 root root 51 Apr 19 09:35 ./ /etc/fluent/configs.d/dynamic: total 16 drwxr-xr-x. 5 root root 50 Apr 19 09:31 ../ -rw-r--r--. 1 root root 548 Apr 19 09:35 input-syslog-default-syslog.conf -rw-r--r--. 1 root root 256 Apr 19 09:35 input-docker-default-docker.conf -rw-r--r--. 1 root root 1 Apr 19 09:35 es-ops-copy-config.conf -rw-r--r--. 1 root root 1 Apr 19 09:35 es-copy-config.conf drwxrwxrwx. 2 root root 144 Apr 19 09:35 ./ /etc/fluent/keys: total 0 drwxr-xr-x. 2 root root 100 Apr 19 09:35 ..4984_19_04_09_35_40.035434562/ lrwxrwxrwx. 1 root root 10 Apr 19 09:35 key -> ..data/key lrwxrwxrwx. 1 root root 11 Apr 19 09:35 cert -> ..data/cert lrwxrwxrwx. 1 root root 9 Apr 19 09:35 ca -> ..data/ca lrwxrwxrwx. 1 root root 31 Apr 19 09:35 ..data -> ..4984_19_04_09_35_40.035434562/ drwxrwxrwt. 3 root root 140 Apr 19 09:35 ./ drwxr-xr-x. 4 root root 54 Apr 19 09:35 ../ /etc/fluent/keys/..4984_19_04_09_35_40.035434562: total 12 -rw-r--r--. 1 root root 1708 Apr 19 09:35 key -rw-r--r--. 1 root root 1249 Apr 19 09:35 cert -rw-r--r--. 1 root root 1078 Apr 19 09:35 ca drwxr-xr-x. 2 root root 100 Apr 19 09:35 ./ drwxrwxrwt. 3 root root 140 Apr 19 09:35 ../
# oc get configmap logging-fluentd -n logging -o yaml apiVersion: v1 data: fluent.conf: |2 @include configs.d/openshift/system.conf @include configs.d/openshift/input-pre-*.conf @include configs.d/dynamic/input-docker-*.conf @include configs.d/dynamic/input-syslog-*.conf @include configs.d/openshift/input-post-*.conf <label @INGRESS> @include configs.d/openshift/filter-pre-*.conf @include configs.d/openshift/filter-retag-journal.conf @include configs.d/openshift/filter-k8s-meta.conf @include configs.d/openshift/filter-kibana-transform.conf @include configs.d/openshift/filter-k8s-record-transform.conf @include configs.d/openshift/filter-syslog-record-transform.conf @include configs.d/openshift/filter-post-*.conf @include configs.d/openshift/output-operations.conf @include configs.d/openshift/output-pre-*.conf @include configs.d/openshift/output-applications.conf </label> throttle-config.yaml: "" kind: ConfigMap metadata: creationTimestamp: 2017-02-28T21:01:23Z labels: logging-infra: support name: logging-fluentd namespace: logging resourceVersion: "42471983" selfLink: /api/v1/namespaces/logging/configmaps/logging-fluentd uid: 1002237d-fdf9-11e6-ac85-1c98ec5f2b08
I'm confused - is this for 3.3 or 3.4? If it is for 3.3, then why does Dockerfile have fluent-plugin-viaq_data_model in it? If this is for 3.3, then the fluentd gem fluent-plugin-flatten-hash, and the config file fluent/configs.d/openshift/filter-k8s-flatten-hash.conf, are all required, and you will need to have this in your fluent.conf: @include configs.d/openshift/filter-k8s-meta.conf @include configs.d/openshift/filter-k8s-flatten-hash.conf @include configs.d/openshift/filter-kibana-transform.conf If this is 3.4, then you must have this in your fluent.conf: @include configs.d/openshift/filter-syslog-record-transform.conf @include configs.d/openshift/filter-viaq-data-model.conf @include configs.d/openshift/filter-post-*.conf
It was recommended as per comment 4 and comment 5 to remove flattern-hash plugin because the docker image failed to build. MESSAGE Try removing fluent-plugin-flatten-hash, and remove fluentd/configs.d/openshift/filter-k8s-flatten-hash.conf, and remove any reference to flatten-hash in any other fluent.conf file. This is not needed any more with 3.4. If that seems too risky, try adding 'serverengine:<2' \ to the list of gem install packages, just after 'activesupport:<5' Please let us know how to build the custom fluentd image for 3.3 with fluent-plugin-flatten-hash plugin because when we built it it was asking for ruby 2.1.0 ERROR: Error installing fluent-plugin-flatten-hash: serverengine requires Ruby version >= 2.1.0. My mistake the version of this OCP was not set to 3.3. My Sincere apologies for that.
Right - https://bugzilla.redhat.com/show_bug.cgi?id=1442951#c4 says "This (flatten_hash) is not needed any more with 3.4." It most definitely is required for 3.3. I guess I should have made that more clear. To build on 3.3: add 'serverengine:<2' \ to the list of gem install packages, just after 'activesupport:<5' That should pull in serverengine < 2, and hopefully fluent-plugin-flatten-hash will be satisfied with that version. If not, then you'll have to change fluent-plugin-flatten-hash \ to 'fluent-plugin-flatten-hash:<$version' \ Where $version is the latest version reported by https://rubygems.org/gems/fluent-plugin-flatten-hash. Then just keep going down the versions until you find one that works.
Miheer can you comment on the status of this issue?
> FYI: both OCP-3.4 and 3.5 do not have filter-k8s-flatten-hash.conf in the fluentd config dirs. We don't need to flatten with the common data model. We switched to that in 3.4. The default cpu and memory limits are not suitable for high volume log traffic. Try editing the fluentd daemonset to increase the cpu and memory limits.
The fluentd buffer_queue_limit and buffer_chunk_limit are configurable in OCP-3.6. (See also bz1413147 - Size of the emitted data exceeds buffer_chunk_limit) Also, the customer case 01831544 is closed. Miheer, may we close this bug as dup of bz1413147?
Closing as looks like is a duplicate and we have a fix in 3.6. Please reopen and work with PM if it needs to be backported to another version *** This bug has been marked as a duplicate of bug 1413147 ***
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days