Description of problem: Couldn't find fluentd pod log file. After a deployment of logging EFK stack, execute "oc logs $fluentd-pod", the output are in the below: ============================= Fluentd logs have been redirected to: /var/log/fluentd/fluentd.log If you want to print out the logs, use command: oc exec <pod_name> /opt/app-root/src/utils/logs ============================= umounts of dead containers will fail. Ignoring... umount: /var/lib/docker/containers/2ed37f4b5460c50224003b667cc8a80772290c296df1262f331cf8022aafc27a/shm: not mounted umount: /var/lib/docker/containers/3bebfa9be927ad765eacc17dbd9a240c3e5034d15d4d7c7ac8d391de8851450c/shm: not mounted umount: /var/lib/docker/containers/4391779ea9805a582a3b28cc9003c45f48777ef2a9303a0f91cb5c7097d9c007/shm: not mounted umount: /var/lib/docker/containers/63ca3d8ab9cc7518e589777e1d981a1615871439b6cebd3fe20689679bd1a61d/shm: not mounted umount: /var/lib/docker/containers/6ce5fb53d7bdd1686d9c70c81397a836b0505a8949a30f7efb4b23abd5920a1f/shm: not mounted umount: /var/lib/docker/containers/829088d8f33a5454badcbae52adedb3b47a9ff77995f6f2c09fc69f54a14353d/shm: not mounted umount: /var/lib/docker/containers/ab0f5b8ad0758c2778319d1327000b5ca6b0ba736a5c7c694457d9bbc5d5da59/shm: not mounted umount: /var/lib/docker/containers/ce952648d783c077db72486f2b56020b990e6aa55ffa6858e6b2b61e6a1fae24/shm: not mounted umount: /var/lib/docker/containers/d1dadd88a755d5163b8f194ddc34300b71d726aa27ff3e6724b5389bd5ec4010/shm: not mounted [DEPRECATION WARNING]: 'merge_json_log' for the fluent-plugin-kubernetes_metadata_filter will no longer be supported in future releases 2018-10-25 22:48:33 -0400 [info]: reading config file path="/etc/fluent/fluent.conf" 2018-10-25 22:48:44 -0400 [warn]: 'block' action stops input process until the buffer full is resolved. Check your pipeline this action is fit or not 2018-10-25 22:48:44 -0400 [warn]: 'block' action stops input process until the buffer full is resolved. Check your pipeline this action is fit or not 2018-10-25 22:49:22 -0400 [warn]: temporarily failed to flush the buffer. next_retry=2018-10-25 22:48:47 -0400 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"logging-es\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"})!" plugin_id="elasticsearch-apps" 2018-10-25 22:49:22 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.17.2/lib/fluent/plugin/out_elasticsearch.rb:228:in `client' 2018-10-25 22:49:22 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.17.2/lib/fluent/plugin/out_elasticsearch.rb:477:in `rescue in send_bulk' 2018-10-25 22:49:22 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.17.2/lib/fluent/plugin/out_elasticsearch.rb:464:in `send_bulk' 2018-10-25 22:49:22 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.17.2/lib/fluent/plugin/out_elasticsearch.rb:382:in `write_objects' 2018-10-25 22:49:22 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/output.rb:490:in `write' 2018-10-25 22:49:22 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/buffer.rb:354:in `write_chunk' 2018-10-25 22:49:22 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/buffer.rb:333:in `pop' 2018-10-25 22:49:22 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/output.rb:342:in `try_flush' 2018-10-25 22:49:22 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/output.rb:149:in `run' 2018-10-25 22:49:58 -0400 [warn]: temporarily failed to flush the buffer. next_retry=2018-10-25 22:48:49 -0400 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"logging-es\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"})!" plugin_id="elasticsearch-apps" 2018-10-25 22:49:58 -0400 [warn]: suppressed same stacktrace 2018-10-25 22:50:08 -0400 [warn]: retry succeeded. plugin_id="elasticsearch-apps" rsh into a fluentd pod, there is no fluentd.log file in /var/log/fluentd directory, and couldn't find /opt/app-root/src/utils/logs either. No output shows after run "oc exec $fluentd-pod -- logs". Version-Release number of selected component (if applicable): logging-fluentd-v3.9.48-1 How reproducible: Always Steps to Reproduce: 1.Deploy logging Actual results: Couldn't find fluentd pod log file. Expected results: Fluentd should log its log to a file by default. Additional info:
Partially resolved by: https://github.com/openshift/origin-aggregated-logging/pull/1411
https://github.com/openshift/origin-aggregated-logging/pull/1450 (release-3.11)
PR https://github.com/openshift/origin-aggregated-logging/pull/1450 is for release-3.11. Is it ok to change the target release of this bz to 3.11.z and change the status of this bug to MODIFIED?
This bug needs to be cloned for every release that is affected. For each clone, do the following: - set the Target Release to the appropriate 3.N.z - set the the appropriate aos-3.N.x Flag - Add a link to the PR for that branch in a comment _and_ add the PR to the External Trackers - Once you have a bz, add a link to the bz in the PR comment - Set the Doc Type and Doc Text fields Once the PR is submitted, change the Status of the bz to POST. Once the PR merges, set the status of the PR to MODIFIED. You will then have to wait until the ART automated merge and build runs to find out which image your fix is in. Once you find out, set the Fixed in Version field to the image name and version. https://mojo.redhat.com/docs/DOC-1178565#jive_content_id_Workflows https://mojo.redhat.com/docs/DOC-1070701
https://github.com/openshift/origin-aggregated-logging/pull/1456 release-3.9] Bug 1643340 - Fluentd pod should log its logs to /var/log/fluentd/fluentd.log by default
Commit pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/11ebbb877ba03241ee88388b4d0621ee47fea24c Bug 1643340 - Fluentd pod should log its logs to /var/log/fluentd/fluentd.log by default Fixing the logs utility path. Making the output of the run.sh script go to the /var/log/fluentd/fluentd.log file. (cherry picked from commit 97e9399cb114637660540323a1a3d24f3d3816f8)
Verified in logging-fluentd-v3.9.55-1.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:3748