Bug 1643341

Summary:	[RFE] [3.10] Fluentd pod should log its logs to /var/log/fluentd/fluentd.log by default
Product:	OpenShift Container Platform	Reporter:	Qiaoling Tang <qitang>
Component:	Logging	Assignee:	Noriko Hosoi <nhosoi>
Status:	CLOSED ERRATA	QA Contact:	Qiaoling Tang <qitang>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	3.10.0	CC:	anli, aos-bugs, jcantril, nhosoi, rmeggins
Target Milestone:	---
Target Release:	3.10.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: Running "oc logs $fluentd_pod" suggests to run "oc exec <pod_name> /opt/app-root/src/utils/logs" which includes the non existing utility logs path. Consequence: "oc exec <pod_name> /opt/app-root/src/utils/logs" fails with "no such file or directory". Fix: Fixed the suggested command line to "oc exec <pod_name> -- logs" since the utility "logs" is now in the PATH and no need to specify the full path. Result: "oc logs $fluentd_pod" suggests the correct command line to show the fluentd logs.	Story Points:	---
Clone Of:	1643340	Environment:
Last Closed:	2018-12-13 17:09:08 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1643340, 1649997, 1650005
Bug Blocks:

Description Qiaoling Tang 2018-10-26 03:40:57 UTC

+++ This bug was initially created as a clone of Bug #1643340 +++

Description of problem:
Couldn't find fluentd pod log file.

After a deployment of logging EFK stack, execute "oc logs $fluentd-pod", the output are in the below:
=============================
Fluentd logs have been redirected to: /var/log/fluentd/fluentd.log
If you want to print out the logs, use command:
oc exec <pod_name> /opt/app-root/src/utils/logs
=============================
umounts of dead containers will fail. Ignoring...
umount: /var/lib/docker/containers/2ed37f4b5460c50224003b667cc8a80772290c296df1262f331cf8022aafc27a/shm: not mounted
umount: /var/lib/docker/containers/3bebfa9be927ad765eacc17dbd9a240c3e5034d15d4d7c7ac8d391de8851450c/shm: not mounted
umount: /var/lib/docker/containers/4391779ea9805a582a3b28cc9003c45f48777ef2a9303a0f91cb5c7097d9c007/shm: not mounted
umount: /var/lib/docker/containers/63ca3d8ab9cc7518e589777e1d981a1615871439b6cebd3fe20689679bd1a61d/shm: not mounted
umount: /var/lib/docker/containers/6ce5fb53d7bdd1686d9c70c81397a836b0505a8949a30f7efb4b23abd5920a1f/shm: not mounted
umount: /var/lib/docker/containers/829088d8f33a5454badcbae52adedb3b47a9ff77995f6f2c09fc69f54a14353d/shm: not mounted
umount: /var/lib/docker/containers/ab0f5b8ad0758c2778319d1327000b5ca6b0ba736a5c7c694457d9bbc5d5da59/shm: not mounted
umount: /var/lib/docker/containers/ce952648d783c077db72486f2b56020b990e6aa55ffa6858e6b2b61e6a1fae24/shm: not mounted
umount: /var/lib/docker/containers/d1dadd88a755d5163b8f194ddc34300b71d726aa27ff3e6724b5389bd5ec4010/shm: not mounted
[DEPRECATION WARNING]: 'merge_json_log' for the fluent-plugin-kubernetes_metadata_filter will no longer be supported in future releases
2018-10-25 22:48:33 -0400 [info]: reading config file path="/etc/fluent/fluent.conf"
2018-10-25 22:48:44 -0400 [warn]: 'block' action stops input process until the buffer full is resolved. Check your pipeline this action is fit or not
2018-10-25 22:48:44 -0400 [warn]: 'block' action stops input process until the buffer full is resolved. Check your pipeline this action is fit or not
2018-10-25 22:49:22 -0400 [warn]: temporarily failed to flush the buffer. next_retry=2018-10-25 22:48:47 -0400 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"logging-es\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"})!" plugin_id="elasticsearch-apps"
  2018-10-25 22:49:22 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.17.2/lib/fluent/plugin/out_elasticsearch.rb:228:in `client'
  2018-10-25 22:49:22 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.17.2/lib/fluent/plugin/out_elasticsearch.rb:477:in `rescue in send_bulk'
  2018-10-25 22:49:22 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.17.2/lib/fluent/plugin/out_elasticsearch.rb:464:in `send_bulk'
  2018-10-25 22:49:22 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.17.2/lib/fluent/plugin/out_elasticsearch.rb:382:in `write_objects'
  2018-10-25 22:49:22 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/output.rb:490:in `write'
  2018-10-25 22:49:22 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/buffer.rb:354:in `write_chunk'
  2018-10-25 22:49:22 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/buffer.rb:333:in `pop'
  2018-10-25 22:49:22 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/output.rb:342:in `try_flush'
  2018-10-25 22:49:22 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/output.rb:149:in `run'
2018-10-25 22:49:58 -0400 [warn]: temporarily failed to flush the buffer. next_retry=2018-10-25 22:48:49 -0400 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"logging-es\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"})!" plugin_id="elasticsearch-apps"
  2018-10-25 22:49:58 -0400 [warn]: suppressed same stacktrace
2018-10-25 22:50:08 -0400 [warn]: retry succeeded. plugin_id="elasticsearch-apps"

rsh into a fluentd pod, there is no fluentd.log file in /var/log/fluentd directory, and couldn't find /opt/app-root/src/utils/logs either.

No output shows after run "oc exec $fluentd-pod -- logs".

Version-Release number of selected component (if applicable):
logging-fluentd-v3.10.64-1

How reproducible:
Always

Steps to Reproduce:
1.Deploy logging 

Actual results:
Couldn't find fluentd pod log file.

Expected results:
Fluentd should log its log to a file by default.

Additional info:

Comment 1 Noriko Hosoi 2018-11-14 23:49:08 UTC

https://github.com/openshift/origin-aggregated-logging/pull/1455

Comment 5 Qiaoling Tang 2018-11-29 07:29:38 UTC

Verified in logging-fluentd-v3.10.79-1.

Comment 7 errata-xmlrpc 2018-12-13 17:09:08 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3750