|Summary:||Failed to forward docker events to remote rsyslog server.|
|Product:||OpenShift Container Platform||Reporter:||Qiaoling Tang <qitang>|
|Component:||Logging||Assignee:||Noriko Hosoi <nhosoi>|
|Status:||CLOSED ERRATA||QA Contact:||Anping Li <anli>|
|Version:||3.10.0||CC:||aos-bugs, jcantril, nhosoi, rmeggins|
|Fixed In Version:||Doc Type:||No Doc Update|
undefined (it was introduced by bz1515715 which is not released yet.)
|Last Closed:||2018-07-30 19:14:38 UTC||Type:||Bug|
|oVirt Team:||---||RHEL 7.3 requirements from Atomic Host:|
|Cloudforms Team:||---||Target Upstream Version:|
Description Qiaoling Tang 2018-05-08 02:31:45 UTC
Description of problem: Deploy logging with "openshift_logging_fluentd_audit_container_engine=true", and enable remote rsyslog server, then check logs in rsyslog server and rsyslogback server, no docker event could be found. However, the docker event could be found in es pod, and other logs could be found in rsyslog server. Version-Release number of selected component (if applicable): oc v3.10.0-0.36.0 kubernetes v1.10.0+b81c8f8 features: Basic-Auth GSSAPI Kerberos SPNEGO openshift v3.10.0-0.36.0 kubernetes v1.10.0+b81c8f8 Image Version: logging-fluentd-v3.10.0-0.36.0.0 logging-elasticsearch-v3.10.0-0.36.0.2 Ansible Version: openshift-ansible-3.10.0-0.36.0.git.0.521f0ef.el7.noarch How reproducible: Always Steps to Reproduce: 1.Deploy logging with "openshift_logging_fluentd_audit_container_engine=true" 2.Enable remote rsyslog server 3.Execute docker commands 4.Check docker event in es and rsyslog server Actual results: Docker event could be found in es pod, but couldn't be found in rsyslog server. Expected results: Docker event could be found in es pod and rsyslog server when remote rsyslog server is enabled. Additional info: # oc get ds logging-fluentd -o yaml|grep -A 2 SYS - name: USE_REMOTE_SYSLOG value: "true" - name: REMOTE_SYSLOG_HOST value: $rsyslog_ip - name: REMOTE_SYSLOG_HOST_BACKUP value: $rsyslogback_ip - name: REMOTE_SYSLOG_PORT_BACKUP value: "514"
Comment 1 Noriko Hosoi 2018-05-08 04:43:40 UTC
(In reply to Qiaoling Tang from comment #0) > Actual results: > Docker event could be found in es pod, but couldn't be found in rsyslog > server. Please attach the following data. 1) a couple of example docker events indexed in the ElasticSearch, 2) output from oc logs <fluentd_pod>, 3) the remote syslog config file in the fluentd pod. /etc/fluent/configs.d/dynamic/output-remote-syslog.conf Also, is ops enabled? What is missing in the rsyslog server is just audit log? Other logs are successfully forwarded? Thanks.
Comment 5 Qiaoling Tang 2018-05-08 06:11:13 UTC
Hi Noriko, The ops wasn't enabled. The journal, docker log can be found in the rsyslog server.
Comment 6 Noriko Hosoi 2018-05-08 16:10:21 UTC
Thank you, @Qiaoling! Looking at the config file , there is no tag_key directive here. There is a known issue  found by @Anping, which is supposed to be fixed  in the next build. Could you please rerun the test when the next build including the fix ? Thanks!  - https://bugzilla.redhat.com/attachment.cgi?id=1432991  - https://bugzilla.redhat.com/show_bug.cgi?id=1515715#c18  - https://github.com/openshift/origin-aggregated-logging/pull/1134
Comment 8 Qiaoling Tang 2018-06-20 08:35:46 UTC
No log could be found in rsyslog server. Seems the changes are not in logging-fluentd/images/v3.10.1-1.
Comment 13 Qiaoling Tang 2018-06-25 03:37:41 UTC
Verified on logging-fluentd/images/v3.10.7-1.
Comment 15 errata-xmlrpc 2018-07-30 19:14:38 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1816