Bug 1575815 - Failed to forward docker events to remote rsyslog server.
Summary: Failed to forward docker events to remote rsyslog server.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.10.0
Hardware: x86_64
OS: Linux
unspecified
low
Target Milestone: ---
: 3.10.0
Assignee: Noriko Hosoi
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-05-08 02:31 UTC by Qiaoling Tang
Modified: 2018-07-30 19:14 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined (it was introduced by bz1515715 which is not released yet.)
Clone Of:
Environment:
Last Closed: 2018-07-30 19:14:38 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:1816 None None None 2018-07-30 19:14:59 UTC

Description Qiaoling Tang 2018-05-08 02:31:45 UTC
Description of problem:
Deploy logging with "openshift_logging_fluentd_audit_container_engine=true", and enable remote rsyslog server, then check logs in rsyslog server and rsyslogback server, no docker event could be found. However, the docker event could be found in es pod, and other logs could be found in rsyslog server.

Version-Release number of selected component (if applicable):
oc v3.10.0-0.36.0
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

openshift v3.10.0-0.36.0
kubernetes v1.10.0+b81c8f8

Image Version:
logging-fluentd-v3.10.0-0.36.0.0
logging-elasticsearch-v3.10.0-0.36.0.2

Ansible Version:
openshift-ansible-3.10.0-0.36.0.git.0.521f0ef.el7.noarch

How reproducible:
Always


Steps to Reproduce:
1.Deploy logging with "openshift_logging_fluentd_audit_container_engine=true"
2.Enable remote rsyslog server
3.Execute docker commands
4.Check docker event in es and rsyslog server

Actual results:
Docker event could be found in es pod, but couldn't be found in rsyslog server.

Expected results:
Docker event could be found in es pod and rsyslog server when remote rsyslog server is enabled.

Additional info:
# oc get ds logging-fluentd -o yaml|grep -A 2 SYS
        - name: USE_REMOTE_SYSLOG
          value: "true"
        - name: REMOTE_SYSLOG_HOST
          value: $rsyslog_ip
        - name: REMOTE_SYSLOG_HOST_BACKUP
          value: $rsyslogback_ip
        - name: REMOTE_SYSLOG_PORT_BACKUP
          value: "514"

Comment 1 Noriko Hosoi 2018-05-08 04:43:40 UTC
(In reply to Qiaoling Tang from comment #0)
> Actual results:
> Docker event could be found in es pod, but couldn't be found in rsyslog
> server.

Please attach the following data.
1) a couple of example docker events indexed in the ElasticSearch,
2) output from oc logs <fluentd_pod>,
3) the remote syslog config file in the fluentd pod.
   /etc/fluent/configs.d/dynamic/output-remote-syslog.conf

Also, is ops enabled?  What is missing in the rsyslog server is just audit log?  Other logs are successfully forwarded?

Thanks.

Comment 5 Qiaoling Tang 2018-05-08 06:11:13 UTC
Hi Noriko, The ops wasn't enabled. The journal, docker log can be found in the rsyslog server.

Comment 6 Noriko Hosoi 2018-05-08 16:10:21 UTC
Thank you, @Qiaoling!

Looking at the config file [1], there is no tag_key directive here.  There is a known issue [2] found by @Anping, which is supposed to be fixed [3] in the next build.

Could you please rerun the test when the next build including the fix [3]?
Thanks!

[1] - https://bugzilla.redhat.com/attachment.cgi?id=1432991
[2] - https://bugzilla.redhat.com/show_bug.cgi?id=1515715#c18
[3] - https://github.com/openshift/origin-aggregated-logging/pull/1134

Comment 8 Qiaoling Tang 2018-06-20 08:35:46 UTC
No log could be found in rsyslog server.

Seems the changes are not in logging-fluentd/images/v3.10.1-1.

Comment 13 Qiaoling Tang 2018-06-25 03:37:41 UTC
Verified on logging-fluentd/images/v3.10.7-1.

Comment 15 errata-xmlrpc 2018-07-30 19:14:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1816


Note You need to log in before you can comment on or make changes to this bug.