Bug 1575815

Summary: Failed to forward docker events to remote rsyslog server.
Product: OpenShift Container Platform Reporter: Qiaoling Tang <qitang>
Component: LoggingAssignee: Noriko Hosoi <nhosoi>
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: low Docs Contact:
Priority: unspecified    
Version: 3.10.0CC: aos-bugs, jcantril, nhosoi, rmeggins
Target Milestone: ---   
Target Release: 3.10.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined (it was introduced by bz1515715 which is not released yet.)
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-30 19:14:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Qiaoling Tang 2018-05-08 02:31:45 UTC
Description of problem:
Deploy logging with "openshift_logging_fluentd_audit_container_engine=true", and enable remote rsyslog server, then check logs in rsyslog server and rsyslogback server, no docker event could be found. However, the docker event could be found in es pod, and other logs could be found in rsyslog server.

Version-Release number of selected component (if applicable):
oc v3.10.0-0.36.0
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

openshift v3.10.0-0.36.0
kubernetes v1.10.0+b81c8f8

Image Version:
logging-fluentd-v3.10.0-0.36.0.0
logging-elasticsearch-v3.10.0-0.36.0.2

Ansible Version:
openshift-ansible-3.10.0-0.36.0.git.0.521f0ef.el7.noarch

How reproducible:
Always


Steps to Reproduce:
1.Deploy logging with "openshift_logging_fluentd_audit_container_engine=true"
2.Enable remote rsyslog server
3.Execute docker commands
4.Check docker event in es and rsyslog server

Actual results:
Docker event could be found in es pod, but couldn't be found in rsyslog server.

Expected results:
Docker event could be found in es pod and rsyslog server when remote rsyslog server is enabled.

Additional info:
# oc get ds logging-fluentd -o yaml|grep -A 2 SYS
        - name: USE_REMOTE_SYSLOG
          value: "true"
        - name: REMOTE_SYSLOG_HOST
          value: $rsyslog_ip
        - name: REMOTE_SYSLOG_HOST_BACKUP
          value: $rsyslogback_ip
        - name: REMOTE_SYSLOG_PORT_BACKUP
          value: "514"

Comment 1 Noriko Hosoi 2018-05-08 04:43:40 UTC
(In reply to Qiaoling Tang from comment #0)
> Actual results:
> Docker event could be found in es pod, but couldn't be found in rsyslog
> server.

Please attach the following data.
1) a couple of example docker events indexed in the ElasticSearch,
2) output from oc logs <fluentd_pod>,
3) the remote syslog config file in the fluentd pod.
   /etc/fluent/configs.d/dynamic/output-remote-syslog.conf

Also, is ops enabled?  What is missing in the rsyslog server is just audit log?  Other logs are successfully forwarded?

Thanks.

Comment 5 Qiaoling Tang 2018-05-08 06:11:13 UTC
Hi Noriko, The ops wasn't enabled. The journal, docker log can be found in the rsyslog server.

Comment 6 Noriko Hosoi 2018-05-08 16:10:21 UTC
Thank you, @Qiaoling!

Looking at the config file [1], there is no tag_key directive here.  There is a known issue [2] found by @Anping, which is supposed to be fixed [3] in the next build.

Could you please rerun the test when the next build including the fix [3]?
Thanks!

[1] - https://bugzilla.redhat.com/attachment.cgi?id=1432991
[2] - https://bugzilla.redhat.com/show_bug.cgi?id=1515715#c18
[3] - https://github.com/openshift/origin-aggregated-logging/pull/1134

Comment 8 Qiaoling Tang 2018-06-20 08:35:46 UTC
No log could be found in rsyslog server.

Seems the changes are not in logging-fluentd/images/v3.10.1-1.

Comment 13 Qiaoling Tang 2018-06-25 03:37:41 UTC
Verified on logging-fluentd/images/v3.10.7-1.

Comment 15 errata-xmlrpc 2018-07-30 19:14:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1816