Bug 1504583

Summary: Fluent failed to gather docker event logs
Product: OpenShift Container Platform Reporter: Anping Li <anli>
Component: LoggingAssignee: Josef Karasek <jkarasek>
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: high Docs Contact:
Priority: high    
Version: 3.7.0CC: aos-bugs, jcantril, rmeggins, wsun
Target Milestone: ---   
Target Release: 3.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
This was an internal only bug found by our QE. It was never released to any customer. Fluentd plugin that reads the audit.log file enriched log lines with a value sourced from the environment. This value was inherently immutable, which caused a conversion problem during further processing.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-28 22:18:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
The fluent logs none

Description Anping Li 2017-10-20 09:28:13 UTC 
Created attachment 1341178 [details]
The fluent logs

Description of problem:
The flunet report error when gather docker event logs.

Version-Release number of selected component (if applicable):
A fluent images build from origin-aggregated-logging:master

How reproducible:
Always

Steps to Reproduce:
1. Enable docker event
openshift_logging_fluentd_audit_container_engine=true
openshift_logging_fluentd_audit_file=/var/log/audit/audit.log
openshift_logging_fluentd_audit_pos_file=/var/log/audit/audit.log.pos

2. check the fluned logs after deployed

Actual results:
  2017-10-20 05:09:59 -0400 [warn]: suppressed same stacktrace
2017-10-20 05:10:01 -0400 [warn]: emit transaction failed: error_class=RuntimeError error="can't modify frozen String" tag="audit.log"
  2017-10-20 05:10:01 -0400 [warn]: suppressed same stacktrace
2017-10-20 05:10:01 -0400 [warn]: emit transaction failed: error_class=RuntimeError error="can't modify frozen String" tag="audit.log"
  2017-10-20 05:10:01 -0400 [warn]: suppressed same stacktrace
2017-10-20 05:10:06 -0400 [warn]: emit transaction failed: error_class=RuntimeError error="can't modify frozen String" tag="audit.log"
  2017-10-20 05:10:06 -0400 [warn]: suppressed same stacktrace
2017-10-20 05:10:06 -0400 [warn]: emit transaction failed: error_class=RuntimeError error="can't modify frozen String" tag="audit.log"
  2017-10-20 05:10:06 -0400 [warn]: suppressed same stacktrace
2017-10-20 05:10:09 -0400 [warn]: emit transaction failed: error_class=RuntimeError error="can't modify frozen String" tag="audit.log"
  2017-10-20 05:10:09 -0400 [warn]: suppressed same stacktrace
2017-10-20 05:10:09 -0400 [warn]: emit transaction failed: error_class=RuntimeError error="can't modify frozen String" tag="audit.log"
  2017-10-20 05:10:09 -0400 [warn]: suppressed same stacktrace
2017-10-20 05:10:11 -0400 [warn]: emit transaction failed: error_class=RuntimeError error="can't modify frozen String" tag="audit.log"
  2017-10-20 05:10:11 -0400 [warn]: suppressed same stacktrace
2017-10-20 05:10:11 -0400 [warn]: emit transaction failed: error_class=RuntimeError error="can't modify frozen String" tag="audit.log"
  2017-10-20 05:10:11 -0400 [warn]: suppressed same stacktrace


Expected results:
The docker event logs can be gathered and sent to ES.


Additional info:
Comment 1 Josef Karasek 2017-10-23 12:40:45 UTC 
This should be fixed by https://github.com/ViaQ/fluent-plugin-viaq_docker_audit_log_parser/commit/1f5fff420dd96f76068a081d4c48f2b4d712d02d
Comment 2 Rich Megginson 2017-10-25 01:39:45 UTC 
koji_builds:
  https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=612936
repositories:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:rhaos-3.7-rhel-7-docker-candidate-79931-20171025012405
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:latest
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.7.0
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.7.0-0.177.0.1
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.7
Comment 3 Anping Li 2017-10-30 08:36:18 UTC 
The bug have been fixed in logging-fluentd:v3.7.0-0.185.0.0
Comment 6 errata-xmlrpc 2017-11-28 22:18:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188