Bug 1505602

Summary: Aggregated logging unconditionally excludes DEBUG log level messages ("priority 7") when collecting from journald
Product: OpenShift Container Platform Reporter: Peter Portante <pportant>
Component: LoggingAssignee: Lukas Vlcek <lvlcek>
Status: CLOSED CURRENTRELEASE QA Contact: Anping Li <anli>
Severity: high Docs Contact:
Priority: high    
Version: 3.7.0CC: aos-bugs, jcantril, lvlcek, rmeggins
Target Milestone: ---   
Target Release: 3.8.0   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: openshift3/logging-fluentd:v3.8.25-3 Doc Type: No Doc Update
Doc Text:
undefined
 Story Points: ---
Clone Of: Environment:
Last Closed: 2019-11-21 18:37:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Peter Portante 2017-10-24 00:58:13 UTC 
In principle I don't think the aggregated logging sub-system should ever exclude data unless explicitly directed by the administrator.  The exclusion should be opt-in, with the default being to collect everything.

Today we silently drop all DEBUG log level messages when collecting from the journal via fluentd.  See [1].

Now that [2] has landed, this becomes a silent and significant change, as we never dropped any messages from /var/log/messages before hand because we could not tell when the log-level was.

[1] https://github.com/openshift/origin-aggregated-logging/blob/4d4e175ac040eab477b641d4d0c978d1435e1364/fluentd/configs.d/openshift/filter-exclude-journal-debug.conf
[2] https://github.com/openshift/origin-aggregated-logging/commit/829725d65b7139e79373b49d84c3333597287b3b
Comment 1 Peter Portante 2017-10-24 01:03:26 UTC 
See also BZ 1502850 comment 16: https://bugzilla.redhat.com/show_bug.cgi?id=1502850#c16.
Comment 2 Jeff Cantrill 2017-10-24 17:35:30 UTC 
@Rich,  Do we have an explicit reason you know of why we drop these records?
Comment 3 Rich Megginson 2017-10-24 17:38:30 UTC 
https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/configs.d/openshift/filter-retag-journal.conf#L41

"in addition, there are a few other differences about /var/log/messages and journal messages
# * the default rsyslog.conf on el7 will only log messages to /var/log/messages at INFO level
#   or higher
# * the default rsyslog.d includes 21-cloudinit.conf which redirects all cloud-init messages
#   to /var/log/cloud-init.log
#   When using the journal, we keep the cloud-init messages and store
#   them in elasticsearch.  But we filter out DEBUG messages.
#   We should revisit this at a later date, when we have a good grasp of the storage
# requirements, so we can turn on this firehose."

This is now a later date, so we are revisiting.  We need to get a good grasp of the storage and processing requirements to add DEBUG messages to logging.
Comment 4 Jeff Cantrill 2017-12-01 14:53:45 UTC 
Maybe We should address this with configuration allow users to opt-in DEBUG log messages at their own risk.  I think we still need to understand the requirements for resources but at least they have the option.
Comment 5 Rich Megginson 2018-01-04 03:02:10 UTC 
koji_builds:
  https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=636331
repositories:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:rhaos-3.9-rhel-7-docker-candidate-89109-20180103231254
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:latest
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.9.0
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.9.0-0.16.0.1
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.9.0
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.9.0.20180103.170133
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.9

koji_builds:
  https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=636358
repositories:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:rhaos-3.8-rhel-7-docker-candidate-92791-20180103235159
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:latest
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.8.25
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.8.25-3
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.8.25.20180102.220346
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.8
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.8.25
Comment 6 Anping Li 2018-01-10 13:39:24 UTC 
The debug level log can be show in es on openshift3/logging-fluentd/images/v3.8.29-1.