In principle I don't think the aggregated logging sub-system should ever exclude data unless explicitly directed by the administrator. The exclusion should be opt-in, with the default being to collect everything. Today we silently drop all DEBUG log level messages when collecting from the journal via fluentd. See [1]. Now that [2] has landed, this becomes a silent and significant change, as we never dropped any messages from /var/log/messages before hand because we could not tell when the log-level was. [1] https://github.com/openshift/origin-aggregated-logging/blob/4d4e175ac040eab477b641d4d0c978d1435e1364/fluentd/configs.d/openshift/filter-exclude-journal-debug.conf [2] https://github.com/openshift/origin-aggregated-logging/commit/829725d65b7139e79373b49d84c3333597287b3b
See also BZ 1502850 comment 16: https://bugzilla.redhat.com/show_bug.cgi?id=1502850#c16.
@Rich, Do we have an explicit reason you know of why we drop these records?
https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/configs.d/openshift/filter-retag-journal.conf#L41 "in addition, there are a few other differences about /var/log/messages and journal messages # * the default rsyslog.conf on el7 will only log messages to /var/log/messages at INFO level # or higher # * the default rsyslog.d includes 21-cloudinit.conf which redirects all cloud-init messages # to /var/log/cloud-init.log # When using the journal, we keep the cloud-init messages and store # them in elasticsearch. But we filter out DEBUG messages. # We should revisit this at a later date, when we have a good grasp of the storage # requirements, so we can turn on this firehose." This is now a later date, so we are revisiting. We need to get a good grasp of the storage and processing requirements to add DEBUG messages to logging.
Maybe We should address this with configuration allow users to opt-in DEBUG log messages at their own risk. I think we still need to understand the requirements for resources but at least they have the option.
koji_builds: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=636331 repositories: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:rhaos-3.9-rhel-7-docker-candidate-89109-20180103231254 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:latest brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.9.0 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.9.0-0.16.0.1 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.9.0 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.9.0.20180103.170133 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.9 koji_builds: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=636358 repositories: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:rhaos-3.8-rhel-7-docker-candidate-92791-20180103235159 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:latest brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.8.25 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.8.25-3 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.8.25.20180102.220346 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.8 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.8.25
The debug level log can be show in es on openshift3/logging-fluentd/images/v3.8.29-1.