Description of problem: As per documentation[1], we can configure ClusterLogForwarder to send project-specific logs to external/internal log aggregators. [1] https://docs.openshift.com/container-platform/4.6/logging/cluster-logging-external.html#cluster-logging-collector-log-forwarding-about_cluster-logging-external If only single project logs need to be forwarded, the configuration works fine. But if we need to forward multiple project logs to different destinations, the fluentd configuration is messed up. For example, forwarding logs from two different projects to two different external elasticsearch instances, the logs from both the projects are forwarded to both the elasticsearch instances, due to the fluentd configuration getting messed up as below: ~~~ cat /etc/fluentd/fluent.conf -- OUTOUT_EMITTED--- # A log source matcher may be null if no pipeline wants that type of log. <match **_default_** **_kube-*_** **_openshift-*_** **_openshift_** journal.** system.var.log**> @type null </match> <match kubernetes.**_dev-apple_** kubernetes.**_dev-ocp_** > @type relabel @label @_APPLICATION <<<<< Here combined logs are labelled as APPLICATION logs </match> <match kubernetes.** > @type null </match> <match linux-audit.log** k8s-audit.log** openshift-audit.log**> @type null </match> <match **> @type stdout </match> </label> # Relabel specific sources (e.g. logs.apps) to multiple pipelines <<< Then we have this section, I believe this section should be merged/corrected with above section <label @_APPLICATION> <match **> @type copy <store> @type relabel @label @LOGS_NAMESPACES_APPLE </store> <store> @type relabel @label @LOGS_NAMESPACES_OCP </store> --OUTPUT_EMITTED-- ~~~ Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Deploy an OCP 4.6 cluster and install cluster-logging stack on it. 2. Configure clusterlogforwarder to forward logs from two projects to different elasticsearch instance. 3. Check the logs on external elasticsearch instances, it will contain logs from both the projects Actual results: Logs from both the project are available on both elasticsearch instances Expected results: Logs from one project to go to the first external elasticsearch instance and the second projects logs to the second elasticsearch instance. Additional info: [1] Example CLF configuration: ~~~ apiVersion: "logging.openshift.io/v1" kind: ClusterLogForwarder metadata: name: instance namespace: openshift-logging spec: outputs: - name: elasticsearch-ocp type: "elasticsearch" url: http://<FQDN-ES-1>:9200 - name: elasticsearch-apple type: "elasticsearch" url: http://<FQDN-ES-2>:9200 inputs: - name: input-namespaces-ocp application: namespaces: - dev-ocp - name: input-namespaces-apple application: namespaces: - dev-apple pipelines: - name: logs-namespaces-ocp inputRefs: - input-namespaces-ocp outputRefs: - elasticsearch-ocp labels: datacenter: lab-ocpcart-01 - name: logs-namespaces-apple inputRefs: - input-namespaces-apple outputRefs: - elasticsearch-apple labels: datacenter: lab-ocpcart-01 ~~~
Note: before fixing this, check the progress of https://github.com/openshift/cluster-logging-operator/pull/865 That PR can fix this bug as a side-effect.
The fix is actively being worked on now.
Here's the pull request on GitHub, I think you should be able to see it: https://github.com/openshift/cluster-logging-operator/pull/955
Moved to modified as PR merged
Verified on clusterlogging.4.6.0-202104161407.p0
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.6.26 security and extras update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:1230
*** Bug 1953646 has been marked as a duplicate of this bug. ***
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days