Bug 1925361

Description of problem: 

As per documentation[1], we can configure ClusterLogForwarder to send project-specific logs to external/internal log aggregators.

[1] https://docs.openshift.com/container-platform/4.6/logging/cluster-logging-external.html#cluster-logging-collector-log-forwarding-about_cluster-logging-external

If only single project logs need to be forwarded, the configuration works fine.
But if we need to forward multiple project logs to different destinations, the fluentd configuration is messed up.

For example, forwarding logs from two different projects to two different external elasticsearch instances, the logs from both the projects are forwarded to both the elasticsearch instances, due to the fluentd configuration getting messed up as below:

~~~
cat /etc/fluentd/fluent.conf

-- OUTOUT_EMITTED---

  # A log source matcher may be null if no pipeline wants that type of log.
  <match **_default_** **_kube-*_** **_openshift-*_** **_openshift_** journal.** system.var.log**>
    @type null
  </match>
  <match kubernetes.**_dev-apple_** kubernetes.**_dev-ocp_** >       
    @type relabel
    @label @_APPLICATION                                         <<<<< Here combined logs are labelled as APPLICATION logs
  </match>
  <match kubernetes.** >
    @type null
  </match>
  <match linux-audit.log** k8s-audit.log** openshift-audit.log**>
    @type null
  </match>

  <match **>
    @type stdout
  </match>

</label>

# Relabel specific sources (e.g. logs.apps) to multiple pipelines      <<< Then we have this section, I believe this section should be merged/corrected with above section
<label @_APPLICATION> 
  <match **>
    @type copy

    <store>
      @type relabel
      @label @LOGS_NAMESPACES_APPLE
    </store>
    <store>
      @type relabel
      @label @LOGS_NAMESPACES_OCP
    </store>

--OUTPUT_EMITTED--
~~~


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Deploy an OCP 4.6 cluster and install cluster-logging stack on it.
2. Configure clusterlogforwarder to forward logs from two projects to different elasticsearch instance.
3. Check the logs on external elasticsearch instances, it will contain logs from both the projects

Actual results:

Logs from both the project are available on both elasticsearch instances

Expected results:

Logs from one project to go to the first external elasticsearch instance and the second projects logs to the second elasticsearch instance.


Additional info:

[1] Example CLF configuration:

~~~
apiVersion: "logging.openshift.io/v1"
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
   - name: elasticsearch-ocp
     type: "elasticsearch"
     url: http://<FQDN-ES-1>:9200
   - name: elasticsearch-apple
     type: "elasticsearch"
     url: http://<FQDN-ES-2>:9200
  inputs:
   - name: input-namespaces-ocp
     application:
        namespaces:
        - dev-ocp
   - name: input-namespaces-apple
     application:
        namespaces:
        - dev-apple
  pipelines:
   - name: logs-namespaces-ocp
     inputRefs:
      - input-namespaces-ocp
     outputRefs:
      - elasticsearch-ocp
     labels:
       datacenter: lab-ocpcart-01
   - name: logs-namespaces-apple
     inputRefs:
      - input-namespaces-apple
     outputRefs:
      - elasticsearch-apple
     labels:
       datacenter: lab-ocpcart-01
~~~