Bug 1925361 - [4.6] ClusterLogForwarder namespace-specific log forwarding does not work as expected [NEEDINFO]
Summary: [4.6] ClusterLogForwarder namespace-specific log forwarding does not work as ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.6.z
Assignee: Jeff Cantrill
QA Contact: Anping Li
URL:
Whiteboard: logging-core
: 1953646 (view as bug list)
Depends On:
Blocks: 1905615
TreeView+ depends on / blocked
 
Reported: 2021-02-05 02:16 UTC by Devendra Kulkarni
Modified: 2021-04-27 13:16 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-04-27 08:58:26 UTC
Target Upstream Version:
aconway: needinfo? (naygupta)


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-logging-operator pull 977 0 None open Bug 1925361: fix routing of app logs per namespace 2021-04-10 00:19:03 UTC
Red Hat Knowledge Base (Solution) 5778201 0 None None None 2021-02-25 02:47:52 UTC

Description Devendra Kulkarni 2021-02-05 02:16:43 UTC
Description of problem: 

As per documentation[1], we can configure ClusterLogForwarder to send project-specific logs to external/internal log aggregators.

[1] https://docs.openshift.com/container-platform/4.6/logging/cluster-logging-external.html#cluster-logging-collector-log-forwarding-about_cluster-logging-external

If only single project logs need to be forwarded, the configuration works fine.
But if we need to forward multiple project logs to different destinations, the fluentd configuration is messed up.

For example, forwarding logs from two different projects to two different external elasticsearch instances, the logs from both the projects are forwarded to both the elasticsearch instances, due to the fluentd configuration getting messed up as below:

~~~
cat /etc/fluentd/fluent.conf

-- OUTOUT_EMITTED---

  # A log source matcher may be null if no pipeline wants that type of log.
  <match **_default_** **_kube-*_** **_openshift-*_** **_openshift_** journal.** system.var.log**>
    @type null
  </match>
  <match kubernetes.**_dev-apple_** kubernetes.**_dev-ocp_** >       
    @type relabel
    @label @_APPLICATION                                         <<<<< Here combined logs are labelled as APPLICATION logs
  </match>
  <match kubernetes.** >
    @type null
  </match>
  <match linux-audit.log** k8s-audit.log** openshift-audit.log**>
    @type null
  </match>

  <match **>
    @type stdout
  </match>

</label>

# Relabel specific sources (e.g. logs.apps) to multiple pipelines      <<< Then we have this section, I believe this section should be merged/corrected with above section
<label @_APPLICATION> 
  <match **>
    @type copy

    <store>
      @type relabel
      @label @LOGS_NAMESPACES_APPLE
    </store>
    <store>
      @type relabel
      @label @LOGS_NAMESPACES_OCP
    </store>

--OUTPUT_EMITTED--
~~~


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Deploy an OCP 4.6 cluster and install cluster-logging stack on it.
2. Configure clusterlogforwarder to forward logs from two projects to different elasticsearch instance.
3. Check the logs on external elasticsearch instances, it will contain logs from both the projects

Actual results:

Logs from both the project are available on both elasticsearch instances

Expected results:

Logs from one project to go to the first external elasticsearch instance and the second projects logs to the second elasticsearch instance.


Additional info:

[1] Example CLF configuration:

~~~
apiVersion: "logging.openshift.io/v1"
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
   - name: elasticsearch-ocp
     type: "elasticsearch"
     url: http://<FQDN-ES-1>:9200
   - name: elasticsearch-apple
     type: "elasticsearch"
     url: http://<FQDN-ES-2>:9200
  inputs:
   - name: input-namespaces-ocp
     application:
        namespaces:
        - dev-ocp
   - name: input-namespaces-apple
     application:
        namespaces:
        - dev-apple
  pipelines:
   - name: logs-namespaces-ocp
     inputRefs:
      - input-namespaces-ocp
     outputRefs:
      - elasticsearch-ocp
     labels:
       datacenter: lab-ocpcart-01
   - name: logs-namespaces-apple
     inputRefs:
      - input-namespaces-apple
     outputRefs:
      - elasticsearch-apple
     labels:
       datacenter: lab-ocpcart-01
~~~

Comment 2 Alan Conway 2021-02-12 22:46:08 UTC
Note: before fixing this, check the progress of https://github.com/openshift/cluster-logging-operator/pull/865
That PR can fix this bug as a side-effect.

Comment 4 Alan Conway 2021-03-23 15:26:52 UTC
The fix is actively being worked on now.

Comment 6 Alan Conway 2021-03-23 18:09:43 UTC
Here's the pull request on GitHub, I think you should be able to see it: https://github.com/openshift/cluster-logging-operator/pull/955

Comment 10 Jeff Cantrill 2021-04-16 18:13:40 UTC
Moved to modified as PR merged

Comment 13 Anping Li 2021-04-20 12:36:15 UTC
Verified on clusterlogging.4.6.0-202104161407.p0

Comment 15 errata-xmlrpc 2021-04-27 08:58:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.6.26 security and extras update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1230

Comment 16 Jeff Cantrill 2021-04-27 13:16:08 UTC
*** Bug 1953646 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.