Description of problem: While sending specific namespace logs to an external aggregator as well as all application namespace logs to internal OCP elastic search(default). Internal ES could see only the specific namespace(test) logs. Version-Release number of selected component (if applicable): 4.6 How reproducible: Steps to Reproduce: 1. Configure Cluster log forwarder with any external aggregator and also include the internal elastic search. ~~~ apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: instance namespace: openshift-logging spec: inputs: - application: namespaces: - test name: test-app-logs outputs: - name: rsyslog type: syslog syslog: facility: local0 rfc: RFC3164 payloadKey: message severity: informational url: 'tcp://<syslog-ip>:514' pipelines: - inputRefs: - application - audit - infrastructure name: application-audit-infrastructure-logs outputRefs: - default - inputRefs: - test-app-logs name: forward-to-syslog outputRefs: - rsyslog ~~~ 2. 3. Actual results: Internal Elastic search can only see the specific namespace Expected results: All the application namespace log should available internal elastic search while sending specific namespace to external Additional info:
Could this situation have something to do with the way retention is configured according to https://docs.openshift.com/container-platform/4.6/logging/config/cluster-logging-log-store.html#cluster-logging-elasticsearch-retention_cluster-logging-store If you do not specify a retention policy for all three log sources, only logs from the sources with a retention policy are stored. For example, if you set a retention policy for the infrastructure and applicaiton logs, but do not set a retention policy for audit logs, the audit logs will not be retained and there will be no audit- index in Elasticsearch or Kibana.
This issue is actually combination of the following: * https://bugzilla.redhat.com/show_bug.cgi?id=1925361 * https://bugzilla.redhat.com/show_bug.cgi?id=1921263
*** Bug 1900804 has been marked as a duplicate of this bug. ***
Moved to modified as https://github.com/openshift/cluster-logging-operator/pull/977 merged to resolve https://bugzilla.redhat.com/show_bug.cgi?id=1925361
The pod logs in test namespace can not sent to default ES. The other pods logs can be sent to default ES. apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: instance namespace: openshift-logging spec: inputs: - application: namespaces: - test name: test-app-logs outputs: - name: rsyslog type: syslog syslog: facility: local0 severity: informational url: 'udp://rsyslogserver.openshift-logging.svc:514' pipelines: - inputRefs: - application - audit - infrastructure name: application-audit-infrastructure-logs outputRefs: - default - inputRefs: - test-app-logs name: forward-to-syslog outputRefs: - rsyslog
Created attachment 1776873 [details] fluent.conf from master
Created attachment 1776874 [details] fluent.conf from 4.6 gen
@anli, Based on using our config generators for 4.6 and master, the configurations are almost identical. I have attached those generated conf to this issue. Your 4.6 fluentd config should be similar to #c11. Please reconfirm your testing as it was confirmed as verified in both of: * https://issues.redhat.com/browse/LOG-1224 * https://issues.redhat.com/browse/LOG-1222
Yes, that is confused, The fluentd.conf looks good. I will close this one as it do fix the issue Description. And file a new bug if I can find provide more detail.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6.28 extras update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:1489