Bug 1905615
| Summary: | can't forward all application log to internal(OCP ES) while forwarding logs from specific projects to external aggregator | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Mani <mmohan> | ||||||
| Component: | Logging | Assignee: | Jeff Cantrill <jcantril> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Anping Li <anli> | ||||||
| Severity: | medium | Docs Contact: | |||||||
| Priority: | medium | ||||||||
| Version: | 4.6 | CC: | achakrat, anli, aos-bugs, benjamin.merot, periklis, qitang, stwalter | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | 4.6.z | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | logging-core | ||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: |
Cause: The cluster-logging-operator generated the collector configuration incorrectly
Consequence: Log messages were not correctly routing the the desired destinations
Fix: Correct the config generator
Result: Logs from specific namespaces can be selectively targeted to desired outputs using the cluster log forwarding resource
|
Story Points: | --- | ||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2021-05-12 12:16:04 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | 1921263, 1925361, 1928949 | ||||||||
| Bug Blocks: | |||||||||
| Attachments: |
|
||||||||
Could this situation have something to do with the way retention is configured according to https://docs.openshift.com/container-platform/4.6/logging/config/cluster-logging-log-store.html#cluster-logging-elasticsearch-retention_cluster-logging-store If you do not specify a retention policy for all three log sources, only logs from the sources with a retention policy are stored. For example, if you set a retention policy for the infrastructure and applicaiton logs, but do not set a retention policy for audit logs, the audit logs will not be retained and there will be no audit- index in Elasticsearch or Kibana. This issue is actually combination of the following: * https://bugzilla.redhat.com/show_bug.cgi?id=1925361 * https://bugzilla.redhat.com/show_bug.cgi?id=1921263 *** Bug 1900804 has been marked as a duplicate of this bug. *** Moved to modified as https://github.com/openshift/cluster-logging-operator/pull/977 merged to resolve https://bugzilla.redhat.com/show_bug.cgi?id=1925361 The pod logs in test namespace can not sent to default ES. The other pods logs can be sent to default ES.
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
name: instance
namespace: openshift-logging
spec:
inputs:
- application:
namespaces:
- test
name: test-app-logs
outputs:
- name: rsyslog
type: syslog
syslog:
facility: local0
severity: informational
url: 'udp://rsyslogserver.openshift-logging.svc:514'
pipelines:
- inputRefs:
- application
- audit
- infrastructure
name: application-audit-infrastructure-logs
outputRefs:
- default
- inputRefs:
- test-app-logs
name: forward-to-syslog
outputRefs:
- rsyslog
Created attachment 1776873 [details]
fluent.conf from master
Created attachment 1776874 [details]
fluent.conf from 4.6 gen
@anli, Based on using our config generators for 4.6 and master, the configurations are almost identical. I have attached those generated conf to this issue. Your 4.6 fluentd config should be similar to #c11. Please reconfirm your testing as it was confirmed as verified in both of: * https://issues.redhat.com/browse/LOG-1224 * https://issues.redhat.com/browse/LOG-1222 Yes, that is confused, The fluentd.conf looks good. I will close this one as it do fix the issue Description. And file a new bug if I can find provide more detail. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6.28 extras update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:1489 |
Description of problem: While sending specific namespace logs to an external aggregator as well as all application namespace logs to internal OCP elastic search(default). Internal ES could see only the specific namespace(test) logs. Version-Release number of selected component (if applicable): 4.6 How reproducible: Steps to Reproduce: 1. Configure Cluster log forwarder with any external aggregator and also include the internal elastic search. ~~~ apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: instance namespace: openshift-logging spec: inputs: - application: namespaces: - test name: test-app-logs outputs: - name: rsyslog type: syslog syslog: facility: local0 rfc: RFC3164 payloadKey: message severity: informational url: 'tcp://<syslog-ip>:514' pipelines: - inputRefs: - application - audit - infrastructure name: application-audit-infrastructure-logs outputRefs: - default - inputRefs: - test-app-logs name: forward-to-syslog outputRefs: - rsyslog ~~~ 2. 3. Actual results: Internal Elastic search can only see the specific namespace Expected results: All the application namespace log should available internal elastic search while sending specific namespace to external Additional info: