Bug 1771813
| Summary: | Fluentd doesn't use the `shared_key` in the user-managed secret when secure forward enabled. | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Qiaoling Tang <qitang> |
| Component: | Logging | Assignee: | Jeff Cantrill <jcantril> |
| Status: | CLOSED ERRATA | QA Contact: | Anping Li <anli> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.3.0 | CC: | aos-bugs, rmeggins |
| Target Milestone: | --- | ||
| Target Release: | 4.3.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-01-23 11:12:22 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Verified with ose-cluster-logging-operator-v4.3.0-201911191807 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0062 |
Description of problem: Deploy logging via OLM and enable secure forward, the user-managed secret has a key `shared_key`, but the fluentd pod doesn't use it, the shared_key in the fluentd pod is the output name of the secure forward. There have some error messages in the fluentd pod: 2019-11-13 02:13:57 +0000 [warn]: suppressed same stacktrace 2019-11-13 02:13:57 +0000 [warn]: recovered forwarding server '' host="fluentdserver.openshift-logging.svc" port=24224 2019-11-13 02:19:23 +0000 [warn]: connection refused to fluentdserver.openshift-logging.svc: authentication failed: shared_key mismatch 2019-11-13 02:19:23 +0000 [warn]: connection refused to fluentdserver.openshift-logging.svc: authentication failed: shared_key mismatch 2019-11-13 02:19:23 +0000 [warn]: failed to flush the buffer. retry_time=12 next_retry_seconds=2019-11-13 02:23:46 +0000 chunk="597301832331c43f606485cfae9c53b1" error_class=Fluent::Plugin::ForwardOutput::ConnectionClosedError error="failed to establish connection with node " $ oc rsh fluentd-wlr77 sh-4.2# grep -r "shared_key" /etc/fluent/fluent.conf shared_key fluentd-created-by-user sh-4.2# ls /run/ocp-collector/secrets/fluentdserver/ ca-bundle.crt ca.key shared_key tls.crt tls.key sh-4.2# cat /run/ocp-collector/secrets/fluentdserver/shared_key fluentdserversh-4.2# $ oc get logforwarding instance -oyaml apiVersion: logging.openshift.io/v1alpha1 kind: LogForwarding metadata: creationTimestamp: "2019-11-13T02:06:16Z" generation: 1 name: instance namespace: openshift-logging resourceVersion: "72238" selfLink: /apis/logging.openshift.io/v1alpha1/namespaces/openshift-logging/logforwardings/instance uid: 12b0ddde-190e-4808-b3d9-e8774732da24 spec: outputs: - endpoint: fluentdserver.openshift-logging.svc:24224 name: fluentd-created-by-user secret: name: fluentdserver type: forward pipelines: - inputType: logs.app name: clo-default-app-pipeline outputRefs: - fluentd-created-by-user - inputType: logs.infra name: clo-default-infra-pipeline outputRefs: - fluentd-created-by-user - inputType: logs.audit name: clo-default-audit-pipeline outputRefs: - fluentd-created-by-user Version-Release number of selected component (if applicable): ose-cluster-logging-operator-v4.3.0-201911081316 How reproducible: Always Steps to Reproduce: 1.deploy logging via OLM, enable logforwarding 2.create secret for logforwarding 3.deploy fluentd server 4.create logforwarding instance to use secure forward Actual results: The fluentd can't connect to the server due to authentication failed: shared_key mismatch Expected results: Additional info: workaround: change the shared_key in the fluentdserver to `fluentd-created-by-user` then restart fluentdserver and fluentd pods I've tried to update the shared_key in cm/fluentd, but the CLO overwrote it.