Bug 1921263

+++ This bug was initially created as a clone of Bug #1911477 +++

[Description of problem]
In previous version, for example 3.x and until 4.3. When using secure_forward following the documentation [1] the logs were sent to the internal Elasticsearch and to the external instance.

Now, in 4.5 it's only sending to the external instance and it doesn't send more to the internal Elasticsearch.

Verifying the documentation, it has not changed [2]. Then, it's expected that it works like it did in the past:

- Sending logs to the internal Elasticsearch
- Sending logs to the external instance configured in the secure-forward configmap

One thing has changed in the configuration generated for fluentd. In 4.3 the fluentd configuration after configuring secure_forward following the documentation is like this:

~~~
$ oc rsh <fluentd pod> cat /etc/fluent/fluent.conf
...
<label @_LOGS_APP>
        <match **>
        @type copy

                <store>
                @type relabel
                @label @CLO_DEFAULT_APP_PIPELINE
        </store>

                <store>
                @type relabel
                @label @_LEGACY_SECUREFORWARD
        </store>

</match>
<label @_LOGS_INFRA>
        <match **>
        @type copy

                <store>
                @type relabel
                @label @CLO_DEFAULT_INFRA_PIPELINE
        </store>

                <store>
                @type relabel
                @label @_LEGACY_SECUREFORWARD
        </store>

</match>
</label>

# Relabel specific pipelines to multiple, outputs (e.g. ES, kafka stores)

<label @CLO_DEFAULT_APP_PIPELINE>
        <match **>
        @type copy

                <store>
                @type relabel
                @label @CLO_DEFAULT_OUTPUT_ES
        </store>
</match>
</label>

<label @CLO_DEFAULT_INFRA_PIPELINE>
        <match **>
        @type copy

                <store>
                @type relabel
                @label @CLO_DEFAULT_OUTPUT_ES
        </store>
</match>
</label>
...
~~~

As we can see above, it's sending to the CLO_DEFAULT and to the LEGACY_SECUREFORWARD, but, the configuration in OCP 4.5 generated after configuring the secure forward is like this:

~~~
$ oc rsh <fluentd pod> cat /etc/fluent/fluent.conf
...
<label @_LOGS_APP>
  <match **>
    @type copy


    <store>
      @type relabel
      @label @_LEGACY_SECUREFORWARD
    </store>

  </match>
</label>
<label @_LOGS_AUDIT>
  <match **>
    @type copy


    <store>
      @type relabel
      @label @_LEGACY_SECUREFORWARD
    </store>

  </match>
</label>
<label @_LOGS_INFRA>
  <match **>
    @type copy


    <store>
      @type relabel
      @label @_LEGACY_SECUREFORWARD
    </store>

  </match>
</label>
...
~~~

As we can see, it's only relabeling like "_LEGACY_SECUREFORWARD", but it's not possible to see the relabeling to CLO_DEFAULT_XXX.


[Version-Release number of selected component (if applicable):]

Version used for OCP 4.5
~~~
$ oc version
Client Version: 4.5.23
Server Version: 4.5.23
$ oc get csv -n openshift-logging
NAME                                           DISPLAY                  VERSION                 REPLACES   PHASE
clusterlogging.4.5.0-202012120433.p0           Cluster Logging          4.5.0-202012120433.p0              Failed
elasticsearch-operator.4.5.0-202012120433.p0   Elasticsearch Operator   4.5.0-202012120433.p0              Succeeded
~~~

Version used for OCP 4.3:
~~~
$ oc version
Client Version: 4.3.38
Server Version: 4.3.40
Kubernetes Version: v1.16.2+853223d
$ oc get csv -n openshift-logging
NAME                                            DISPLAY                  VERSION                  REPLACES   PHASE
clusterlogging.4.3.40-202010141211.p0           Cluster Logging          4.3.40-202010141211.p0              Succeeded
elasticsearch-operator.4.3.40-202010141211.p0   Elasticsearch Operator   4.3.40-202010141211.p0              Succeeded
~~~

[How reproducible]
Always


Steps to Reproduce:
1. Install Cluster Logging
2. Configure secure_forward
3. Logs are not sent to the internal Elasticsearch

[Actual results]
Logs are not sent to the internal Elasticsearch

[Expected results]
Logs should be sent to the internal Elasticsearch at the same time that to the external instance configured in the secure-forward configmap


We are aware that this is deprecated, but in 4.3 the documentation is saying the same that in 4.4 and 4.5 and it was working in 4.3 and previous versions, the same that in 3.x. Then, it's expected that it continues working and the logs are sent in parallel to the internal Elasticsearch, to the external instance configured.


[1] https://docs.openshift.com/container-platform/4.3/logging/config/cluster-logging-external.html
[2] https://docs.openshift.com/container-platform/4.3/logging/config/cluster-logging-external.html#cluster-logging-collector-fluentd_cluster-logging-external