Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1856435

Summary: logging-fluentd ConfigMap is overwritten during logging stack upgrades
Product: OpenShift Container Platform Reporter: Oscar Casal Sanchez <ocasalsa>
Component: LoggingAssignee: ewolinet
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.11.0CC: aos-bugs, ewolinet, jmalde, mburke, rheinzma
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1875191 (view as bug list) Environment:
Last Closed: 2020-08-26 22:44:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1875191    

Description Oscar Casal Sanchez 2020-07-13 16:10:38 UTC
[Description of problem]
Secure forward setting in the logging-fluent ConfigMap are overwritten during EFK stack upgrades

[Version-Release number of selected component (if applicable)]
OCP 3.11.232

[How reproducible]
Always

[Steps to Reproduce]

1. Set up an OCP 3.11 cluster with the logging stack

~~~
...
openshift_pkg_version=-3.11.188
openshift_image_tag=v3.11.188
...
## Logging
openshift_logging_install_logging=true
openshift_logging_es_nodeselector={"node-role.kubernetes.io/infra":"true"}
openshift_logging_es_memory_limit=4G
openshift_logging_es_cpu_request=500m
...
~~~


2. Modify the logging-fluentd configmap for adding a custom configuration following the documentation here [1]

~~~
$ oc -n openshift-logging cm logging-fluentd
...
<store>
@type secure_forward
self_hostname pod-${HOSTNAME}
shared_key thisisasharedkey
secure yes
enable_strict_verification yes
ca_cert_path /etc/fluent/keys/your_ca_cert
ca_private_key_path /etc/fluent/keys/your_private_key
ca_private_key_passphrase passphrase
<server>
  host ose1.example.com
  port 24284
</server>
...
~~~

3. Upgrade the cluster following the documentation here [2]

~~~
## Edit inventory file
...
openshift_pkg_version=-3.11.188
openshift_image_tag=v3.11.188
...
~~~


4. Upgrade the logging stack

~~~
$ ansible-playbook -i <inventory file> `playbooks/openshift-logging/config.yml`
~~~

[Actual results]
The logging-fluentd cm is overwritten with the default configmap and the custom secure_forward configuration was deleted . 

[Expected results]
The custom configuration for the logging-fluentd cm is maintained


[Additional info]
This issue is the same that was fixed for the curator configmap in the BZ#1841439 and the ansible code existent now in the logging-fluentd configmap is very similar to the code that was changed for the curator.

PR for curator configmap [4]. 

It would be great to have for the logging-fluentd cm the same option available for curator:

~~~
 openshift_logging_curator_replace_configmap: no
~~~

It could be something like:


~~~
 openshift_logging_fluentd_replace_configmap: no
~~~


[1] https://docs.openshift.com/container-platform/3.11/install_config/aggregate_logging.html
[2] https://docs.openshift.com/container-platform/3.11/upgrading/automated_upgrades.html#upgrading-efk-logging-stack
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1841439
[4] https://github.com/openshift/openshift-ansible/pull/12179/commits/f0ad6b6792ffd158187e694de95f872bd626320b

Comment 1 Oscar Casal Sanchez 2020-07-13 16:15:52 UTC
Hello,

This bug was tested and verified for the next upgrading paths:

OCP 3.11.188 -> OCP 3.11.200 -> OCP 3.11.232 and it's always reproducible when you are doing an upgrade.

Comment 4 Anping Li 2020-08-04 11:49:28 UTC
Verified on openshift-ansible-3.11.256

Comment 5 Anping Li 2020-08-04 12:21:42 UTC
Verified on openshift-ansible-3.11.252 too

Comment 7 errata-xmlrpc 2020-08-26 22:44:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 3.11.272 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3245

Comment 8 Oscar Casal Sanchez 2020-08-28 13:41:50 UTC
Hello,

Thank you so much for giving us the fix for this issue, but I'm not able to find the variable openshift_logging_fluentd_replace_configmap reflected in the documentation, am I missing it or was it not added to the documentation? If this latest is true, could you add this in the documentation?

Regards,
Oscar

Comment 9 Michael Burke 2020-09-03 01:06:19 UTC
Adding the parameter to 3.11 docs in https://bugzilla.redhat.com/show_bug.cgi?id=1875191