Bug 1519619

Summary: Elasticsearch configmap is rebuilt removing any customizations made by an admin
Product: OpenShift Container Platform Reporter: Peter Portante <pportant>
Component: LoggingAssignee: ewolinet
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: high Docs Contact:
Priority: high    
Version: 3.8.0CC: anli, aos-bugs, jcantril, rmeggins
Target Milestone: ---Keywords: OpsBlocker
Target Release: 3.9.0   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Feature: When redeploying logging, we will attempt to maintain any changes that were made to the configmaps post installation. Reason: It was difficult to let users specify the contents of a configmap file while still needing to be able to provide configurations required for the different EFK components. Result: We create a patch based on changes made post deployment and apply that to the files provided by the installer.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2018-03-28 14:14:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Configure map prior and post redeploy none

Description Peter Portante 2017-12-01 03:33:59 UTC 
The ES configmap loses changes made to it via oc edit.  Such changes allow an admin to apply various tunings to Elasticsearch's behavior.  Those tunings are lost when the openshift-ansible logging playbook is applied.

We should consider an approach like the following:

1. Consider both the logging.yml and elasticsearch.yml configmap map entries as optional

  If they exist, use them as the basis for what the product builds into the
  final copy given to the ES container.

  If they don't exist, use the logging.yml and elasticsearch.yml maintained in
  the ES image.

2. For each supported DC ENV variable that maps to a field in the logging.yml or elasticsearch.yml file, modify those files to contain the ENV variable value

3. Provide the modified version of logging.yml and elasticsearch.yml to the ES java invocation in the container
Comment 1 Rich Megginson 2017-12-01 03:35:03 UTC 
I believe Eric is working on this or similar.
Comment 2 ewolinet 2017-12-01 14:18:48 UTC 
Functionality in https://github.com/openshift/openshift-ansible/pull/5894 should allow the patching of logging configmaps to preserve customizations as much as possible
Comment 3 Peter Portante 2017-12-01 14:23:11 UTC 
Will it behave as described in this BZ?  If not, can we say why should not implement the suggested course of action?
Comment 4 ewolinet 2017-12-01 15:39:04 UTC 
It will use the content in the currently deployed configmaps and build a patch based on what we provide in the deployer. We then patch in any customizations made on the deployed configmap onto what we are trying to deploy so we can pick up any required changes/additions that we may need to make while maintaining any changes a customer may have made post installation.

We also whitelist certain entries within the configmap to ensure we don't lose that value if it differs from the role default if it is not provided as an inventory entry.
Comment 6 Anping Li 2018-01-23 02:46:58 UTC 
Created attachment 1384657 [details]
Configure map prior and post redeploy

Some configure map values aren't preserved.

openshift-ansible:v3.9.0-0.22

For example:
The secure-forward.conf

Recreate step:
1. deploy logging and save the logging-fluentd configmap.
2. Modify logging-fluentd configmap to support secure-forward
3. redeploy logging again and save the logging-fluentd configmap.
4. diff the logging-fluentd configmap
Comment 8 Anping Li 2018-01-24 05:04:20 UTC 
Which version of openshift-ansible was you using? I am using openshift-ansible-3.9.0-0.23.0.git.0.d53d7ed.el7.noarch.

Below is the inventory variable I used.

openshift_logging_install_logging=true
openshift_logging_es_cluster_size=1
openshift_logging_es_nodeselector={"logging-node":"es"}
openshift_logging_es_memory_limit=2Gi

openshift_logging_namespace=openshift-logging
openshift_logging_image_prefix=regiustry.example.com/openshift3/
openshift_logging_image_version=v3.9



I patched the configmap by the following scripts.

oc patch configmap logging-fluentd -p'{"data": {"secure-forward.conf": "\u003cstore\u003e\n@type secure_forward\nshared_key sharedkey\nself_hostname ${HOSTNAME}\nca_cert_path /etc/fluentd/forward/cert\nca_private_key_passphrase passphrase\nenable_strict_verification yes\nsecure yes\n\n\u003cserver\u003e\nhost 192.168.1.221\nport 24284\n\u003c/server\u003e\n\n\n\u003c/store\u003e\n"'}}
Comment 9 ewolinet 2018-01-24 14:46:39 UTC 
I was able to see it correctly keep the contents with using your oc patch command, so I still am unable to recreate this locally.

I am using openshift-ansible-3.9.0-0.23.0
Comment 11 Anping Li 2018-02-01 12:14:24 UTC 
The configmap are kept. No sure what is wrong in my prior test. Maybe the openshift-ansible version is not the right version.

Verified using both ose-ansible/images/v3.9.0-0.23.0.0 and ose-ansible:v3.9.0-0.35.0.0
Comment 14 errata-xmlrpc 2018-03-28 14:14:24 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489