Created attachment 1269815[details]
ElasticSearch Trace provided by customer
+++ This bug was initially created as a clone of Bug #1439554 +++
Description of problem:
When upgrading from 3.4.0 to 3.4.1-17 the following error is shown and the deployment fails.
Could not resolve placeholder 'MIN_MASTERS'
Version-Release number of selected component (if applicable):
Openshift 3.4.1.10
openshift3-logging-elasticsearch-3.4.1-17
How reproducible:
Always
Steps to Reproduce:
1. oc new-app logging-deployer-template --param IMAGE_VERSION=3.4.1 --param MODE=upgrade
Actual results:
The metrics fail to deploy because the ENV variable MIN_MASTERS is not present in the deployment config for logging-es-xxxx
Expected results:
The metrics are upgraded to the latest version and the deployment config includes the MIN_MASTERS environment variable defined.
Additional info:
By manually adding the variable, the deployment succeeds.
The most reliable workaround I found was to reinstall the metrics. I.E. MODE=reinstall
--- Additional comment from Jeff Cantrill on 2017-04-07 08:15:12 EDT ---
The work around to resolve this issue is:
Determine the number of min masters by:
min_masters = floor("# ES cluster nodes" / 2) + 1
1. oc edit dc $ES_DC_NAME
2. add env entry to container with name: 'MIN_MASTERS' value: $min_master
3. oc rollout latest $DC_NAME
Apply change to each deploymentconfig.
Fix for the deployer is forthcoming.
--- Additional comment from Boris Kurktchiev on 2017-04-07 08:59:54 EDT ---
So just tested this and it worked... sorta. Now there is a new error but that may have to go in its own bug?
> SEE ATTACHMENT
Also I want to point out that ES_JAVA_OPTS: '-Dmapper.allow_dots_in_name=true -Xms128M -Xmx2048m' even though INSTANCE_RAM=4GB in the DC
--- Additional comment from Jeff Cantrill on 2017-04-07 11:01:10 EDT ---
The issue is the migrate script was not updated to account for the min master change. Subsequently you see the reported issue. The resolution is to remove the 'zen.minimum_master_nodes' setting from the logging-elasticsearch configmap or edit each DC as described in #8.