Bug 1460564

Summary: Change the Elasticsearch setting "node.max_local_storage_nodes" to 1 to prevent sharing EBS volumes
Product: OpenShift Container Platform Reporter: Peter Portante <pportant>
Component: LoggingAssignee: Jeff Cantrill <jcantril>
Status: CLOSED ERRATA QA Contact: Xia Zhao <xiazhao>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.6.0CC: aos-bugs, juzhao, pweil, rmeggins, rromerom
Target Milestone: ---   
Target Release: 3.7.0   
Hardware: All   
OS: All   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Elasticsearch default value for sharing storage between ES instances was wrong Consequence: The incorrect default value allowed an ES pod starting up (when another ES pod was shutting down, e.g. during dc redeployments) to create a new location on the PV for managing the storage volume, duplicating data, and in some instances, potentially causing data loss. Fix: All ES pods now run with "node.max_local_storage_nodes" set to 1. Result: The ES pods starting up/shutting down will no longer share the same storage and prevent the data duplication and/or data loss.
Story Points: ---
Clone Of:
: 1462277 (view as bug list) Environment:
Last Closed: 2017-11-28 21:56:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1462277, 1462281, 1463046    

Description Peter Portante 2017-06-12 02:38:46 UTC
Change the setting for node.max_local_storage_nodes to 1 for all ES pods, as this would prevent us from seeing problems where two ES pods end up sharing the same EBS volume if one pod does not shut down properly.

For an example of this, see https://bugzilla.redhat.com/show_bug.cgi?id=1443350#c33

See discussion from https://discuss.elastic.co/t/multiple-folders-inside-nodes-folder/85358, and the documentation at https://www.elastic.co/guide/en/elasticsearch/reference/2.4/modules-node.html#max-local-storage-nodes.

Comment 1 openshift-github-bot 2017-06-16 22:05:45 UTC
Commit pushed to master at https://github.com/openshift/openshift-ansible

Ensure only one ES pod per PV

bug 1460564. Fixes [BZ #1460564](https://bugzilla.redhat.com/show_bug.cgi?id=1460564).

Unfortunately, the defaults for Elasticsearch prior to v5 allow more
than one "node" to access the same configured storage volume(s).

This change forces this value to 1 to ensure we don't have an ES pod
starting up accessing a volume while another ES pod is shutting down
when reploying. This can lead to "1" directories being created in
By default ES uses a "0" directory there when only one node is accessing

Comment 3 Junqi Zhao 2017-06-26 03:05:00 UTC
max_local_storage_nodes is 1 now
# oc get configmap logging-elasticsearch -n logging -o yaml | grep -i max_local_storage_nodes
      max_local_storage_nodes: 1

Testing env:
# openshift version
openshift v3.6.122
kubernetes v1.6.1+5115d708d7
etcd 3.2.0

Images from brew registry
# docker images | grep logging
logging-kibana          v3.6                fd67e351dadf        2 days ago          342.4 MB
logging-elasticsearch   v3.6                1006eb106849        2 days ago          404.6 MB
logging-auth-proxy      v3.6                301fd39f57e0        2 days ago          229.6 MB
logging-fluentd         v3.6                dba31f5b54ba        2 days ago          232.5 MB
logging-curator         v3.6                a0148dd96b8d        2 weeks ago         221.5 MB

Comment 7 errata-xmlrpc 2017-11-28 21:56:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.