Change the setting for node.max_local_storage_nodes to 1 for all ES pods, as this would prevent us from seeing problems where two ES pods end up sharing the same EBS volume if one pod does not shut down properly. For an example of this, see https://bugzilla.redhat.com/show_bug.cgi?id=1443350#c33 See discussion from https://discuss.elastic.co/t/multiple-folders-inside-nodes-folder/85358, and the documentation at https://www.elastic.co/guide/en/elasticsearch/reference/2.4/modules-node.html#max-local-storage-nodes.
Commit pushed to master at https://github.com/openshift/openshift-ansible https://github.com/openshift/openshift-ansible/commit/fd165fe201abb5fbd76306a16febaf1cb3c8ad0b Ensure only one ES pod per PV bug 1460564. Fixes [BZ #1460564](https://bugzilla.redhat.com/show_bug.cgi?id=1460564). Unfortunately, the defaults for Elasticsearch prior to v5 allow more than one "node" to access the same configured storage volume(s). This change forces this value to 1 to ensure we don't have an ES pod starting up accessing a volume while another ES pod is shutting down when reploying. This can lead to "1" directories being created in `/elasticsearch/persistent/${CLUSTER_NAME}/data/${CLUSTER_NAME}/nodes/`. By default ES uses a "0" directory there when only one node is accessing it.
max_local_storage_nodes is 1 now # oc get configmap logging-elasticsearch -n logging -o yaml | grep -i max_local_storage_nodes max_local_storage_nodes: 1 Testing env: # openshift version openshift v3.6.122 kubernetes v1.6.1+5115d708d7 etcd 3.2.0 Images from brew registry # docker images | grep logging logging-kibana v3.6 fd67e351dadf 2 days ago 342.4 MB logging-elasticsearch v3.6 1006eb106849 2 days ago 404.6 MB logging-auth-proxy v3.6 301fd39f57e0 2 days ago 229.6 MB logging-fluentd v3.6 dba31f5b54ba 2 days ago 232.5 MB logging-curator v3.6 a0148dd96b8d 2 weeks ago 221.5 MB
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:3188