Bug 1582232 - The .searchguard indices end up with 2 replicas by default
Summary: The .searchguard indices end up with 2 replicas by default
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.7.0
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
: 3.9.z
Assignee: Jeff Cantrill
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-05-24 15:23 UTC by Peter Portante
Modified: 2018-07-19 01:39 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The defaults for the searchguard index are set to autoexpand to number of nodes - 1 Consequence: the number of replicas for .searchguard indices expand to the number of nodes in the cluster, if any one node goes down, the cluster will never return to a green state without all nodes coming back. Fix: Use sgadmin tool to disable replica expansion, update replicas to 0. Modify index setting to allocate to a named node Result: The Searchguard index has auto replica expansion disabled, replicas set to 0 and is allocated to a specific node.
Clone Of:
Environment:
Last Closed: 2018-07-18 09:18:55 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin-aggregated-logging pull 1194 0 None None None 2018-06-01 14:53:51 UTC
Github openshift origin-aggregated-logging pull 1207 0 None None None 2018-06-08 01:23:13 UTC
Red Hat Product Errata RHBA-2018:2213 0 None None None 2018-07-18 09:19:28 UTC

Description Peter Portante 2018-05-24 15:23:45 UTC
Here is an example from the "quay" cluster below:

{".searchguard.logging-es-data-master-0irqvvyj":{"settings":{"index":{"creation_date":"1525733378290","number_of_shards":"1","number_of_replicas":"2","uuid":"gXq7tLTNTZKhABqTplF24A","version":{"created":"2040499"}}}}}

Comment 1 Jeff Cantrill 2018-05-31 15:20:55 UTC
Peter,

Can you define why this is an 'OPSBLOCKER'?  Given the current deployment model, this is undesirable, but is the overhead associated with these indices truly an issue?  The indices are small in both number of documents and size.

We can modify SG to not autoexpand these indices though I would prefer to resolve with a single SG index

Comment 2 Peter Portante 2018-05-31 17:08:19 UTC
The overhead is not a problem, but the replica count is.  Because the number of replicas for .searchguard indices expand to the number of nodes in the cluster, if any one node goes down, we can never return to a green state without all nodes coming back.

Solving this with 1 searchguard index would be ideal, then you could replicate it to a quorum of nodes in the cluster.

Solving this each searchguard index routed to the node it is dedicated to, removing replicas is the other way to solve this.

Comment 4 openshift-github-bot 2018-06-08 00:16:03 UTC
Commits pushed to master at https://github.com/openshift/origin-aggregated-logging

https://github.com/openshift/origin-aggregated-logging/commit/4a0f75d898236e5e75d144c7042317ab5452e74e
bug 1582232. Disable shard allocation for searchguard

https://github.com/openshift/origin-aggregated-logging/commit/6f46de8d13b1be666a21c51bd89accdc881682af
Merge pull request #1194 from jcantrill/1582232_update_replica_settings

bug 1582232. Disable shard allocation for searchguard

Comment 7 Anping Li 2018-07-10 07:58:40 UTC
Verified in v3.9.33
curl https://localhost:9200/_cat/indices?v
health status index                                                                        pri rep docs.count docs.deleted store.size pri.store.size 
green  open   project.kube-service-catalog.e709d7e5-840f-11e8-9c45-42010af00072.2018.07.10   2   1       5749            0        8mb          3.9mb 
green  open   .kibana                                                                        1   1          1            0      6.3kb          3.1kb 
green  open   .searchguard.logging-es-data-master-updhb7w4                                   1   0          5            0     34.4kb         34.4kb 
green  open   .operations.2018.07.10                                                         3   0     818417            0    636.6mb        636.6mb 
green  open   .searchguard.logging-es-data-master-nbnl2x52                                   1   0          5            0     34.4kb         34.4kb 
green  open   project.install-test.2c4e687f-8410-11e8-9c45-42010af00072.2018.07.10           2   1        788            0      1.2mb          674kb 
green  open   .searchguard.logging-es-data-master-bru5xq6a                                   1   0          5            0     34.4kb         34.4kb

Comment 9 errata-xmlrpc 2018-07-18 09:18:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2213


Note You need to log in before you can comment on or make changes to this bug.