Bug 1582232
| Summary: | The .searchguard indices end up with 2 replicas by default | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Peter Portante <pportant> |
| Component: | Logging | Assignee: | Jeff Cantrill <jcantril> |
| Status: | CLOSED ERRATA | QA Contact: | Anping Li <anli> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 3.7.0 | CC: | aos-bugs, pportant, rmeggins |
| Target Milestone: | --- | Keywords: | OpsBlocker |
| Target Release: | 3.9.z | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: The defaults for the searchguard index are set to autoexpand to number of nodes - 1
Consequence: the number of replicas for .searchguard indices expand to the number of nodes in the cluster, if any one node goes down, the cluster will never return to a green state without all nodes coming back.
Fix: Use sgadmin tool to disable replica expansion, update replicas to 0. Modify index setting to allocate to a named node
Result: The Searchguard index has auto replica expansion disabled, replicas set to 0 and is allocated to a specific node.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-07-18 09:18:55 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Peter, Can you define why this is an 'OPSBLOCKER'? Given the current deployment model, this is undesirable, but is the overhead associated with these indices truly an issue? The indices are small in both number of documents and size. We can modify SG to not autoexpand these indices though I would prefer to resolve with a single SG index The overhead is not a problem, but the replica count is. Because the number of replicas for .searchguard indices expand to the number of nodes in the cluster, if any one node goes down, we can never return to a green state without all nodes coming back. Solving this with 1 searchguard index would be ideal, then you could replicate it to a quorum of nodes in the cluster. Solving this each searchguard index routed to the node it is dedicated to, removing replicas is the other way to solve this. Commits pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/4a0f75d898236e5e75d144c7042317ab5452e74e bug 1582232. Disable shard allocation for searchguard https://github.com/openshift/origin-aggregated-logging/commit/6f46de8d13b1be666a21c51bd89accdc881682af Merge pull request #1194 from jcantrill/1582232_update_replica_settings bug 1582232. Disable shard allocation for searchguard Verified in v3.9.33 curl https://localhost:9200/_cat/indices?v health status index pri rep docs.count docs.deleted store.size pri.store.size green open project.kube-service-catalog.e709d7e5-840f-11e8-9c45-42010af00072.2018.07.10 2 1 5749 0 8mb 3.9mb green open .kibana 1 1 1 0 6.3kb 3.1kb green open .searchguard.logging-es-data-master-updhb7w4 1 0 5 0 34.4kb 34.4kb green open .operations.2018.07.10 3 0 818417 0 636.6mb 636.6mb green open .searchguard.logging-es-data-master-nbnl2x52 1 0 5 0 34.4kb 34.4kb green open project.install-test.2c4e687f-8410-11e8-9c45-42010af00072.2018.07.10 2 1 788 0 1.2mb 674kb green open .searchguard.logging-es-data-master-bru5xq6a 1 0 5 0 34.4kb 34.4kb Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2213 |
Here is an example from the "quay" cluster below: {".searchguard.logging-es-data-master-0irqvvyj":{"settings":{"index":{"creation_date":"1525733378290","number_of_shards":"1","number_of_replicas":"2","uuid":"gXq7tLTNTZKhABqTplF24A","version":{"created":"2040499"}}}}}