Bug 1510697
| Summary: | The readiness probe defined for the Elasticsearch pods in aggregated logging prevents cluster recovery | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Peter Portante <pportant> | |
| Component: | Logging | Assignee: | Jan Wozniak <jwozniak> | |
| Status: | CLOSED ERRATA | QA Contact: | Anping Li <anli> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 3.6.1 | CC: | aos-bugs, jcantril, jokerman, jwozniak, mmccomas, pweil, rmeggins | |
| Target Milestone: | --- | Keywords: | OpsBlocker | |
| Target Release: | 3.7.z | |||
| Hardware: | All | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: |
Cause: ES Clusters that take a long time recovering data do not reach YELLOW state fast enough
Consequence: Openshift cluster restarts the pod because the readiness probe fails which starts the ES node recovery again
Fix: Check only for the ES cluster to be listening on the desired port
Result: The Openshift Cluster does not terminate the ES node early which allows it to complete its recovery. The cluster may be in RED state at this time but is able to accept queries and writes.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1523807 1523808 (view as bug list) | Environment: | ||
| Last Closed: | 2018-01-23 17:58:09 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1523807, 1523808 | |||
|
Description
Peter Portante
2017-11-08 02:34:09 UTC
We need a much longer default timeout? What about 24 hours? I believe the readiness probe is not being applied correctly. What does it mean to be "not-ready"? The intention is an ES pod does not participate in the Elasticsearch "service" until that the ES pod is ready. But that has nothing to do with the formation of an Elasticsearch cluster and the internal housing keeping that has to be performed to get ready. So the readiness check ends up causing the deployment to be killed when it was actually deployed successfully and doing what it was supposed to do. There will be times when 24 hours might not be enough time depending on the size of the data sets in play during a recovery. Instead, I think we need to leverage the notion of client, master, and data nodes here as Anton has been suggesting now for a while applying appropriate readiness probes for each. Master nodes would be one DC with a replica count, with each "readiness" probe just being that the Java process is up and running and responding to HTTPS requests. Master nodes would use a Java HEAP matching the pod memory limits since it does not handle on-disk data. Data nodes would be one DC per PV as they are today, each with a simple readiness probe that it is responding to HTTPS requests. Client nodes would be one DC with a replica count, Java HEAP matching the pod memory limits since they also don't handle data, with each "readiness" probe verifying that the client has joined the cluster properly. I don't think readiness probes should reflect the state of the cluster, as ES clients get that information in response to API requests. Peter, which part of the readiness probe checks for the ES "yellow" state? Or is it meant transitively that ES run script tries to insert index_templates and SG files and they do not get inserted into a red state index, therefore, readiness probe is unable to fetch them? Backport to 3.7->3.6 Commits pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/a316868a2b16b4813642bfec0ebd5efb2ab38665 Bug 1510697 - Simplify ES readiness probe https://bugzilla.redhat.com/show_bug.cgi?id=1510697 https://github.com/openshift/origin-aggregated-logging/commit/4be0c5293b38337281bf681cc7823a035243f7d2 Merge pull request #812 from wozniakjan/bz1510697/simplify_es_rp Automatic merge from submit-queue. Bug 1510697 - Simplify ES readiness probe The fix is in openshift3/logging-elasticsearch/images/v3.7.14-5. move bug to verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0113 Commit pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/168a33b9cbc8e04dc9fbd828d40f3308a94b28af Bug 1510697 - Simplify ES readiness probe https://bugzilla.redhat.com/show_bug.cgi?id=1510697 (cherry picked from commit a316868a2b16b4813642bfec0ebd5efb2ab38665) |