Bug 1397256

Summary: [RFE] Support external elasticsearch for curator
Product: OpenShift Container Platform Reporter: Takayoshi Tanaka <tatanaka>
Component: RFEAssignee: Jeff Cantrill <jcantril>
Status: CLOSED DEFERRED QA Contact: Xiaoli Tian <xtian>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.3.0CC: aos-bugs, drettori, erich, jokerman, mmccomas, myllynen, tatanaka
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-01-09 11:02:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Takayoshi Tanaka 2016-11-22 04:28:21 UTC 
From the RFE template:

3. What is the nature and description of the request?  
The customer wants to use external elasticsearch (especially AWS managed elasticsearch service) instead of embedded one. Red Hat support shipping logs to external elasticsearch with fluentd template, but curator only support embedded elasticsearch. It's reasonable to support external elasticsearch.

4. Why does the customer need this? (List the business requirements here)  
Some customers want to AWS elasticsearch or other external elasticsearches for some business reason. AWS elasticsearch is a managed service, so it has a merit to use. 

5. How would the customer like to achieve this? (List the functional requirements here)  
Same as 3.  

6. For each functional requirement listed, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented.  
Example scenario: Create AWS elasticseach instance and ship logs from fluentd to elasticsearch. Then the curator can manage this external elasticssearch as the same as the embedded one.  

10. List any affected packages or components.  
Logging template. Especially, logging-curator-template and logging-curator image.
Comment 1 Takayoshi Tanaka 2016-11-22 04:31:29 UTC 
I forgot the customer's comment 
6.
I guess the way to validate it would be to make sure that AWS ES indices are deleted/rotated as per the curator settings. Basically the same way functionality curator currently implements with the embedded ES instance.
Comment 3 Jeff Cantrill 2016-11-22 18:22:06 UTC 
What exactly is the enhancement for this request?  It is currently possible to deploy 3.3 curator image to manage an external ES cluster, assuming the cluster is visible to openshift and has access to the certs: https://github.com/openshift/origin-aggregated-logging/blob/v1.3.1/curator/run_cron.py#L80 and configuration file: https://github.com/openshift/origin-aggregated-logging/blob/v1.3.1/curator/run_cron.py#L53.   Alternatively, I expect with minimal modifications, this script could be added as a cron job to the host running the ES cluster.