Bug 1469918

Summary: The searchguard plugin script is missing in the latest elasticsearch image
Product: OpenShift Container Platform Reporter: Xia Zhao <xiazhao>
Component: LoggingAssignee: Jeff Cantrill <jcantril>
Status: CLOSED ERRATA QA Contact: Xia Zhao <xiazhao>
Severity: high Docs Contact:
Priority: high    
Version: 3.6.0CC: aos-bugs, pportant, rmeggins
Target Milestone: ---Keywords: Regression, TestBlocker
Target Release: 3.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-28 22:00:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
es_log none

Description Xia Zhao 2017-07-12 05:39:14 UTC
Created attachment 1296734 [details]
es_log

Description of problem:
elasticsearch not able to start up since searchguard not yet initialized :
$ oc logs logging-es-data-master-bru9a1e0-1-9rp80
...
[2017-07-12 02:43:13,002][INFO ][container.run            ] Seeding the searchguard ACL index.  Will wait up to 604800 seconds.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
IOError: [Errno 2] No such file or directory: '/usr/share/elasticsearch/config/elasticsearch.yml'
/usr/local/bin/es_seed_acl: line 16: /usr/share/elasticsearch/plugins/openshift-elasticsearch/sgadmin.sh: No such file or directory
0
[2017-07-12 02:43:13,116][WARN ][container.run            ] Error seeding the searchguard ACL index... retrying in 10 seconds - 0 retries so far
[2017-07-12 02:43:13,117][WARN ][container.run            ] Seeding will continue to fail until the cluster status is YELLOW
[2017-07-12 02:43:13,227][INFO ][container.run            ] Remaining red indices: 0
/usr/local/bin/es_seed_acl: line 16: /usr/share/elasticsearch/plugins/openshift-elasticsearch/sgadmin.sh: No such file or directory
1
[2017-07-12 05:17:24,754][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized (you may need to run sgadmin)
/usr/local/bin/es_seed_acl: line 16: /usr/share/elasticsearch/plugins/openshift-elasticsearch/sgadmin.sh: No such file or directory
25

Went into container, found that directory "/usr/share/elasticsearch/plugins/" is missing:
# oc rsh logging-es-data-master-bru9a1e0-1-9rp80
sh-4.2$ cd /usr/share/elasticsearch/    
sh-4.2$ ls -al
total 4
drwxr-xr-x.  6 root root  110 Jul 11 17:19 .
drwxr-xr-x. 74 root root 4096 Jul 11 17:19 ..
lrwxrwxrwx.  1 root root   29 Jul 11 17:19 elasticsearch -> /usr/share/java/elasticsearch
drwxr-xr-x.  2 root root   58 Jul 11 17:17 index_patterns
drwxr-xr-x.  2 root root  164 Jul 11 17:17 index_templates
drwxr-xr-x.  2 root root   86 Jul 11 17:17 kibana_ui_objects
drwxr-xr-x.  2 root root   26 Jul 11 17:17 probe

Version-Release number of selected component (if applicable):
logging-elasticsearch   v3.6                bcacb3cf0505        12 hours ago        404.2 MB

# openshift version
openshift v3.6.140
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

How reproducible:
always

Steps to Reproduce:
1.Deploy logging 3.6.0
2.Check efk status

Actual results:
elasticsearch not able to start up since searchguard not yet initialized:
$ oc get po
NAME                                      READY     STATUS    RESTARTS   AGE
logging-curator-1-j8qwf                   1/1       Running   29         2h
logging-es-data-master-bru9a1e0-1-9rp80   1/1       Running   0          2h
logging-fluentd-dngrh                     1/1       Running   0          2h
logging-fluentd-jbhb4                     1/1       Running   1          2h
logging-kibana-1-4mdqf                    2/2       Running   0          2h

Expected results:
es should work well

Additional info:
Full es log attached

Comment 1 Xia Zhao 2017-07-12 05:47:35 UTC
Blocks all logging 3.6.0 tests.

Comment 4 Xia Zhao 2017-07-13 02:32:37 UTC
It's fixed and logging system are back working fine. Set to verified:

Images verified with:
logging-elasticsearch   v3.6                ca1c9074bf99        6 hours ago         404.7 MB

Comment 9 errata-xmlrpc 2017-11-28 22:00:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188