Created attachment 1296734 [details]
es_log

Description of problem:
elasticsearch not able to start up since searchguard not yet initialized :
$ oc logs logging-es-data-master-bru9a1e0-1-9rp80
...
[2017-07-12 02:43:13,002][INFO ][container.run            ] Seeding the searchguard ACL index.  Will wait up to 604800 seconds.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
IOError: [Errno 2] No such file or directory: '/usr/share/elasticsearch/config/elasticsearch.yml'
/usr/local/bin/es_seed_acl: line 16: /usr/share/elasticsearch/plugins/openshift-elasticsearch/sgadmin.sh: No such file or directory
0
[2017-07-12 02:43:13,116][WARN ][container.run            ] Error seeding the searchguard ACL index... retrying in 10 seconds - 0 retries so far
[2017-07-12 02:43:13,117][WARN ][container.run            ] Seeding will continue to fail until the cluster status is YELLOW
[2017-07-12 02:43:13,227][INFO ][container.run            ] Remaining red indices: 0
/usr/local/bin/es_seed_acl: line 16: /usr/share/elasticsearch/plugins/openshift-elasticsearch/sgadmin.sh: No such file or directory
1
[2017-07-12 05:17:24,754][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized (you may need to run sgadmin)
/usr/local/bin/es_seed_acl: line 16: /usr/share/elasticsearch/plugins/openshift-elasticsearch/sgadmin.sh: No such file or directory
25

Went into container, found that directory "/usr/share/elasticsearch/plugins/" is missing:
# oc rsh logging-es-data-master-bru9a1e0-1-9rp80
sh-4.2$ cd /usr/share/elasticsearch/    
sh-4.2$ ls -al
total 4
drwxr-xr-x.  6 root root  110 Jul 11 17:19 .
drwxr-xr-x. 74 root root 4096 Jul 11 17:19 ..
lrwxrwxrwx.  1 root root   29 Jul 11 17:19 elasticsearch -> /usr/share/java/elasticsearch
drwxr-xr-x.  2 root root   58 Jul 11 17:17 index_patterns
drwxr-xr-x.  2 root root  164 Jul 11 17:17 index_templates
drwxr-xr-x.  2 root root   86 Jul 11 17:17 kibana_ui_objects
drwxr-xr-x.  2 root root   26 Jul 11 17:17 probe

Version-Release number of selected component (if applicable):
logging-elasticsearch   v3.6                bcacb3cf0505        12 hours ago        404.2 MB

# openshift version
openshift v3.6.140
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

How reproducible:
always

Steps to Reproduce:
1.Deploy logging 3.6.0
2.Check efk status

Actual results:
elasticsearch not able to start up since searchguard not yet initialized:
$ oc get po
NAME                                      READY     STATUS    RESTARTS   AGE
logging-curator-1-j8qwf                   1/1       Running   29         2h
logging-es-data-master-bru9a1e0-1-9rp80   1/1       Running   0          2h
logging-fluentd-dngrh                     1/1       Running   0          2h
logging-fluentd-jbhb4                     1/1       Running   1          2h
logging-kibana-1-4mdqf                    2/2       Running   0          2h

Expected results:
es should work well

Additional info:
Full es log attached