Description of problem: The admin user authorized by the admin key/cert is supposed to have access to everything in our Elasticsearch, but doesn't. It seems to be a permissions error. Steps to Reproduce: 1. Deploy logging solution 2. Find an ES pod from the deployment, for example below "logging-es-uhwvmy8x-1-gszqu" 3. Attempt to request various _cat endpoints (https://www.elastic.co/guide/en/elasticsearch/reference/1.5/cat.html), for example: $ oc exec logging-es-uhwvmy8x-1-gszqu -- curl --key /etc/elasticsearch/keys/admin-key --cert /etc/elasticsearch/keys/admin-cert --cacert /etc/elasticsearch/keys/admin-ca -XGET "https://localhost:9200/_cat/indices?v" Actual results: {"error":"RuntimeException[java.lang.NullPointerException]; nested: NullPointerException; ","status":500} In ES logs: [2016-09-03 07:38:57,866][ERROR][com.floragunn.searchguard.filter.SearchGuardActionFilter] Error while apply() due to java.lang.NullPointerException for action indices:monitor/stats java.lang.NullPointerException at java.util.Objects.requireNonNull(Objects.java:203) at java.util.Arrays$ArrayList.<init>(Arrays.java:3813) at java.util.Arrays.asList(Arrays.java:3800) at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply0(SearchGuardActionFilter.java:191) [...]
Similar problems were reported in https://github.com/openshift/origin-aggregated-logging/issues/59
We expect this to be fixed with OSE 3.4. It's not clear the level of effort to patch for this in previous versions - would be a patch for Search Guard.
Their openshift version version is 3.1.0 the elasticsearch image stream reference is image: 'registry.access.redhat.com/openshift3/logging-elasticsearch:3.1.1' , is this the right version ?
3.1.1-9+ should have the fix to accept the admin cert. It's a little hard to tell which release a system has - there are multiple releases for some versions and you basically have to compare the image hash. What is the image hash for this image according to docker images? Have them try oc import-image logging-elasticsearch:3.1.1 ...before going for the full upgrade.
I'm waiting to learn if updating the 3.1.1 image solved this.
Verification work blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1388753
Verification work blocked by new test blocker https://bugzilla.redhat.com/show_bug.cgi?id=1390854
Blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1391318
Possibly blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1391803, I'm confirming it there.
@Luke Should I test this with curl ES from kibana? Because when I do # oc exec logging-es-e1bxs3th-1-htc5i -- curl --key /etc/elasticsearch/keys/admin-key --cert /etc/elasticsearch/keys/admin-cert --cacert /etc/elasticsearch/keys/admin-ca -XGET "https://localhost:9200/_cat/indices?v" I got these error: curl: (77) Problem with the SSL CA cert (path? access rights?) since directory /etc/elasticsearch/keys/ did not actually exist inside ES pod.
Blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1391803
Hi Luke, Could you please help to take a look at my question in comment #30? Thanks, Xia
(In reply to Xia Zhao from comment #30) > @Luke > > Should I test this with curl ES from kibana? > > Because when I do > > # oc exec logging-es-e1bxs3th-1-htc5i -- curl --key > /etc/elasticsearch/keys/admin-key --cert /etc/elasticsearch/keys/admin-cert > --cacert /etc/elasticsearch/keys/admin-ca -XGET > "https://localhost:9200/_cat/indices?v" > > I got these error: > curl: (77) Problem with the SSL CA cert (path? access rights?) > > since directory /etc/elasticsearch/keys/ did not actually exist inside ES > pod. Right - it's /etc/elasticsearch/secret
I'm not sure if the kibana user has access to see that API endpoint. But we want to use the admin cert on ES anyway. As Rich said, the location of the keys has just changed (which means we need to update https://docs.openshift.org/latest/install_config/aggregate_logging.html#aggregate-logging-performing-elasticsearch-maintenance-operations) So: # oc exec logging-es-e1bxs3th-1-htc5i -- curl \ --key /etc/elasticsearch/secret/admin-key \ --cert /etc/elasticsearch/secret/admin-cert \ --cacert /etc/elasticsearch/secret/admin-ca \ -XGET "https://localhost:9200/_cat/indices?v"
Verified with latest images on brew, it's fixed: $ oc exec logging-es-h6a5k1eh-1-5yceg -- curl \ > --key /etc/elasticsearch/secret/admin-key \ > --cert /etc/elasticsearch/secret/admin-cert \ > --cacert /etc/elasticsearch/secret/admin-ca \ > -XGET "https://localhost:9200/_cat/indices?v" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1420 100 1420 0 0 10624 0 --:--:-- --:--:-- --:--:-- 10676 health status index pri rep docs.count docs.deleted store.size pri.store.size green open project.install-test.627931ab-a619-11e6-912c-fa163e4c9831.2016.11.09 1 0 15572 0 3.9mb 3.9mb green open .operations.2016.11.10 1 0 470325 0 202.6mb 202.6mb green open .kibana 1 0 1 0 3.1kb 3.1kb green open .operations.2016.11.09 1 0 1574648 0 677.2mb 677.2mb green open .searchguard.logging-es-h6a5k1eh-1-5yceg 1 0 4 0 28.2kb 28.2kb green open project.xiazhao.19f21915-a623-11e6-b108-fa163e4c9831.2016.11.09 1 0 1603 0 641.1kb 641.1kb green open project.install-test.627931ab-a619-11e6-912c-fa163e4c9831.2016.11.10 1 0 5192 0 1.4mb 1.4mb green open .kibana.91938315022b77cf223d212e426080092f1aafcf 1 0 2 1 18.3kb 18.3kb green open project.xiazhao.19f21915-a623-11e6-b108-fa163e4c9831.2016.11.10 1 0 241 0 275.8kb 275.8kb
Docs PR to address this: https://github.com/openshift/openshift-docs/pull/3206
*** This bug has been marked as a duplicate of bug 1369924 ***