Bug 1467963

Commit pushed to master at https://github.com/openshift/origin-aggregated-logging

https://github.com/openshift/origin-aggregated-logging/commit/b0044ec9023bc6472d125df0a1dee6746ad27b18
Bug 1467963 - There are no Kibana dashboards for container admins

https://bugzilla.redhat.com/show_bug.cgi?id=1467963

The script es_load_kibana_ui_objects is used to load dashboards and
other Kibana UI objects for the given user.  Usage:

oc exec $espod -- es_load_kibana_ui_objects user-name

That is, it exists inside the Elasticsearch and ES-OPS pod, and must
be run inside those pods.  Additionally, it requires some indices and
other objects set up by the OpenShift Elasticsearch plugin, so the user
must login to Kibana (or Elasticsearch) before using this script.

This will also add an index pattern for `project.*` and load the
necessary index pattern file.

This relies on some additions to the viaq schema which is provided
by https://github.com/ViaQ/elasticsearch-templates/pull/50

new build - openshift3/logging-elasticsearch:v3.6.140-2

Blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1469918

Hi Rich,

* During testing, when I tried to load dashboards and other Kibana UI objects for cluster-admin user xiazhao (I followed the guide to log in to kibana UI with her before trying this), meet this error:

$ oc get po
NAME                                      READY     STATUS    RESTARTS   AGE
logging-curator-1-6gj7l                   1/1       Running   0          1h
logging-es-data-master-5ywqx64o-1-p2rzb   1/1       Running   0          1h
logging-fluentd-gvjd7                     1/1       Running   0          1h
logging-fluentd-pczkz                     1/1       Running   0          1h
logging-kibana-1-f95tp                    2/2       Running   0          1h

$ oc exec logging-es-data-master-5ywqx64o-1-p2rzb -- es_load_kibana_ui_objects xiazhao
rpc error: code = 13 desc = invalid header field value "oci runtime error: exec failed: container_linux.go:247: starting container process caused \"exec: \\\"es_load_kibana_ui_objects\\\": executable file not found in $PATH\"\n"

* After oc rsh into es pod, I can't find the script " es_load_kibana_ui_objects":
# oc rsh logging-es-data-master-5ywqx64o-1-p2rzb
sh-4.2$ find . -name es_load_kibana_ui_objects
sh-4.2$


* From your PR I found the script here, but don't know how should it run:
$ curl -O https://raw.githubusercontent.com/openshift/origin-aggregated-logging/master/elasticsearch/utils/es_load_kibana_ui_objects

$  oc exec logging-es-data-master-5ywqx64o-1-p2rzb -- es_load_kibana_ui_objects xiazhao
rpc error: code = 13 desc = invalid header field value "oci runtime error: exec failed: container_linux.go:247: starting container process caused \"exec: \\\"es_load_kibana_ui_objects\\\": executable file not found in $PATH\"\n"
command terminated with exit code 126

Could you please help point out what else steps should I do? 

Thanks,
Xia

Confirmed that dev is rebuilding the elasticsearch image, set back to assigned, thanks!

koji_builds:
  https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=574393
repositories:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:rhaos-3.6-rhel-7-docker-candidate-85872-20170717180328
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:latest
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.152.0
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.152.0-2

Tested with the latest v3.6 es image which is 12 hours newer than v3.6.152.0-2, issue is reproduced:
# docker images | grep elasticsearch
${brew-registry}/openshift3/logging-elasticsearch   v3.6                9b0f4b948e79        3 hours ago         404.2 MB
${brew-registry}/openshift3/logging-elasticsearch   v3.6.152.0-2        1e210744feb3        15 hours ago        404.7 MB

Test steps in detail:
1. Login kibana with user "xiazhao" who is cluster-admin

2. When executing the adding dashboard script, get this error:
# oc exec logging-es-data-master-x3pw9820-1-v7k4p -- es_load_kibana_ui_objects xiazhao
[2017-07-18 09:15:32,266][INFO ][container.run            ] Adding Kibana dashboards and other UI objects for user xiazhao index .kibana.f7724d98466ed7391e970202dc54a6460046aadb
[2017-07-18 09:15:32,267][INFO ][container.run            ] Adding the index pattern for project.* . . .
cat: /usr/share/java/elasticsearch/index_patterns/com.redhat.viaq-openshift.index-pattern.json: No such file or directory
command terminated with exit code 1

3. Relogin kibana with user "xiazhao", went into the "dashboard" tab, and searched for the kibana visualizations with pattern "*", get 0 foundings

@Rich, please help review and point out if any of the above test steps were wrong, thanks!

(In reply to Xia Zhao from comment #7)
> Tested with the latest v3.6 es image which is 12 hours newer than
> v3.6.152.0-2, 

@Xia can you explain this comment?  If you tested an image is 12 hours newer then it is not the same image as:

issue is reproduced:
> # docker images | grep elasticsearch
> ${brew-registry}/openshift3/logging-elasticsearch   v3.6               
> 9b0f4b948e79        3 hours ago         404.2 MB
> ${brew-registry}/openshift3/logging-elasticsearch   v3.6.152.0-2       
> 1e210744feb3        15 hours ago        404.7 MB
>

Or are you stating you tested 2 sets of images:

1. 12 hours newer
2.  v3.6.152.0-2

koji_builds:
  https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=574625
repositories:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:rhaos-3.6-rhel-7-docker-candidate-57374-20170718145154
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:latest
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.153
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.153-2

(In reply to Jeff Cantrill from comment #8)
> (In reply to Xia Zhao from comment #7)
> > Tested with the latest v3.6 es image which is 12 hours newer than
> > v3.6.152.0-2, 
> 
> @Xia can you explain this comment?  If you tested an image is 12 hours newer
> then it is not the same image as:
> 
> issue is reproduced:
> > # docker images | grep elasticsearch
> > ${brew-registry}/openshift3/logging-elasticsearch   v3.6               
> > 9b0f4b948e79        3 hours ago         404.2 MB
> > ${brew-registry}/openshift3/logging-elasticsearch   v3.6.152.0-2       
> > 1e210744feb3        15 hours ago        404.7 MB
> >
> 
> Or are you stating you tested 2 sets of images:
> 
> 1. 12 hours newer
> 2.  v3.6.152.0-2

Hi Jeff,

By specifying openshift_logging_image_version=v3.6 in the inventory file when deploy the logging tsacks, I tested with this image:
> > ${brew-registry}/openshift3/logging-elasticsearch   v3.6               
> > 9b0f4b948e79        3 hours ago         404.2 MB

Thanks,
Xia

Verified with elasticsearch image v3.6.153-2, get this error message while loading the kibana dashboards, and checked on kibana UI, 0 visualizations was found under the dashboard tab for cluster-admin user:

# oc exec logging-es-data-master-odlf264i-1-bc62n -- es_load_kibana_ui_objects xiazhao
[2017-07-19 06:35:54,131][INFO ][container.run            ] Adding Kibana dashboards and other UI objects for user xiazhao index .kibana.f7724d98466ed7391e970202dc54a6460046aadb
[2017-07-19 06:35:54,132][INFO ][container.run            ] Adding the index pattern for project.* . . .
[2017-07-19 06:35:54,306][INFO ][container.run            ] Adding the Kibana UI objects . . .
cat: /usr/share/java/elasticsearch/kibana_ui_objects/*.json: No such file or directory
Traceback (most recent call last):
  File "<string>", line 4, in <module>
  File "/usr/lib64/python2.7/json/__init__.py", line 290, in load
    **kw)
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
command terminated with exit code 1

Commit pushed to master at https://github.com/openshift/origin-aggregated-logging

https://github.com/openshift/origin-aggregated-logging/commit/a4dd522cd1b55a3303343cdf6ab7694ab5954dae
Bug 1467963 There are no Kibana dashboards for container admins

https://bugzilla.redhat.com/show_bug.cgi?id=1467963
Additional fixes:
* Add kibana_ui_objects directory to downstream Dockerfile
* Other fixes to sync upstream and downstream Dockerfiles
* use ES_HOME instead of hardcoded path
* check for missing files and directories

koji_builds:
  https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=575170
repositories:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:rhaos-3.6-rhel-7-docker-candidate-80341-20170719194850
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:latest
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.153
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.153-3

It's fixed: 

1. The adding script can now be run successfully:
# oc exec logging-es-data-master-ccham3fm-1-tx33k -- es_load_kibana_ui_objects xiazhao
[2017-07-20 06:48:36,553][INFO ][container.run            ] Adding Kibana dashboards and other UI objects for user xiazhao index .kibana.f7724d98466ed7391e970202dc54a6460046aadb
[2017-07-20 06:48:36,553][INFO ][container.run            ] Adding the index pattern for project.* . . .
[2017-07-20 06:48:36,701][INFO ][container.run            ] Adding the Kibana UI objects . . .
[2017-07-20 06:48:36,899][INFO ][container.run            ] Success

2. Login kibana after step 1) is done, found there are 2 pre-saved visualizations for cluster-admin user, and normal user not able to view them:

    Kubernetes Logs by Namespace, DC, Pod, Container
    Kubernetes Logs over Time w/ Container Name 

Image tested with: elasticsearch v3.6.153-3
# docker images | grep elasticsearch
${brew-registry}/openshift3/logging-elasticsearch      v3.6                f6e798e43e8f        10 hours ago        434.5 MB
${brew-registry}/openshift3/logging-elasticsearch      v3.6.153-3          f6e798e43e8f        10 hours ago        434.5 MB

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188